Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
en:services:application_services:high_performance_computing:transfer_data [2019/01/10 11:01]
tehlers
en:services:application_services:high_performance_computing:transfer_data [2020/06/15 13:17]
mboden [Using FileZilla]
Line 3: Line 3:
 One often wants to copy data to or from the Scientific Compute Cluster. In this article we show options to copy from inside or outside of GÖNET, the network of the University of Göttingen and Max Planck institutes in Göttingen. One often wants to copy data to or from the Scientific Compute Cluster. In this article we show options to copy from inside or outside of GÖNET, the network of the University of Göttingen and Max Planck institutes in Göttingen.
  
-The Transfernodes ​are ''​transfer-scc.gwdg.de''​ (reachable inside the GÖNET) and ''​transfer.gwdg.de''​ (world accessible). ​+The transfernodes ​are ''​transfer-scc.gwdg.de''​ (reachable inside the GÖNET) and ''​transfer.gwdg.de''​ (world accessible). ​
  
 ''​transfer-scc.gwdg.de''​ has homes and /scratch. ''​transfer-scc.gwdg.de''​ has homes and /scratch.
Line 9: Line 9:
 ''​transfer.gwdg.de''​ has only homes. /scratch is reachable with ssh port-forward to transfer-scc;​ see below. ''​transfer.gwdg.de''​ has only homes. /scratch is reachable with ssh port-forward to transfer-scc;​ see below.
  
-For the filesystem /scratch2 we don't have a dedicated ​Transfernode. For this filesystem you need to use the frontend gwdu103.gwdg.de (instead of ''​transfer-scc.gwdg.de''​).+For the filesystem /scratch2 we don't have a dedicated ​transfernode. For this filesystem you need to use the frontend ​''​gwdu103.gwdg.de'' ​(instead of ''​transfer-scc.gwdg.de''​). 
 + 
 +=== Transfering Large Quantaties of Data === 
 + 
 +If you need to transfer large quantities of data to our system, it may be advisable to copy directly to /scratch to increase the transfer rate, since the write-performance of our home file systems can sometimes be the limiting factor. Transfering to /scratch directly will not have this limitation. 
 =====  Within the GÖNET ​ ===== =====  Within the GÖNET ​ =====
 +
 +==== Using FileZilla ====
 +
 +[[https://​filezilla-project.org/​|FileZilla]] is a feature rich, easy to use, cross-platform file transfer utility. If you are not keen on using the command line to transfer your data, this is the right tool for you. Here is short introduction.
 +{{ :​en:​services:​application_services:​high_performance_computing:​filezilla-key.png?​direct&​800 |}}
 +Go to "​File"​ -> "Site Manger"​ and click on "New Site". Put the required information into the fields on the right. Host either transfer.gwdg.de,​ transfer-scc.gwdg.de or just gwdu103.gwdg.de if you need access to scratch2. From the "​Protocol"​-drop-down menu choose SFTP.\\
 +We disabled the password-only authentication so you have to change the "Logon Type" to "Key file" and supply an ssh key. See [[en:​services:​application_services:​high_performance_computing:​connect_with_ssh|Connect with SSH]] for details on how to generate and upload one, if you haven'​t already.
 +{{ :​en:​services:​application_services:​high_performance_computing:​fingerprint.png?​direct&​500 |}}
 +If you connect for the first time, you may be asked to confirm the fingerprint.\\
 +You can now transfer data between your local machine and your home on our cluster by drag-and-dropping it from the left pane (or anywhere from your PC) to the right pane. You can also copy from our cluster to your machine.
 +{{ :​en:​services:​application_services:​high_performance_computing:​scratch.png?​direct&​800 |}}
 +To transfer data to /scratch, make sure you are connected to transfer-scc.gwdg.de or if you need access to /scratch2, gwdu103.gwdg.de. You can type your scratch directory into the location bar in the right pane.
 +
 +==== Using rsync or scp ====
  
 If your computer is located //inside the GÖNET//, you can use ''​transfer-scc.gwdg.de''​ to copy data from and to: If your computer is located //inside the GÖNET//, you can use ''​transfer-scc.gwdg.de''​ to copy data from and to:
Line 18: Line 37:
 **/​scratch** **/​scratch**
  
-using ssh (scp or rsync tunneled through ssh). Here is an example for copying data from your computer to scratch (please replace {SRC-DIR},​{DST-DIR} and {USERID}):+using ssh (scp or rsync tunneled through ssh). Here is an example for copying data from your computer to scratch (please replace ​''​{SRC-DIR}''​''​{DST-DIR}'' ​and ''​{USERID}''​):
 <​code>​ <​code>​
 scp -rp {SRC-DIR} {USERID}@transfer-scc.gwdg.de:/​scratch/​{USERID}/​{DST-DIR} scp -rp {SRC-DIR} {USERID}@transfer-scc.gwdg.de:/​scratch/​{USERID}/​{DST-DIR}
Line 41: Line 60:
 =====  Outside of the GÖNET (from the Internet) ​ ===== =====  Outside of the GÖNET (from the Internet) ​ =====
  
-If your computer is located //outside the GÖNET//, you can use the server ''​transfer-scc.gwdg.de''​ to copy data from and to:+If your computer is located //outside the GÖNET//, you can use the server ''​transfer.gwdg.de''​ to copy data from and to:
  
 **your Home directory** **your Home directory**
Line 49: Line 68:
 For security reasons this node is not located inside the SCC network and therefor only has your Home directory but not /scratch accessible. For security reasons this node is not located inside the SCC network and therefor only has your Home directory but not /scratch accessible.
  
-====  Getting Access to Scratch from Outside the GÖNET ​ (ssh agent method)====+====  Getting Access to Scratch from Outside the GÖNET ​ via VPN (recommended) ==== 
 + 
 +The easiest way to get access to /scratch is to use our [[en:​services:​network_services:​vpn:​start|VPN]]. After activating the VPN connection as described in the documentation and you will be inside the GÖNET and therefore be able to use transfer-scc.gwdg.de as described above. 
 + 
 +====  Getting Access to Scratch from Outside the GÖNET ​ (ssh proxy method)==== 
 + 
 +If you want to copy data to scratch, you need to "​hop"​ over transfer.gwdg.de to ''​transfer-scc.gwdg.de''​. This is not necessary for the transfer from and to your home directory (see above)! The easiest way is to create a section in your ssh config ''​~/​.ssh/​config''​ on your local system (don't forget to substitute your GWDG-UserID):​ 
 + 
 +<​code>​ 
 +Host transfer-scc.gwdg.de 
 +        User {USERID} 
 +        Hostname transfer-scc.gwdg.de 
 +        ProxyCommand ssh -W %h:%p {USERID}@transfer.gwdg.de 
 +</​code>​ 
 +\\ 
 +This configures ssh to reach ''​transfer-scc.gwdg.de''​ not directly, but via ''​transfer.gwdg.de''​. An ''​ssh transfer-scc.gwdg.de''​ or ''​scp''​ or ''​rsync''​ will now tunnel through ''​transfer.gwdg.de''​.
  
 ====  Getting Access to Scratch from Outside the GÖNET ​ (port tunnel method)==== ====  Getting Access to Scratch from Outside the GÖNET ​ (port tunnel method)====
Line 104: Line 138:
  
 [[Kategorie:​ Scientific Computing]] [[Kategorie:​ Scientific Computing]]
 +
 +===== ownCloud =====
 +
 +You can directly transfer data from and to the GWDG ownCloud by the WebDAV client ''​cadaver''​. This program provides a command line interface to the ownCloud similar to command line FTP applications. Connect to the server with:
 +<​code>​
 +$ cadaver https://​owncloud.gwdg.de/​remote.php/​nonshib-webdav
 +Authentication required for ownCloud on server `owncloud.gwdg.de':​
 +Username: your-user-name
 +Password: ​
 +dav:/​remote.php/​nonshib-webdav/> ​
 +</​code>​
 +You can use the usual commands such as ''​cd'',​ ''​ls'',​ ''​cp'',​ etc. to navigate and copy/move files within you ownCloud. Uploading and downloading files can be done with ''​put''​ and ''​get''​ respectively (or ''​mput''​ and ''​mget''​ to transfer multiple files at once).
 +
 +To change your directory on the cluster itself and list files, use ''​lcd''​ and ''​lls''​ (think of it as local-cd and local-ls). After you are done, you can quit with ''​quit'',​ ''​exit''​ or with ''​CTRL-D''​. Here are a few examples:
 +<​code>​
 +dav:/​remote.php/​nonshib-webdav/>​ ls
 +[... contents of ownCloud ...]
 +dav:/​remote.php/​nonshib-webdav/>​ lls
 +exampledir ​ file1
 +dav:/​remote.php/​nonshib-webdav/>​ lpwd
 +Local directory: /​scratch/​mboden/​dav-example
 +dav:/​remote.php/​nonshib-webdav/>​ mkdir example
 +Creating `example':​ succeeded.
 +dav:/​remote.php/​nonshib-webdav/>​ cd example
 +dav:/​remote.php/​nonshib-webdav/​example/>​ ls
 +Listing collection `/​remote.php/​nonshib-webdav/​example/':​ collection is empty.
 +dav:/​remote.php/​nonshib-webdav/​example/>​ put file1
 +Uploading file1 to `/​remote.php/​nonshib-webdav/​example/​file1':​ succeeded.
 +dav:/​remote.php/​nonshib-webdav/​example/>​ put exampledir
 +Uploading exampledir to `/​remote.php/​nonshib-webdav/​example/​exampledir':​
 +Progress: [                              ]   0.0% of 1 bytes failed:
 +Failed reading request body file: Is a directory
 +</​code>​
 +Copying whole directories is not possible. You should create a tar or zip archive first and copy that.
 +<​code>​
 +dav:/​remote.php/​nonshib-webdav/>​ cd example ​
 +dav:/​remote.php/​nonshib-webdav/​example/>​ lls -la
 +total 1
 +drwx------ 2 mboden GWDG 0 Dec  9 10:17 .
 +drwxr-xr-x 5 mboden GWDG 4 Dec  9 10:17 ..
 +dav:/​remote.php/​nonshib-webdav/​example/>​ ls
 +Listing collection `/​remote.php/​nonshib-webdav/​example/':​ succeeded.
 +        file1                                  0  Dec  9 10:10
 +dav:/​remote.php/​nonshib-webdav/​example/>​ get file1
 +Downloading `/​remote.php/​nonshib-webdav/​example/​file1'​ to file1: [.] succeeded.
 +dav:/​remote.php/​nonshib-webdav/​example/>​ lls -la
 +total 1
 +drwx------ 2 mboden GWDG 0 Dec  9 10:19 .
 +drwxr-xr-x 5 mboden GWDG 4 Dec  9 10:17 ..
 +-rw------- 1 mboden GWDG 0 Dec  9 10:19 file1
 +</​code>​
 +
 +