Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
en:services:application_services:high_performance_computing:transfer_data [2019/01/10 11:16]
tehlers [Transfer Data]
en:services:application_services:high_performance_computing:transfer_data [2020/06/15 13:17]
mboden [Using FileZilla]
Line 10: Line 10:
  
 For the filesystem /scratch2 we don't have a dedicated transfernode. For this filesystem you need to use the frontend ''​gwdu103.gwdg.de''​ (instead of ''​transfer-scc.gwdg.de''​). For the filesystem /scratch2 we don't have a dedicated transfernode. For this filesystem you need to use the frontend ''​gwdu103.gwdg.de''​ (instead of ''​transfer-scc.gwdg.de''​).
 +
 +=== Transfering Large Quantaties of Data ===
 +
 +If you need to transfer large quantities of data to our system, it may be advisable to copy directly to /scratch to increase the transfer rate, since the write-performance of our home file systems can sometimes be the limiting factor. Transfering to /scratch directly will not have this limitation.
 +
 =====  Within the GÖNET ​ ===== =====  Within the GÖNET ​ =====
 +
 +==== Using FileZilla ====
 +
 +[[https://​filezilla-project.org/​|FileZilla]] is a feature rich, easy to use, cross-platform file transfer utility. If you are not keen on using the command line to transfer your data, this is the right tool for you. Here is short introduction.
 +{{ :​en:​services:​application_services:​high_performance_computing:​filezilla-key.png?​direct&​800 |}}
 +Go to "​File"​ -> "Site Manger"​ and click on "New Site". Put the required information into the fields on the right. Host either transfer.gwdg.de,​ transfer-scc.gwdg.de or just gwdu103.gwdg.de if you need access to scratch2. From the "​Protocol"​-drop-down menu choose SFTP.\\
 +We disabled the password-only authentication so you have to change the "Logon Type" to "Key file" and supply an ssh key. See [[en:​services:​application_services:​high_performance_computing:​connect_with_ssh|Connect with SSH]] for details on how to generate and upload one, if you haven'​t already.
 +{{ :​en:​services:​application_services:​high_performance_computing:​fingerprint.png?​direct&​500 |}}
 +If you connect for the first time, you may be asked to confirm the fingerprint.\\
 +You can now transfer data between your local machine and your home on our cluster by drag-and-dropping it from the left pane (or anywhere from your PC) to the right pane. You can also copy from our cluster to your machine.
 +{{ :​en:​services:​application_services:​high_performance_computing:​scratch.png?​direct&​800 |}}
 +To transfer data to /scratch, make sure you are connected to transfer-scc.gwdg.de or if you need access to /scratch2, gwdu103.gwdg.de. You can type your scratch directory into the location bar in the right pane.
 +
 +==== Using rsync or scp ====
  
 If your computer is located //inside the GÖNET//, you can use ''​transfer-scc.gwdg.de''​ to copy data from and to: If your computer is located //inside the GÖNET//, you can use ''​transfer-scc.gwdg.de''​ to copy data from and to:
Line 18: Line 37:
 **/​scratch** **/​scratch**
  
-using ssh (scp or rsync tunneled through ssh). Here is an example for copying data from your computer to scratch (please replace {SRC-DIR},​{DST-DIR} and {USERID}):+using ssh (scp or rsync tunneled through ssh). Here is an example for copying data from your computer to scratch (please replace ​''​{SRC-DIR}''​''​{DST-DIR}'' ​and ''​{USERID}''​):
 <​code>​ <​code>​
 scp -rp {SRC-DIR} {USERID}@transfer-scc.gwdg.de:/​scratch/​{USERID}/​{DST-DIR} scp -rp {SRC-DIR} {USERID}@transfer-scc.gwdg.de:/​scratch/​{USERID}/​{DST-DIR}
Line 41: Line 60:
 =====  Outside of the GÖNET (from the Internet) ​ ===== =====  Outside of the GÖNET (from the Internet) ​ =====
  
-If your computer is located //outside the GÖNET//, you can use the server ''​transfer-scc.gwdg.de''​ to copy data from and to:+If your computer is located //outside the GÖNET//, you can use the server ''​transfer.gwdg.de''​ to copy data from and to:
  
 **your Home directory** **your Home directory**
Line 48: Line 67:
  
 For security reasons this node is not located inside the SCC network and therefor only has your Home directory but not /scratch accessible. For security reasons this node is not located inside the SCC network and therefor only has your Home directory but not /scratch accessible.
 +
 +====  Getting Access to Scratch from Outside the GÖNET ​ via VPN (recommended) ====
 +
 +The easiest way to get access to /scratch is to use our [[en:​services:​network_services:​vpn:​start|VPN]]. After activating the VPN connection as described in the documentation and you will be inside the GÖNET and therefore be able to use transfer-scc.gwdg.de as described above.
  
 ====  Getting Access to Scratch from Outside the GÖNET ​ (ssh proxy method)==== ====  Getting Access to Scratch from Outside the GÖNET ​ (ssh proxy method)====
  
-If you want to copy data to scratch, you need to "​hop"​ over transfer.gwdg.de to ''​transfer-scc.gwdg.de''​. This is not necessary for the transfer from and to your home directory (see above)! The easiest way is to create a section in your ssh config ''​~/​.ssh/​config''​ (don't forget to substitute your GWDG-UserID):​+If you want to copy data to scratch, you need to "​hop"​ over transfer.gwdg.de to ''​transfer-scc.gwdg.de''​. This is not necessary for the transfer from and to your home directory (see above)! The easiest way is to create a section in your ssh config ''​~/​.ssh/​config'' ​on your local system ​(don't forget to substitute your GWDG-UserID):​
  
 <​code>​ <​code>​
Line 115: Line 138:
  
 [[Kategorie:​ Scientific Computing]] [[Kategorie:​ Scientific Computing]]
 +
 +===== ownCloud =====
 +
 +You can directly transfer data from and to the GWDG ownCloud by the WebDAV client ''​cadaver''​. This program provides a command line interface to the ownCloud similar to command line FTP applications. Connect to the server with:
 +<​code>​
 +$ cadaver https://​owncloud.gwdg.de/​remote.php/​nonshib-webdav
 +Authentication required for ownCloud on server `owncloud.gwdg.de':​
 +Username: your-user-name
 +Password: ​
 +dav:/​remote.php/​nonshib-webdav/> ​
 +</​code>​
 +You can use the usual commands such as ''​cd'',​ ''​ls'',​ ''​cp'',​ etc. to navigate and copy/move files within you ownCloud. Uploading and downloading files can be done with ''​put''​ and ''​get''​ respectively (or ''​mput''​ and ''​mget''​ to transfer multiple files at once).
 +
 +To change your directory on the cluster itself and list files, use ''​lcd''​ and ''​lls''​ (think of it as local-cd and local-ls). After you are done, you can quit with ''​quit'',​ ''​exit''​ or with ''​CTRL-D''​. Here are a few examples:
 +<​code>​
 +dav:/​remote.php/​nonshib-webdav/>​ ls
 +[... contents of ownCloud ...]
 +dav:/​remote.php/​nonshib-webdav/>​ lls
 +exampledir ​ file1
 +dav:/​remote.php/​nonshib-webdav/>​ lpwd
 +Local directory: /​scratch/​mboden/​dav-example
 +dav:/​remote.php/​nonshib-webdav/>​ mkdir example
 +Creating `example':​ succeeded.
 +dav:/​remote.php/​nonshib-webdav/>​ cd example
 +dav:/​remote.php/​nonshib-webdav/​example/>​ ls
 +Listing collection `/​remote.php/​nonshib-webdav/​example/':​ collection is empty.
 +dav:/​remote.php/​nonshib-webdav/​example/>​ put file1
 +Uploading file1 to `/​remote.php/​nonshib-webdav/​example/​file1':​ succeeded.
 +dav:/​remote.php/​nonshib-webdav/​example/>​ put exampledir
 +Uploading exampledir to `/​remote.php/​nonshib-webdav/​example/​exampledir':​
 +Progress: [                              ]   0.0% of 1 bytes failed:
 +Failed reading request body file: Is a directory
 +</​code>​
 +Copying whole directories is not possible. You should create a tar or zip archive first and copy that.
 +<​code>​
 +dav:/​remote.php/​nonshib-webdav/>​ cd example ​
 +dav:/​remote.php/​nonshib-webdav/​example/>​ lls -la
 +total 1
 +drwx------ 2 mboden GWDG 0 Dec  9 10:17 .
 +drwxr-xr-x 5 mboden GWDG 4 Dec  9 10:17 ..
 +dav:/​remote.php/​nonshib-webdav/​example/>​ ls
 +Listing collection `/​remote.php/​nonshib-webdav/​example/':​ succeeded.
 +        file1                                  0  Dec  9 10:10
 +dav:/​remote.php/​nonshib-webdav/​example/>​ get file1
 +Downloading `/​remote.php/​nonshib-webdav/​example/​file1'​ to file1: [.] succeeded.
 +dav:/​remote.php/​nonshib-webdav/​example/>​ lls -la
 +total 1
 +drwx------ 2 mboden GWDG 0 Dec  9 10:19 .
 +drwxr-xr-x 5 mboden GWDG 4 Dec  9 10:17 ..
 +-rw------- 1 mboden GWDG 0 Dec  9 10:19 file1
 +</​code>​
 +
 +