Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
en:services:application_services:high_performance_computing:running_jobs_for_experienced_users [2019/09/27 09:46]
vend [SMP jobs] ->https://projects.gwdg.de/projects/parallelrechnerbeschaffung-2012-13/wiki/outdated-running-jobs-for-experienced-users
en:services:application_services:high_performance_computing:running_jobs_for_experienced_users [2019/09/27 09:47]
vend [Memory selection] ->https://projects.gwdg.de/projects/parallelrechnerbeschaffung-2012-13/wiki/outdated-running-jobs-for-experienced-users
Line 5: Line 5:
  
  
-===== Using the fat+ queue ===== 
  
-Nodes with a lot of memory are very expensive and should not normally be used for jobs which could also run on our other nodes. Therefore, please note the following policies: 
  
-  * Your job must need more than 250 GB RAM. 
-  * Your job must use at least a full 512 GB node or half a 1.5 TB or 2 TB node: 
- 
-  * For a full 512 GB node: 
-<​code>​ 
-#BSUB -x 
-#BSUB -R "​maxmem < 600000"​ 
-</​code>​ 
- 
-  * For half a 1.5 TB node (your job needs more than 500 GB RAM): 
-<​code>​ 
-#BSUB -n 20 
-#BSUB -R span[hosts=1] 
-#BSUB -R "​maxmem < 1600000 && maxmem > 600000"​ 
-</​code>​ 
- 
-  * For a full 1.5 TB node (your job needs more than 700 GB RAM): 
-<​code>​ 
-#BSUB -x 
-#BSUB -R "​maxmem < 1600000 && maxmem > 600000"​ 
-</​code>​ 
- 
-  * For half a 2 TB node (your job needs more than 700 GB RAM): 
-<​code>​ 
-#BSUB -n 16 
-#BSUB -R span[hosts=1] 
-#BSUB -R "​maxmem > 1600000"​ 
-</​code>​ 
- 
-  * For a full 2 TB node (your job needs more than 1.5 TB RAM): 
-<​code>​ 
-#BSUB -x 
-#BSUB -R "​maxmem > 1600000"​ 
-</​code>​ 
- 
-The 512 GB nodes are also available in the fat queue, without these restrictions. However, fat jobs on these nodes have a lower priority compared to fat+ jobs. 
- 
-===== CPU architecture selection ===== 
- 
-Our cluster provides four generations of Intel CPUs and two generations of AMD CPUs. However, the main difference between these CPU types is whether they support Intel'​s AVX2 or not. For selecting this we have introduced the x64inlvl (for x64 instruction level) label: 
- 
-<​code>​ 
-x64inlvl=1 : Supports only AVX 
-x64inlvl=2 : Supports AVX and AVX2 
-</​code>​ 
- 
-In order to choose an AVX2 capable node you therefore have to include 
-<​code>​ 
-#BSUB -R "​x64inlvl=2"​ 
-</​code>​ 
-in your submission script. 
- 
-If you need to be more specific, you can also directly choose the CPU generation: 
- 
-<​code>​ 
-amd=1 : Interlagos 
-amd=2 : Abu Dhabi 
- 
-intel=1 : Sandy Bridge 
-intel=2 : Ivy Bridge 
-intel=3 : Haswell 
-intel=4 : Broadwell 
-</​code>​ 
- 
-So, in order to choose any AMD CPU: 
-<​code>​ 
-#BSUB -R amd 
-</​code>​ 
-In order to choose an Intel CPU of at least Haswell generation: 
-<​code>​ 
-#BSUB -R "​intel>​=3"​ 
-</​code>​ 
-This is equivalent to ''​x64inlvl=2''​. 
- 
-===== GPU selection ===== 
- 
-In order to use a GPU you should submit your job to the ''​gpu''​ queue, and request GPU shares. Each node equipped with a GPU provides as many GPU shares as it has cores, independent of how many GPUs are built in. So for example, on the nodes, which have 24 CPU cores, the following would give you exclusive access to GPUs: 
-<​code>​ 
-#BSUB -R "​rusage[ngpus_shared=24]"​ 
-</​code>​ 
-Note that you need not necessarily also request 24 cores with ''​-n 24'',​ as jobs from the MPI queue may utilize free CPU cores if you do not need them. The latest "​gpu"​ nodes have two GPUs each, and you should use both, if possible. 
- 
-If you request less shares than cores available, other jobs may also utilize the GPUs. However, we have currently no mechanism to select a specific one for a job. This would have to be handled in the application or your job script. 
- 
-A good way to use the nodes which have 2 GPUs with jobs only working on one GPU would be to put two together in one job script and preselect a GPU for each one. 
- 
-Currently we have several generations of NVidia GPUs in the cluster, selectable in the same way as CPU generations:​ 
- 
-<​code>​ 
-nvgen=1 : Kepler 
-nvgen=2 : Maxwell 
-nvgen=3 : Pascal 
-</​code>​ 
- 
-Most GPUs are commodity graphics cards, and only provide good performance for single precision calculations. If you need double precision performance,​ or error correcting memory (ECC RAM), you can select the Tesla GPUs with 
-<​code>​ 
-#BSUB -R tesla 
-</​code>​ 
-Our Tesla K40 are of the Kepler generation (nvgen=1). 
- 
-If you want to make sure to run on a node equipped with two GPUs use: 
-<​code>​ 
-#BSUB -R "​ngpus=2"​ 
-</​code>​ 
  
 ===== Memory selection ===== ===== Memory selection =====