{"id":22815,"date":"2023-04-17T09:12:13","date_gmt":"2023-04-17T07:12:13","guid":{"rendered":"https:\/\/info.gwdg.de\/news\/?p=22815"},"modified":"2023-04-27T14:45:02","modified_gmt":"2023-04-27T12:45:02","slug":"what-scientists-should-know-to-efficiently-use-the-scientific-computing-cluster","status":"publish","type":"post","link":"https:\/\/info.gwdg.de\/news\/what-scientists-should-know-to-efficiently-use-the-scientific-computing-cluster\/","title":{"rendered":"What scientists should know to efficiently use the Scientific Computing Cluster"},"content":{"rendered":"
Author: Dorothea Sommer<\/pre>\n
Introduction<\/h3>\n
Switching from executing code locally to running it on a compute cluster can be a daunting and sometimes also confusing experience. This article provides a brief overview of fundamental concepts to efficiently run code on the cluster. It is not<\/strong> a step-by-step guide on how to set up a particular environment or run a specific software. Rather, it gives an introductory explanation about concepts and ideas to keep in mind while using the cluster, particularly if you have little to no prior experience.<\/p>\n
Find also information regarding this article here.<\/a><\/p>\n
Overview<\/h3>\n
    \n
  1. General Workflow<\/li>\n
  2. Hardware Components<\/li>\n
  3. Requesting Hardware via Slurm<\/li>\n
  4. Cluster Connections<\/li>\n
  5. Using Environment Variables in Slurm<\/li>\n<\/ol>\n
    <\/a>General Workflow<\/h3>\n
    We will start with a simplified cluster concept and add details when they become relevant. For now, a cluster consists of some computing units (called nodes<\/strong>) that are connected to each other. There are some nodes reserved for login (green<\/strong> in Figure 1) and some reserved for the actual computations (blue<\/strong> in Figure 1).<\/p>\n
    \"\"<\/a><\/p>\n
    Once you logged into a frontend node (1<\/strong>), do not directly execute code that you would like to run. (If some users run heavy computations on the frontend nodes, other users will have problems to login in!). Instead, there is a scheduler<\/strong> installed on the cluster. This is a programme that coordinates who gets to calculate when on which nodes. The scheduler on the scientific compute cluster is called Slurm<\/a>. Thus, to run code, you prepare a Slurm script (2<\/strong>). It includes both which programme you would like to run and which computing resources (hardware) you need. Such a script is called a job<\/strong> script. It is taken by the Slurm controller and, after some waiting (3<\/strong>) you get the requested resources and your code is executed automatically. All nodes have access to a shared file system, here called shared scratch<\/strong>. You can save the output of your computations in this file system. As for many users, the most unintuitive part of this workflow is to understand which hardware resources they need, we will cover this topic next.<\/p>\n
    \n
    TAKEAWAY:<\/em><\/strong> Please schedule your programme as a Slurm job instead of blocking the frontend with heavy computations.<\/p>\n<\/blockquote>\n
    <\/a>Hardware Components<\/h3>\n
    So, which resources should you request? This depends on your computation, so for instance whether the code uses multithreading or multiprocessing. We have a brief overview<\/a> of the available hardware. However, to understand how<\/em> software components map to the hardware and then decide which you need, let us start with the an amazing hardware-software overview, which is adapted from Smilei<\/a> in Figure 2<\/p>\n
    \"\"<\/a><\/p>\n
    The software (bottom<\/strong> in Figure 2) consists of processes and threads (i.e., lines of execution). It can be mapped to hardware components (top<\/strong>), which are the physical elements of the cluster. The hardware elements consist of:<\/p>\n