{"id":23632,"date":"2024-02-01T09:22:17","date_gmt":"2024-02-01T08:22:17","guid":{"rendered":"https:\/\/info.gwdg.de\/news\/?p=23632"},"modified":"2024-02-01T09:22:17","modified_gmt":"2024-02-01T08:22:17","slug":"best-practices-for-machine-learning-with-hpc","status":"publish","type":"post","link":"https:\/\/info.gwdg.de\/news\/best-practices-for-machine-learning-with-hpc\/","title":{"rendered":"Best Practices for Machine Learning with HPC"},"content":{"rendered":"

The GWDG offers data scientists various services and training courses to support them in their work throughout their entire workflow. As the success of a machine learning project often depends on the available computing resources, the working group „Computing“ operates various HPC systems with suitable accelerators to train deep learning models. This article explains model training using the highly energy-efficient HPC cluster „Grete“. The example is an excerpt from the course „Deep learning with GPU cores,“ offered every six months.<\/p>\n

Introduction<\/h2>\n

The GWDG offers several services and trainings to support researchers in their machine-learning projects1<\/a><\/sup>. Deep Learning success is heavily compute-bound2<\/a><\/sup>3<\/a><\/sup>; accelerators integrated into our HPC systems can help overcome this bottleneck. The GWDG offers several HPC systems4<\/a><\/sup> with modern GPU nodes and other accelerators5<\/a><\/sup> accessible to different user groups6<\/a><\/sup>. If you are wondering whether you have access to our systems, look at our science domains blog article from June 6<\/a><\/sup>. Additionally, the KISSKI project 7<\/a><\/sup> is not limited to researchers, but small and medium-sized enterprises are highly welcome.<\/p>\n

Further, we aim to support different science domains in the scope of the NHR alliance 8<\/a><\/sup>. Here we focus on researchers in the fields of artificial intelligence 9<\/a><\/sup>, digital humanities, bioinformatics 10<\/a><\/sup>, and forest science 11<\/a><\/sup>.<\/p>\n

<\/a>Machine learning workflow<\/h2>\n

A typical machine-learning workflow can be seen below (see figure 1). However, this article will only focus on model development and evaluation best practices. Regardless, the GWDG offers services to support you in every step of this workflow 1<\/a><\/sup>. If you need our assistance with anything, feel free to contact us.<\/p>\n

\"\"<\/a>
Figure 1: „A typical machine learning workflow involves data collection, pre-processing, dataset building, model training and refinement, evaluation, deployment, and continuous monitoring and updating. This process is iterative and may require revisiting previous steps based on the results at later stages[^1][^2][^4][^5].“, explained by a transformer model deployed by Perplexity [^perplexity].<\/p>\n

The figure is adapted from https:\/\/fullstackdeeplearning.com\/course\/2022\/<\/a> .<\/p>\n<\/figcaption><\/figure>\n

<\/a>HPC Cluster<\/h3>\n

A high-performance computing (HPC) cluster consists of different essential components, which are critical to understand to use its resources efficiently. Here, we will briefly explain the basic terminology. If you are interested in a more detailed article on this topic, please read our science domain blog article „What scientists should know to efficiently use the Scientific Computing Cluster“ 12<\/a><\/sup>.<\/p>\n

First, a user can access the login node of the HPC cluster via an SSH (Secure Shell Protocol) connection (1 in figure 2).<\/p>\n

In the example workflow below, you can see that glogin9<\/code> is the login node of our GPU system Grete13<\/a><\/sup>. On the login node, jobs are submitted with the job scheduler Slurm14<\/a><\/sup> (2 in figure 2) A job on SLURM is a combination of which program you would like to run and which computing resources (hardware) you need. Compute nodes take care of the actual computing of your jobs, similar to your personal computer. Those compute nodes consist of cores and memory (RAM). Cores are processing units, such as CPU or GPU. Each node can have a different number of cores. Each node also has its own (temporary) random-access memory (RAM) used for temporary computations. Various storage systems with different characteristics help to handle and organize data efficiently (3 in figure 2). While using the shared scratch<\/code> for your computation has great results, it is not great for long-term storage. This type of file system has no backup! However, your home folder has a backup. As indicated in the figure by the connection to the tape archive (right center). These are literal tapes that, after being written, are stored as physical components not connected to any electricity. Lastly, connecting S3 buckets to the whole cluster (bottom right) is also possible.<\/p>\n

\"\"<\/a>
Figure 2: Schematic visualization of an HPC system, similar to the system Grete. On the left is the user and the front nodes that are used for login. In the center, the compute nodes handle the computation of the submitted jobs. Those are connected with different storage systems on the right.<\/p>\n

The figure was taken from our deep learning with GPU course offered every half year.15<\/a><\/sup><\/p>\n<\/figcaption><\/figure>\n

<\/a>Coding<\/h2>\n

In order to use the HPC resources, you will need to write code and develop programs. The programming language depends on the target application. For machine learning and deep learning, Python is often the best choice due to its moderate learning curve and availability of comprehensive packages and frameworks, such as PyTorch, JAX (with Flux) or Tensorflow.<\/p>\n

<\/a>Tools and Frameworks<\/h3>\n

For development, we recommend VSCode16<\/a><\/sup> as it supports many features and programming languages including Python through its vast selection of extensions17<\/a><\/sup>. VSCode can also be configured to connect to the cluster directly which makes development simple and straightforward18<\/a><\/sup>.<\/p>\n

When working on a Python project, it is best to use a virtual environment or container with a tool such as conda<\/code>, singularity<\/code> or apptainer<\/code>19<\/a><\/sup>, to ensure reproducibility and robustness of the code, as packages are constantly updated and your code may require specific versions.<\/p>\n

There are several widely-used frameworks for deep learning in Python. Three more common ones are PyTorch, TensorFlow, and JAX. Tensorflow is the first major deep learning framework and was developed by Google in 2015. PyTorch was developed in 2016 by Facebook, and is currently the most dominant framework in academia winning 77% of the competitions. JAX20<\/a><\/sup> is a relatively new framework developed by Google that is aimed towards a more general auto-differentiation and vectorisation framework.<\/p>\n

For most cases, we recommend using PyTorch.<\/p>\n

<\/a>Documentation and Code Handling<\/h3>\n

Code should be well-documented and clean; comments and README files should be included when necessary and repetitive code must be converted into functions and packages. It is also recommended to treat raw data as immutable; build the code around the data instead of editing the raw data itself. Cookiecutter is a tool that can get you started with a decent project structure and has some good recommendations as well21<\/a><\/sup><\/p>\n

It is crucial to use a Version Control System (VCS) such as git to avoid accidents and disasters when altering code and in general to gain better access and control over the code history. GWDG offers a GitLab server for this purpose. A good example is the deep learning with GPU workshop repository located in GWDG’s own GitLab server15<\/a><\/sup>.<\/p>\n

<\/a>Job Scheduling with SLURM<\/h3>\n

Running complex programs is forbidden on the login nodes. It is important to run complex tasks, e.g. data processing, model training and testing, only on the compute nodes. As mentioned before, the login nodes are strictly for low-cost tasks and job submission. You must use our job scheduler slurm to run jobs on the various nodes in the cluster. When submitting a job via slurm, you must set the job configuration, e.g. requested partition\/node and GPUs, and you may also add additional settings such as the amount of RAM, time limit, email address to be notified when the job is complete, and a location to store the job’s log file produced by slurm. It is recommended to specify a location to store the log file and regularly check it for any issues. An overview of the configurations in SLURM can be found in our documentation 22<\/a><\/sup>.<\/p>\n

During model development, you may want to run interactive jobs in the terminal with srun<\/code>, e.g., for testing purposes. In this case it is better to use nodes in partitions designated for this use case such as grete-interactive<\/code>. Finally, you may train the model by writing a slurm batch script and running it with sbatch<\/code> on a dedicated GPU node, e.g. on partition grete<\/code>. 23<\/a><\/sup><\/p>\n

You must also make sure that the requested node configuration is available and suitable for the job. For example, during development if the job is more likely to fail, it is better to set a low time limit and run the job on a test node if applicable. Similarly, the number of nodes and resources must be sufficient for your job while not being too excessive. It is better to split a long job into smaller jobs to avoid time limit issues and potentially wasting resources. To chain consecutive jobs with automated start and easier checkpointing we recommend snakemake24<\/a><\/sup>.<\/p>\n

<\/a>Monitoring<\/h2>\n

While your program is running, it is important to monitor the resource utilization on the nodes to identify bottlenecks, errors, and possible improvements.<\/p>\n

<\/a>CPU vs GPU<\/h3>\n

CPUs and GPUs are both powerful processing hardware that can perform complex calculations. Figure 3 depicts the inner structure of CPUs and GPUs. While CPUs are capable of handling a larger variety of instructions, GPUs are more efficient when it comes to specific simple instructions. It is important to know when using a GPU improves performance and when it does not.<\/p>\n

\"\"<\/a>
Figure 3. Internal comparison between a CPU and a GPU. This figure was taken from https:\/\/docs.nvidia.com\/cuda\/cuda-c-programming-guide\/<\/a><\/p>\n<\/figcaption><\/figure>\n

A thread is a series of sequential operations. A CPU is capable of processing multiple threads performing different tasks simultaneously. GPUs, on the other hand, are generally suitable for tasks that run the same operation on a large number of elements, i.e., running many similar threads at the same time.<\/p>\n

\"\"<\/a><\/figure>\n

Figure 4. Role of CPU and GPU in typical deep learning task.<\/p>\n

Figure 4 depicts a typical deep learning task running on a GPU node. The CPU is responsible for starting and running the program, beginning by initiliazing the model and storing it in the RAM. We would now like to take advantage of the GPU’s fast processing power in order to train the model, however, the GPU cannot directly access data stored in the RAM. Instead, it has its own memory (usually called VRAM, short for Video-RAM), which can vary in size depending on the GPU, but is typically smaller than the RAM.<\/p>\n

During training, the model parameters and data must be copied from the RAM into the GPU’s VRAM. The GPU can then feed the input data into the model, compute the result, and update the model. The result may then be transferred back to the CPU for further processing.<\/p>\n

Transferring data between the CPU and GPU is not immediate, and the time it takes should be taken into account. If the processing task is too simple, running it on a GPU would have little advantage over a CPU. To make full use of a GPU, you must ensure that lots of computation is performed on the GPU and the amount of data transfer is low in comparison. This can be verified using our recommended monitoring tools.<\/p>\n

<\/a>CPU and RAM Usage<\/h3>\n

It is possible to view the hardware resource usage of a node by simply logging into it and running top<\/code> (for CPU nodes) or module load nvitop<\/code> then nvitop<\/code> (for GPU nodes) in the terminal.<\/p>\n

\"\"<\/a>
Figure 5. Example output from `nvitop`.<\/p>\n

A typical output of nvitop<\/code> is shown in Figure 5. MEM depicts the amount of allocated GPU memory and UTL shows the GPU utilization. A constant low utilization of the GPU means that the GPU is mostly idle and waiting for more data.<\/p>\n<\/figcaption><\/figure>\n

The GPU memory should be large enough to support the model and batch of data simultaneously. There are different types of GPUs available on the cluster which can be selected depending on the use case. For newer GPUs, you may run your job on a GPU slice using NVIDIA-MIG when the full GPU power is not required. An overview of the available GPUs and usage can be found here: https:\/\/www.hlrn.de\/doc\/display\/PUB\/GPU+Usage<\/a><\/p>\n

<\/a>Example<\/h3>\n

An example of deep learning with GPU on the cluster and a recorded video series is available. 15<\/a><\/sup> 25<\/a><\/sup><\/p>\n

\n
    \n
  1. \n

    https:\/\/gwdg.de\/en\/services\/<\/a> \u21a9<\/a> \u21a92<\/sup><\/a><\/p>\n<\/li>\n

  2. \n

    https:\/\/openai.com\/research\/ai-and-compute#addendum<\/a> \u21a9<\/a><\/p>\n<\/li>\n

  3. \n

    https:\/\/academy.gwdg.de\/academy<\/a> \u21a9<\/a><\/p>\n<\/li>\n

  4. \n

    https:\/\/gwdg.de\/hpc\/systems\/<\/a> \u21a9<\/a><\/p>\n<\/li>\n

  5. \n

    Both graphcore systems and neuromorphic chips are being acquired as part of the KISSKI project. \u21a9<\/a><\/p>\n<\/li>\n

  6. \n

    https:\/\/gitlab-ce.gwdg.de\/hpc-team-public\/science-domains-blog\/-\/blob\/main\/20230623_hpc-access.md?ref_type=heads<\/a> \u21a9<\/a> \u21a92<\/sup><\/a><\/p>\n<\/li>\n

  7. \n

    https:\/\/kisski.gwdg.de\/<\/a> \u21a9<\/a><\/p>\n<\/li>\n

  8. \n

    https:\/\/www.nhr-verein.de\/en<\/a> \u21a9<\/a><\/p>\n<\/li>\n

  9. \n

    https:\/\/gwdg.de\/en\/community-pages\/ai-intro\/<\/a> \u21a9<\/a><\/p>\n<\/li>\n

  10. \n

    https:\/\/gwdg.de\/en\/community-pages\/bioinfo-intro\/<\/a> \u21a9<\/a><\/p>\n<\/li>\n

  11. \n

    https:\/\/gwdg.de\/en\/community-pages\/forestry-intro\/<\/a> \u21a9<\/a><\/p>\n<\/li>\n

  12. \n

    https:\/\/gitlab-ce.gwdg.de\/hpc-team-public\/science-domains-blog\/-\/blob\/main\/20230417_cluster-practical.md<\/a> \u21a9<\/a><\/p>\n<\/li>\n

  13. \n

    https:\/\/www.hlrn.de\/doc\/display\/PUB\/GPU+Usage<\/a> \u21a9<\/a><\/p>\n<\/li>\n

  14. \n

    https:\/\/slurm.schedmd.com<\/a> \u21a9<\/a><\/p>\n<\/li>\n

  15. \n

    https:\/\/gitlab-ce.gwdg.de\/dmuelle3\/deep-learning-with-gpu-cores<\/a> \u21a9<\/a> \u21a92<\/sup><\/a> \u21a93<\/sup><\/a><\/p>\n<\/li>\n

  16. \n

    https:\/\/code.visualstudio.com\/<\/a> \u21a9<\/a><\/p>\n<\/li>\n

  17. \n

    https:\/\/marketplace.visualstudio.com\/VSCode<\/a> \u21a9<\/a><\/p>\n<\/li>\n

  18. \n

    https:\/\/info.gwdg.de\/news\/en\/configuring-vscode-to-access-gwdgs-hpc-cluster\/<\/a> \u21a9<\/a><\/p>\n<\/li>\n

  19. \n

    https:\/\/gitlab-ce.gwdg.de\/hpc-team-public\/science-domains-blog\/-\/blob\/main\/20230907_python-apptainer.md?ref_type=heads<\/a> \u21a9<\/a><\/p>\n<\/li>\n

  20. \n

    https:\/\/jax.readthedocs.io\/en\/latest\/index.html<\/a> \u21a9<\/a><\/p>\n<\/li>\n

  21. \n

    https:\/\/drivendata.github.io\/cookiecutter-data-science\/<\/a> \u21a9<\/a><\/p>\n<\/li>\n

  22. \n

    https:\/\/www.hlrn.de\/doc\/display\/PUB\/Slurm<\/a> \u21a9<\/a><\/p>\n<\/li>\n

  23. \n

    https:\/\/www.hlrn.de\/doc\/display\/PUB\/GPU+Usage<\/a> \u21a9<\/a><\/p>\n<\/li>\n

  24. \n

    https:\/\/snakemake.github.io\/<\/a> \u21a9<\/a><\/p>\n<\/li>\n

  25. \n

    https:\/\/www.youtube.com\/playlist?list=PLvcoSsXFNRblM4AG5PZwY1AfYEW3EbD9O<\/a> \u21a9<\/a><\/p>\n<\/li>\n<\/ol>\n<\/section>\n

    Author<\/h3>\n

    Ali Doost Hosseini<\/a> | Hauke Kirchner<\/a> | Dorothea Sommer<\/p>\n","protected":false},"excerpt":{"rendered":"

    The GWDG offers data scientists various services and training courses to support them in their work throughout their entire workflow. As the success of a machine learning project often depends on the available computing resources, the working group „Computing“ operates various HPC systems with suitable accelerators to train deep learning models. This article explains model … Weiterlesen<\/a><\/p>\n","protected":false},"author":166,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1,133,129],"tags":[],"class_list":["post-23632","post","type-post","status-publish","format-standard","hentry","category-alle","category-kuenstliche-intelligenz","category-wissenschaftliche-domaenen"],"_links":{"self":[{"href":"https:\/\/info.gwdg.de\/news\/wp-json\/wp\/v2\/posts\/23632","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/info.gwdg.de\/news\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/info.gwdg.de\/news\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/info.gwdg.de\/news\/wp-json\/wp\/v2\/users\/166"}],"replies":[{"embeddable":true,"href":"https:\/\/info.gwdg.de\/news\/wp-json\/wp\/v2\/comments?post=23632"}],"version-history":[{"count":4,"href":"https:\/\/info.gwdg.de\/news\/wp-json\/wp\/v2\/posts\/23632\/revisions"}],"predecessor-version":[{"id":23647,"href":"https:\/\/info.gwdg.de\/news\/wp-json\/wp\/v2\/posts\/23632\/revisions\/23647"}],"wp:attachment":[{"href":"https:\/\/info.gwdg.de\/news\/wp-json\/wp\/v2\/media?parent=23632"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/info.gwdg.de\/news\/wp-json\/wp\/v2\/categories?post=23632"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/info.gwdg.de\/news\/wp-json\/wp\/v2\/tags?post=23632"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}