Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Next revision Both sides next revision
en:services:application_services:jupyter:hpc [2020/04/03 11:10]
akhuziy [Installing additional Python modules]
en:services:application_services:jupyter:hpc [2020/07/03 10:45]
akhuziy [Installing additional Python modules]
Line 3: Line 3:
  
 **Note** that for using Jupyter-Hub on the Scientific Compute Cluster (SCC) you need a GWDG account which is activated for the use of the compute cluster. If your GWDG account is not yet activated you can do it by sending an informal email to <​hpc@gwdg.de>​. **Note** that for using Jupyter-Hub on the Scientific Compute Cluster (SCC) you need a GWDG account which is activated for the use of the compute cluster. If your GWDG account is not yet activated you can do it by sending an informal email to <​hpc@gwdg.de>​.
 +
 ===== How to use Jupyter-Hub on the SCC ===== ===== How to use Jupyter-Hub on the SCC =====
-Jupyter-Hub on the SCC can be used identically to the [[en:​services:​application_services:​jupyter:​start|Jupyter / Jupyter-Hub]] and currently supports ''​Python''​ and ''​R''​ kernels. After a successful authentication,​ there are options of spawning Jupyter notebooks available:+Jupyter-Hub on the SCC can be used identically to the [[en:​services:​application_services:​jupyter:​start|Jupyter / Jupyter-Hub]] and currently supports ''​Python''​ and ''​R''​ kernels. After a successful authentication,​ there are options of spawning Jupyter notebooks available:
   - GWDG HPC   - GWDG HPC
   - GWDG HPC with IPython Parallel   - GWDG HPC with IPython Parallel
 +  - GWDG HPC with Own Container
  
-If you just need a Jupyter notebook, then select the 1st option. If you want to use [[https://​ipython.org/​ipython-doc/​3/​parallel/​|IPython Parallel]] then select the 2nd option.+If you just need a Jupyter notebook, then select the 1st option. If you want to use [[https://​ipython.org/​ipython-doc/​3/​parallel/​|IPython Parallel]] then select the 2nd option. In case if you have your own Singularity container and want to use the notebook from that container, then select the 3rd option. ​
  
 IPython Parallel allows to increase computational resources and spawn compute workers on any nodes of the HPC cluster (not in the interactive queue) IPython Parallel allows to increase computational resources and spawn compute workers on any nodes of the HPC cluster (not in the interactive queue)
  
-A Jupyter notebook and IPython Parallel workers run as normal jobs in the cluster, ​therefore ​you will receive an email with the output ​after the jobs are finished +A Jupyter notebook and IPython Parallel workers run as normal jobs in the cluster
 + 
 +There are several options you can set on the spawning page to adjust the resourseslike number of cores and amount of memory. 
 + 
 +===== Options ===== 
 +**Job profile**: this option sets which notebook ​you want to use, normal one, with IPython Parallel or your own. 
 + 
 +**Singularity container location**: if you selected your own Container, then you have to provide ​the full path (or using ''​$HOME''​ variable) to the container you want to spawn. More on that further in the documentation. 
 + 
 +**Duration**:​ The duration of the job, note that after that time, the job will be killed and you should spawn the notebook once again. 
 + 
 +**Number of cores**: This option sets the number of cores accessible by the notebook. Note that cores are not for exclusive usage and might be shared if there are more notebooks spawned than available resources.  
 + 
 +**Amount of memory**: The option sets the amount of memory accessible by the notebook. Same as the amount of cores, memory might be shared if there are more notebooks than resources.  
 + 
 +**Notebook'​s Home directory**:​ You can provide some custom location for a default home of the Jupyter. This path will be a default path which will be opened when the notebook is spawned 
 ===== Resources ===== ===== Resources =====
 Jupyter notebooks in [[https://​jupyter-hpc.gwdg.de|Jupyter-Hub on HPC]] service are launched in the [[en:​services:​application_services:​high_performance_computing:​interactive_queue|interactive queue]] of the [[en:​services:​application_services:​high_performance_computing:​|High Performance Computing Cluster]]. ​ Jupyter notebooks in [[https://​jupyter-hpc.gwdg.de|Jupyter-Hub on HPC]] service are launched in the [[en:​services:​application_services:​high_performance_computing:​interactive_queue|interactive queue]] of the [[en:​services:​application_services:​high_performance_computing:​|High Performance Computing Cluster]]. ​
Line 21: Line 39:
  
 <WRAP center round important 90%> <WRAP center round important 90%>
-Jupyter notebooks start not in the root of your HOME directory but in the folder ''​~/​jupyterhub-gwdg''​. Place your notebooks and files in the corresponding folder.+Jupyter notebooks ​by default ​start not in the root of your HOME directory but in the folder ''​~/​jupyterhub-gwdg''​. Place your notebooks and files in the corresponding folder. Or set the **Home directory** option described above.
 </​WRAP>​ </​WRAP>​
- 
  
 Currently one session of Jupyter notebook can run maximum **8 hours**, after that it will be killed, but your files will stay intact. ​ Currently one session of Jupyter notebook can run maximum **8 hours**, after that it will be killed, but your files will stay intact. ​
Line 58: Line 75:
 You also can use self defined kernels and install conda environments on non-parallel notebook. Please refer to [[en:​services:​application_services:​jupyter:​start#​installation_of_additional_packages_and_environments_via_conda|Installing additional environments via conda]] You also can use self defined kernels and install conda environments on non-parallel notebook. Please refer to [[en:​services:​application_services:​jupyter:​start#​installation_of_additional_packages_and_environments_via_conda|Installing additional environments via conda]]
  
 +===== Running your own Singularity Container =====
 +You can build your own Singularity container with Jupyter notebook and run it. It will allow you to not only be independent of our Jupyter notebooks, but also easily spawn the same notebook in your local environment for development or tests. ​
 +    ​
 +Here is an example of the Singularity container you might use for your own Jupyter notebook:
 +    ​
 +<code bash>
 +Bootstrap: docker
 +From: ubuntu:​18.04
  
 +%post
 +
 +apt-get -y update
 +apt-get -y install python3-pip net-tools
 +apt-get -y install curl wget
 +
 +# Install conda and check the md5 sum provided on the download site
 +export CONDA_DIR=/​opt/​conda
 +export PATH=$CONDA_DIR/​bin:​$PATH
 +export MINICONDA_VERSION=4.7.12.1
 +
 +cd /tmp && \
 +wget --quiet https://​repo.continuum.io/​miniconda/​Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh
 +echo "​81c773ff87af5cfac79ab862942ab6b3 *Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh"​ | md5sum -c -
 +/bin/bash Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh -f -b -p $CONDA_DIR
 +rm Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh
 +$CONDA_DIR/​bin/​conda config --system --prepend channels conda-forge
 +$CONDA_DIR/​bin/​conda config --system --set auto_update_conda false
 +$CONDA_DIR/​bin/​conda config --system --set show_channel_urls true
 +$CONDA_DIR/​bin/​conda install --quiet --yes conda="​${MINICONDA_VERSION%.*}.*"​
 +conda update --all --quiet --yes
 +conda clean -tipsy
 +
 +conda install --quiet --yes \
 +    '​notebook=6.0.3'​ \
 +    '​jupyterhub=1.0.0'​ \
 +    '​jupyterlab=2.1.5'​
 +
 +
 +curl -sL https://​deb.nodesource.com/​setup_8.x | bash -
 +apt-get install -y nodejs
 +
 +jupyter labextension install @jupyterlab/​hub-extension
 +
 +%environment
 +
 +XDG_RUNTIME_DIR=""​
 +PATH=${PATH}:/​opt/​conda/​bin
 +</​code>​
 +    ​
 +You can extend that notebooks as much as you want. The important thing is to make ''​jupyterhub-singleuser''​ binary which comes with ''​jupyterhub''​ package available in PATH (done by last line). This binary is called when the notebook starts. Also it is better to use these versions of ''​notebook'',​ ''​jupyterhub''​ and ''​jupyterlab''​ for compatibility with the JupyterHub on SCC.
 +    ​
 +Unfortunately you cannot build containers in SCC. Please install [[https://​sylabs.io/​singularity|Singularity]] on your local machine, build the image and then [[en:​services:​application_services:​high_performance_computing:​transfer_data|transfer]] the image on SCC.
 +    ​
 +    ​
 +    ​
 +    ​