Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
en:services:application_services:jupyter:start [2019/05/10 14:26]
bwegman1 [What is Jupyter / Jupyter-Hub?]
en:services:application_services:jupyter:start [2020/03/04 08:56] (current)
bbrauns [Install addition kernel with pipenv]
Line 3: Line 3:
 GWDG offers [[https://​jupyter.gwdg.de|Jupyter-Hub]] as a [[en:​services:​application_services:​jupyter:​beta|beta service]] for users of Python, Julia or R. GWDG offers [[https://​jupyter.gwdg.de|Jupyter-Hub]] as a [[en:​services:​application_services:​jupyter:​beta|beta service]] for users of Python, Julia or R.
  
 +<WRAP round important 60% >
 +Due to the limited scalability of the service jupyter.gwdg.de,​ it will be replaced by jupyter-cloud.gwdg.de on **March 31, 2020**. The functionality of the new system is identical and user can have their files transferred by visiting [[https://​jupyter-cloud.gwdg.de/​migrate|jupyter-cloud.gwdg.de/​migrate]].
 +
 +Further details and answers can be found at our [[en:​services:​application_services:​jupyter:​migration|information page]].
 +</​WRAP>​
 ===== What is Jupyter / Jupyter-Hub?​ ===== ===== What is Jupyter / Jupyter-Hub?​ =====
  
Line 9: Line 14:
 Jupyter-Hub is the portal users log in to and start and manage their notebooks and associated files. Jupyter-Hub is the portal users log in to and start and manage their notebooks and associated files.
  
-Please note that you can also use [[https://​info.gwdg.de/​docs/​doku.php?​id=en:​services:​application_services:​high_performance_computing:​jupyter|Jupyter on our SCC]] if you do need direct access to data on the SCC and have medium to high compute resource requirements.+Please note that you can also use [[https://​info.gwdg.de/​docs/​doku.php?​id=en:​services:​application_services:​jupyter:hpc|Jupyter on our SCC]] if you do need direct access to data on the SCC and have medium to high compute resource requirements.
  
 ===== How to use Jupyter / Jupyter-Hub?​ ===== ===== How to use Jupyter / Jupyter-Hub?​ =====
Line 40: Line 45:
 ==== Installing additional python modules ==== ==== Installing additional python modules ====
 Additional Python modules can be installed via the terminal and the Python package manager "​pip"​. To do this, a terminal must be opened via the menu "​New"​ -> "​Terminal"​. Afterwards <​code>​pip install --user <​module></​code>​ installs a new module in the home directory. Additional Python modules can be installed via the terminal and the Python package manager "​pip"​. To do this, a terminal must be opened via the menu "​New"​ -> "​Terminal"​. Afterwards <​code>​pip install --user <​module></​code>​ installs a new module in the home directory.
 +
 +=== Installing large python modules and disk space ===
 +
 +The installation of large Python modules like "​tensorflow"​ may fail with a message "No space left on device"​. This is caused by the temporary space under "/​tmp"​ being too small for pip to work the downloaded packages. The following steps use a temporary directory in the much larger user home directory for this one installation:​
 +
 +<code bash>
 +mkdir -v ~/​.user-temp
 +TEMP=~/​.user-temp pip install --user <​module>​
 +</​code>​
 +
 +Prefixing the installation with the TEMP variable makes pip use that location for this one installation.
 +
 +=== Notebook fails to start after package installation or update, error 500 ===
 +
 +If every notebook fails to start after a package installation or upgrade (error 500) the issue can be resolved by renaming the folder ''​.local'':​
 +<code bash>
 +mv -v .local/ .local.gwdg-disable
 +</​code>​
 +Afterwards the notebook server should be restarted. In the upper right corner click on "​Control Panel" - "Stop My Server"​ -> "My Server"​. This deactivates all additionally installed packages. The notebook should start again normally.
 +
 +==== Installation of additional packages and environments via Conda ====
 +
 +Management of software packages and environments with Conda requires a terminal session started from the notebook server. The terminal ist available after login via ''​New -> Terminal''​.
 +
 +Before working with ''​conda''​ commands the necessary conda functions need to be loaded (**mind the dot at the beginning!**):​
 +<code bash>
 +. /​opt/​conda/​etc/​profile.d/​conda.sh
 +</​code>​
 +
 +=== Creating a new environment ===
 +
 +The following describes the creation of a new, simple environment ''​wikidoku'',​ the installation of the package ''​jinja2''​ as an example and how to make the environment available in the kernel selection of the notebook.
 +
 +Creating and activating the environment:​
 +<code bash>
 +conda create -y --prefix ./wikidoku
 +conda activate ./wikidoku
 +</​code>​
 +
 +As an example the package ''​jinja2''​ will be installed next. This is the step to install the desired software packages from various Conda channels.
 +<code bash>
 +conda install -y jinja2
 +</​code>​
 +
 +Next the new environment will be registered with the notebook. Terms for ''​--name''​ and ''​--display-name''​ (optional) can be chosen freely, the later being shown in the kernel selection of the notebook. The second command lists all known environments. ​ The last command exists the environment.
 +<code bash>
 +python3 -m ipykernel install --user --name wikidoku --display-name "​Python (wikidoku)"​
 +jupyter kernelspec list
 +conda deactivate
 +</​code>​
 +
 +If installation of the kernel fails with the message ''/​usr/​bin/​python:​ No module named ipykernel''​ the additional package ''​jupyter''​ needs to be installed and installation of the kernel repeated:
 +<code bash>
 +python3 -m pip install jupyter
 +</​code>​
 +
 +=== Selecting the new environment ===
 +
 +== Restarting of the notebook server ==
 +
 +After installation of a new environment it is recommended to restart the notebook server. Leave all existing terminals and close all open notebooks. In the Jupyter overview page click on ''​Control Panel''​ and stop the server by clicking ''​Stop My Server''​ and restart it with ''​Start My Server''​.
 +
 +Via ''​New ->''​ a new notebook with the new environment can be started. In existing notebooks the kernel can be changed after starting the notebook and selecting ''​Kernel -> Change Kernel''​.
 +
 +=== Installing additional kernels in an Conda environment ===
 +
 +Installing a new, independent Python kernel für the current environment is possible. As an example an older Python 2.7 kernel will be installed next.
 +
 +A new environment needs to be created and activated as per the steps above.
 +Next follows the installation of the kernel, the ''​jupyter''​ module for that kernel and finally the kernel is made available for selection in the kernel list.
 +<code bash>
 +conda install -y python=2.7
 +python3 -m pip install jupyter
 +python3 -m ipykernel install --user --name oldpython --display-name "​Python 2.7 (oldpython)"​
 +</​code>​
 +
 +The new kernel is now available for new and existing notebooks after restarting the notebook server.
 +The current kernel version can be queried from within Python:
 +<code python>
 +import sys
 +print (sys.version)
 +</​code>​
 +
 +=== Removing an environment ===
 +
 +In order to remove an environment it has to be de-registered from the notebook server and then its files removed (optional but recommended). We list the installed kernels, de-register and remove the environment'​s files:
 +
 +<code bash>
 +jupyter kernelspec list
 +jupyter kernelspec remove wikidoku
 +rm -rf ./wikidoku
 +</​code>​
  
 ==== Installing additional R packages==== ==== Installing additional R packages====
Line 100: Line 197:
   * Stop and restart server via control panel   * Stop and restart server via control panel
   * Afterwards "​projectname"​ is usable as new kernel ​   * Afterwards "​projectname"​ is usable as new kernel ​
 +
 +==== Install additional julia packages with an extra kernel ====
 +** !experimental! **
 +
 +The jupyter docker stacks image sets the variable JULIA_DEPOT_PATH to the path /opt/julia. However, this is volatile, since only the home directory is kept persistent. The following describes the installation of a new julia kernel, which has its package directory pointed to the home directory:
 +
 +<​code>​
 +Start terminal  ​
 +Temporarily change julia package directories:  ​
 +
 +export JULIA_DEPOT_PATH=/​home/​jovyan/​.julia-depot
 +export JULIA_PKGDIR=/​home/​jovyan/​.julia-depot
 +
 +Create directory for custom packages and new julia kernel:
 +
 +> mkdir /​home/​jovyan/​.julia-depot
 +> julia
 +julia > # switch to pkg with '​]'​ character
 +pkg > add IJulia # switch back to julia with CTRL+C
 +julia > using IJulia
 +    ​installkernel("​My-Julia-kernel",​ env=Dict("​JULIA_DEPOT_PATH"​=>"/​home/​jovyan/​.julia-depot"​))
 +Restart notebook server
 +Create new notebook with "​My-Julia-kernel"​ kernel
 +Add package example: ​
 +
 +using Pkg
 +Pkg.add("​DataFrames"​)
 +</​code>​
 +
 +
 +
  
 ===== More information about the service being in beta ===== ===== More information about the service being in beta =====
  
 To learn more about the practical implications of a [[en:​services:​application_services:​jupyter:​beta|beta service]] see the explanation of a [[en:​services:​application_services:​jupyter:​beta|beta service]]. To learn more about the practical implications of a [[en:​services:​application_services:​jupyter:​beta|beta service]] see the explanation of a [[en:​services:​application_services:​jupyter:​beta|beta service]].