Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision
Previous revision
en:services:application_services:high_performance_computing:hail [2018/08/03 15:21]
ckoehle2 created
en:services:application_services:high_performance_computing:hail [2019/04/15 15:33]
ckoehle2 update reserved core count for interactive sessions
Line 23: Line 23:
 <​code>​ <​code>​
 #!/bin/bash #!/bin/bash
-#BSUB -q mpi +#SBATCH ​-p medium 
-#BSUB -+#SBATCH ​-
-#BSUB -R span[ptile=1] +#SBATCH ​--ntasks-per-node=1 
-#BSUB -01:00+#SBATCH ​-01:00:00
  
 lsf-spark-submit.sh $SPARK_ARGS lsf-spark-submit.sh $SPARK_ARGS
 </​code>​ </​code>​
-where ''​spark-submit''​ has been replaced by ''​lsf-spark-submit.sh''​ and ''​$SPARK_ARGS''​ are the submit arguments without the ''​--master''​ argument - this will be added automatically,​ depending on which cluster node the master has been launched on. Because of ''​-4''​ there are 4 workers ​in total and ''​-R span[ptile=1]''​ ensures that one worker per node is started.+where ''​spark-submit''​ has been replaced by ''​lsf-spark-submit.sh''​ and ''​$SPARK_ARGS''​ are the submit arguments without the ''​--master''​ argument - this will be added automatically,​ depending on which cluster node the master has been launched on. Because of ''​-4''​ there are 4 nodes in total and ''​<​nowiki>​--ntasks-per-node=1</​nowiki>​''​ ensures that one worker per node is started.
  
 ==== Interactive Sessions ==== ==== Interactive Sessions ====
 A Spark cluster to be used with Scala from the [[https://​spark.apache.org/​docs/​latest/​quick-start.html|interactive console]] can be spawned in a similar fashion, except we start an interactive LSF job and use the wrapper script ''​lsf-spark-shell.py''​ instead: A Spark cluster to be used with Scala from the [[https://​spark.apache.org/​docs/​latest/​quick-start.html|interactive console]] can be spawned in a similar fashion, except we start an interactive LSF job and use the wrapper script ''​lsf-spark-shell.py''​ instead:
 <​code>​ <​code>​
-bsub -int -4 -R span[ptile=1] -01:00 -ISs lsf-spark-shell.sh+srun -int -4 --ntasks-per-node=20 -01:00:00 lsf-spark-shell.sh
 </​code>​ </​code>​
 ===== Running Hail ===== ===== Running Hail =====
Line 71: Line 71:
 An LSF job running the ''​pyspark''​-based console for Hail can then be submitted as follows: An LSF job running the ''​pyspark''​-based console for Hail can then be submitted as follows:
 <​code>​ <​code>​
-bsub -int -4 -R span[ptile=1] -01:00 -ISs lsf-pyspark-hail.sh+srun -int -4 --ntasks-per-node=20 -01:00:00 lsf-pyspark-hail.sh
 </​code>​ </​code>​
 Once the console is running, initialize hail with the global Spark context ''​sc''​ in the following way: Once the console is running, initialize hail with the global Spark context ''​sc''​ in the following way: