Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
en:services:application_services:high_performance_computing:hail [2019/04/09 10:37]
tehlers [Submitting Spark Applications]
en:services:application_services:high_performance_computing:hail [2019/04/15 15:33] (current)
ckoehle2 update reserved core count for interactive sessions
Line 30: Line 30:
 lsf-spark-submit.sh $SPARK_ARGS lsf-spark-submit.sh $SPARK_ARGS
 </​code>​ </​code>​
-where ''​spark-submit''​ has been replaced by ''​lsf-spark-submit.sh''​ and ''​$SPARK_ARGS''​ are the submit arguments without the ''​--master''​ argument - this will be added automatically,​ depending on which cluster node the master has been launched on. Because of ''​-N 4''​ there are 4 nodes in total and ''​--ntasks-per-node=1''​ ensures that one worker per node is started.+where ''​spark-submit''​ has been replaced by ''​lsf-spark-submit.sh''​ and ''​$SPARK_ARGS''​ are the submit arguments without the ''​--master''​ argument - this will be added automatically,​ depending on which cluster node the master has been launched on. Because of ''​-N 4''​ there are 4 nodes in total and ''​<​nowiki>​--ntasks-per-node=1</​nowiki>​''​ ensures that one worker per node is started.
  
 ==== Interactive Sessions ==== ==== Interactive Sessions ====
 A Spark cluster to be used with Scala from the [[https://​spark.apache.org/​docs/​latest/​quick-start.html|interactive console]] can be spawned in a similar fashion, except we start an interactive LSF job and use the wrapper script ''​lsf-spark-shell.py''​ instead: A Spark cluster to be used with Scala from the [[https://​spark.apache.org/​docs/​latest/​quick-start.html|interactive console]] can be spawned in a similar fashion, except we start an interactive LSF job and use the wrapper script ''​lsf-spark-shell.py''​ instead:
 <​code>​ <​code>​
-bsub -int -4 -R span[ptile=1] -01:00 -ISs lsf-spark-shell.sh+srun -int -4 --ntasks-per-node=20 -01:00:00 lsf-spark-shell.sh
 </​code>​ </​code>​
 ===== Running Hail ===== ===== Running Hail =====
Line 71: Line 71:
 An LSF job running the ''​pyspark''​-based console for Hail can then be submitted as follows: An LSF job running the ''​pyspark''​-based console for Hail can then be submitted as follows:
 <​code>​ <​code>​
-bsub -int -4 -R span[ptile=1] -01:00 -ISs lsf-pyspark-hail.sh+srun -int -4 --ntasks-per-node=20 -01:00:00 lsf-pyspark-hail.sh
 </​code>​ </​code>​
 Once the console is running, initialize hail with the global Spark context ''​sc''​ in the following way: Once the console is running, initialize hail with the global Spark context ''​sc''​ in the following way: