![So what is Quest like?
SSH as:
[netid]@quest.it.northwestern.edu](https://image.slidesharecdn.com/quest-overview-150205103618-conversion-gate01/85/Quest-overview-9-320.jpg)
![So what is Quest like?
SSH as:
[netid]@quest.it.northwestern.edu](https://image.slidesharecdn.com/quest-overview-150205103618-conversion-gate01/85/Quest-overview-10-320.jpg)
![So what is Quest like?
You will have a user directory, it is located at:
/home/[netid]
Your user directory is limited to 50 GB and not backed up](https://image.slidesharecdn.com/quest-overview-150205103618-conversion-gate01/85/Quest-overview-11-320.jpg)
Quest overview
- 1. Quest Cluster Computing for the Amaral lab Adam Pah 8.21.14
- 2. What is Quest • Quest is the computing cluster that is administered by Northwestern • Quest is shared by all of Northwestern • There are two ways to get time on Quest: • Get an allocation of hours • Purchase hardware to be added to Quest • We purchased hardware to be added to Quest
- 3. What did we purchase • We purchased 9 nodes to be added to Quest • Since we purchased hardware we have a project group that will be everyone in the lab • Each node has: • 12 cores per node • 128 GB RAM per node • So we can run 108 simultaneous jobs as a project group
- 4. How is Quest set up to handle all these users? There are log-in nodes
- 5. How is Quest set up to handle all these users? There are log-in nodes And then compute nodes that actually execute jobs
- 6. How is Quest set up to handle all these users? There are log-in nodes And then compute nodes that actually execute jobs Jobs are submitted on log-in nodes to a “scheduler”
- 7. How is Quest set up to handle all these users? There are log-in nodes And then compute nodes that actually execute jobs Jobs are submitted on log-in nodes to a “scheduler” The “scheduler” finds a free compute node and runs the job
- 8. How is Quest set up to handle all these users? There are log-in nodes And then compute nodes that actually execute jobs Jobs are submitted on log-in nodes to a “scheduler” The “scheduler” finds a free compute node and runs the job For us, it will find a free spot in the hardware that we purchased
- 9. So what is Quest like? SSH as: [netid]@quest.it.northwestern.edu
- 10. So what is Quest like? SSH as: [netid]@quest.it.northwestern.edu
- 11. So what is Quest like? You will have a user directory, it is located at: /home/[netid] Your user directory is limited to 50 GB and not backed up
- 12. So what is Quest like? We also have a project directory, it is at: /projects/b1022 Our project directory is limited to 432 GB and it is shared by *EVERYONE* in the lab
- 13. So how do I set up my user on Quest? Quest isn’t the same as every other computer we have in the lab and there are some adjustments I already created a bashrc, some shortcuts, and a script to make it as much like using phoenix as possible So the first thing to do is get the bashrc that the lab will use
- 14. Getting the lab bashrc Copyable: $ cd ~ $ curl https://gist.githubusercontent.com/adamrpah/d0133fc9b36ad5854efe/raw/ aa282448ae924592452c952f25632a27361deff6/amaral_quest_bashrc > .bashrc $ source ~/.bashrc This puts LabTools on your path (LabTools is our projects folder) It also sets up aliases so mercurial and python will work
- 15. What python are we using? Quest has a number of prepared “modules” for researchers to use
- 16. What python are we using? Quest has a number of prepared “modules” for researchers to use These “modules” are prepackaged programming environments for a number of languages/ use cases
- 17. What python are we using? Quest has a number of prepared “modules” for researchers to use These “modules” are prepackaged programming environments for a number of languages/ use cases We’re using Anaconda
- 18. What python are we using? This anaconda environment already has a lot of what we use installed
- 19. What python are we using? This anaconda environment already has a lot of what we use installed I created a virtual environment and added in the few missing packages that we use
- 20. What python are we using? This anaconda environment already has a lot of what we use installed I created a virtual environment and added in the few missing packages that we use To use this python on a log-in node or test to make sure a package is installed just do: Note that the “actpy” command is unique and requires our lab bashrc
- 21. What if I need a package installed? Let myself, Nick, Joao, or June know and we’ll take care of it to the best of our ability Anaconda uses pip to install packages, so almost all packages are good to go However, this means that any package that won’t install cleanly with pip is now suspect! We can ask the Quest team to take care of it, but that will take time
- 22. How do I get files to Quest? Two choices: 1) check out a repository
- 23. How do I get files to Quest? Two choices: 1) check out a repository 2) SCP files
- 24. How do I submit a job to Quest to run? Actually, it’s pretty simple. I wrote a script that is in LabTools that abstracts away any differences on Quest 1) Navigate to directory with python code
- 25. How do I submit a job to Quest to run? Actually, it’s pretty simple. I wrote a script that is in LabTools that abstracts away any differences on Quest 1) Navigate to directory with python code 2) Submit the job with the “generate_qsub” script
- 26. How do I submit a job to Quest to run? Actually, it’s pretty simple. I wrote a script that is in LabTools that abstracts away any differences on Quest 1) Navigate to directory with python code 2) Submit the job with the “generate_qsub” script 3) Check that the job is running with “qstat”
- 27. How do I submit a job to Quest to run? Actually, it’s pretty simple. I wrote a script that is in LabTools that abstracts away any differences on Quest 1) Navigate to directory with python code 2) Submit the job with the “generate_qsub” script 3) Check that the job is running with “qstat” 4) Enjoy!
- 28. But what did all those options do? The basics of that script are explained here: Torque Job Submission Options: • —msub Quest uses the Moab scheduler, this makes a moab submission file. This option is MANDATORY when using Quest • —execute With this option the generate_qsub script will submit the job to the scheduler automatically • —temporary This option will remove the submission scripts
- 29. But what did all those options do? There are other options (that have Phoenix in mind) but the other important one is: • —kwargs This option takes in command arguments that your code uses
- 30. That’s it! Life’s good!