SlideShare a Scribd company logo

How to save the environment

Aaron Zauner, profile picture
Aaron Zauner

The document discusses the use of environment modules in high-performance computing (HPC) to manage software dependencies and versions effectively. It highlights the history, benefits, and implementations of environment modules, particularly focusing on 'lmod' as a modern tool for module management. Additionally, the document critiques certain version management tools like RVM for their complexity and environmental impacts.

Technology
1 of 36
Downloaded 18 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
How to save the environment ..and get rid of virtualenv, rvm, pythonbrew, rbenv, pythonz (...) FOSDEM 2014 - Brussels, Belgium, Old World Earth, Orion–Cygnus Arm, Milky Way Aaron Zauner azet@azet.org @a_z_e_t
`whoami` Self-employed engineer currently involved in IT-Infrastructure, HPC and security stuff primarily working for biology institutes of the austrian academy of sciences Used to do Web/Mail and Datacenter Operations Front/Backend Dev. Security auditing FOSS/Community (BSD) […]
Slide intentionally left blank for your enjoyment.
Text ‘Environment Modules’ First officially published at LISA V back in 1991 by John L. Furlani
Environment Modules In use ever since in High Performance and Scientific Computing Ease of administration Ease of use
Environment Modules by most just refered to as “Modules” originally intended to enable multiple users to have software in different versions available traditionally most HPC environments do not use system packages special compilers, toolchains, math libraries, optimisation, interconnect-specifics […] and often scientific software is not to be found in $distro repo (because it’s crap || not used outside the sci. community) somewhat like Gentoo or Arch - but a lot more cumbersome for sysadmins and users a system to manage multiple versions of the same software with different toolchains is needed that is easy to maintain and easy for users (developers and scientists, usually) to learn and use
Environment Modules “modulefiles” dynamically set an environment specific to a software may load/unload other modulefiles if there is a dependency present examples include but are not limited to PATH MANPATH LD_LIBRARY_PATH LD_PRELOAD
Environment Modules (with Lmod - more on that later) ! ! whatis("Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. - Homepage: http://git-scm.com/“) ! conflict("git") ! prepend_path(“LD_LIBRARY_PATH","/software/git/1.8.2/lib") prepend_path("LD_LIBRARY_PATH","/software/git/1.8.2/lib64") prepend_path(“MANPATH","/software/git/1.8.2/share/man") ! […]
Environment Modules: Implementations http://xkcd.com/927/ Creative Commons Attribution-NonCommercial 2.5 License
Environment Modules: Implementations “Classical Modules” (Furlani et. al.): C/Tcl still in use at arguably most HPC sites modules.sourceforge.net ! modules-tcl (Kent Mein of UMN): Tcl http://sourceforge.net/p/modules/modules-tcl Cmod (Per Cederqvist et. al.): C http://www.lysator.liu.se/cmod/ Lmod (Robert McLay of TACC): Lua https://www.tacc.utexas.edu/tacc-projects/lmod similar systems: ! SoftEnv (by and in use at Argonne National Lab): Csh http://www.mcs.anl.gov/hs/software/systems/ softenv/softenv-admin.html dotkit (by and in use at Lawrence Livermore National Lab): *Shell https://computing.llnl.gov/? set=jobs&page=dotkit
Environment Modules: Implementations “Classical Modules” (Furlani et. al.): C/Tcl still in use at arguably most HPC sites modules.sourceforge.net ! modules-tcl (Kent Mein of UMN): Tcl http://sourceforge.net/p/modules/modules-tcl Cmod (Per Cederqvist et. al.): C http://www.lysator.liu.se/cmod/ Lmod (Robert McLay of TACC): Lua https://www.tacc.utexas.edu/tacc-projects/lmod similar systems: ! SoftEnv (by and in use at Argonne National Lab): Csh http://www.mcs.anl.gov/hs/software/systems/ softenv/softenv-admin.html dotkit (by and in use at Lawrence Livermore National Lab): *Shell https://computing.llnl.gov/? set=jobs&page=dotkit
Modules in the 21st century: Lmod I’ll focus on Lmod from now on, though most things will work with other systems as well Very actively maintained by one of the most awesome maintainers we’ve seen in an open-source project Active community Cool new features: automatic swapping of modules, if applicable conflict detection (i.e. two compilers loaded at the same time) loading of modules with “atleast”, “between” and “latest” w.r.t software versions Spider (searching) and Keywords Caching
Modules in the 21st century: Lmod I’ll focus on Lmod from now on, though most things will work with other systems as well Very actively maintained by one of the most awesome maintainers we’ve seen in an open-source project Active community Cool new features: Hooks (e.g. function call every time a module is loaded) tcl2lua wrapper for conversion of old module files shell script/environment to Lua wrapper GPU and Xeon Phi specifics (colors :D) […]
Environment Modules: In Action module avail ----------------- /net/gmi.oeaw.ac.at/software/mendel/intel-x86_64-sandybridge-avx/modules/devel -----------------Autoconf/2.69-goolf-1.4.10 SCons/2.3.0-ictce-5.3.0-Python-2.7.3 (D) Automake/1.13.1-goolf-1.4.10 SWIG/2.0.4-goolf-1.4.10-Python-2.7.3 Boost/1.51.0-goolf-1.4.10-Python-2.7.3 ZeroMQ/2.2.0-goolf-1.4.10 Boost/1.51.0-goolf-1.4.10 (D) ant/1.8.4-Java-1.7.0_10 CMake/2.8.4-goolf-1.4.10 gperf/3.0.4-goolf-1.4.10 CMake/2.8.11-goolf-1.4.10 guile/1.8.8-goolf-1.4.10 CMake/2.8.12-GCC-4.7.2 (D) ncurses/5.9-GCC-4.7.2 Greenlet/0.4.1-goolf-1.4.10-Python-2.7.3 ncurses/5.9-goalf-1.1.0-no-OFED JUnit/4.10-Java-1.7.0_10 ncurses/5.9-goolf-1.4.10 LZO/2.06-goolf-1.4.10 ncurses/5.9-ictce-5.2.0 LZO/2.06-ictce-5.3.0 (D) ncurses/5.9-ictce-5.3.0 (D) M4/1.4.16-goolf-1.4.10 pkg-config/0.27.1-goolf-1.4.10 PCRE/8.12-goolf-1.4.10 setuptools/0.6c11-goolf-1.4.10-Python-2.7.3 PCRE/8.12-ictce-5.3.0 (D) setuptools/0.7.8-goolf-1.4.10-Python-2.7.3 PyZMQ/2.2.0.1-goolf-1.4.10-Python-2.7.3-zmq2 setuptools/2.0.1-goolf-1.4.10-Python-2.7.3 (D) SCons/2.3.0-goolf-1.4.10-Python-2.7.3 ! (only a small subset of the modules at this site)
Environment Modules: In Action module load aaron.zauner@login0 [~]> python --version Python 2.6.8 aaron.zauner@login0 [~]> module load Python/2.7.3-ictce-5.3.0 aaron.zauner@login0 [~]> python --version Python 2.7.3
Environment Modules: In Action module swap ! aaron.zauner@login0 [~]> module swap Python Python/2.7.5-goolf-1.4.10 The following have been reloaded with a version change: 1) bzip2/1.0.6-ictce-5.3.0 => bzip2/1.0.6-goolf-1.4.10 2) libreadline/6.2-ictce-5.3.0 => libreadline/6.2-goolf-1.4.10 3) ncurses/5.9-ictce-5.3.0 => ncurses/5.9-goolf-1.4.10 4) zlib/1.2.7-ictce-5.3.0 => zlib/1.2.8-goolf-1.4.10 ! aaron.zauner@login0 [~]> python --version Python 2.7.5
Environment Modules: In Action module spider ! ! ! ! ! ! ! ! aaron.zauner@login0 [~]> module spider setuptools --------------------------------------------------------------------------------------------------------------setuptools: --------------------------------------------------------------------------------------------------------------Versions: setuptools/0.6c11-goolf-1.4.10-Python-2.7.3 setuptools/0.7.8-goolf-1.4.10-Python-2.7.3 setuptools/2.0.1-goolf-1.4.10-Python-2.7.3 --------------------------------------------------------------------------------------------------------------To find detailed information about setuptools please enter the full name. For example: $ module spider setuptools/2.0.1-goolf-1.4.10-Python-2.7.3 ---------------------------------------------------------------------------------------------------------------
Environment Modules: In Action module list ! aaron.zauner@login0 [~]> module list Currently Loaded Modules: 1) std_env 2) gmi_term_color 3) gmi_resources 4) ictce/5.3.0 5) icc/2013.3.163 6) ifort/2013.3.163 7) impi/4.1.0.030 8) imkl/11.0.3.163 9) Python/2.7.5-goolf-1.4.10 10) goolf/1.4.10 11) GCC/4.7.2 ! ! ! ! ! 12) 13) 14) 15) 16) 17) 18) 19) 20) 21) OpenMPI/1.6.4-GCC-4.7.2 hwloc/1.6.2-GCC-4.7.2 OpenBLAS/0.2.6-gompi-1.4.10-LAPACK-3.4.2 gompi/1.4.10 FFTW/3.3.3-gompi-1.4.10 ScaLAPACK/2.0.2-gompi-1.4.10-OpenBLAS-0.2.6-LAPACK-3.4.2 bzip2/1.0.6-goolf-1.4.10 zlib/1.2.8-goolf-1.4.10 libreadline/6.2-goolf-1.4.10 ncurses/5.9-goolf-1.4.10
Environment Modules: In Action module display ! aaron.zauner@login0 [~]> module display GCC/4.7.2 --------------------------------------------------------------------------------------------------------------/net/gmi.oeaw.ac.at/software/mendel/intel-x86_64-sandybridge-avx/modules/compiler/GCC/4.7.2: --------------------------------------------------------------------------------------------------------------whatis("The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada, as well as libraries for these languages (libstdc++, libgcj,...). - Homepage: http://gcc.gnu.org/") conflict("GCC") prepend_path("CPATH","/net/gmi.oeaw.ac.at/software/mendel/29_04_2013/software/GCC/4.7.2/include") prepend_path("LD_LIBRARY_PATH","/net/gmi.oeaw.ac.at/software/mendel/29_04_2013/software/GCC/4.7.2/lib") prepend_path("LD_LIBRARY_PATH","/net/gmi.oeaw.ac.at/software/mendel/29_04_2013/software/GCC/4.7.2/lib64") prepend_path("LD_LIBRARY_PATH","/net/gmi.oeaw.ac.at/software/mendel/29_04_2013/software/GCC/4.7.2/lib/gcc/x86_64unknown-linux-gnu/4.7.2") prepend_path("MANPATH","/net/gmi.oeaw.ac.at/software/mendel/29_04_2013/software/GCC/4.7.2/share/man") prepend_path("PATH","/net/gmi.oeaw.ac.at/software/mendel/29_04_2013/software/GCC/4.7.2/bin") setenv("EBROOTGCC","/net/gmi.oeaw.ac.at/software/mendel/29_04_2013/software/GCC/4.7.2") setenv("EBVERSIONGCC","4.7.2") setenv("EBDEVELGCC","/net/gmi.oeaw.ac.at/software/mendel/29_04_2013/software/GCC/4.7.2/easybuild/GCC-4.7.2easybuild-devel") help([[ The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada, as well as libraries for these languages (libstdc++, libgcj,...). - Homepage: http://gcc.gnu.org/ ! ]]) ! !
Environment Modules: In Action ..A lot more to show but time is limited.. !
Modules for FOSS Devs. Still largely unkown and unused in the FOSS community Could be a reasonable replacement for the vast number of language specific environment abstraction tools as in use today There’s really no reason not to use it (that I know of) I’ll demonstrate on two prominent examples
Examples: RVM “Ruby enVironment (Version) Manager” [sic!] 20k lines of shell script, now being rewritten in Ruby basically installs and manages multiple versions of Ruby also does a couple of other cool things recently did a fundraiser for 50k USD to do the rewrite to Ruby actually has corporate sponsoring Sorry to say this: I develop Ruby code and It’s a bloated piece of shit that fucks up your environment beyond repair. Won’t ever use it again, ever. ever. Of course web developers love the hell out of it
Examples: RVM rvm install ruby-2 installs Ruby2 from source, downloads dependencies. wow. such magic. many hax. much sophisticate rvm list rvm use ruby-2 rvm use 1.9.2 —default does any of that sound familiar? right.
Examples: RVM azet@debian:~$ rvm use ruby-2 Using /home/azet/.rvm/gems/ruby-2.1.0 azet@debian:~$ set -v; rvm use system &>gimmethepain azet@debian:~$ wc -l gimmethepain 6458 gimmethepain azet@debian:~$ ! Yes. It takes ~6.5k lines of bash code to switch the Ruby version in the current environment with RVM.
“Always code as if the person who ends up maintaining your code is a violent psychopath who knows where you live.”
Examples: RVM Environmental pollution fact: RVM screws with `cd`. azet@debian:~$ set -x azet@debian:~$ cd ~ + cd /home/azet + builtin cd /home/azet + [[ -n '' ]] + true + __rvm_cd_functions_set + __rvm_do_with_env_before + [[ -n '' ]] + [[ -n /home/azet/.rvm ]] + source /home/azet/.rvm/scripts/initialize ++ : rvm_trace_flag:0 ++ (( rvm_trace_flag > 0 )) ++ [[ -n 4.2.45(1)-release ]] ++ shopt -s extglob ++ export __rvm_env_loaded ++ : __rvm_env_loaded:0: ++ : __rvm_env_loaded:1: ++ [[ -z '' ]] ++ typeset -f __rvm_cleanse_variables ++ __rvm_cleanse_variables ++ __rvm_unset_ruby_variables ++ unset rvm_env_string rvm_ruby_string rvm_ruby_strings rvm_ruby_binary rvm_ruby_gem_home rvm_ruby_gem_path rvm_ruby_home rvm_ruby_interpreter rvm_ruby_irbrc rvm_ruby_log_path rvm_ruby_major_version rvm_ruby_minor_version rvm_ruby_package_name rvm_ruby_patch_level rvm_ruby_release_version rvm_ruby_repo_url rvm_ruby_repo_branch rvm_ruby_revision rvm_ruby_selected_flag rvm_ruby_tag rvm_ruby_version rvm_head_flag rvm_ruby_package_file rvm_ruby_configure rvm_ruby_name rvm_ruby_url rvm_ruby_global_gems_path rvm_ruby_args rvm_ruby_name rvm_llvm_flag ++ __rvm_load_rvmrc ++ typeset _file ++ typeset -a rvm_rvmrc_files ++ (( 0 == 1 )) ..this goes on for another 148 lines of executed bullshit. 170 total 
 for every `cd` you type after installing RVM. yup. isn’t that great?
Examples: RVM Environmental pollution fact: RVM screws with `cd`. azet@debian:~$ set -x azet@debian:~$ cd ~ + cd /home/azet + builtin cd /home/azet + [[ -n '' ]] + true + __rvm_cd_functions_set + __rvm_do_with_env_before + [[ -n '' ]] + [[ -n /home/azet/.rvm ]] + source /home/azet/.rvm/scripts/initialize ++ : rvm_trace_flag:0 ++ (( rvm_trace_flag > 0 )) ++ [[ -n 4.2.45(1)-release ]] ++ shopt -s extglob ++ export __rvm_env_loaded ++ : __rvm_env_loaded:0: ++ : __rvm_env_loaded:1: ++ [[ -z '' ]] ++ typeset -f __rvm_cleanse_variables ++ __rvm_cleanse_variables ++ __rvm_unset_ruby_variables ++ unset rvm_env_string rvm_ruby_string rvm_ruby_strings rvm_ruby_binary rvm_ruby_gem_home rvm_ruby_gem_path rvm_ruby_home rvm_ruby_interpreter rvm_ruby_irbrc rvm_ruby_log_path rvm_ruby_major_version rvm_ruby_minor_version rvm_ruby_package_name rvm_ruby_patch_level rvm_ruby_release_version rvm_ruby_repo_url rvm_ruby_repo_branch rvm_ruby_revision rvm_ruby_selected_flag rvm_ruby_tag rvm_ruby_version rvm_head_flag rvm_ruby_package_file rvm_ruby_configure rvm_ruby_name rvm_ruby_url rvm_ruby_global_gems_path rvm_ruby_args rvm_ruby_name rvm_llvm_flag ++ __rvm_load_rvmrc ++ typeset _file ++ typeset -a rvm_rvmrc_files ++ (( 0 == 1 )) ..this goes on for another 148 lines of executed bullshit. 170 total 
 for every `cd` you type after installing RVM. yup. isn’t that great?
Examples: RVM azet@debian:~$ time cd ! real 0m0.009s user 0m0.004s sys 0m0.004s azet@debian:~$ rvm implode […] azet@debian:~$ time cd ! real 0m0.000s user 0m0.000s sys 0m0.000s ! ..if you really do not like modules, please use rbenv instead
 a lot more to tell about RVM but lets move on,..
Examples: RVM Ahh by the way, almost forgot..
 
 
 
 
 
 
 
 
 azet@debian:~$ cd project/ -bash: /home/azet/.rvm/scripts/initialize: No such file or directory !
Examples: virtualenv Basically the same story here “virtualenv is a tool to create isolated Python environments.”
 azet@debian:~/python$ virtualenv MyLittlePythonSandbox New python executable in MyLittlePythonSandbox/bin/python Installing setuptools, pip...done. azet@debian:~/python$ cd MyLittlePythonSandbox/ azet@debian:~/python/MyLittlePythonSandbox$ source bin/activate (MyLittlePythonSandbox)azet@debian:~/python/MyLittlePythonSandbox$ pip install XYZFUBAR ! [write some code] People tend to do this a lot and actually ship stuff in that way. So projects end up with this huge requirements.txt files that you might have noticed ..if thats not bad development practice I’m not sure what is bin/activate is actually a lot cleaner (~15 LoC Bash) than the RVM way
Examples: virtualenv It’s a pain installing, using, maintaining and upgrading projects that someone has prepared using virtualenv in a production environment Often Webprojects use it, which means that your HTTP(S) daemon needs to be aware of
 virtualenv and switch to the directory and source stuff to get things working.. m( From a good recent blog post* on why you should not do it that way: 
 
 
 
 
 
 
 
 
 * http://pythonrants.wordpress.com/2013/12/06/why-i-hate-virtualenv-and-pip/
Handling IPhyton with Modules devbox$ module display ipython/0.13.2-Python-2.7.3 ! whatis("The IPython project provides an enhanced interactive environment that includes, among other features, support for data visualization and facilities for distributed and parallel computation. - Homepage: http://ipython.org/index.html”) ! conflict("ipython") ! load("Python/2.7.3") ! prepend_path("LD_LIBRARY_PATH","/software/ipython/0.13.2-Python-2.7.3/lib") prepend_path("MANPATH","/software/ipython/0.13.2-Python-2.7.3/share/man") prepend_path(“PATH","/software/ipython/0.13.2-Python-2.7.3/bin") ! prepend_path(“PYTHONPATH","/software/ipython/0.13.2-Python-2.7.3/lib/python2.7/ site-packages") ! help([[ The IPython project provides an enhanced interactive environment that includes, among other features, support for data visualization and facilities for distributed and parallel computation. - Homepage: http://ipython.org/index.html ! ]]) ! devbox$ module load ipython/0.13.2-Python-2.7.3 devbox$ ipython —version 0.13.2 devbox$
What if I tell you that.. You can do all this things quite easily with modules Save time & effort maintaining stuff Have reproducibility ..and more fun & time to concentrate on that is important developing good software ?
..but modules does not install stuff for me I take it? Back in the 1970ies at Bell Labs Ken Thompson put forward a small set of radical ideas for software engineering that are often ignored by the FOSS community today (who shouted systemd?), those have been summarised by Doug Mcllroy of Bell Labs as:
 
 “Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.”
..but modules does not install stuff for me I take it? Back in the 1970ies at Bell Labs Ken Thompson put forward a small set of radical ideas for software engineering that are often ignored by the FOSS community today (who shouted systemd?), those have been summarised by Doug Mcllroy of Bell Labs as:
 
 “Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.” There are a lot of different ways to install binary or source distributions of software. They are all compatible with the idea of environment modules. For an excellent approach that fits HPC/comp.sci. sites well talk to the EasyBuild guys (the adjacent talk will introduce you to their concept). EasyBuild not only does optimised builds but outputs modulefiles as well!
Thank you for your time! E-Mail: azet@azet.org XMPP/Jabber: azet@jabber.ccc.de Twitter: @a_z_e_t GitHub: azet GPG Fingerprint: 0xFDB9B5D5

More Related Content

PDF
Implement Runtime Environments for HSA using LLVM
National Cheng Kung University
 
PDF
F9: A Secure and Efficient Microkernel Built for Deeply Embedded Systems
National Cheng Kung University
 
PDF
淺談探索 Linux 系統設計之道
National Cheng Kung University
 
PDF
Embedded Virtualization applied in Mobile Devices
National Cheng Kung University
 
PDF
Hands on with embedded linux using zero hardware
Rajesh Sola
 
PDF
Kernel Recipes 2013 - ARM support in the Linux kernel
Anne Nicolas
 
PPTX
Linux Kernel Module - For NLKB
shimosawa
 
PPT
Building Embedded Linux
Sherif Mousa
 
Implement Runtime Environments for HSA using LLVM
National Cheng Kung University
 
F9: A Secure and Efficient Microkernel Built for Deeply Embedded Systems
National Cheng Kung University
 
淺談探索 Linux 系統設計之道
National Cheng Kung University
 
Embedded Virtualization applied in Mobile Devices
National Cheng Kung University
 
Hands on with embedded linux using zero hardware
Rajesh Sola
 
Kernel Recipes 2013 - ARM support in the Linux kernel
Anne Nicolas
 
Linux Kernel Module - For NLKB
shimosawa
 
Building Embedded Linux
Sherif Mousa
 

What's hot (14)

PPTX
Linux internals v4
Liran Ben Haim
 
PDF
Building Embedded Linux Systems Introduction
Sherif Mousa
 
PDF
Yocto Project Kernel Lab hands-on
Trevor Woerner
 
PDF
Embedded Linux from Scratch to Yocto
Sherif Mousa
 
PDF
Embedded Linux On A R M
Emanuele Bonanni
 
PDF
Device Tree for Dummies (ELC 2014)
Thomas Petazzoni
 
PDF
Embedded linux system development (slides)
Jaime Barragan
 
PDF
Unix v6 Internals
National Cheng Kung University
 
PPTX
Introduction to Embedded Linux
Hossain Reja
 
PPT
Basic Linux Internals
mukul bhardwaj
 
PDF
Linux device driver
chatsiri
 
PDF
Hints for L4 Microkernel
National Cheng Kung University
 
PDF
Fuse'ing python for rapid development of storage efficient FS
Chetan Giridhar
 
PDF
olibc: Another C Library optimized for Embedded Linux
National Cheng Kung University
 
Linux internals v4
Liran Ben Haim
 
Building Embedded Linux Systems Introduction
Sherif Mousa
 
Yocto Project Kernel Lab hands-on
Trevor Woerner
 
Embedded Linux from Scratch to Yocto
Sherif Mousa
 
Embedded Linux On A R M
Emanuele Bonanni
 
Device Tree for Dummies (ELC 2014)
Thomas Petazzoni
 
Embedded linux system development (slides)
Jaime Barragan
 
Unix v6 Internals
National Cheng Kung University
 
Introduction to Embedded Linux
Hossain Reja
 
Basic Linux Internals
mukul bhardwaj
 
Linux device driver
chatsiri
 
Hints for L4 Microkernel
National Cheng Kung University
 
Fuse'ing python for rapid development of storage efficient FS
Chetan Giridhar
 
olibc: Another C Library optimized for Embedded Linux
National Cheng Kung University
 
Ad

Viewers also liked (14)

PDF
Banking ppt
J. Sen
 
PPTX
Save environment , save life
Vivek Rathi
 
PPTX
Rahul
rahulnarola
 
PDF
Unit 01business environment pdf
J. Sen
 
PPT
Healthy Heart for a long happy life
HHC Healthcare Pvt. Ltd.
 
PDF
Warning signs of a heart attack
Victor Go
 
PPTX
50 things we can do to save our environment
Roi Fernandez
 
PPTX
Group 6 dissolved oxygen determination ppt
April Marcelino
 
PPTX
Project oxygen ppt
Gorla Sindhura Sindhura
 
PPT
Save Environment and Control Pollution
Gourav Upadhayay
 
PPTX
Save Environment PPT
VikrantPatankar
 
PPTX
The evolution of mobile phones
Olivia2590
 
PPTX
Operating Systems - File Management
Damian T. Gordon
 
PDF
How to Become a Thought Leader in Your Niche
Leslie Samuel
 
Banking ppt
J. Sen
 
Save environment , save life
Vivek Rathi
 
Rahul
rahulnarola
 
Unit 01business environment pdf
J. Sen
 
Healthy Heart for a long happy life
HHC Healthcare Pvt. Ltd.
 
Warning signs of a heart attack
Victor Go
 
50 things we can do to save our environment
Roi Fernandez
 
Group 6 dissolved oxygen determination ppt
April Marcelino
 
Project oxygen ppt
Gorla Sindhura Sindhura
 
Save Environment and Control Pollution
Gourav Upadhayay
 
Save Environment PPT
VikrantPatankar
 
The evolution of mobile phones
Olivia2590
 
Operating Systems - File Management
Damian T. Gordon
 
How to Become a Thought Leader in Your Niche
Leslie Samuel
 
Ad

Similar to How to save the environment (20)

PPTX
Packaging computational biology tools for broad distribution and ease-of-reuse
Matthew Vaughn
 
PDF
14_Ed_Symp_Open_Source
Steve Arnold
 
PDF
Free/Open Source Software for Science & Engineering
Kinshuk Sunil
 
PPTX
DataArt Odessa IT talk: Ansible use-cases and custom modules
KharchenkoKaterina
 
ODP
Linux and Open Source in Math, Science and Engineering
PDE1D
 
PDF
Package Management via Spack on SJTU π Supercomputer
Jianwen Wei
 
PDF
The Yocto Project
rossburton
 
PDF
Jump into Squeak - Integrate Squeak projects with Docker & Github
hubx
 
PPTX
spack_hpc.pptx
Object Automation
 
PPTX
DevConf.US 2019 - How to Jump Start Your Career in Open Source
Eric D. Schabell
 
PPTX
A Tour of Open Source on the Mainframe
All Things Open
 
PDF
[HKOSCON][20200613][ Ansible: From VM to Kubernetes]
Wong Hoi Sing Edison
 
KEY
Eclipse Virgo
glynnormington
 
PDF
Deploying Hadoop-Based Bigdata Environments
Puppet
 
PDF
Deploying Hadoop-based Bigdata Environments
buildacloud
 
PDF
Kitware: Qt and Scientific Computing
account inactive
 
PDF
Deep Dive into the Microsoft OpenStack CI Infrastructure (Alessandro Pilotti)
ITCamp
 
PPTX
Openslava 2017 - Are developers the real emerging technology?
Eric D. Schabell
 
ODP
Introducing OpenStack for Beginners
openstackindia
 
PPTX
Building Immutable Machine Images with Packer and Ansible
Jason Harley
 
Packaging computational biology tools for broad distribution and ease-of-reuse
Matthew Vaughn
 
14_Ed_Symp_Open_Source
Steve Arnold
 
Free/Open Source Software for Science & Engineering
Kinshuk Sunil
 
DataArt Odessa IT talk: Ansible use-cases and custom modules
KharchenkoKaterina
 
Linux and Open Source in Math, Science and Engineering
PDE1D
 
Package Management via Spack on SJTU π Supercomputer
Jianwen Wei
 
The Yocto Project
rossburton
 
Jump into Squeak - Integrate Squeak projects with Docker & Github
hubx
 
spack_hpc.pptx
Object Automation
 
DevConf.US 2019 - How to Jump Start Your Career in Open Source
Eric D. Schabell
 
A Tour of Open Source on the Mainframe
All Things Open
 
[HKOSCON][20200613][ Ansible: From VM to Kubernetes]
Wong Hoi Sing Edison
 
Eclipse Virgo
glynnormington
 
Deploying Hadoop-Based Bigdata Environments
Puppet
 
Deploying Hadoop-based Bigdata Environments
buildacloud
 
Kitware: Qt and Scientific Computing
account inactive
 
Deep Dive into the Microsoft OpenStack CI Infrastructure (Alessandro Pilotti)
ITCamp
 
Openslava 2017 - Are developers the real emerging technology?
Eric D. Schabell
 
Introducing OpenStack for Beginners
openstackindia
 
Building Immutable Machine Images with Packer and Ansible
Jason Harley
 

More from Aaron Zauner (13)

PDF
Because "use urandom" isn't everything: a deep dive into CSPRNGs in Operating...
Aaron Zauner
 
PDF
[BlackHat USA 2016] Nonce-Disrespecting Adversaries: Practical Forgery Attack...
Aaron Zauner
 
PDF
No need for Black Chambers: Testing TLS in the E-Mail Ecosystem at Large (hac...
Aaron Zauner
 
PDF
State of Transport Security in the E-Mail Ecosystem at Large
Aaron Zauner
 
PDF
Javascript Object Signing & Encryption
Aaron Zauner
 
PDF
Introduction to and survey of TLS security (BsidesHH 2014)
Aaron Zauner
 
PDF
Beautiful Bash: Let's make reading and writing bash scripts fun again!
Aaron Zauner
 
PDF
Introduction to and survey of TLS Security
Aaron Zauner
 
PDF
[IETF Part] BetterCrypto Workshop @ Hack.lu 2014
Aaron Zauner
 
PDF
[Attacks Part] BetterCrypto Workshop @ Hack.lu 2014
Aaron Zauner
 
PDF
Introduction to and survey of TLS Security
Aaron Zauner
 
PDF
BetterCrypto: Applied Crypto Hardening
Aaron Zauner
 
PDF
Sc12 workshop-writeup
Aaron Zauner
 
Because "use urandom" isn't everything: a deep dive into CSPRNGs in Operating...
Aaron Zauner
 
[BlackHat USA 2016] Nonce-Disrespecting Adversaries: Practical Forgery Attack...
Aaron Zauner
 
No need for Black Chambers: Testing TLS in the E-Mail Ecosystem at Large (hac...
Aaron Zauner
 
State of Transport Security in the E-Mail Ecosystem at Large
Aaron Zauner
 
Javascript Object Signing & Encryption
Aaron Zauner
 
Introduction to and survey of TLS security (BsidesHH 2014)
Aaron Zauner
 
Beautiful Bash: Let's make reading and writing bash scripts fun again!
Aaron Zauner
 
Introduction to and survey of TLS Security
Aaron Zauner
 
[IETF Part] BetterCrypto Workshop @ Hack.lu 2014
Aaron Zauner
 
[Attacks Part] BetterCrypto Workshop @ Hack.lu 2014
Aaron Zauner
 
Introduction to and survey of TLS Security
Aaron Zauner
 
BetterCrypto: Applied Crypto Hardening
Aaron Zauner
 
Sc12 workshop-writeup
Aaron Zauner
 

Recently uploaded (20)

PDF
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Libreria ERP
 
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
PPTX
Cloud computing and distributed systems.
kkkkkhan
 
PDF
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
PDF
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
PPTX
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
PPTX
sap open course for s4hana steps from ECC to s4
sreeni2106invites
 
PDF
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
PPTX
Programs and apps: productivity, graphics, security and other tools
4mqw9zch22
 
PDF
Encapsulation theory and applications.pdf
gurumoop
 
PDF
Approach and Philosophy of On baking technology
30anthuong17
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
PDF
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
PDF
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Libreria ERP
 
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
Cloud computing and distributed systems.
kkkkkhan
 
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
sap open course for s4hana steps from ECC to s4
sreeni2106invites
 
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
Programs and apps: productivity, graphics, security and other tools
4mqw9zch22
 
Encapsulation theory and applications.pdf
gurumoop
 
Approach and Philosophy of On baking technology
30anthuong17
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
Electronic commerce courselecture one. Pdf
OmerMohamed64
 

How to save the environment

  • 1. How to save the environment ..and get rid of virtualenv, rvm, pythonbrew, rbenv, pythonz (...) FOSDEM 2014 - Brussels, Belgium, Old World Earth, Orion–Cygnus Arm, Milky Way Aaron Zauner azet@azet.org @a_z_e_t
  • 2. `whoami` Self-employed engineer currently involved in IT-Infrastructure, HPC and security stuff primarily working for biology institutes of the austrian academy of sciences Used to do Web/Mail and Datacenter Operations Front/Backend Dev. Security auditing FOSS/Community (BSD) […]
  • 3. Slide intentionally left blank for your enjoyment.
  • 4. Text ‘Environment Modules’ First officially published at LISA V back in 1991 by John L. Furlani
  • 5. Environment Modules In use ever since in High Performance and Scientific Computing Ease of administration Ease of use
  • 6. Environment Modules by most just refered to as “Modules” originally intended to enable multiple users to have software in different versions available traditionally most HPC environments do not use system packages special compilers, toolchains, math libraries, optimisation, interconnect-specifics […] and often scientific software is not to be found in $distro repo (because it’s crap || not used outside the sci. community) somewhat like Gentoo or Arch - but a lot more cumbersome for sysadmins and users a system to manage multiple versions of the same software with different toolchains is needed that is easy to maintain and easy for users (developers and scientists, usually) to learn and use
  • 7. Environment Modules “modulefiles” dynamically set an environment specific to a software may load/unload other modulefiles if there is a dependency present examples include but are not limited to PATH MANPATH LD_LIBRARY_PATH LD_PRELOAD
  • 8. Environment Modules (with Lmod - more on that later) ! ! whatis("Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. - Homepage: http://git-scm.com/“) ! conflict("git") ! prepend_path(“LD_LIBRARY_PATH","/software/git/1.8.2/lib") prepend_path("LD_LIBRARY_PATH","/software/git/1.8.2/lib64") prepend_path(“MANPATH","/software/git/1.8.2/share/man") ! […]
  • 10. Environment Modules: Implementations “Classical Modules” (Furlani et. al.): C/Tcl still in use at arguably most HPC sites modules.sourceforge.net ! modules-tcl (Kent Mein of UMN): Tcl http://sourceforge.net/p/modules/modules-tcl Cmod (Per Cederqvist et. al.): C http://www.lysator.liu.se/cmod/ Lmod (Robert McLay of TACC): Lua https://www.tacc.utexas.edu/tacc-projects/lmod similar systems: ! SoftEnv (by and in use at Argonne National Lab): Csh http://www.mcs.anl.gov/hs/software/systems/ softenv/softenv-admin.html dotkit (by and in use at Lawrence Livermore National Lab): *Shell https://computing.llnl.gov/? set=jobs&page=dotkit
  • 11. Environment Modules: Implementations “Classical Modules” (Furlani et. al.): C/Tcl still in use at arguably most HPC sites modules.sourceforge.net ! modules-tcl (Kent Mein of UMN): Tcl http://sourceforge.net/p/modules/modules-tcl Cmod (Per Cederqvist et. al.): C http://www.lysator.liu.se/cmod/ Lmod (Robert McLay of TACC): Lua https://www.tacc.utexas.edu/tacc-projects/lmod similar systems: ! SoftEnv (by and in use at Argonne National Lab): Csh http://www.mcs.anl.gov/hs/software/systems/ softenv/softenv-admin.html dotkit (by and in use at Lawrence Livermore National Lab): *Shell https://computing.llnl.gov/? set=jobs&page=dotkit
  • 12. Modules in the 21st century: Lmod I’ll focus on Lmod from now on, though most things will work with other systems as well Very actively maintained by one of the most awesome maintainers we’ve seen in an open-source project Active community Cool new features: automatic swapping of modules, if applicable conflict detection (i.e. two compilers loaded at the same time) loading of modules with “atleast”, “between” and “latest” w.r.t software versions Spider (searching) and Keywords Caching
  • 13. Modules in the 21st century: Lmod I’ll focus on Lmod from now on, though most things will work with other systems as well Very actively maintained by one of the most awesome maintainers we’ve seen in an open-source project Active community Cool new features: Hooks (e.g. function call every time a module is loaded) tcl2lua wrapper for conversion of old module files shell script/environment to Lua wrapper GPU and Xeon Phi specifics (colors :D) […]
  • 14. Environment Modules: In Action module avail ----------------- /net/gmi.oeaw.ac.at/software/mendel/intel-x86_64-sandybridge-avx/modules/devel -----------------Autoconf/2.69-goolf-1.4.10 SCons/2.3.0-ictce-5.3.0-Python-2.7.3 (D) Automake/1.13.1-goolf-1.4.10 SWIG/2.0.4-goolf-1.4.10-Python-2.7.3 Boost/1.51.0-goolf-1.4.10-Python-2.7.3 ZeroMQ/2.2.0-goolf-1.4.10 Boost/1.51.0-goolf-1.4.10 (D) ant/1.8.4-Java-1.7.0_10 CMake/2.8.4-goolf-1.4.10 gperf/3.0.4-goolf-1.4.10 CMake/2.8.11-goolf-1.4.10 guile/1.8.8-goolf-1.4.10 CMake/2.8.12-GCC-4.7.2 (D) ncurses/5.9-GCC-4.7.2 Greenlet/0.4.1-goolf-1.4.10-Python-2.7.3 ncurses/5.9-goalf-1.1.0-no-OFED JUnit/4.10-Java-1.7.0_10 ncurses/5.9-goolf-1.4.10 LZO/2.06-goolf-1.4.10 ncurses/5.9-ictce-5.2.0 LZO/2.06-ictce-5.3.0 (D) ncurses/5.9-ictce-5.3.0 (D) M4/1.4.16-goolf-1.4.10 pkg-config/0.27.1-goolf-1.4.10 PCRE/8.12-goolf-1.4.10 setuptools/0.6c11-goolf-1.4.10-Python-2.7.3 PCRE/8.12-ictce-5.3.0 (D) setuptools/0.7.8-goolf-1.4.10-Python-2.7.3 PyZMQ/2.2.0.1-goolf-1.4.10-Python-2.7.3-zmq2 setuptools/2.0.1-goolf-1.4.10-Python-2.7.3 (D) SCons/2.3.0-goolf-1.4.10-Python-2.7.3 ! (only a small subset of the modules at this site)
  • 15. Environment Modules: In Action module load aaron.zauner@login0 [~]> python --version Python 2.6.8 aaron.zauner@login0 [~]> module load Python/2.7.3-ictce-5.3.0 aaron.zauner@login0 [~]> python --version Python 2.7.3
  • 16. Environment Modules: In Action module swap ! aaron.zauner@login0 [~]> module swap Python Python/2.7.5-goolf-1.4.10 The following have been reloaded with a version change: 1) bzip2/1.0.6-ictce-5.3.0 => bzip2/1.0.6-goolf-1.4.10 2) libreadline/6.2-ictce-5.3.0 => libreadline/6.2-goolf-1.4.10 3) ncurses/5.9-ictce-5.3.0 => ncurses/5.9-goolf-1.4.10 4) zlib/1.2.7-ictce-5.3.0 => zlib/1.2.8-goolf-1.4.10 ! aaron.zauner@login0 [~]> python --version Python 2.7.5
  • 17. Environment Modules: In Action module spider ! ! ! ! ! ! ! ! aaron.zauner@login0 [~]> module spider setuptools --------------------------------------------------------------------------------------------------------------setuptools: --------------------------------------------------------------------------------------------------------------Versions: setuptools/0.6c11-goolf-1.4.10-Python-2.7.3 setuptools/0.7.8-goolf-1.4.10-Python-2.7.3 setuptools/2.0.1-goolf-1.4.10-Python-2.7.3 --------------------------------------------------------------------------------------------------------------To find detailed information about setuptools please enter the full name. For example: $ module spider setuptools/2.0.1-goolf-1.4.10-Python-2.7.3 ---------------------------------------------------------------------------------------------------------------
  • 18. Environment Modules: In Action module list ! aaron.zauner@login0 [~]> module list Currently Loaded Modules: 1) std_env 2) gmi_term_color 3) gmi_resources 4) ictce/5.3.0 5) icc/2013.3.163 6) ifort/2013.3.163 7) impi/4.1.0.030 8) imkl/11.0.3.163 9) Python/2.7.5-goolf-1.4.10 10) goolf/1.4.10 11) GCC/4.7.2 ! ! ! ! ! 12) 13) 14) 15) 16) 17) 18) 19) 20) 21) OpenMPI/1.6.4-GCC-4.7.2 hwloc/1.6.2-GCC-4.7.2 OpenBLAS/0.2.6-gompi-1.4.10-LAPACK-3.4.2 gompi/1.4.10 FFTW/3.3.3-gompi-1.4.10 ScaLAPACK/2.0.2-gompi-1.4.10-OpenBLAS-0.2.6-LAPACK-3.4.2 bzip2/1.0.6-goolf-1.4.10 zlib/1.2.8-goolf-1.4.10 libreadline/6.2-goolf-1.4.10 ncurses/5.9-goolf-1.4.10
  • 19. Environment Modules: In Action module display ! aaron.zauner@login0 [~]> module display GCC/4.7.2 --------------------------------------------------------------------------------------------------------------/net/gmi.oeaw.ac.at/software/mendel/intel-x86_64-sandybridge-avx/modules/compiler/GCC/4.7.2: --------------------------------------------------------------------------------------------------------------whatis("The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada, as well as libraries for these languages (libstdc++, libgcj,...). - Homepage: http://gcc.gnu.org/") conflict("GCC") prepend_path("CPATH","/net/gmi.oeaw.ac.at/software/mendel/29_04_2013/software/GCC/4.7.2/include") prepend_path("LD_LIBRARY_PATH","/net/gmi.oeaw.ac.at/software/mendel/29_04_2013/software/GCC/4.7.2/lib") prepend_path("LD_LIBRARY_PATH","/net/gmi.oeaw.ac.at/software/mendel/29_04_2013/software/GCC/4.7.2/lib64") prepend_path("LD_LIBRARY_PATH","/net/gmi.oeaw.ac.at/software/mendel/29_04_2013/software/GCC/4.7.2/lib/gcc/x86_64unknown-linux-gnu/4.7.2") prepend_path("MANPATH","/net/gmi.oeaw.ac.at/software/mendel/29_04_2013/software/GCC/4.7.2/share/man") prepend_path("PATH","/net/gmi.oeaw.ac.at/software/mendel/29_04_2013/software/GCC/4.7.2/bin") setenv("EBROOTGCC","/net/gmi.oeaw.ac.at/software/mendel/29_04_2013/software/GCC/4.7.2") setenv("EBVERSIONGCC","4.7.2") setenv("EBDEVELGCC","/net/gmi.oeaw.ac.at/software/mendel/29_04_2013/software/GCC/4.7.2/easybuild/GCC-4.7.2easybuild-devel") help([[ The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada, as well as libraries for these languages (libstdc++, libgcj,...). - Homepage: http://gcc.gnu.org/ ! ]]) ! !
  • 20. Environment Modules: In Action ..A lot more to show but time is limited.. !
  • 21. Modules for FOSS Devs. Still largely unkown and unused in the FOSS community Could be a reasonable replacement for the vast number of language specific environment abstraction tools as in use today There’s really no reason not to use it (that I know of) I’ll demonstrate on two prominent examples
  • 22. Examples: RVM “Ruby enVironment (Version) Manager” [sic!] 20k lines of shell script, now being rewritten in Ruby basically installs and manages multiple versions of Ruby also does a couple of other cool things recently did a fundraiser for 50k USD to do the rewrite to Ruby actually has corporate sponsoring Sorry to say this: I develop Ruby code and It’s a bloated piece of shit that fucks up your environment beyond repair. Won’t ever use it again, ever. ever. Of course web developers love the hell out of it
  • 23. Examples: RVM rvm install ruby-2 installs Ruby2 from source, downloads dependencies. wow. such magic. many hax. much sophisticate rvm list rvm use ruby-2 rvm use 1.9.2 —default does any of that sound familiar? right.
  • 24. Examples: RVM azet@debian:~$ rvm use ruby-2 Using /home/azet/.rvm/gems/ruby-2.1.0 azet@debian:~$ set -v; rvm use system &>gimmethepain azet@debian:~$ wc -l gimmethepain 6458 gimmethepain azet@debian:~$ ! Yes. It takes ~6.5k lines of bash code to switch the Ruby version in the current environment with RVM.
  • 25. “Always code as if the person who ends up maintaining your code is a violent psychopath who knows where you live.”
  • 26. Examples: RVM Environmental pollution fact: RVM screws with `cd`. azet@debian:~$ set -x azet@debian:~$ cd ~ + cd /home/azet + builtin cd /home/azet + [[ -n '' ]] + true + __rvm_cd_functions_set + __rvm_do_with_env_before + [[ -n '' ]] + [[ -n /home/azet/.rvm ]] + source /home/azet/.rvm/scripts/initialize ++ : rvm_trace_flag:0 ++ (( rvm_trace_flag > 0 )) ++ [[ -n 4.2.45(1)-release ]] ++ shopt -s extglob ++ export __rvm_env_loaded ++ : __rvm_env_loaded:0: ++ : __rvm_env_loaded:1: ++ [[ -z '' ]] ++ typeset -f __rvm_cleanse_variables ++ __rvm_cleanse_variables ++ __rvm_unset_ruby_variables ++ unset rvm_env_string rvm_ruby_string rvm_ruby_strings rvm_ruby_binary rvm_ruby_gem_home rvm_ruby_gem_path rvm_ruby_home rvm_ruby_interpreter rvm_ruby_irbrc rvm_ruby_log_path rvm_ruby_major_version rvm_ruby_minor_version rvm_ruby_package_name rvm_ruby_patch_level rvm_ruby_release_version rvm_ruby_repo_url rvm_ruby_repo_branch rvm_ruby_revision rvm_ruby_selected_flag rvm_ruby_tag rvm_ruby_version rvm_head_flag rvm_ruby_package_file rvm_ruby_configure rvm_ruby_name rvm_ruby_url rvm_ruby_global_gems_path rvm_ruby_args rvm_ruby_name rvm_llvm_flag ++ __rvm_load_rvmrc ++ typeset _file ++ typeset -a rvm_rvmrc_files ++ (( 0 == 1 )) ..this goes on for another 148 lines of executed bullshit. 170 total 
 for every `cd` you type after installing RVM. yup. isn’t that great?
  • 27. Examples: RVM Environmental pollution fact: RVM screws with `cd`. azet@debian:~$ set -x azet@debian:~$ cd ~ + cd /home/azet + builtin cd /home/azet + [[ -n '' ]] + true + __rvm_cd_functions_set + __rvm_do_with_env_before + [[ -n '' ]] + [[ -n /home/azet/.rvm ]] + source /home/azet/.rvm/scripts/initialize ++ : rvm_trace_flag:0 ++ (( rvm_trace_flag > 0 )) ++ [[ -n 4.2.45(1)-release ]] ++ shopt -s extglob ++ export __rvm_env_loaded ++ : __rvm_env_loaded:0: ++ : __rvm_env_loaded:1: ++ [[ -z '' ]] ++ typeset -f __rvm_cleanse_variables ++ __rvm_cleanse_variables ++ __rvm_unset_ruby_variables ++ unset rvm_env_string rvm_ruby_string rvm_ruby_strings rvm_ruby_binary rvm_ruby_gem_home rvm_ruby_gem_path rvm_ruby_home rvm_ruby_interpreter rvm_ruby_irbrc rvm_ruby_log_path rvm_ruby_major_version rvm_ruby_minor_version rvm_ruby_package_name rvm_ruby_patch_level rvm_ruby_release_version rvm_ruby_repo_url rvm_ruby_repo_branch rvm_ruby_revision rvm_ruby_selected_flag rvm_ruby_tag rvm_ruby_version rvm_head_flag rvm_ruby_package_file rvm_ruby_configure rvm_ruby_name rvm_ruby_url rvm_ruby_global_gems_path rvm_ruby_args rvm_ruby_name rvm_llvm_flag ++ __rvm_load_rvmrc ++ typeset _file ++ typeset -a rvm_rvmrc_files ++ (( 0 == 1 )) ..this goes on for another 148 lines of executed bullshit. 170 total 
 for every `cd` you type after installing RVM. yup. isn’t that great?
  • 28. Examples: RVM azet@debian:~$ time cd ! real 0m0.009s user 0m0.004s sys 0m0.004s azet@debian:~$ rvm implode […] azet@debian:~$ time cd ! real 0m0.000s user 0m0.000s sys 0m0.000s ! ..if you really do not like modules, please use rbenv instead
 a lot more to tell about RVM but lets move on,..
  • 29. Examples: RVM Ahh by the way, almost forgot..
 
 
 
 
 
 
 
 
 azet@debian:~$ cd project/ -bash: /home/azet/.rvm/scripts/initialize: No such file or directory !
  • 30. Examples: virtualenv Basically the same story here “virtualenv is a tool to create isolated Python environments.”
 azet@debian:~/python$ virtualenv MyLittlePythonSandbox New python executable in MyLittlePythonSandbox/bin/python Installing setuptools, pip...done. azet@debian:~/python$ cd MyLittlePythonSandbox/ azet@debian:~/python/MyLittlePythonSandbox$ source bin/activate (MyLittlePythonSandbox)azet@debian:~/python/MyLittlePythonSandbox$ pip install XYZFUBAR ! [write some code] People tend to do this a lot and actually ship stuff in that way. So projects end up with this huge requirements.txt files that you might have noticed ..if thats not bad development practice I’m not sure what is bin/activate is actually a lot cleaner (~15 LoC Bash) than the RVM way
  • 31. Examples: virtualenv It’s a pain installing, using, maintaining and upgrading projects that someone has prepared using virtualenv in a production environment Often Webprojects use it, which means that your HTTP(S) daemon needs to be aware of
 virtualenv and switch to the directory and source stuff to get things working.. m( From a good recent blog post* on why you should not do it that way: 
 
 
 
 
 
 
 
 
 * http://pythonrants.wordpress.com/2013/12/06/why-i-hate-virtualenv-and-pip/
  • 32. Handling IPhyton with Modules devbox$ module display ipython/0.13.2-Python-2.7.3 ! whatis("The IPython project provides an enhanced interactive environment that includes, among other features, support for data visualization and facilities for distributed and parallel computation. - Homepage: http://ipython.org/index.html”) ! conflict("ipython") ! load("Python/2.7.3") ! prepend_path("LD_LIBRARY_PATH","/software/ipython/0.13.2-Python-2.7.3/lib") prepend_path("MANPATH","/software/ipython/0.13.2-Python-2.7.3/share/man") prepend_path(“PATH","/software/ipython/0.13.2-Python-2.7.3/bin") ! prepend_path(“PYTHONPATH","/software/ipython/0.13.2-Python-2.7.3/lib/python2.7/ site-packages") ! help([[ The IPython project provides an enhanced interactive environment that includes, among other features, support for data visualization and facilities for distributed and parallel computation. - Homepage: http://ipython.org/index.html ! ]]) ! devbox$ module load ipython/0.13.2-Python-2.7.3 devbox$ ipython —version 0.13.2 devbox$
  • 33. What if I tell you that.. You can do all this things quite easily with modules Save time & effort maintaining stuff Have reproducibility ..and more fun & time to concentrate on that is important developing good software ?
  • 34. ..but modules does not install stuff for me I take it? Back in the 1970ies at Bell Labs Ken Thompson put forward a small set of radical ideas for software engineering that are often ignored by the FOSS community today (who shouted systemd?), those have been summarised by Doug Mcllroy of Bell Labs as:
 
 “Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.”
  • 35. ..but modules does not install stuff for me I take it? Back in the 1970ies at Bell Labs Ken Thompson put forward a small set of radical ideas for software engineering that are often ignored by the FOSS community today (who shouted systemd?), those have been summarised by Doug Mcllroy of Bell Labs as:
 
 “Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.” There are a lot of different ways to install binary or source distributions of software. They are all compatible with the idea of environment modules. For an excellent approach that fits HPC/comp.sci. sites well talk to the EasyBuild guys (the adjacent talk will introduce you to their concept). EasyBuild not only does optimised builds but outputs modulefiles as well!
  • 36. Thank you for your time! E-Mail: azet@azet.org XMPP/Jabber: azet@jabber.ccc.de Twitter: @a_z_e_t GitHub: azet GPG Fingerprint: 0xFDB9B5D5