Cyberinfrastructure Experiences with Apache Airavata
- 1. CI Software experiences with Apache Airavata Suresh Marru, Marlon Pierce Sudhakar Pamidighantam, Chathuri Wimalasena Research Technologies Pervasive Technology Institute/UITS, Indiana University
- 2. Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services Outline 1. Motivation - SciGaP • Operational Sustainability of Science Gateways 2. Reality vs Fantasy with emerging trends • Micro-Services based architecture, DevOps, Cloud Native, Big Data Tools…..
- 3. Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services Acknowledgement: SciGaP project funded by NSF ACI Awards: 1339774, 1339856, and 1339649 PI: Marlon Pierce Co-I: Suresh Marru PI: Mark Miller Co-I: Amit Majumdar PI: Borries Demeler GridChem Science Gateway CIPRES Science Gateway Ultrascan Science Gateway IU Cyberinfrastructure Gateway Neuroscience Gateway
- 4. Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services Knowledge and Expertise Computational Resources Scientific Instruments Algorithms and Models Archived Data and Metadata Advanced Science Tools Science Gateways: Enabling & Democratizing Scientific Research
- 5. Scale number of gateways without having to scale FTE’s needed to support them
- 6. Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services
- 7. Science Gateway Platform as a Service (SciGaP) User Identity Management Information, Monitoring & Auditing Application Programmer Interface CIPRES Science Gateways Neuro Science Ultrascan BioVLABGAAMP DES SimWG Param Chem Graphical Interfaces Admin Dashboards XSEDE OSG Future Grid Data Nets Campus Clusters Academic & Commercial Clouds International Grids Data & Provenance Management Scalable SecureLoad Balanced Configurable Fault Tolerant Maintainable Performance Job & Workflow Management Improve sustainability by converging on a single set of hosted infrastructure services
- 9. Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services Airavata PMC • Aleksander Slominski, IBM • Amila Jayasekara, IU • Ate Douma, One Hippo • Chathura Herath, Knight Capital • Chathuri Wimalasena, IU • Chris A. Mattmann, NASA JPL • Eran Chinthaka, Comprehend • Heshan Suriyaarachchi, • Lahiru Gunathilake, HighFive • Marlon Pierce, IU • Milinda Pathirage, IU • Patanachai Tangchaisin, Ipsy • Raminder Singh, IU • Saminda Wijeratne, Georgia Tech • Shahani Weerawarana, Auxenta • Shameera Rathnayaka, IU • Srinath Perera, WSO2 • Supun Nakandala, UoM • Suresh Marru (Chair), IU • Thilina Gunarathne, KPMG
- 10. Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services Apache Community: Extend Airavata from your project or extend your project from Airavata
- 11. Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services Plans for Continuous Integration & Deployment • Use Ansible to setup a Apache Mesos Cluster • To be deployed on Quarry Gateway Hosting at IU, Amazon EC2, Rackspace and in future on JetStream • Dockerize all Airavata and dependent services • Use Marathon to deploy and maintain docker containers on the mesos cluster.
- 12. Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services Airavata: A simple and open architecture to facilitate external contributions
- 13. Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services • Modern distributed applications are rarely composed of modules written in a single language. • Weaving together innovations made in a range of languages is a core competency of successful enterprises. • Cross language communications are a necessity, not a luxury. Gateways are Polyglot * source: The Programmer’s Guide to Apache Thrift.
- 14. Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services Apache Airavata API’s built over Apache Thrift Clean way to define IDLs with richer data structures
- 15. Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services API’s and CPI’s • External clients interact with Airavata API (based on Apache Thrift). • Internally, components interact with each other through Component Programming Interfaces (CPIs).
- 16. Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services Experiences • Complex use cases evolve over time which challenges the need to preserve a cleaner design. • Support native clients in various languages. • Thrift implementation of API/CPI services provide robust and scalable servicers. Different implementations to switch based on QoS (TSimpleServer, TThreadPoolServer TNonblockingServer, THsHaServer). • No need to marshall / unmarshall data – objects generated from models are used internally as well. • Auto-generation of Client-SDK’s makes it easy to do modifications to the models. • Convenient way to achieve backward compatibility.
- 17. Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services What we learn from fellow Apache Projects: • Working collaboratively. • Constructive criticism is a blessing. • Commit early and commit often. • Test driven development https://cwiki.apache.org/confluence/display/AIRA VATA/Tests+in+Airavata • Don’t be afraid to make a change, after all version control systems (git, svn) can be trusted.
- 18. Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services Community helps find New Stakeholders • Current stakeholders have been slow to adopt cloud computing. • Scientific “big data” could change this. • And we want Airavata to move beyond academic computing. • Where does Airavata fit in the Apache stack?
- 19. Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services Architecture Feedback: architecture@airavata.apache.org Not so fast, My friend
- 20. Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services Community Hangout Mailing lists: • architecture@airavata.apache.org • dev@airavata.apache.org • users@airavata.apache.org Direct Contact: Suresh Marru (smarru@iu.edu) ?? Questions ??