"In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014
4 likes6,474 views
The document presents insights from the OSConf 2014, focusing on the evolution and impact of open source software. It outlines the author's personal journey in open source, detailing experiences and projects in web operations and data platforms, as well as key technologies used, including programming languages and big data stacks. The overarching vision is to use open source technology to address global issues and improve lives.
"In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014
- 1. OSDConf 2014 “In love with Open Source : Past, Present and Future…“ ~/Piyush http://piyush.me
- 2. In real open source, you have the right to control your own destiny. ~Linus Torvalds OSDConf 2014 : http://piyush.me 2
- 3. Mostly Everyone Uses/Loves Open Source Directly or Indirectly • Linux • Android • Mozilla Firefox • LibreOffice/OpenOffice • Apache HTTP server • Asterisk (PBX) • VLC media player • Wordpress OSDConf 2014 : http://piyush.me 3
- 4. 4 C’s of Open Source • Community • Collaboration • Collective Ownership of intellectual property rights – Distributed Credit • Continuous, public process – Software releases part of the process – Open peer review – Public Bug Tracking – Testing OSDConf 2014 : http://piyush.me 4
- 5. My entry to open source… • I started with using Linux • then started using a classic LAMP stack as Dev • And then in sea of Infrastructure (WebOps) it was open source all around • Now playing with BigData Stack again … its OSS … making our life easier ! OSDConf 2014 : http://piyush.me 5
- 6. Journey…. • 2003-2004 -> Linux User back there in my college days • 2005 -> became first Red Hat Certified Engineer of college • 2006-2008 -> NexTag Journey as QE • 2008-2013 -> MakeMyTrip, Website Operations Team • 2013-till now -> BI / Lead Data Platform Team OSDConf 2014 : http://piyush.me 6
- 7. 2006 -> 2008 • Started as QE – Quality Engineer at NexTag.com (27 months) part of QA- Infrastructure team having worked on managing QA Servers, Testing of NexTag Application (Search Indexes, Cache System, Monitoring Applications) and Automation • Performance Testing of OS, Apache HTTP server, Tomcat + Every New Release • Managing F5 BigIP LTM in QA & Google Mini (Search Engine) OSDConf 2014 : http://piyush.me 7
- 8. Key Learning's • COMMUNITY : IRC #centos channel freenode – Inspired by awesome z00dax : Karanbir Singh • HABIT : IRC #noc channel for production updates @ NexTag Infra • Learned the art of – ./configure && make && make install • Love for BASH #!/bin/bash • SOA – Service Oriented Architecture • Performance Testing - Jmeter/Replay ( Load / Stress ) • Automated Testing – Sahi / Selenium • Code Coverage - Cobertura • Importance of managing Infrastructure like your web applications – Continuous updates / improvements – Automation – Monitoring OSDConf 2014 : http://piyush.me 8
- 9. 2008 -> till date • I played various roles in Web Operations (which demanded multidisciplinary skills ) – System Engineer – Security Operations – Lead Infrastructure – ISM – Information Security Manager – Lead DataPlatformTeam OSDConf 2014 : http://piyush.me 9 hacker@makemytrip
- 10. DDoS Protection Firewall - ASA IPS – Intrusion Detection / Prevention System WAF – Web Application Firewall Load Balancer Web Server Engine / SOA / APIs RDBMS - Transactional Database NoSQL Caching Technologies Virtualization / Cloud Stack Email / DNS / AUTH / Config Management / CI/Deployment CDN – content delivery network Log Management - SIEM Typical Web Infra Stack Obviously including Hardware, Network & Power OSDConf 2014 : http://piyush.me 10
- 11. MMT OSS powered Tech Stack • Programming Languages – JAVA / PHP / Python • Operating System – CentOS 5/6 • Caching proxy – ATS – Apache Traffic Server • Webserver – Apache HTTPD – Nginx • Application Server – Apache Tomcat / Jetty – JBoss • RDBMS – MySQL OSDConf 2014 : http://piyush.me 11
- 12. • NoSQL – SOLR / ElasticSearch (Lucene) – Search – Cassandra – Couchbase • Monitoring – Graphite / Diamond – Nagios – NOCMATE-Rx • Lmetric – Logging Infrastructure – Syslog – Logstash – ElasticSearch – Kibana – Web Dashboard OSDConf 2014 : http://piyush.me 12
- 13. • Cheetah – Selenium , TestNG , Pyvnc2swf – Jenkins • Neerikshan – Observium (network monitoring) • Version Control – GIT, SVN , CVS – GitLab – Git Management • Continuous Integration – Jenkins • Performance Testing – Jmeter / ab / soapui • IDE – Eclipse / vim OSDConf 2014 : http://piyush.me 13
- 14. • Virtualization – XenServer • Configuration Management - Puppet • Application Frameworks – Spring (JAVA) – Laravel (PHP) – Django / Flask (Python) • Knowledge Management – Twiki / Wordpress • CMS – Drupal • Security – OSSIM / AlienVault – OWASP ZAP • …..and many more …. OSDConf 2014 : http://piyush.me 14
- 15. Cheetah • cheetah : Test automation and business service monitoring tool – uses Selenium and Jenkins(Hudson) integration for Real Time monitoring of website with nice Reporting and Alerting – record sessions using Pyvnc2swf OSDConf 2014 : http://piyush.me 15
- 16. Few other hacks • BizEye SynC Tool • MyClient • Image Thumbnail Processing – ImageMagick • MyIdeas – Drupal OSDConf 2014 : http://piyush.me 16
- 17. LBManager • Managing LTM and GTM using Django / PyControl • Single interface for managing the LoadBalancers hosted in multi-colocations / IDCs OSDConf 2014 : http://piyush.me 17
- 18. Graphite/Diamond & NOCMATE-Rx • Monitoring for Operational Excellence – Easy interface to collect data / Metrics – Trends, Dashboards, Alerting , Reporting – User specific/LOB specific dashboards • Desi principle: If it moves, we track it :) • “You cannot manage what you do not measure and to measure you need to monitor” • 5 Categories of Metrics – System | Network | Security | Business | Application • Poller’s – DBPoller, SNMP, Script etc. OSDConf 2014 : http://piyush.me 18
- 19. Graphite : Trends / Analysis / Dashboards OSDConf 2014 : http://piyush.me 19
- 20. Lmetric (Logging Metric) • Centralized Log and event storage and search ! • Powered by : – Logstash – ElasticSearch – Kibana – Web Dashboard OSDConf 2014 : http://piyush.me 20
- 21. Personare : BigData Stack • To collect valuable user interaction information for subsequent use in personalization, recommendations, analysis and reporting. • Layered Architecture (Lambda): – Batch Layer : Hadoop M/R – Speed Layer : Storm NRT Computation – Serving Layer OSDConf 2014 : http://piyush.me 21
- 22. Personare : Architecture OSDConf 2014 : http://piyush.me 22
- 23. OSS powered BigData Stack • Log Collection: Logstash, Syslog-NG • Distributed messaging system : Kafka , RabbitMQ • ETL : Pentaho/ Talend • Hadoop Ecosystem • NRT Computation: Twitter Storm • Document-Oriented NoSQL DB : Couchbase • Distributed Search: ElasticSearch • Distributed messaging system : Kafka , RabbitMQ • NoSQL : Cassandra, Redis, Neo4J (Graph) • API Management : WSO2 API Manager • Programming Languages : Java , Python, R OSDConf 2014 : http://piyush.me 23
- 24. My next hope ! …goal ? • Would I be able to apply what all I have learned in technology through OSS to solve some issues faced by masses around the world ..? – Using technology to make world a better place for everyone ! and I am sure OSS will have major role to play in it !! OSDConf 2014 : http://piyush.me 24
- 25. Thanks! Questions (If any ) ? Contact: @piykumar http://piyush.me OSDConf 2014 : http://piyush.me 25