SlideShare a Scribd company logo
Dev Ops in Big Data
Sujesh Chirackkal
github: https://github.com/sujeshchirackkal
LinkedIn: https://in.linkedin.com/in/sujesh-chirackkal-60270b51
Dev OPS Eco System
Continuous
Improvement
Continuous
Delivery
Process
Simplification
Agile
Continuous
Integration
“It is painful to upgrade, let us not do
it”
Are we ready to upgrade our cluster and applications any time
“It is going to impact business, let us live with current version for some more
time”
Challenges
• Environment readiness
• Changes without backward compatibility
• Time, Resource constraints etc.
Automated system tests using
• Create a pseudo cluster, as a Docker container, of the new version to be
used.
• Write code to perform automated system/integration test along with unit test
cases.
What we did
/**
* testSettings to invoke as "it:run"
*/
def testSettings(): Seq[Setting[_]] = {
// override the run settings, run command on sbt will invoke the tests
run := {
val dockerImg = docker.dockerImage.value
val containerId = dockerImg.startContainer().substring(0, 12)
if(dockerImg.checkIfContainerReady(containerId)) {
val result = dockerImg.runJob(containerId)
} else {
System.exit(-1)
}
dockerImg.stopContainer(containerId)
}
}
Leveraging SBT(Scala Build Tool) test settings
class DockerImage(dockerImageName: String, dockerTag: String, mountDir: String, baseDir: File) {
val localDir = (baseDir)
def startContainer(): String = {
Process(
"docker" :: "run" :: "-d" :: ”-P” ::
"-v" :: s"${localDir}:${mountDir}" :: s"${dockerImageName}:${dockerTag}" :: Nil
) !!
}
def runJob(containerId: String) = {
Process ("docker" :: "exec" :: "-t" :: s"$containerId" :: "/bin/bash" :: "/var/demo/src/main/scripts/run.sh" :: Nil)!
}
def stopContainer(containerId: String): Unit = {
Process("docker" :: "stop" :: s"$containerId" :: Nil)!
}
def checkIfContainerReady(containerId: String): Boolean = {
val status =
Process("docker" :: "exec" :: "-t" :: s"$containerId" :: "/bin/bash" ::
"/var/demo/src/main/scripts/checkContainer.sh" :: Nil)!
if (status != 0) {
stopContainer(containerId)
false
}
else
true
}
}
Managing Docker lifecycle
Linux package mapping plugins in SBT(Scala Build Tool)
• Final deployable is a single file (an rpm file)
• A single command to deploy the application (yum install or yum
update)
• Application artifacts are immutable on the deployed server (only root
can access), also you need root to deploy the same.
Linux package mapping plugins in SBT
linuxPackageMappings <<= (assembly, name, version, sourceDirectory, target) map {
(fatJar: File, name: String, version: String, src: File, target: File) => Seq(
packageMapping(
fatJar -> s"/usr/share/myapp/${version}/${fatJar.getName}"
).withPerms("0755"))}
lazy val myRunScript = TaskKey[File] (”my-run-script","Run Script for the application")
myRunScript <<= (assembly, version, sourceDirectory, target) map { (fatJar: File, version:
String, src: File, target: File) =>
createScript(target / ”my_run_script",
IO.read(src / "main/scripts/my_run_script.sh")
.replace("%myjar%", fatJar.getName))}
linuxPackageMappings <++= (myRunScript , version) map { (script: File, version: String) =>
Seq(
packageMapping(
script -> s”/usr/share/myapp/${version}/${script.getName}"
).withPerms("0755")
)}
Questions ???
Thank you

More Related Content

PPTX
Hadoop enhancements using next gen IA technologies
PPTX
Automating aws infrastructure and code deployments using Ansible @WebEngage
PPTX
Cyansible
PDF
DevOps in a Regulated World - aka 'Ansible, AWS, and Jenkins'
PDF
[OpenInfra Days Korea 2018] Day 2 - E5-1: "Invited Talk: Kubicorn - Building ...
PPTX
Enable IPv6 on Route53 AWS ELB, docker and node App
PDF
Ansible 2 and Ansible Galaxy 2
PDF
Ansible with AWS
Hadoop enhancements using next gen IA technologies
Automating aws infrastructure and code deployments using Ansible @WebEngage
Cyansible
DevOps in a Regulated World - aka 'Ansible, AWS, and Jenkins'
[OpenInfra Days Korea 2018] Day 2 - E5-1: "Invited Talk: Kubicorn - Building ...
Enable IPv6 on Route53 AWS ELB, docker and node App
Ansible 2 and Ansible Galaxy 2
Ansible with AWS

What's hot (20)

PDF
Kube-AWS
PDF
LAMP Stack (Reloaded) - Infrastructure as Code with Terraform & Packer
PDF
Scripting Embulk Plugins
PPTX
Ceph Day San Jose - Enable Fast Big Data Analytics on Ceph with Alluxio
PDF
Ansible and AWS
PDF
OpenNebulaConf 2016 - Icinga2 - APIFY them all by Achim Ledermüller, Netways ...
PDF
The Monitoring Playground
PDF
Continuous Testing with Molecule, Ansible, and GitHub Actions
PDF
Solr on Docker - the Good, the Bad and the Ugly
PDF
Automating Workflows for Analytics Pipelines
PDF
To AWS with Ansible
PDF
Managing Your Cisco Datacenter Network with Ansible
PPTX
MySQL Head-to-Head
PDF
Ansible Intro - June 2015 / Ansible Barcelona User Group
PDF
Monitoring and Log Management for
PPTX
Basics of Ansible - Sahil Davawala
PPTX
ILM - Pipeline in the cloud
ODP
Playing with Hadoop 2013-10-31
PDF
SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...
PPTX
Which Hypervisor is Best?
Kube-AWS
LAMP Stack (Reloaded) - Infrastructure as Code with Terraform & Packer
Scripting Embulk Plugins
Ceph Day San Jose - Enable Fast Big Data Analytics on Ceph with Alluxio
Ansible and AWS
OpenNebulaConf 2016 - Icinga2 - APIFY them all by Achim Ledermüller, Netways ...
The Monitoring Playground
Continuous Testing with Molecule, Ansible, and GitHub Actions
Solr on Docker - the Good, the Bad and the Ugly
Automating Workflows for Analytics Pipelines
To AWS with Ansible
Managing Your Cisco Datacenter Network with Ansible
MySQL Head-to-Head
Ansible Intro - June 2015 / Ansible Barcelona User Group
Monitoring and Log Management for
Basics of Ansible - Sahil Davawala
ILM - Pipeline in the cloud
Playing with Hadoop 2013-10-31
SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...
Which Hypervisor is Best?
Ad

Viewers also liked (20)

PDF
Building a distributed data-platform - A perspective on current trends in co...
PDF
Data Driven Innovation: New Business Models, Products and Services
PPTX
Renouveau et Futures Performances du Décisionnel
PPT
Competitive advantage from Data Mining: some lessons learnt ...
PPTX
Jupyter Ascending: a practical hand guide to galactic scale, reproducible dat...
PPTX
INT2016 Keynote - Emil Pawlowski & Vesna Gordon (Gemius) - Data Science Revol...
PDF
"Where's the data?" The role of metadata in enabling the transformation to a ...
PPTX
Deteo. Data science, Big Data expertise
PDF
SQL PASS BA London 2014 - Data Culture & Future of Analytics
PPTX
SplunkLive! London 2016 Splunk for Devops
PDF
XebiConFr 15 - Voyages-sncf.com - Les apports de la Data Science à la connais...
PDF
Ellie Mirman - Creating an Agile, Data-Driven Marketing Team
PDF
Yhat - Applied Data Science - Feb 2016
PDF
Open Data Science Conference Agile Data
PPTX
Du craft chez les OPS
PDF
Resilient Predictive Data Pipelines (GOTO Chicago 2016)
PDF
software architecture cant fight lean startup
PDF
Microsoft DevOps Day 2015 02122015 - L'expérience du groupe produit Visual St...
PDF
The Nitty Gritty of Advanced Analytics Using Apache Spark in Python
PPTX
Alexander Sokolov “How Data Science and Big Data are changing the World”
Building a distributed data-platform - A perspective on current trends in co...
Data Driven Innovation: New Business Models, Products and Services
Renouveau et Futures Performances du Décisionnel
Competitive advantage from Data Mining: some lessons learnt ...
Jupyter Ascending: a practical hand guide to galactic scale, reproducible dat...
INT2016 Keynote - Emil Pawlowski & Vesna Gordon (Gemius) - Data Science Revol...
"Where's the data?" The role of metadata in enabling the transformation to a ...
Deteo. Data science, Big Data expertise
SQL PASS BA London 2014 - Data Culture & Future of Analytics
SplunkLive! London 2016 Splunk for Devops
XebiConFr 15 - Voyages-sncf.com - Les apports de la Data Science à la connais...
Ellie Mirman - Creating an Agile, Data-Driven Marketing Team
Yhat - Applied Data Science - Feb 2016
Open Data Science Conference Agile Data
Du craft chez les OPS
Resilient Predictive Data Pipelines (GOTO Chicago 2016)
software architecture cant fight lean startup
Microsoft DevOps Day 2015 02122015 - L'expérience du groupe produit Visual St...
The Nitty Gritty of Advanced Analytics Using Apache Spark in Python
Alexander Sokolov “How Data Science and Big Data are changing the World”
Ad

Similar to Dev ops meetup (20)

PDF
(Declarative) Jenkins Pipelines
PDF
DevOps Odessa #TechTalks 21.01.2020
PDF
DevOPS training - Day 2/2
PDF
Jenkins Pipelines
PPTX
Docker Starter Pack
PDF
Server(less) Swift at SwiftCloudWorkshop 3
PDF
Antons Kranga Building Agile Infrastructures
PDF
Test-Driven Infrastructure with Chef
PDF
Deploying Docker (Provisioning /w Docker + Chef/Puppet) - DevopsDaysPGH
PDF
Continuous Integration/Deployment with Docker and Jenkins
PDF
Scala, docker and testing, oh my! mario camou
PDF
Integration tests: use the containers, Luke!
PDF
The Fairy Tale of the One Command Build Script
PDF
PDXPortland - Dockerize Django
PPTX
Moving from Jenkins 1 to 2 declarative pipeline adventures
PPT
Play!ng with scala
PDF
Fargate 를 이용한 ECS with VPC 1부
PPTX
Continuous Integration & Development with Gitlab
PDF
Advanced debugging  techniques in different environments
PDF
Excelian hyperledger walkthrough-feb17
(Declarative) Jenkins Pipelines
DevOps Odessa #TechTalks 21.01.2020
DevOPS training - Day 2/2
Jenkins Pipelines
Docker Starter Pack
Server(less) Swift at SwiftCloudWorkshop 3
Antons Kranga Building Agile Infrastructures
Test-Driven Infrastructure with Chef
Deploying Docker (Provisioning /w Docker + Chef/Puppet) - DevopsDaysPGH
Continuous Integration/Deployment with Docker and Jenkins
Scala, docker and testing, oh my! mario camou
Integration tests: use the containers, Luke!
The Fairy Tale of the One Command Build Script
PDXPortland - Dockerize Django
Moving from Jenkins 1 to 2 declarative pipeline adventures
Play!ng with scala
Fargate 를 이용한 ECS with VPC 1부
Continuous Integration & Development with Gitlab
Advanced debugging  techniques in different environments
Excelian hyperledger walkthrough-feb17

Recently uploaded (20)

PPTX
mbdjdhjjodule 5-1 rhfhhfjtjjhafbrhfnfbbfnb
PPTX
advance b rammar.pptxfdgdfgdfsgdfgsdgfdfgdfgsdfgdfgdfg
PPTX
climate analysis of Dhaka ,Banglades.pptx
PPTX
The THESIS FINAL-DEFENSE-PRESENTATION.pptx
PPTX
iec ppt-1 pptx icmr ppt on rehabilitation.pptx
PPT
Chapter 2 METAL FORMINGhhhhhhhjjjjmmmmmmmmm
PPT
Miokarditis (Inflamasi pada Otot Jantung)
PPTX
Introduction to Knowledge Engineering Part 1
PPTX
DISORDERS OF THE LIVER, GALLBLADDER AND PANCREASE (1).pptx
PPTX
Introduction-to-Cloud-ComputingFinal.pptx
PPT
Chapter 3 METAL JOINING.pptnnnnnnnnnnnnn
PPT
Quality review (1)_presentation of this 21
PPTX
05. PRACTICAL GUIDE TO MICROSOFT EXCEL.pptx
PDF
.pdf is not working space design for the following data for the following dat...
PPTX
Major-Components-ofNKJNNKNKNKNKronment.pptx
PPTX
oil_refinery_comprehensive_20250804084928 (1).pptx
PPTX
IB Computer Science - Internal Assessment.pptx
PPTX
1_Introduction to advance data techniques.pptx
PDF
Recruitment and Placement PPT.pdfbjfibjdfbjfobj
mbdjdhjjodule 5-1 rhfhhfjtjjhafbrhfnfbbfnb
advance b rammar.pptxfdgdfgdfsgdfgsdgfdfgdfgsdfgdfgdfg
climate analysis of Dhaka ,Banglades.pptx
The THESIS FINAL-DEFENSE-PRESENTATION.pptx
iec ppt-1 pptx icmr ppt on rehabilitation.pptx
Chapter 2 METAL FORMINGhhhhhhhjjjjmmmmmmmmm
Miokarditis (Inflamasi pada Otot Jantung)
Introduction to Knowledge Engineering Part 1
DISORDERS OF THE LIVER, GALLBLADDER AND PANCREASE (1).pptx
Introduction-to-Cloud-ComputingFinal.pptx
Chapter 3 METAL JOINING.pptnnnnnnnnnnnnn
Quality review (1)_presentation of this 21
05. PRACTICAL GUIDE TO MICROSOFT EXCEL.pptx
.pdf is not working space design for the following data for the following dat...
Major-Components-ofNKJNNKNKNKNKronment.pptx
oil_refinery_comprehensive_20250804084928 (1).pptx
IB Computer Science - Internal Assessment.pptx
1_Introduction to advance data techniques.pptx
Recruitment and Placement PPT.pdfbjfibjdfbjfobj

Dev ops meetup

  • 1. Dev Ops in Big Data Sujesh Chirackkal github: https://github.com/sujeshchirackkal LinkedIn: https://in.linkedin.com/in/sujesh-chirackkal-60270b51
  • 2. Dev OPS Eco System Continuous Improvement Continuous Delivery Process Simplification Agile Continuous Integration
  • 3. “It is painful to upgrade, let us not do it” Are we ready to upgrade our cluster and applications any time “It is going to impact business, let us live with current version for some more time”
  • 4. Challenges • Environment readiness • Changes without backward compatibility • Time, Resource constraints etc. Automated system tests using • Create a pseudo cluster, as a Docker container, of the new version to be used. • Write code to perform automated system/integration test along with unit test cases. What we did
  • 5. /** * testSettings to invoke as "it:run" */ def testSettings(): Seq[Setting[_]] = { // override the run settings, run command on sbt will invoke the tests run := { val dockerImg = docker.dockerImage.value val containerId = dockerImg.startContainer().substring(0, 12) if(dockerImg.checkIfContainerReady(containerId)) { val result = dockerImg.runJob(containerId) } else { System.exit(-1) } dockerImg.stopContainer(containerId) } } Leveraging SBT(Scala Build Tool) test settings
  • 6. class DockerImage(dockerImageName: String, dockerTag: String, mountDir: String, baseDir: File) { val localDir = (baseDir) def startContainer(): String = { Process( "docker" :: "run" :: "-d" :: ”-P” :: "-v" :: s"${localDir}:${mountDir}" :: s"${dockerImageName}:${dockerTag}" :: Nil ) !! } def runJob(containerId: String) = { Process ("docker" :: "exec" :: "-t" :: s"$containerId" :: "/bin/bash" :: "/var/demo/src/main/scripts/run.sh" :: Nil)! } def stopContainer(containerId: String): Unit = { Process("docker" :: "stop" :: s"$containerId" :: Nil)! } def checkIfContainerReady(containerId: String): Boolean = { val status = Process("docker" :: "exec" :: "-t" :: s"$containerId" :: "/bin/bash" :: "/var/demo/src/main/scripts/checkContainer.sh" :: Nil)! if (status != 0) { stopContainer(containerId) false } else true } } Managing Docker lifecycle
  • 7. Linux package mapping plugins in SBT(Scala Build Tool) • Final deployable is a single file (an rpm file) • A single command to deploy the application (yum install or yum update) • Application artifacts are immutable on the deployed server (only root can access), also you need root to deploy the same.
  • 8. Linux package mapping plugins in SBT linuxPackageMappings <<= (assembly, name, version, sourceDirectory, target) map { (fatJar: File, name: String, version: String, src: File, target: File) => Seq( packageMapping( fatJar -> s"/usr/share/myapp/${version}/${fatJar.getName}" ).withPerms("0755"))} lazy val myRunScript = TaskKey[File] (”my-run-script","Run Script for the application") myRunScript <<= (assembly, version, sourceDirectory, target) map { (fatJar: File, version: String, src: File, target: File) => createScript(target / ”my_run_script", IO.read(src / "main/scripts/my_run_script.sh") .replace("%myjar%", fatJar.getName))} linuxPackageMappings <++= (myRunScript , version) map { (script: File, version: String) => Seq( packageMapping( script -> s”/usr/share/myapp/${version}/${script.getName}" ).withPerms("0755") )}

Editor's Notes

  • #7: Docker start, stop and remove