SlideShare a Scribd company logo
Aug. 30. 2018
Quality Engineering in an AI & IoT World
Madhusudan Shekar
Principal Technical Evangelist - Amazon
@madhushekar23
Why are we here today?
OLD WORLD IT
Employees at work
Factories + supply chainSales channels
Marketing analytics
Employees at work
Factories + supply chainSales channels
Marketing analytics
OLD WORLD IT
NEW WORLD IT
NEW WORLD IT Employees at work
Factories +
supply chain
IoT connected
things
Online
marketing
Continuous supply
tracking
Just in time
production
Online sales
+ delivery
Social media
Dimensionality
• Functional
• Performance
• Context Sensitivity
• User Experience
• …
“Earth’s most customer-centric company”
“We see our customers as invited guests to a party, and we are
the hosts. It’s our job every day to make every important aspect
of the customer experience a little bit better.”
Jeff Bezos
• Single-purpose
Connected through
APIs
• Highly decoupled
• “Microservices”
• Circa 2009
DevOps Cycle…
DevOps = efficiencies that speed up this lifecycle
developers customers
releasetestbuild
plan monitor
delivery pipeline
feedback loop
Software development lifecycle
Microservice development lifecycle
developers delivery pipelinesservices
releasetestbuild
releasetestbuild
releasetestbuild
releasetestbuild
releasetestbuild
releasetestbuild
= 50 million deployments a year
Thousands of teams
× Microservice architecture
× Continuous delivery
× Multiple environments
Role Augmentation using ML & IoT
• Code Analysis
• Predict Failures
• Instrument through Data
• Observability
• Statistical
• GAN Networks
Disruption
Quality engineering in a world with AI and IoT
Software – Everywhere…
Fire drills save lives in
the event of a real fire,
because people are
trained how to react
People
Training
Infrastructure
Switching
Application
People
Who runs the
“fire drill” ?
Infrastructure
Switching
Application
People
Chaos
Engineering
Team
Infrastructure
Switching
Application
People
Chaos
Engineering
Team
Infrastructure
Switching
Application
People
Tools
Chaos
Engineering
Team
Infrastructure
Switching
Application
People
Chaos
Engineering
Team
Game days
Tools
Simian Army
chaostoolkit
ChAP
Gremlin
Infrastructure
Switching
Application
People
Tools
Chaos
Engineering
Team
Security
Red
Team
Infrastructure
Switching
Application
People
Tools
Tools
Chaos
Engineering
Team
Security
Red
Team
Infrastructure
Switching
Application
People
Chaos
Engineering
Team
Tools
Security
Red
Team
Safestack
AVA
Infection
Monkey
Chaosslingr
AttackIQ
Tools
SafeBreach
Infrastructure
Switching
Application
People
Chaos
Engineering
Team
Tools
Security
Red
Team
Tools
Four layers
Two teams
An attitude—
Break it to make it better
Chaos
Architecture
Break it to make it safer
For more on the “New View” of Safety see:
Todd Conklin’s Pre-accident podcast
John Allspaw’s stella.report
Synoptic Illegibility
You can’t write down exactly what really
happens, so you can’t write a synopsis or
run-book. System safety is an emergent
property.
The Safety Anarchist
Sydney Decker
Failures are a
system problem—
lack of safety margin
Not something with a root cause
of component or human error
Blindfolded on a cliff edge,
what would you do?
Hypothesis testing
• We think we have safety margin in this
dimension, let’s carefully test to be sure
• In production
• Without causing an issue
Dimensionality ++
• Mechanical
• Electrical & Sensors
• Life Cycle
• Business Impact
• Social Impact
“You can’t legislate against
failure, focus on fast
detection and response.”
—Chris Pinkham
Thank you!!
There’s Never Been A
Better Time
@madhushekar23

More Related Content

PDF
AI driven classification framework for advanced Test Automation
PPTX
THE (IR)RATIONAL INCIDENT RESPONSE: HOW PSYCHOLOGICAL BIASES AFFECT INCIDENT ...
PPTX
ATAGTR2017 Security Testing / IoT Testing in Real World
PPTX
How to Develop and Simulate Models with No Coding Experience
PPTX
ATAGTR2017 Batch Workload Modelling and Performance Optimization
PPTX
Py data scikit-production
PPTX
How to Design, Build and Map IT and Biz Services Breakout Session
PDF
How to Leverage Machine Learning (R, Hadoop, Spark, H2O) for Real Time Proces...
AI driven classification framework for advanced Test Automation
THE (IR)RATIONAL INCIDENT RESPONSE: HOW PSYCHOLOGICAL BIASES AFFECT INCIDENT ...
ATAGTR2017 Security Testing / IoT Testing in Real World
How to Develop and Simulate Models with No Coding Experience
ATAGTR2017 Batch Workload Modelling and Performance Optimization
Py data scikit-production
How to Design, Build and Map IT and Biz Services Breakout Session
How to Leverage Machine Learning (R, Hadoop, Spark, H2O) for Real Time Proces...

What's hot (20)

PPTX
Devops Powered by Splunk
PPTX
Digital Product Security
PDF
Threat hunting with Elastic APM
PPTX
Splunk for Developers
PDF
Software Quality without Testing
PDF
ANALYZER_whole
PPTX
DevOps Powered by Splunk
PPTX
IoT Best Practices: Unit Testing
PDF
Scaling security in a cloud environment v0.5 (Sep 2017)
PPTX
Simulating Different-Network Speeds using JMETER
PDF
Managing Quality of Service for Containerized Microservice Applications
PDF
Towards Scalable Validation of Low-Code System Models: Mapping EVL to VIATRA ...
PPTX
AWS Loft London: Finding the signal in the noise - Effective SecOps with Soph...
PPTX
Splitting the Check on Compliance and Security
PPTX
One Lifecycle One Tool webinar
PDF
Implement Combinatorial Test Patterns for Better Mobile and IoT Testing
PPTX
Splunk for Developers
PPTX
Application performance monitoring with Elastic APM and the ELK stack
PPTX
Containers: DevOp Enablers of Technical Solutions
PPTX
ATAGTR2017 Performance Testing and Non-Functional Testing Strategy for Big Da...
Devops Powered by Splunk
Digital Product Security
Threat hunting with Elastic APM
Splunk for Developers
Software Quality without Testing
ANALYZER_whole
DevOps Powered by Splunk
IoT Best Practices: Unit Testing
Scaling security in a cloud environment v0.5 (Sep 2017)
Simulating Different-Network Speeds using JMETER
Managing Quality of Service for Containerized Microservice Applications
Towards Scalable Validation of Low-Code System Models: Mapping EVL to VIATRA ...
AWS Loft London: Finding the signal in the noise - Effective SecOps with Soph...
Splitting the Check on Compliance and Security
One Lifecycle One Tool webinar
Implement Combinatorial Test Patterns for Better Mobile and IoT Testing
Splunk for Developers
Application performance monitoring with Elastic APM and the ELK stack
Containers: DevOp Enablers of Technical Solutions
ATAGTR2017 Performance Testing and Non-Functional Testing Strategy for Big Da...
Ad

Similar to Quality engineering in a world with AI and IoT (20)

PPTX
Chaos Engineering: Why Breaking Things Should Be Practised.
PPTX
Shoumen_IOT_Tampere
PPTX
Keynote - Chaos Engineering: Why breaking things should be practiced
PPTX
Lunch Keynote
PDF
Don’t Let Technology Slow Down Your Digital Transformation
PDF
Don't Let Technology Slow Down Your Digital Transformation
PDF
Don’t Let Technology Slow Down Your Digital Transformation
PDF
Converged IoT Systems: Bringing the Data Center to the Edge of Everything
PPTX
Digital transformation and AI @Edge
PDF
2019 Global Report Digital transformation | Schneider Electric
PDF
Keynote Roberto Delamora - AWS Cloud Experience Argentina
PPTX
Safety and security in mission critical IoT systems
PDF
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
PDF
Migliorare la Developer Experience in un mondo Cloud Native
PDF
How to make existing business applications io t ready
PDF
Industrial Control System Applications go Mobile in the Cloud 20150825 v1
PDF
Industrial Control Systems Go Mobile in the Cloud
PPTX
Tendências na Transformação Digital
PPTX
AWS Initiate - Tendências da Transformação Digital
PPTX
Industry 4.0 and Digital Transformation
Chaos Engineering: Why Breaking Things Should Be Practised.
Shoumen_IOT_Tampere
Keynote - Chaos Engineering: Why breaking things should be practiced
Lunch Keynote
Don’t Let Technology Slow Down Your Digital Transformation
Don't Let Technology Slow Down Your Digital Transformation
Don’t Let Technology Slow Down Your Digital Transformation
Converged IoT Systems: Bringing the Data Center to the Edge of Everything
Digital transformation and AI @Edge
2019 Global Report Digital transformation | Schneider Electric
Keynote Roberto Delamora - AWS Cloud Experience Argentina
Safety and security in mission critical IoT systems
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
Migliorare la Developer Experience in un mondo Cloud Native
How to make existing business applications io t ready
Industrial Control System Applications go Mobile in the Cloud 20150825 v1
Industrial Control Systems Go Mobile in the Cloud
Tendências na Transformação Digital
AWS Initiate - Tendências da Transformação Digital
Industry 4.0 and Digital Transformation
Ad

More from STePINForum (12)

PPTX
Am I a Social Monster?
PPTX
Code to Release using Artificial Intelligence and Machine Learning
PPTX
Predictive Analytics based Regression Test Optimization
PPTX
Blockchain and AI - Match made in Heaven
PPTX
Machine Learning & Artificial Intelligence - Machine Controlled Data Dispensa...
PPTX
Self Healing - Bringing Intelligence into Automation
PPTX
How accurate are the Wearable fitness tracker showing 10000 steps in a day: A...
PPTX
Performance Testing Internet of Things
PPTX
Continuous Testing: Preparing for DevOps
PPTX
Blockchain and AI: Powering the future
PPTX
Docker–Grid (A On demand and Scalable dockerized selenium grid architecture)
PDF
Jugaadu Automation - The Real Desi Test Automation Story
Am I a Social Monster?
Code to Release using Artificial Intelligence and Machine Learning
Predictive Analytics based Regression Test Optimization
Blockchain and AI - Match made in Heaven
Machine Learning & Artificial Intelligence - Machine Controlled Data Dispensa...
Self Healing - Bringing Intelligence into Automation
How accurate are the Wearable fitness tracker showing 10000 steps in a day: A...
Performance Testing Internet of Things
Continuous Testing: Preparing for DevOps
Blockchain and AI: Powering the future
Docker–Grid (A On demand and Scalable dockerized selenium grid architecture)
Jugaadu Automation - The Real Desi Test Automation Story

Recently uploaded (20)

PPTX
Programs and apps: productivity, graphics, security and other tools
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
PPTX
Machine Learning_overview_presentation.pptx
PDF
Network Security Unit 5.pdf for BCA BBA.
PPTX
sap open course for s4hana steps from ECC to s4
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
PDF
A comparative analysis of optical character recognition models for extracting...
PDF
Chapter 3 Spatial Domain Image Processing.pdf
PPTX
Big Data Technologies - Introduction.pptx
PDF
Building Integrated photovoltaic BIPV_UPV.pdf
PPTX
Cloud computing and distributed systems.
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
PDF
Review of recent advances in non-invasive hemoglobin estimation
PDF
Spectral efficient network and resource selection model in 5G networks
PDF
Unlocking AI with Model Context Protocol (MCP)
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
PPTX
ACSFv1EN-58255 AWS Academy Cloud Security Foundations.pptx
PDF
NewMind AI Weekly Chronicles - August'25-Week II
Programs and apps: productivity, graphics, security and other tools
20250228 LYD VKU AI Blended-Learning.pptx
Machine Learning_overview_presentation.pptx
Network Security Unit 5.pdf for BCA BBA.
sap open course for s4hana steps from ECC to s4
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
A comparative analysis of optical character recognition models for extracting...
Chapter 3 Spatial Domain Image Processing.pdf
Big Data Technologies - Introduction.pptx
Building Integrated photovoltaic BIPV_UPV.pdf
Cloud computing and distributed systems.
Agricultural_Statistics_at_a_Glance_2022_0.pdf
The Rise and Fall of 3GPP – Time for a Sabbatical?
Review of recent advances in non-invasive hemoglobin estimation
Spectral efficient network and resource selection model in 5G networks
Unlocking AI with Model Context Protocol (MCP)
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
ACSFv1EN-58255 AWS Academy Cloud Security Foundations.pptx
NewMind AI Weekly Chronicles - August'25-Week II

Quality engineering in a world with AI and IoT

Editor's Notes

  • #3: Why are we here today? DevOps buzzword – no formal definition Our view of devops, history of amazon, …
  • #8: Established 1994 (seattle)
  • #9: company’s logo is subtly designed to portray a customer’s smile
  • #10: - we took the monolith and broke it apart into a service oriented architecture - factored the app into small, focused, single-purpose services, which we call "primitives" - for example, we had a primitive for displaying the buy button on a product page, and we had one for calculating taxes - every primitive was packaged as a standalone web service, and got an HTTP interface - these building blocks only communicated to each other through the web service interfaces - this created a highly decoupled architecture where these services could be iterated on independently as long as they adhered to their web service interface - to give you an idea of the scope of these small services, I've included this graphic - this is the constellation of services that deliver the Amazon.com website back in 2009, 8 years ago - this term didn't exist back then, but today you'd call this a microservice architecture
  • #11: - I have a confession to make – I used to hate the term "DevOps" - it bugged me because it's a very fuzzy term - people use it in many different ways to mean many different things, so no one really knew what other people are talking about when they hear it - earlier this year, I finally caved to the momentum, and started using it in my talks - I had to admit that "DevOps" came the closest to describing this new modern style of rapid cloud development and delivery that I talked about today - since I'm using this fuzzy term, it's now my responsibility to define it, so at least we're on the same page - but rather than try to define it directly, I'm going to put it in the context of something that all of us are familiar with - the software development lifecycle - here's the general development lifecycle for a web application or service - on one side is the development team, and on the other side are the customers - every new feature or bug fix goes through this same process - developer writes code, code is built and unit tested, app is deployed to a testing environment for deeper testing, finally given a thumbs up and deployed to production where customers can use it - after that happens, the company can collect feedback from customers, make decisions, and continue to iterate and improve the product - there are a few important things to note here - the speed of completing this loop determines your business agility: to go from an idea, to a delivered feature, to learning about it and coming up with the next idea - the faster you can complete that loop, the faster you can innovate - if you can only complete this cycle once a month, you will be outmaneuvered by competitors that can do this every day - another point is that you're only adding value when you're writing code for new changes - the effort you spend in this middle section is lost time - don't get me wrong, you need to ensure high quality releases, but the less your team spends releasing software, the more time they can be writing code - to me, that's the essence of DevOps – to make this process as efficient as possible, and speed up the learning cycle - this is why DevOps is fuzzy, because there are many different ways to optimize this cycle - you can make process changes, organization changes, culture changes, tool changes - to me, they all count, and I think it's fine to classify them all as DevOps
  • #12: - with these new tools, we completed the puzzle - the teams were decoupled and they had the tools necessary to efficiently release on their own
  • #13: - what does success look like - there are a lot of ways that you can measure the process, and no one way is perfect - but here's one data point - when you have thousands of independent teams - producing highly-factored microservices - that are deployed across multiple dev, test, and production environments - in a continuous delivery process - you get a lot of deployments - at Amazon in 2014, we ran over 50M deployments - that's an average of 1.5 deployments every second
  • #15: Why are we here today? DevOps buzzword – no formal definition Our view of devops, history of amazon, …
  • #16: Amazon Go is a new kind of store with no checkout required. We created the world’s most advanced shopping technology so you never have to wait in line. With our Just Walk Out Shopping experience, simply use the Amazon Go app to enter the store, take the products you want, and go! No lines, no checkout. (No, seriously.) No lines, no checkout Our checkout-free shopping experience is made possible by the same types of technologies used in self-driving cars: computer vision, sensor fusion, and deep learning. Our Just Walk Out Technology automatically detects when products are taken from or returned to the shelves and keeps track of them in a virtual cart. When you’re done shopping, you can just leave the store. Shortly after, we’ll charge your Amazon account and send you a receipt.
  • #35: The truth is, the future is always in the making. It’s engineers and wild thinkers that are helping to create the world of tomorrow. It’s customers like tusimple, revolutionizing the commerical trucking industry, arterys saving lives through 4d medical imaging, duolingo teaching language around the world.. It’s customers like you that are helping to build this. Together, we can tackle some of the worlds most challenging problems, and together we can continue to build a smater future.