mabl's Machine Learning Implementation on Google Cloud Platform
2 likes237 views
The document discusses mabl's implementation of machine learning for software testing using Google Cloud Platform, highlighting challenges in software quality assurance due to complexity and slow processes. It outlines areas of focus such as anomaly detection and auto-healing tests, alongside the architecture of model feedback loops and the impact of serverless and event-driven designs. Overall, mabl has rapidly developed its product to process a significant volume of data efficiently with a small team.
mabl's Machine Learning Implementation on Google Cloud Platform
- 1. mabl’s ML Implementation Testing @ Scale with Google Cloud Platform Joseph Lust mabl engineer @lustcoder
- 2. About Me: Joseph Lust Engineer ■ Building the web for two decades ■ Cloud native since 2011 ■ Currently building mabl ML cloud on GCP ■ Co-organizer GDG Cloud Boston Meetup
- 3. Agenda ■ mabl’s problem statement ■ ML on Google Cloud ■ Cloud best practices at mabl ■ Q & A
- 5. Developing Quality Software is Slow Image Source: IBM SAGE ■ Software is complex ■ Testing software is slow ■ Humans have low clock speeds ■ Humans are expensive
- 6. Continuous Delivery breaks QA ■ Change happens infrequently ■ Weeks or months to write tests ■ Weeks to execute tests ■ Change is constant ■ Hours to write tests ■ Minutes to execute tests
- 7. The Doughnut Hole: Continuous Testing (CT) Source: Google Trends Hudson Released Facebook CD Article
- 8. Continuous Testing Roadmap ■ Highly parallel testing (fail fast) ■ Automatically heal broken scripts ■ Automatically generate test scripts Manual Testing Manually Scripted Testing Playback Testing Auto Test Generation Auto-healing Testing ✔ ✔ ✔ ✔
- 9. the ML @ mabl
- 10. Machine Learning Areas @ mabl ■ Anomaly detection ▲ Timing anomalies ▲ Visual change anomalies ■ Auto-healing tests ▲ Alternative element candidate election ▲ Multi-branch evaluation
- 14. Why anomaly detection isn’t quite that easy ■ Domain variability one app’s anomaly is another’s normal ■ Temporal variability today’s anomaly is tomorrow’s normal ■ Detection quality not all anomalies are bad (or important) ■ Initial quality what can we detect with minimal app-specific data
- 15. Approach ■ Modeling the app ▲ domain variability, quality ■ Incremental learning ▲ temporal variability, initial quality ■ User feedback ▲ use case variability, quality
- 16. Analysis pipeline Observations ▲ Raw data/events Measurements ▲ Calculated features State detections ▲ Aggregate measurements ▲ Apply detection models (abstraction) ▲ Integrate detections (higher abstraction) Insights ▲ Recognize significant state changes
- 17. Signals for anomaly detection ■ Visual appearance ■ Performance ■ Failures ■ Errors (non-fatal) ■ Page structure changes
- 18. visual anomalies
- 30. model training and feedback
- 31. Pieces of the puzzle ■ Cloud ML Engine ▲ Tensorflow machine learning models ▲ Managed model training ▲ Online prediction service ■ Dataflow ▲ Streaming data processing ▲ I/O connectors for PubSub, etc. ■ Datastore ▲ Augment trained model with mutable data ▲ Fast query by type and filter (e.g., by key)
- 32. Architecture of model feedback loops
- 33. harnessing google cloud for ML
- 34. mabl cloud design tenets ■ Serverless ■ Decoupled sub systems ■ Event driven architecture
- 35. Serverless just works ■ No Provisioning ■ Transparent Scaling ■ Event Driven ■ Pay only for Use 35
- 36. e.g. Kubernetes Engine ■ NoOps - developers’ containers in prod ■ Language agnostic ■ Massively scale ▲ running millions of containers a month ■ “Set it and forget it!” 36 Kubernetes Engine
- 37. Decoupled Architecture ■ Message passing ■ Polyglot development ■ Independent deployment ■ Minimize blast radius Cloud Pub/Sub Cloud Functions
- 38. Event Driven Architecture ■ No “cron jobs” or batch processes ■ Continuous pipelines ■ Surge buffers ■ 100s ms to 1-2s E2E handling 38 Cloud Dataflow Cloud Functions
- 39. Google Cloud’s Impact on mabl ■ From zero to alpha product in ~6mo with 8 developers ▲ Processing ~100M pages per month ■ Systems designed for scale on day one ■ No dreaded “rewrite” ■ Product families work very well together out of the box
- 40. Questions?