Abstract image of a woman sitting on books and studying on a laptop

2013.10 Operating * by the Numbers

Allison Miller, profile picture
Allison Miller

The document discusses the importance of using models in operations to enhance process improvements, emphasizing the transformative nature of operations in converting inputs into outputs. It explores various modeling techniques, risk management, and decision-making tools while highlighting the significance of data-driven strategies in optimizing performance and automating processes. Ultimately, it advocates for leveraging analytics and metrics to inform operational decisions and improve efficiency in various domains.

Technology
1 of 67
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
OPERATING * BY THE NUMBERS Allison Miller! @selenakyle
Overview ! How we got here! ! Improving systems using models! ! Model building! ! Back to the Numbers! ! Beg, Borrow, Steal
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
2013.10 Operating * by the Numbers
A Shift to Operations ! Life at Layer 8! ! The modern operating environment! High complexity! High stakes! ! Operations! Process of transforming inputs into outputs Layer 8 Say You Would What Here? You Do Transport Session Presentation Application Physical Network Data Link Business Logic
The Modern World ! Buzzword Bingo! Big Data / NoSQL / Graph DB’s! Machine Learning! Agile development/delivery (aka Dev Ops)! Cloud / Anything...as a Service! ! The New Hotness is Old School! Management science ! Operations research! Decision Science! Six Sigma / TQM / Kai Zen
Who Cares ! Relevant to control systems! ! Tools to improve running an operation/business! Automation! Optimization! Prediction / Forecasting! ! Modeling as an operations tool
I’m no model lady. A model’s just an imitation of the real thing. ! –Mae West
Improving Systems Using Models ! What are models! Not reality, but an approximation! 90% likelihood vs 90% of behavior observed! ! Why do we employ models! Design (how to build/design a system)! Management (goal setting & performance monitoring)! Live / Production / Operations (automation)! ! How do we know if they work? Abstraction Realism Prescriptive Descriptive Combat Modeling Spectrum Washburn & Kress, Combat Modeling, International Series in Operations Research & Management
Quality cannot be improved by trying harder. ! –W.E. Deming
Operating Better Systems ! Operations – a transformative process that converts inputs into outputs
Example: Data Driven Defense ! What’s a risk decisioning system?! ! Where do you put it?! ! What does it cost?! ! What do you need to build it?! ! How do you build it?! ! Operating Risk by the numbers! Forecasting / Prediction! Automation! Optimization
Big Data & Little Loops
123.123.123.123 - - [26/Apr/2000:00:23:48 -0400] "GET /pics/wpaper.gif HTTP/1.0" 200 6248 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:47 -0400] "GET /asctortf/ HTTP/1.0" 200 8130 "http://search.netscape.com/Computers/Data_Formats/Document/Text/ RTF" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:48 -0400] "GET /pics/5star2000.gif HTTP/1.0" 200 4005 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" [Tue Mar 9 22:02:41 2004] [info] created shared memory segment #10813446[Tue Mar 9 22:02:41 2004] [notice] Apache/1.3.29 (Unix) mod_ssl/ 2.8.16 OpenSSL/0.9.7c configured -- resuming normal operations[Tue Mar 9 22:02:41 2004] [info] Server built: Mar 7 2004 13:38:59pausing [http:// xmlrevenue.com/s.php?username=jenneypan&keywords=Online+Gambling] for 50000 ms[Tue Mar 9 22:04:16 2004] [error] [client 218.93.92.137] mod_security: Access denied with code 200. Pattern match "Basic" at HEADER.[Tue Mar 9 22:07:16 2004] [error] [client 203.121.182.190] mod_security: Invalid character detected [4] 123.123.123.123 - - [26/Apr/2000:00:23:50 -0400] "GET /pics/5star.gif HTTP/ 1.0" 200 1031 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:51 -0400] "GET /pics/a2hlogo.jpg HTTP/1.0" 200 4282 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:51 -0400] "GET /cgi-bin/newcount? jafsof3&width=4&font=digital&noshow HTTP/1.0" 200 36 "http:// www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" [Tue Mar 9 22:02:41 2004] [notice] Accept mutex: sysvsem (Default: sysvsem)[Tue Mar 9 22:03:26 2004] [error] [client 218.93.92.137] mod_security: [Tue Mar 9 22:07:16 2004] [error] [client 203.121.182.190] mod_security: Invalid character detected [4] 123.123.123.123 - - [26/Apr/2000:00:23:50 -0400] "GET /pics/5star.gif HTTP/ 1.0" 200 1031 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:51 -0400] "GET /pics/a2hlogo.jpg HTTP/1.0" 200 4282 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:51 -0400] "GET /cgi-bin/newcount? jafsof3&width=4&font=digital&noshow HTTP/1.0" 200 36 "http:// www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" [Tue Mar 9 22:02:41 2004] [notice] Accept mutex: sysvsem (Default: sysvsem) Big Data & Little Loops
123.123.123.123 - - [26/Apr/2000:00:23:48 -0400] "GET /pics/wpaper.gif HTTP/1.0" 200 6248 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:47 -0400] "GET /asctortf/ HTTP/1.0" 200 8130 "http://search.netscape.com/Computers/Data_Formats/Document/Text/ RTF" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:48 -0400] "GET /pics/5star2000.gif HTTP/1.0" 200 4005 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" [Tue Mar 9 22:02:41 2004] [info] created shared memory segment #10813446[Tue Mar 9 22:02:41 2004] [notice] Apache/1.3.29 (Unix) mod_ssl/ 2.8.16 OpenSSL/0.9.7c configured -- resuming normal operations[Tue Mar 9 22:02:41 2004] [info] Server built: Mar 7 2004 13:38:59pausing [http:// xmlrevenue.com/s.php?username=jenneypan&keywords=Online+Gambling] for 50000 ms[Tue Mar 9 22:04:16 2004] [error] [client 218.93.92.137] mod_security: Access denied with code 200. Pattern match "Basic" at HEADER.[Tue Mar 9 22:07:16 2004] [error] [client 203.121.182.190] mod_security: Invalid character detected [4] 123.123.123.123 - - [26/Apr/2000:00:23:50 -0400] "GET /pics/5star.gif HTTP/ 1.0" 200 1031 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:51 -0400] "GET /pics/a2hlogo.jpg HTTP/1.0" 200 4282 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:51 -0400] "GET /cgi-bin/newcount? jafsof3&width=4&font=digital&noshow HTTP/1.0" 200 36 "http:// www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" [Tue Mar 9 22:02:41 2004] [notice] Accept mutex: sysvsem (Default: sysvsem)[Tue Mar 9 22:03:26 2004] [error] [client 218.93.92.137] mod_security: [Tue Mar 9 22:07:16 2004] [error] [client 203.121.182.190] mod_security: Invalid character detected [4] 123.123.123.123 - - [26/Apr/2000:00:23:50 -0400] "GET /pics/5star.gif HTTP/ 1.0" 200 1031 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:51 -0400] "GET /pics/a2hlogo.jpg HTTP/1.0" 200 4282 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:51 -0400] "GET /cgi-bin/newcount? jafsof3&width=4&font=digital&noshow HTTP/1.0" 200 36 "http:// www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" [Tue Mar 9 22:02:41 2004] [notice] Accept mutex: sysvsem (Default: sysvsem) Big Data & Little Loops
* Loop Disposition: Logic, Human, or Other?
Big Data & Little Loops Why are you picking on me?Boo-yah! Still getting away with it. <Sigh> Nobody understands me.
SHALL WE PLAY A GAME? (SINCE WE CAN’T PLAY “CLUE” FOR EVERY LOGIN TRANSACTION NEW USER MESSAGE FRIEND REQUEST ATTACHMENT PACKET WINK POKE CLICK BIT WE BUILD RISK MODELS)
Applying Decisions Risk management is decision management ACTOR ATTEMPTS ACTION SuSUBMIT WHAT IS THE REQUEST HOW TO HONOR THE REQUEST SHOULD WE HONOR? RESULT ACTION OCCURS
Applied where? Where risks manifest in observable behavior Where system owners make decisions Where controls can be optimized by better recognizing identity, intent, or change
Decisions, Decisions Authorize Block Good false positive Bad false negative RESPONSE POPULATION Incorrect decisions have a cost Correct decisions are free (usually) Good Action Gets Blocked Bad Action Gets Through Downstream Impacts
Such as... Populations - Users, Transactions, Messages, Packets, API calls, Files! Actions - Allow, Block, Challenge, Review, Retry, Quarantine, Add privileges, Upgrade privileges, Make Offer! Costs - Fraud, Data leakage, Customer churn, Customer contacts, Downstream liability
For example: ACTOR ATTEMPTS Payment p (actor attempting payment is accountholder) Decision Authorize Review Refer Request Authentication Decline f(variable A + Variable B + ...) SuSUBMIT
Flavors of Risk Models I deviate significantly from a normal (good) pattern I summarize a known bad pattern fa(x), fb(x), fc(x) fq(x), fr(x), fs(x)
What is normal? http://en.wikipedia.org/wiki/Normal_distribution WHAT IS BAD? WHAT IS GOOD?
Model Development Process Target ! Yes/No questions best Find Data, Variable Creation ! Best part Data Prep ! Worst part Model Training ! Pick an algorithm Assessment ! Catch vs FP rate Deployment ! Decisioning vs Detection
User IP Country <> Billing Country Buying prepaid mobile phones Add new shipping address in cart Buyer = Phone reseller, static machine ID How much $$ is at risk? What is “normal” for this customer? What “bad” profiles does this match? Geolocate IP Convert geo to country code Flag on Mismatch Cart Category Merch Risk Level Date Added Address Type String Matching Customer Profile Device ID Device HistoryTXN-$-AMT Churn Risk, CLV, ... TXNs, logins, ... Stolen CC, Collusion
Model Training Some algorithms: - Regression: Determines the best equation describe relationship between control variable and independent variables! Linear Regression: Best equation is a line! Logistic Regression: Best equation is a curve (exponential properties)! - Bayesian: Used to estimate regression models, useful when working w/small data sets ! - Neural Nets: Can approximate any type of non-linear function, often highly predictive, but doesn’t explain the relationship between control and independent variables
LOGISTIC <DEPVAR> <VAR1> <VAR2>...
p-value of significance, throw out if > .05 Variance in dependent variable explained by independent variables Dependent Variable Independent Variables Factor odds of dependent go up when independent var incremented p-value should be < significance level (.05)
2013.10 Operating * by the Numbers
Operating a Risk System Disposition & Time Email CC# Items Total ! Submit Maybe ! ! No! ! ! Yes!! ! ! SuOutcomeSuAttempt Black & Whitelists Machine Learning Velocity & Spend caps Geo & IP Logic Linking Data • Reporting • Metrics • Analysis • Modeling Good Bad Indeterminate
The Better Mousetrap Automates defensive action x-platform - Fast ! - Accurate! - Cheap In Real Time In Time to Minimize Loss Reasonable False Positives As good as a human specialistReduces More Loss than Cost Created Cheaper than Manual intervention
GAIN More gain/lift = more efficient predictions Catch as much as possible (as much of the “bads”) Minimize the overall affected % of population
Cost Number of Defects Produced Cost of Control Cost of Defects Total Cost “Alice: Which way should I go? Cat: That depends on where you are going. Alice: I don’t know. Cat: Then it doesn’t matter which way you go.” ― Lewis Carroll, Alice in Wonderland
% of population Cost Number of Defects Produced Cost of Control Cost of Defects CV Total CostCV
Finding the * approach in the wild ! Operating * by the numbers in many disciplines! Automation! Optimization! Forecasting / Prediction! ! Such as…! Science ! Finance! Marketing / Advertising! Software Development! Site/Network Ops! Manufacturing! Military Is all fun and game until you are need of put it in production – @devopsborat
Beg, Borrow, Steal ! A/B Testing! ! Control Charts! ! Highly engaged change management! ! Sample strategy! ! Instrument everything! ! Poka-Yoke
Recap Operating systems effectively means: - Using data to understand and improve performance! - Using tools to:! - Automate (Efficiency, Scale, Standardization)! - Optimize (Set goals cognizant of tradeoffs)! - Forecast / Predict (Plan, course correct)! Designing data-driven defenses - Decisions that can be automated w/data! - Where/what data sets to use! - Business drivers to keep in mind ! Numbers, Numbers, Numbers p (bad) f(variable A + Variable B + ...)
Prediction is very difficult, especially about the future Niels Bohr Allison Miller @selenakyle
Metrics vs Analytics METRICS ANALYTICS
Such as... Metrics Analytics $ Loss Txns Purchase trends of high loss users # Compromised Accts IP Sources of bad login attempts % of Spam Messages Delivered Spam subject lines generating most clicks Minutes of downtime Most process-intensive applications # Customer Contacts Generated Highest-contact exception flows
The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. " The second is that automation applied to an inefficient operation will magnify the inefficiency." –Bill Gates

More Related Content

PDF
2012.09 A Million Mousetraps: Using Big Data and Little Loops to Build Better...
Allison Miller
 
PDF
2011.04 How to Isotope Tag a Ghost
Allison Miller
 
PDF
Boomtime: Risk as Economics (Allison Miller, SiRAcon15)
Allison Miller
 
PPTX
You Spent All That Money And Still Got Owned
Joe McCray
 
PDF
DSS ITSEC Conference 2012 - Radware WAF
Andris Soroka
 
PPT
Data Driven Security, from Gartner Security Summit 2012
Nick Galbreath
 
PDF
Web Application Scanning 101
Sasha Nunke
 
PDF
conf2014_PeterLam_Splunk_Security
peter lam
 
2012.09 A Million Mousetraps: Using Big Data and Little Loops to Build Better...
Allison Miller
 
2011.04 How to Isotope Tag a Ghost
Allison Miller
 
Boomtime: Risk as Economics (Allison Miller, SiRAcon15)
Allison Miller
 
You Spent All That Money And Still Got Owned
Joe McCray
 
DSS ITSEC Conference 2012 - Radware WAF
Andris Soroka
 
Data Driven Security, from Gartner Security Summit 2012
Nick Galbreath
 
Web Application Scanning 101
Sasha Nunke
 
conf2014_PeterLam_Splunk_Security
peter lam
 

Similar to 2013.10 Operating * by the Numbers (20)

PPTX
It security the condensed version
Brian Pichman
 
PDF
web application security
ahmed sami
 
KEY
Do it-yourself-audits
Johann-Peter Hartmann
 
PDF
Cybersecurity Roadmap for Beginners
Sanjeev Kumar Jaiswal
 
PDF
Real life hacking101
Florent Batard
 
PDF
Web Security
Gerald Villorente
 
PPTX
Hacking WebApps for fun and profit : how to approach a target?
Yassine Aboukir
 
PDF
BSides Lisbon - Data science, machine learning and cybersecurity
Tiago Henriques
 
PDF
Layer one 2011-joe-mccray-you-spent-all-that-money-and-still-got-0wned
fangjiafu
 
PPS
An introduction to Digital Security - Rishabh Dangwal
Rishabh Dangwal
 
ODP
A journey through an INFOSEC labyrinth
Avădănei Andrei
 
PDF
Application security enterprise strategies
Network Intelligence India
 
PPT
Bsides-Philly-2016-Finding-A-Companys-BreakPoint
Zack Meyers
 
PPT
The Rise and Rise of Web Fraud
David Jones
 
PDF
Drupal Camp Bristol 2017 - Website insecurity
George Boobyer
 
PDF
Web Application Security: Winning When The Odds Are Against You
bendechrai
 
PPT
Security Vulnerabilities
Marius Vorster
 
PPTX
Implementing security for your library | PLAN Tech Day Conference
Brian Pichman
 
PDF
Is the Web at Risk?
Carlos Serrao
 
PPTX
Hardening Enterprise Apache
guestd9aa5
 
It security the condensed version
Brian Pichman
 
web application security
ahmed sami
 
Do it-yourself-audits
Johann-Peter Hartmann
 
Cybersecurity Roadmap for Beginners
Sanjeev Kumar Jaiswal
 
Real life hacking101
Florent Batard
 
Web Security
Gerald Villorente
 
Hacking WebApps for fun and profit : how to approach a target?
Yassine Aboukir
 
BSides Lisbon - Data science, machine learning and cybersecurity
Tiago Henriques
 
Layer one 2011-joe-mccray-you-spent-all-that-money-and-still-got-0wned
fangjiafu
 
An introduction to Digital Security - Rishabh Dangwal
Rishabh Dangwal
 
A journey through an INFOSEC labyrinth
Avădănei Andrei
 
Application security enterprise strategies
Network Intelligence India
 
Bsides-Philly-2016-Finding-A-Companys-BreakPoint
Zack Meyers
 
The Rise and Rise of Web Fraud
David Jones
 
Drupal Camp Bristol 2017 - Website insecurity
George Boobyer
 
Web Application Security: Winning When The Odds Are Against You
bendechrai
 
Security Vulnerabilities
Marius Vorster
 
Implementing security for your library | PLAN Tech Day Conference
Brian Pichman
 
Is the Web at Risk?
Carlos Serrao
 
Hardening Enterprise Apache
guestd9aa5
 
Ad

More from Allison Miller (7)

PDF
Something Wicked
Allison Miller
 
PPTX
When Algorithms Are Our Co-Pilots
Allison Miller
 
PDF
2010.08 Applied Threat Modeling: Live (Hutton/Miller)
Allison Miller
 
PDF
2013.05 Games We Play: Payoffs & Chaos Monkeys
Allison Miller
 
PDF
2014.06 Defending Debit
Allison Miller
 
PDF
2014.04 Bit, Bit, Coin
Allison Miller
 
PDF
2012.12 Games We Play: Defenses & Disincentives
Allison Miller
 
Something Wicked
Allison Miller
 
When Algorithms Are Our Co-Pilots
Allison Miller
 
2010.08 Applied Threat Modeling: Live (Hutton/Miller)
Allison Miller
 
2013.05 Games We Play: Payoffs & Chaos Monkeys
Allison Miller
 
2014.06 Defending Debit
Allison Miller
 
2014.04 Bit, Bit, Coin
Allison Miller
 
2012.12 Games We Play: Defenses & Disincentives
Allison Miller
 
Ad

Recently uploaded (20)

PDF
Univ-Connecticut-ChatGPT-Presentaion.pdf
ry7r52mzb4
 
PDF
A Late Bloomer's Guide to GenAI: Ethics, Bias, and Effective Prompting - Boha...
Mindy Bohannon
 
PDF
Hindi spoken digit analysis for native and non-native speakers
IAESIJAI
 
PDF
Assigned Numbers - 2025 - Bluetooth® Document
Bloombase
 
PDF
1 - Historical Antecedents, Social Consideration.pdf
tuazon2030627
 
PDF
Zenith AI: Advanced Artificial Intelligence
Biz Preneurs
 
PPT
Geologic Time for studying geology for geologist
ssuser738f5a
 
PDF
Transform Your ITIL® 4 & ITSM Strategy with AI in 2025.pdf
PDCA Consulting
 
PPTX
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
PDF
sustainability-14-14877-v2.pddhzftheheeeee
VenkateshGovindaraja9
 
PDF
A contest of sentiment analysis: k-nearest neighbor versus neural network
IAESIJAI
 
PDF
How ambidextrous entrepreneurial leaders react to the artificial intelligence...
IAESIJAI
 
PDF
DASA ADMISSION 2024_FirstRound_FirstRank_LastRank.pdf
rameswartudu05
 
PDF
Getting started with AI Agents and Multi-Agent Systems
Maxim Salnikov
 
PPTX
observCloud-Native Containerability and monitoring.pptx
anupsingh76937
 
PPTX
Final SEM Unit 1 for mit wpu at pune .pptx
anishtilekar08
 
PDF
Architecture types and enterprise applications.pdf
ChristopherTHyatt
 
PDF
Enhancing emotion recognition model for a student engagement use case through...
IAESIJAI
 
PDF
Taming the Chaos: How to Turn Unstructured Data into Decisions
Safe Software
 
PDF
DP Operators-handbook-extract for the Mautical Institute
LeonelMejiaLarios
 
Univ-Connecticut-ChatGPT-Presentaion.pdf
ry7r52mzb4
 
A Late Bloomer's Guide to GenAI: Ethics, Bias, and Effective Prompting - Boha...
Mindy Bohannon
 
Hindi spoken digit analysis for native and non-native speakers
IAESIJAI
 
Assigned Numbers - 2025 - Bluetooth® Document
Bloombase
 
1 - Historical Antecedents, Social Consideration.pdf
tuazon2030627
 
Zenith AI: Advanced Artificial Intelligence
Biz Preneurs
 
Geologic Time for studying geology for geologist
ssuser738f5a
 
Transform Your ITIL® 4 & ITSM Strategy with AI in 2025.pdf
PDCA Consulting
 
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
sustainability-14-14877-v2.pddhzftheheeeee
VenkateshGovindaraja9
 
A contest of sentiment analysis: k-nearest neighbor versus neural network
IAESIJAI
 
How ambidextrous entrepreneurial leaders react to the artificial intelligence...
IAESIJAI
 
DASA ADMISSION 2024_FirstRound_FirstRank_LastRank.pdf
rameswartudu05
 
Getting started with AI Agents and Multi-Agent Systems
Maxim Salnikov
 
observCloud-Native Containerability and monitoring.pptx
anupsingh76937
 
Final SEM Unit 1 for mit wpu at pune .pptx
anishtilekar08
 
Architecture types and enterprise applications.pdf
ChristopherTHyatt
 
Enhancing emotion recognition model for a student engagement use case through...
IAESIJAI
 
Taming the Chaos: How to Turn Unstructured Data into Decisions
Safe Software
 
DP Operators-handbook-extract for the Mautical Institute
LeonelMejiaLarios
 

2013.10 Operating * by the Numbers

  • 1. OPERATING * BY THE NUMBERS Allison Miller! @selenakyle
  • 2. Overview ! How we got here! ! Improving systems using models! ! Model building! ! Back to the Numbers! ! Beg, Borrow, Steal
  • 29. A Shift to Operations ! Life at Layer 8! ! The modern operating environment! High complexity! High stakes! ! Operations! Process of transforming inputs into outputs Layer 8 Say You Would What Here? You Do Transport Session Presentation Application Physical Network Data Link Business Logic
  • 30. The Modern World ! Buzzword Bingo! Big Data / NoSQL / Graph DB’s! Machine Learning! Agile development/delivery (aka Dev Ops)! Cloud / Anything...as a Service! ! The New Hotness is Old School! Management science ! Operations research! Decision Science! Six Sigma / TQM / Kai Zen
  • 31. Who Cares ! Relevant to control systems! ! Tools to improve running an operation/business! Automation! Optimization! Prediction / Forecasting! ! Modeling as an operations tool
  • 32. I’m no model lady. A model’s just an imitation of the real thing. ! –Mae West
  • 33. Improving Systems Using Models ! What are models! Not reality, but an approximation! 90% likelihood vs 90% of behavior observed! ! Why do we employ models! Design (how to build/design a system)! Management (goal setting & performance monitoring)! Live / Production / Operations (automation)! ! How do we know if they work? Abstraction Realism Prescriptive Descriptive Combat Modeling Spectrum Washburn & Kress, Combat Modeling, International Series in Operations Research & Management
  • 35. Operating Better Systems ! Operations – a transformative process that converts inputs into outputs
  • 36. Example: Data Driven Defense ! What’s a risk decisioning system?! ! Where do you put it?! ! What does it cost?! ! What do you need to build it?! ! How do you build it?! ! Operating Risk by the numbers! Forecasting / Prediction! Automation! Optimization
  • 38. 123.123.123.123 - - [26/Apr/2000:00:23:48 -0400] "GET /pics/wpaper.gif HTTP/1.0" 200 6248 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:47 -0400] "GET /asctortf/ HTTP/1.0" 200 8130 "http://search.netscape.com/Computers/Data_Formats/Document/Text/ RTF" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:48 -0400] "GET /pics/5star2000.gif HTTP/1.0" 200 4005 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" [Tue Mar 9 22:02:41 2004] [info] created shared memory segment #10813446[Tue Mar 9 22:02:41 2004] [notice] Apache/1.3.29 (Unix) mod_ssl/ 2.8.16 OpenSSL/0.9.7c configured -- resuming normal operations[Tue Mar 9 22:02:41 2004] [info] Server built: Mar 7 2004 13:38:59pausing [http:// xmlrevenue.com/s.php?username=jenneypan&keywords=Online+Gambling] for 50000 ms[Tue Mar 9 22:04:16 2004] [error] [client 218.93.92.137] mod_security: Access denied with code 200. Pattern match "Basic" at HEADER.[Tue Mar 9 22:07:16 2004] [error] [client 203.121.182.190] mod_security: Invalid character detected [4] 123.123.123.123 - - [26/Apr/2000:00:23:50 -0400] "GET /pics/5star.gif HTTP/ 1.0" 200 1031 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:51 -0400] "GET /pics/a2hlogo.jpg HTTP/1.0" 200 4282 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:51 -0400] "GET /cgi-bin/newcount? jafsof3&width=4&font=digital&noshow HTTP/1.0" 200 36 "http:// www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" [Tue Mar 9 22:02:41 2004] [notice] Accept mutex: sysvsem (Default: sysvsem)[Tue Mar 9 22:03:26 2004] [error] [client 218.93.92.137] mod_security: [Tue Mar 9 22:07:16 2004] [error] [client 203.121.182.190] mod_security: Invalid character detected [4] 123.123.123.123 - - [26/Apr/2000:00:23:50 -0400] "GET /pics/5star.gif HTTP/ 1.0" 200 1031 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:51 -0400] "GET /pics/a2hlogo.jpg HTTP/1.0" 200 4282 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:51 -0400] "GET /cgi-bin/newcount? jafsof3&width=4&font=digital&noshow HTTP/1.0" 200 36 "http:// www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" [Tue Mar 9 22:02:41 2004] [notice] Accept mutex: sysvsem (Default: sysvsem) Big Data & Little Loops
  • 39. 123.123.123.123 - - [26/Apr/2000:00:23:48 -0400] "GET /pics/wpaper.gif HTTP/1.0" 200 6248 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:47 -0400] "GET /asctortf/ HTTP/1.0" 200 8130 "http://search.netscape.com/Computers/Data_Formats/Document/Text/ RTF" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:48 -0400] "GET /pics/5star2000.gif HTTP/1.0" 200 4005 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" [Tue Mar 9 22:02:41 2004] [info] created shared memory segment #10813446[Tue Mar 9 22:02:41 2004] [notice] Apache/1.3.29 (Unix) mod_ssl/ 2.8.16 OpenSSL/0.9.7c configured -- resuming normal operations[Tue Mar 9 22:02:41 2004] [info] Server built: Mar 7 2004 13:38:59pausing [http:// xmlrevenue.com/s.php?username=jenneypan&keywords=Online+Gambling] for 50000 ms[Tue Mar 9 22:04:16 2004] [error] [client 218.93.92.137] mod_security: Access denied with code 200. Pattern match "Basic" at HEADER.[Tue Mar 9 22:07:16 2004] [error] [client 203.121.182.190] mod_security: Invalid character detected [4] 123.123.123.123 - - [26/Apr/2000:00:23:50 -0400] "GET /pics/5star.gif HTTP/ 1.0" 200 1031 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:51 -0400] "GET /pics/a2hlogo.jpg HTTP/1.0" 200 4282 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:51 -0400] "GET /cgi-bin/newcount? jafsof3&width=4&font=digital&noshow HTTP/1.0" 200 36 "http:// www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" [Tue Mar 9 22:02:41 2004] [notice] Accept mutex: sysvsem (Default: sysvsem)[Tue Mar 9 22:03:26 2004] [error] [client 218.93.92.137] mod_security: [Tue Mar 9 22:07:16 2004] [error] [client 203.121.182.190] mod_security: Invalid character detected [4] 123.123.123.123 - - [26/Apr/2000:00:23:50 -0400] "GET /pics/5star.gif HTTP/ 1.0" 200 1031 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:51 -0400] "GET /pics/a2hlogo.jpg HTTP/1.0" 200 4282 "http://www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" 123.123.123.123 - - [26/Apr/2000:00:23:51 -0400] "GET /cgi-bin/newcount? jafsof3&width=4&font=digital&noshow HTTP/1.0" 200 36 "http:// www.jafsoft.com/asctortf/" "Mozilla/4.05 (Macintosh; I; PPC)" [Tue Mar 9 22:02:41 2004] [notice] Accept mutex: sysvsem (Default: sysvsem) Big Data & Little Loops
  • 40. * Loop Disposition: Logic, Human, or Other?
  • 41. Big Data & Little Loops Why are you picking on me?Boo-yah! Still getting away with it. <Sigh> Nobody understands me.
  • 42. SHALL WE PLAY A GAME? (SINCE WE CAN’T PLAY “CLUE” FOR EVERY LOGIN TRANSACTION NEW USER MESSAGE FRIEND REQUEST ATTACHMENT PACKET WINK POKE CLICK BIT WE BUILD RISK MODELS)
  • 43. Applying Decisions Risk management is decision management ACTOR ATTEMPTS ACTION SuSUBMIT WHAT IS THE REQUEST HOW TO HONOR THE REQUEST SHOULD WE HONOR? RESULT ACTION OCCURS
  • 44. Applied where? Where risks manifest in observable behavior Where system owners make decisions Where controls can be optimized by better recognizing identity, intent, or change
  • 45. Decisions, Decisions Authorize Block Good false positive Bad false negative RESPONSE POPULATION Incorrect decisions have a cost Correct decisions are free (usually) Good Action Gets Blocked Bad Action Gets Through Downstream Impacts
  • 46. Such as... Populations - Users, Transactions, Messages, Packets, API calls, Files! Actions - Allow, Block, Challenge, Review, Retry, Quarantine, Add privileges, Upgrade privileges, Make Offer! Costs - Fraud, Data leakage, Customer churn, Customer contacts, Downstream liability
  • 47. For example: ACTOR ATTEMPTS Payment p (actor attempting payment is accountholder) Decision Authorize Review Refer Request Authentication Decline f(variable A + Variable B + ...) SuSUBMIT
  • 48. Flavors of Risk Models I deviate significantly from a normal (good) pattern I summarize a known bad pattern fa(x), fb(x), fc(x) fq(x), fr(x), fs(x)
  • 50. Model Development Process Target ! Yes/No questions best Find Data, Variable Creation ! Best part Data Prep ! Worst part Model Training ! Pick an algorithm Assessment ! Catch vs FP rate Deployment ! Decisioning vs Detection
  • 51. User IP Country <> Billing Country Buying prepaid mobile phones Add new shipping address in cart Buyer = Phone reseller, static machine ID How much $$ is at risk? What is “normal” for this customer? What “bad” profiles does this match? Geolocate IP Convert geo to country code Flag on Mismatch Cart Category Merch Risk Level Date Added Address Type String Matching Customer Profile Device ID Device HistoryTXN-$-AMT Churn Risk, CLV, ... TXNs, logins, ... Stolen CC, Collusion
  • 52. Model Training Some algorithms: - Regression: Determines the best equation describe relationship between control variable and independent variables! Linear Regression: Best equation is a line! Logistic Regression: Best equation is a curve (exponential properties)! - Bayesian: Used to estimate regression models, useful when working w/small data sets ! - Neural Nets: Can approximate any type of non-linear function, often highly predictive, but doesn’t explain the relationship between control and independent variables
  • 54. p-value of significance, throw out if > .05 Variance in dependent variable explained by independent variables Dependent Variable Independent Variables Factor odds of dependent go up when independent var incremented p-value should be < significance level (.05)
  • 56. Operating a Risk System Disposition & Time Email CC# Items Total ! Submit Maybe ! ! No! ! ! Yes!! ! ! SuOutcomeSuAttempt Black & Whitelists Machine Learning Velocity & Spend caps Geo & IP Logic Linking Data • Reporting • Metrics • Analysis • Modeling Good Bad Indeterminate
  • 57. The Better Mousetrap Automates defensive action x-platform - Fast ! - Accurate! - Cheap In Real Time In Time to Minimize Loss Reasonable False Positives As good as a human specialistReduces More Loss than Cost Created Cheaper than Manual intervention
  • 58. GAIN More gain/lift = more efficient predictions Catch as much as possible (as much of the “bads”) Minimize the overall affected % of population
  • 59. Cost Number of Defects Produced Cost of Control Cost of Defects Total Cost “Alice: Which way should I go? Cat: That depends on where you are going. Alice: I don’t know. Cat: Then it doesn’t matter which way you go.” ― Lewis Carroll, Alice in Wonderland
  • 60. % of population Cost Number of Defects Produced Cost of Control Cost of Defects CV Total CostCV
  • 61. Finding the * approach in the wild ! Operating * by the numbers in many disciplines! Automation! Optimization! Forecasting / Prediction! ! Such as…! Science ! Finance! Marketing / Advertising! Software Development! Site/Network Ops! Manufacturing! Military Is all fun and game until you are need of put it in production – @devopsborat
  • 62. Beg, Borrow, Steal ! A/B Testing! ! Control Charts! ! Highly engaged change management! ! Sample strategy! ! Instrument everything! ! Poka-Yoke
  • 63. Recap Operating systems effectively means: - Using data to understand and improve performance! - Using tools to:! - Automate (Efficiency, Scale, Standardization)! - Optimize (Set goals cognizant of tradeoffs)! - Forecast / Predict (Plan, course correct)! Designing data-driven defenses - Decisions that can be automated w/data! - Where/what data sets to use! - Business drivers to keep in mind ! Numbers, Numbers, Numbers p (bad) f(variable A + Variable B + ...)
  • 64. Prediction is very difficult, especially about the future Niels Bohr Allison Miller @selenakyle
  • 66. Such as... Metrics Analytics $ Loss Txns Purchase trends of high loss users # Compromised Accts IP Sources of bad login attempts % of Spam Messages Delivered Spam subject lines generating most clicks Minutes of downtime Most process-intensive applications # Customer Contacts Generated Highest-contact exception flows
  • 67. The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. " The second is that automation applied to an inefficient operation will magnify the inefficiency." –Bill Gates

Editor's Notes

  • #7: Guardsmark (Book published in Reader’s Digest in 1997), Half a billion in revenue in 2005 In 1992, Time magazine cited Guardsmark as the company ". . .which many security experts consider the best national firm in the business," and in his bestseller Liberation Management, Tom Peters praised Guardsmark as the "Tiffany’s" of the security industry. The Committee for Economic Development (CED) honored Guardsmark and Mr. Lipman with the 2002 Corporate Citizenship Award, and in 1996, Guardsmark received the national American Business Ethics Award as the private company recipient. 
  • #14: Basil H. Liddell Hart, British soldier, military historian and leading military theorist “A complacent satisfaction with present knowledge is the chief bar to the pursuit of knowledge.” B. H. Liddell Hart
  • #33: Time Series Considers a Baseline level Trend Seasonality Cyclicality Error y(t) = (a+bt)[f(t)]+e Moving average Maintain N periods of data, use a simple average Weighted moving average: Each of the N periods re-weighted Exponential smoothing: Old average combined with most recent observations Errors can be computed (to monitor erratic demand, determine if forecasting no longer tracking, to determine which parameters least/most prone to error) to set safety capacity) Adaptive exponential smoothing: Smoothing coefficient varied at each forecast to find forecast w/lowest error Box-Jenkins method... Causal Forecasting Method Cause-and-effect between demand and OTHER variables (e.g. demand for ice cream and summer temperatures) Important feature: can be used to predict turning points in the demand function Regression Econometric model (interdependent regression equations) Input-output model (describes flows across sectors) Simulations model (simulates distribution system) Note: in selecting a forecasting method, many data factors (availability, patterns) and usage (time, resources, what decisions will be made using the method) important along w/user & system sophistication “generally speaking, managers are reluctant to use results from techniques they do not understand”. Fit versus predictiveness