SlideShare a Scribd company logo
Global Software
Development
Kevin Loken
Group Technical Director, EA
2
Electronic Arts
20+
Development
studios
worldwide
#1 publisher on
PlayStation® 4
and Xbox One
8,400+
employees
http://ir.ea.com
3
Electronic Arts & Perforce
5,001
EA
Licenses
15+
Years
Usage
10,000+
check-ins
In a day
160
Perforce
Servers
350+
TB
Data
180
Proxy
Servers
Global Software
Development
5
Global Software Development
 It’s not 2 guys in a garage anymore
 Hundreds of staff
• Art & Animation
• Audio
• Code
• Online services & infrastructure
• QA, Certification, Manufacturing
 Not in one geographic location anymore
6
Global Software Development
 Fun is proportional to productivity
 Productivity is about making the remote experience the
same as the local experience
 Perforce offers a number of features that help:
• Proxy servers
• Edge servers
• DVCS
7
Global Software Development
 Perforce Proxy server
• Easy to setup
• Essentially zero maintenance
- Cull old files on a regular basis
• Extremely robust
• Great if all you do is sync,
edit & submit
• Terrible if you diff, revert,
switch streams
8
Global Software Development
 Perforce Edge server
• Complicated to setup – too complicated to show a movie 
• Needs backup, checkpoints
- There is data that is specific and unique to the edge server
• p4 submit is longer than local
• All other operations just like local
9
Global Software Development
 Still limited by TCP
protocol and network
latency
 But I want it to go faster!
• Enable parallel sync
• WAN acceleration
10
Global Software Development
• Art content
• Engine development
• Game development
• Online services
• Sensitive data
• Localization
• Binary data stores
• Tools & CI
 Proxy and Edge servers are great
 Assumes a single Perforce Helix server
 Lots of Perforce Helix servers for a game:
11
Global Software Development
 Risk of moving to a single repository is too high
 Human error is still a factor in our work
• Bad triggers
• Bad upgrades
 Time to execute commands increases
• Checkpoints
• Restoration
 Finding a window to upgrade would be impossible
• World-wide operations, games on different cycles, never a good time
12
Global Software Development
 It would be nice to share file history between servers
 Especially between game engine and game team
 Over the years we have developed a ton of scripts to do this
 But there is something new …
13
Global Software Development
 Perforce Helix DVCS Technology
• All documentation points to individual usage
 Also works server-server!
• This allows us to preserve history between servers
• It simplifies integration work flows for harvesting changes
• Custom scripts are gone, replaced by DVCS features
14
Perforce Helix DVCS
15
Perforce Helix DVCS
16
Perforce Helix DVCS
17
Perforce Helix DVCS
18
Perforce Helix DVCS
19
Perforce Helix DVCS
20
Perforce Helix DVCS
21
Perforce Helix DVCS
 Enables sharing of code across servers
 First-class feature in Perforce … no more custom scripts
 Works through proxy and edge servers
 A way of mitigating risk if a huge mono-repo is not for you
Security
23
EA is a target
24
EA is a target
25
Security
 EA Security model
• Old days – ‘outside the network’ - bad, ‘inside the network’ - safe
• Today – zero trust model
26
Security
 Perforce security models
• Protect table
• User management
• AD Security
- Starting to directly embed AD security groups into Perforce groups
- Simplifies management – “one source of truth”
- Potential downside: Not everyone is aware that adding a user to ‘this group’,
means they get Perforce access
27
Security
 Perforce Helix Threat Detection
• Relies on audit logs
• Take a lot of space!
• Intersects with our Employee Monitoring & Privacy Policy
• Here come the lawyers
28
Perforce Helix Threat Detection
 Understand privacy issues of your jurisdiction
• Switzerland: No, can’t monitor employees.
• Germany: Need approval of work council
• France: Need approval of individual employees
• Canada: Need to notify employees
• US: Don’t need to notify employees, but do anyway
• Contractors: Notify contracting agency in addition to country specific rules
29
Perforce Helix Threat Detection
 Handling the privacy issues
• Build a system that intercepts and cleans all audit logs
• Integrated with HR systems
• Custom rule sets that allow filtering based on local rules
• Only pass the cleaned audit logs on to Threat Detection
30
Perforce Helix Threat Detection
 Proof of Concept
• 3 months of audit logs from 2 servers
• Post processing
• Identified 10 unusual or suspicious users
• Individually handle each case, most of them explainable
• Due to post-process nature some instances couldn’t be fully investigated
• Results made the investment in Threat Detection a no-brainer for us
31
Perforce Helix Threat Detection
 Going live
• Near real-time processing of audit logs from all servers
• Intrusion detection vs. Incident response
• Present case to legal department to get investigation authorization
• Investigate once authorization is achieved
 Future
• Integrate with other data sources like QRadar
32
Perforce Helix Threat Detection
 Results
• Proof of Concept results made investment in Threat Detection a no-brainer
for EA
• Wide ranging legal issues involved, especially for multi-national companies
• Becomes exponentially more powerful as you integrate with other data
sources
Thank you!
Contact:
Kevin Loken
kloken@ea.com

More Related Content

PPTX
Software Engineering
PPTX
Chapter 5 Agile Software development
PPTX
MODEL DRIVEN DEVELOPMENT (1).pptx
PPTX
The Extreme Programming (XP) Model
PPT
Formal Specification in Software Engineering SE9
PPTX
Requirements engineering for agile methods
PPTX
Software matrics and measurement
PPTX
Software quality assurance
Software Engineering
Chapter 5 Agile Software development
MODEL DRIVEN DEVELOPMENT (1).pptx
The Extreme Programming (XP) Model
Formal Specification in Software Engineering SE9
Requirements engineering for agile methods
Software matrics and measurement
Software quality assurance

What's hot (20)

PPT
Rad model
PPT
Architecture design in software engineering
PPTX
Software Process Models
PPTX
Fundamentals of Enterprise Networks
PPTX
Software project management- Software Engineering
PPT
OSI Model
PPTX
Component based software engineering
PPTX
Quality and productivity factors
PDF
Software requirements
PPT
Ian Sommerville, Software Engineering, 9th Edition Ch1
PPTX
Water fall model
PPTX
Cohesion and coupling
PPT
Software development slides
PPTX
software metrics(process,project,product)
PPTX
Waterfall model in SDLC
PPT
Ian Sommerville, Software Engineering, 9th Edition Ch2
PPTX
Validation testing
PPT
Slides chapter 2
PPT
Software Verification & Validation
Rad model
Architecture design in software engineering
Software Process Models
Fundamentals of Enterprise Networks
Software project management- Software Engineering
OSI Model
Component based software engineering
Quality and productivity factors
Software requirements
Ian Sommerville, Software Engineering, 9th Edition Ch1
Water fall model
Cohesion and coupling
Software development slides
software metrics(process,project,product)
Waterfall model in SDLC
Ian Sommerville, Software Engineering, 9th Edition Ch2
Validation testing
Slides chapter 2
Software Verification & Validation
Ad

Viewers also liked (8)

PDF
PDF
Distributed Software Development in Brazil: A Historical Review
PDF
Distributed Software Development with Scrum and Social Coding
PPTX
Introduction to Global Software Engineering TDT4140
PPTX
GSD24 Testing
PDF
Distributed Software Development Process, Initiatives and Key Factors: A Syst...
PPTX
Mapping GSD and PMBOK Guide
PDF
Software testing
Distributed Software Development in Brazil: A Historical Review
Distributed Software Development with Scrum and Social Coding
Introduction to Global Software Engineering TDT4140
GSD24 Testing
Distributed Software Development Process, Initiatives and Key Factors: A Syst...
Mapping GSD and PMBOK Guide
Software testing
Ad

Similar to Global Software Development powered by Perforce (20)

PPTX
Troubleshooting K1000
PDF
Mitigate potential compliance risks
PDF
Challenges Scaling DevOps
PDF
DevOpsCon 2015 - DevOps in Mobile Games
PPTX
Cognos Performance Tuning Tips & Tricks
PDF
Gartner Infrastructure and Operations Summit Berlin 2015 - DevOps Journey
PDF
Sullivan red october-oscon-2014
PPTX
High Performance Continuous Delivery - Versioning and Release Management Aligned
PDF
Continuos Integration and Delivery: from Zero to Hero with TeamCity, Docker a...
PDF
Going Remote: Build Up Your Game Dev Team
PDF
Flintstones or Jetsons? Jump Start Your Virtual Test Lab
PDF
[UC4] Version and Automate Everything
PPTX
DevOps Days Ohio
PPT
FLIGHT Amsterdam Presentation - From Protex to Hub
PDF
Technical and Business Considerations for DNSSEC Deployment
PPTX
Securing the Helix Platform at Citrix
PPTX
Sergey Dzyuban "To Build My Own Cloud with Blackjack…"
PPTX
Game Development Best Practices
PPTX
Compliance Automation with InSpec - Chef NYC Meetup - April 2017
PPTX
Developing for Industrial IoT with Linux OS on DragonBoard™ 410c: Session 4
Troubleshooting K1000
Mitigate potential compliance risks
Challenges Scaling DevOps
DevOpsCon 2015 - DevOps in Mobile Games
Cognos Performance Tuning Tips & Tricks
Gartner Infrastructure and Operations Summit Berlin 2015 - DevOps Journey
Sullivan red october-oscon-2014
High Performance Continuous Delivery - Versioning and Release Management Aligned
Continuos Integration and Delivery: from Zero to Hero with TeamCity, Docker a...
Going Remote: Build Up Your Game Dev Team
Flintstones or Jetsons? Jump Start Your Virtual Test Lab
[UC4] Version and Automate Everything
DevOps Days Ohio
FLIGHT Amsterdam Presentation - From Protex to Hub
Technical and Business Considerations for DNSSEC Deployment
Securing the Helix Platform at Citrix
Sergey Dzyuban "To Build My Own Cloud with Blackjack…"
Game Development Best Practices
Compliance Automation with InSpec - Chef NYC Meetup - April 2017
Developing for Industrial IoT with Linux OS on DragonBoard™ 410c: Session 4

More from Perforce (20)

PDF
How to Organize Game Developers With Different Planning Needs
PDF
Regulatory Traceability: How to Maintain Compliance, Quality, and Cost Effic...
PDF
Efficient Security Development and Testing Using Dynamic and Static Code Anal...
PDF
Understanding Compliant Workflow Enforcement SOPs
PDF
Branching Out: How To Automate Your Development Process
PDF
How to Do Code Reviews at Massive Scale For DevOps
PDF
How to Spark Joy In Your Product Backlog
PDF
Shift to Remote: How to Manage Your New Workflow
PPTX
Hybrid Development Methodology in a Regulated World
PPTX
Better, Faster, Easier: How to Make Git Really Work in the Enterprise
PDF
Easier Requirements Management Using Diagrams In Helix ALM
PDF
How To Master Your Mega Backlog
PDF
Achieving Software Safety, Security, and Reliability Part 3: What Does the Fu...
PDF
How to Scale With Helix Core and Microsoft Azure
PDF
Achieving Software Safety, Security, and Reliability Part 2
PDF
Should You Break Up With Your Monolith?
PDF
Achieving Software Safety, Security, and Reliability Part 1: Common Industry ...
PDF
What's New in Helix ALM 2019.4
PDF
Free Yourself From the MS Office Prison
PDF
5 Ways to Accelerate Standards Compliance with Static Code Analysis
How to Organize Game Developers With Different Planning Needs
Regulatory Traceability: How to Maintain Compliance, Quality, and Cost Effic...
Efficient Security Development and Testing Using Dynamic and Static Code Anal...
Understanding Compliant Workflow Enforcement SOPs
Branching Out: How To Automate Your Development Process
How to Do Code Reviews at Massive Scale For DevOps
How to Spark Joy In Your Product Backlog
Shift to Remote: How to Manage Your New Workflow
Hybrid Development Methodology in a Regulated World
Better, Faster, Easier: How to Make Git Really Work in the Enterprise
Easier Requirements Management Using Diagrams In Helix ALM
How To Master Your Mega Backlog
Achieving Software Safety, Security, and Reliability Part 3: What Does the Fu...
How to Scale With Helix Core and Microsoft Azure
Achieving Software Safety, Security, and Reliability Part 2
Should You Break Up With Your Monolith?
Achieving Software Safety, Security, and Reliability Part 1: Common Industry ...
What's New in Helix ALM 2019.4
Free Yourself From the MS Office Prison
5 Ways to Accelerate Standards Compliance with Static Code Analysis

Recently uploaded (20)

PPTX
Operating system designcfffgfgggggggvggggggggg
PDF
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
PDF
Design an Analysis of Algorithms II-SECS-1021-03
PDF
How Creative Agencies Leverage Project Management Software.pdf
PPTX
Agentic AI Use Case- Contract Lifecycle Management (CLM).pptx
PDF
PTS Company Brochure 2025 (1).pdf.......
PDF
Raksha Bandhan Grocery Pricing Trends in India 2025.pdf
PPTX
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
PPTX
CHAPTER 12 - CYBER SECURITY AND FUTURE SKILLS (1) (1).pptx
PDF
System and Network Administration Chapter 2
PPTX
L1 - Introduction to python Backend.pptx
PPTX
history of c programming in notes for students .pptx
PPTX
Transform Your Business with a Software ERP System
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 42 Updates Latest 2025
PDF
Nekopoi APK 2025 free lastest update
PDF
Adobe Illustrator 28.6 Crack My Vision of Vector Design
PDF
Claude Code: Everyone is a 10x Developer - A Comprehensive AI-Powered CLI Tool
PDF
Understanding Forklifts - TECH EHS Solution
PPTX
Online Work Permit System for Fast Permit Processing
PPTX
ISO 45001 Occupational Health and Safety Management System
Operating system designcfffgfgggggggvggggggggg
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
Design an Analysis of Algorithms II-SECS-1021-03
How Creative Agencies Leverage Project Management Software.pdf
Agentic AI Use Case- Contract Lifecycle Management (CLM).pptx
PTS Company Brochure 2025 (1).pdf.......
Raksha Bandhan Grocery Pricing Trends in India 2025.pdf
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
CHAPTER 12 - CYBER SECURITY AND FUTURE SKILLS (1) (1).pptx
System and Network Administration Chapter 2
L1 - Introduction to python Backend.pptx
history of c programming in notes for students .pptx
Transform Your Business with a Software ERP System
Internet Downloader Manager (IDM) Crack 6.42 Build 42 Updates Latest 2025
Nekopoi APK 2025 free lastest update
Adobe Illustrator 28.6 Crack My Vision of Vector Design
Claude Code: Everyone is a 10x Developer - A Comprehensive AI-Powered CLI Tool
Understanding Forklifts - TECH EHS Solution
Online Work Permit System for Fast Permit Processing
ISO 45001 Occupational Health and Safety Management System

Global Software Development powered by Perforce

  • 2. 2 Electronic Arts 20+ Development studios worldwide #1 publisher on PlayStation® 4 and Xbox One 8,400+ employees http://ir.ea.com
  • 3. 3 Electronic Arts & Perforce 5,001 EA Licenses 15+ Years Usage 10,000+ check-ins In a day 160 Perforce Servers 350+ TB Data 180 Proxy Servers
  • 5. 5 Global Software Development  It’s not 2 guys in a garage anymore  Hundreds of staff • Art & Animation • Audio • Code • Online services & infrastructure • QA, Certification, Manufacturing  Not in one geographic location anymore
  • 6. 6 Global Software Development  Fun is proportional to productivity  Productivity is about making the remote experience the same as the local experience  Perforce offers a number of features that help: • Proxy servers • Edge servers • DVCS
  • 7. 7 Global Software Development  Perforce Proxy server • Easy to setup • Essentially zero maintenance - Cull old files on a regular basis • Extremely robust • Great if all you do is sync, edit & submit • Terrible if you diff, revert, switch streams
  • 8. 8 Global Software Development  Perforce Edge server • Complicated to setup – too complicated to show a movie  • Needs backup, checkpoints - There is data that is specific and unique to the edge server • p4 submit is longer than local • All other operations just like local
  • 9. 9 Global Software Development  Still limited by TCP protocol and network latency  But I want it to go faster! • Enable parallel sync • WAN acceleration
  • 10. 10 Global Software Development • Art content • Engine development • Game development • Online services • Sensitive data • Localization • Binary data stores • Tools & CI  Proxy and Edge servers are great  Assumes a single Perforce Helix server  Lots of Perforce Helix servers for a game:
  • 11. 11 Global Software Development  Risk of moving to a single repository is too high  Human error is still a factor in our work • Bad triggers • Bad upgrades  Time to execute commands increases • Checkpoints • Restoration  Finding a window to upgrade would be impossible • World-wide operations, games on different cycles, never a good time
  • 12. 12 Global Software Development  It would be nice to share file history between servers  Especially between game engine and game team  Over the years we have developed a ton of scripts to do this  But there is something new …
  • 13. 13 Global Software Development  Perforce Helix DVCS Technology • All documentation points to individual usage  Also works server-server! • This allows us to preserve history between servers • It simplifies integration work flows for harvesting changes • Custom scripts are gone, replaced by DVCS features
  • 21. 21 Perforce Helix DVCS  Enables sharing of code across servers  First-class feature in Perforce … no more custom scripts  Works through proxy and edge servers  A way of mitigating risk if a huge mono-repo is not for you
  • 23. 23 EA is a target
  • 24. 24 EA is a target
  • 25. 25 Security  EA Security model • Old days – ‘outside the network’ - bad, ‘inside the network’ - safe • Today – zero trust model
  • 26. 26 Security  Perforce security models • Protect table • User management • AD Security - Starting to directly embed AD security groups into Perforce groups - Simplifies management – “one source of truth” - Potential downside: Not everyone is aware that adding a user to ‘this group’, means they get Perforce access
  • 27. 27 Security  Perforce Helix Threat Detection • Relies on audit logs • Take a lot of space! • Intersects with our Employee Monitoring & Privacy Policy • Here come the lawyers
  • 28. 28 Perforce Helix Threat Detection  Understand privacy issues of your jurisdiction • Switzerland: No, can’t monitor employees. • Germany: Need approval of work council • France: Need approval of individual employees • Canada: Need to notify employees • US: Don’t need to notify employees, but do anyway • Contractors: Notify contracting agency in addition to country specific rules
  • 29. 29 Perforce Helix Threat Detection  Handling the privacy issues • Build a system that intercepts and cleans all audit logs • Integrated with HR systems • Custom rule sets that allow filtering based on local rules • Only pass the cleaned audit logs on to Threat Detection
  • 30. 30 Perforce Helix Threat Detection  Proof of Concept • 3 months of audit logs from 2 servers • Post processing • Identified 10 unusual or suspicious users • Individually handle each case, most of them explainable • Due to post-process nature some instances couldn’t be fully investigated • Results made the investment in Threat Detection a no-brainer for us
  • 31. 31 Perforce Helix Threat Detection  Going live • Near real-time processing of audit logs from all servers • Intrusion detection vs. Incident response • Present case to legal department to get investigation authorization • Investigate once authorization is achieved  Future • Integrate with other data sources like QRadar
  • 32. 32 Perforce Helix Threat Detection  Results • Proof of Concept results made investment in Threat Detection a no-brainer for EA • Wide ranging legal issues involved, especially for multi-national companies • Becomes exponentially more powerful as you integrate with other data sources

Editor's Notes

  • #4: EA has been using Perforce for over 15 years now. The first server was setup for our Tools & Libraries group with 5 users. Then we added it to FIFA, with split development between UK and EA Vancouver, Proxy server helping out significantly We currently have over 160 perforce servers setup, 180 proxy servers, and about half a dozen edge servers Total data under storage is over 350 TB, growing every day Our high traffic servers have meta-data databases that are approaching 1 TB (800 MB)
  • #10: WAN acceleration – SilverPeak, Riverbed, SonicWALL, MACH5
  • #15: The basic flow of code begins from the common ancestry point on frostbite. Having establish a common history, we `p4 fetch` to a branch on the game perforce server. This new branch (//mule/integ/…) is integrated into the dev line (//mule/dev/...). We finish the box by executing `p4 fetch` on frostbite (or a `p4 push` mule perforce server) And integrate into the origin common ancestor (//fmule/integ/...) The red boxes indicate that these branches should be consider read-only on their server. Only the user/service account that does the fetch should have write access to those branches.
  • #31: Funny story, SRM gets tagged as moochers because they only sync code