[NetherRealm Studios] Game Studio Perforce Architecture
1 like512 views
The document outlines the infrastructure and architectural challenges faced by NetherRealm Studios, highlighting their need for reliability, speed, flexibility, and effective management in game development processes. It discusses various strategies and solutions implemented, including the transition to NFS for better performance, the establishment of a VM farm, and emphasizing the importance of redundancy and management tools. Ultimately, it concludes that while Perforce can scale to meet the demands of data-intensive development, ongoing investments in engineering and infrastructure are essential.
[NetherRealm Studios] Game Studio Perforce Architecture
- 1. 1 Â Game Studio Perforce Architecture Ryan Mensching Senior Network Administrator NetherRealm Studios/WB Games Logo area
- 2. 2  The Setup •  Established Studio with track record •  Mortal Kombat (1992 – 2011) •  DC Comics – Injustice (2013), Arkham City Lockdown •  AAA Games on multiple platforms •  360 •  PS3 •  WiiU •  Vita •  iOS
- 3. 3  Historical Perspective •  Team has been making games for 20 plus years. •  “Things will get done.” •  Outages cost money, ship dates, and sanity. •  “Crunch” is a reality. •  Multi-Year development cycle, two years •  Used other tools in the past, Source Safe
- 4. 4  Statistics •  Games are Code and Binary Art Assets •  Textures •  Animations •  Cinematics •  Lighting •  Sync of current project at head is 20G •  Sync of source art assets is 545G
- 5. 5  Visuals •  460+ CL’s per day at peek intervals (1000’s of builds) •  Some 20G of changes per day at peek intervals
- 6. 6  Products •  As a studio we have 3 software products that are being produced at any given time. •  Engine •  Games •  Art Tool/Game Editor
- 7. 7  Challenges •  As IT we had several hurdles to overcome •  Aging Hardware •  Aging processes •  Management woes •  Insight into systems •  Lack of agility to meet business demand
- 9. 9  Strategic Goals •  Game development support is a balancing of immediate gratification and resources. •  IT business driven goals/pillars •  Space •  Speed •  Flexibility •  Reliability •  Management
- 10. 10  Space •  Available, can’t be used if its locked up, i.e. thin provisioning •  Abstraction, array layer should not need configuration to allocate space and at volume layer should not require reconfig for volume expansion.
- 11. 11  Speed •  Entire VM platform, OS, and mass storage. •  Performance needs to be maintained in relation to consumption.
- 12. 12  Flexibility •  Works in conjunction with other pillars. •  Stands alone as a goal in games environment. •  Freedom of protocol choices. •  Ease of VM and hardware additions.
- 13. 13  Reliability •  Redundancy should be inherent. •  Should not require special plans or procedures. •  If the reliability path is to complex issues will arise.
- 14. 14  Management •  IT Glue •  Proper management makes all the pillars come together. •  Self documenting
- 15. 15 Â Solutions How as an IT organization do we accomplish these goals and vision?
- 16. 16  Solutions •  Solid Layer 2 and 3 core with 10G distribution. •  Previous experience with VM, Storage over IP and infrastructure testing.
- 17. 17  Research •  Built VM farm with existing hardware and repurposed array for iSCSI. •  Benchmarks were in the top on the Perforce Benchmark DB. •  Ran real world project on systems.
- 18. 18 Â
- 19. 19  Storage Platform •  Key building block of infrastructure design. •  iSCSI management proved not agile enough, LUN management chores. Redundancy requires complex configurations. •  NFS provided equal or better performance.
- 20. 20  NFS •  True thin provisioning, space consumed and given back. •  No LUN changes. •  Volumes size can be adjusted on the fly. •  Strong VMware support. •  NetApp engine •  Snapshots •  Native NFS among other protocols •  Key to P4 revision file storage design.
- 21. 21  Virtualization •  VM hardware needs to be “wide and fast”. •  As much RAM as possible to improve VM operations and better disk caching. •  Cores should have highest clock speed possible. •  Perforce uses 2 vCPUs. •  Cisco UCS •  Hardware abstraction •  Native Unified Fabric, 10G •  VM Passthrough
- 22. 22  OS •  RHEL for all Perforce servers and many applications. •  Linux provides best usage of resources for VM’s and disk caching. •  Current project Perforce servers have 32GB of RAM. •  Able to achieve 440+Mb sustained transfer to p4 clients.
- 23. 23  Reliability •  Redundancy at array level with NetApp SnapMirror. •  Volume and VM level snapshots. •  Provides ability to single file restore directly back to depots/servers. •  UCS blades multi-homed to core; dual chassis uplinks, dual fabric uplinks, dual core switches. •  VMware HA clustering •  Perforce uses proxies for load balancing and checkpoints for integrity. •  Moving to replicas.
- 24. 24  Management Tools •  NetApp OnCommand •  UCS Manager •  VMware •  Plugins, SMVI •  Active monitoring of P4 services and disk health.
- 25. 25 Â
- 26. 26  Conclusion •  Perforce is very capable of scaling to speed and data intense development. •  Requires investments in engineering and infrastructure. •  Each environment is different, but can share philosophies. •  Abstraction, abstraction, abstraction
- 27. 27 Â Thank you Questions? Ryan Mensching rmensching@wbgames.com