How hard can it be - Ui development at keen games
2 likes2,174 views
Keen Games has over 10 years of experience developing user interfaces for games. They previously used various approaches like separating UI design and code, using scripting languages, and third-party middleware like Scaleform. For their game Portal Knights, they decided to rewrite the UI system from scratch in C++ using an immediate mode approach to address issues with prior systems like performance, bugs, and flexibility. They utilize a multi-pass system of layout, input, and render and take advantage of C++ features while simplifying aspects like memory management to improve productivity.
How hard can it be - Ui development at keen games
- 1. “How hard can it be” 10 years of Ui development at keen games Julien Koenen - Technical Director keen core GmbH This is a presentation I gave at the Quo Vadis 2018 conference in Berlin (24.04.2018)
- 2. Agenda ● Introduction ● History ● Portal Knights ● Lessons learned ● Q&A ● Bonus Short overview of the Content:
- 3. Keen? Independent game development studio from frankfurt - Founded 2005 - Before: NEON Studios (1992-2005) - Sold to Jowood - Most of the founders of ‘92 are still active in the company - Split up into 3 companies: - keen flare (Part of flaregames) - Mobile F2P Games (Royal Revolt 1+2, Olympus Rising, Monsters with Attitude) - keen games (productions) - PC/Console games - currently: Portal Knights - keen core - Engine + Tools + Middleware - Currently around 50 people - Keen games: 20 - Keen core: 5 - Keen flare: 26
- 4. Keen! - More than 30 games over the last 25 years - Tunnel B1 (DOS,PS1,Saturn) - Legend of Kay (PS2) -> one of the few big PS2 games produced in germany - Anno: Dawn of Discovery (WiiU/NDS) - one of the few games with > 80 metacritic from germany - TNT Racers -> the Only Wii-Game with DLC Support in Europe - Sacred 3 (PS3,Xb360,PC) - Royal Revolt 1+2 -> f2p games - Portal Knights -> Early access / shipped on 6 platforms / continuous development (> 1MPlayers) - Now mostly working on Games-as-a-Service and F2P Mobile games
- 5. Keen! - Shipped games on more than 20 platforms - PS1, Saturn, PC/DOS, GameBoy Color, GBA, PS2, Wii, PSP, PC/Windows, PS3, Xbox, Xb360, NDS, WiiU, iOS, Android, WP8, UWP, PS4, Xb1, … - Custom tech all the time - One short adventure trying to use RenderWare on PS2
- 6. Game UI is... ● The interface to between player and game code ● Design and Logic ● Often underestimated ● Changing all the time ● Lots of Approaches ● Reinvented all the time - Ui is the way for the player(s) to interact with the gameplay code ○ That means it always has to share / interpret data from the game ○ Tight coupling to the gameplay features (which changes all the time) - Traditionally two separate roles: ○ Ui Artist / Designer - responsible for the ‘look’ ○ Ui Coder/Scripter - responsible for connecting the Ui Art with the gameplay code - Look&Feel / Design of the UI ○ What is placed where / how do thinks look ○ How do elements move / animate over time ○ Each game usually has its own unique design ○ Lots of iterations required for good results - Logic / Functionality ○ How do the elements react to input ○ How is the game data transformed into changes in the Ui - “It’s just Ui Code” - usually the complexity of the Ui Code was wildly underestimated ○ Therefore the UI Code is done by the least experienced programmer in the team - No common solution for the last games at keen. Some historic examples:...
- 7. Dance Dance Revolution Disney Channel Edition Platform: PS2 - New Version of the Popular DDR IP - Complete rewrite (used the source of other versions as reference) - Except for some background video and 1-2 Dancing characters everything visible is Ui - Very flashy/animated -> everything is moving all the time - Single Platform
- 8. Dance Dance Revolution Disney Channel Edition Platform: PS2 ● Design in 3DS Max ● Code in C++ ● Strict separation ● Pro: Familiar Tools ● Pro: Animations ● Pro: Iteration speed on some things ● Con: Dependency on Coder ● Con: Iteration speed for most things ● Con: Ui ‘Feel’ in ‘wrong’ hands - The UI Artist (who was doing the Character Models as well) was pretty familiar with 3D Studio Max.. so we created a workflow based on MAX - Based on ‘dialogs’ and ‘elements’ - Whole dialog layouts where defined in MAX (with baked fade in / fade out / idle animations) - Connection of ui code with ui layout was done through name lookup - Strict separation of UI Code and UI Design - Ui Code was C++ written by the gameplay programmer - Ui Design was done in 3D Studio Max - Pro: - Each party used familiar tools - Animations where no problem at all - Ui Designer could quickly iterate on design-only changes - Cons: - Ui Designer had to wait for the Programmer to see anything new in the game - Iterations involving code-changes were slow - All of the Ui “Feel” was developed by the ‘wrong’ person
- 9. Sacred 3 Platforms: PS3, XB360, PC/Steam - Action RPG for Consoles - “Gauntlet” Style Combat - Lots of iterations because the game changed a whole lot during development
- 10. Sacred 3 Platforms: PS3, XB360, PC/Steam ● Ui Code in Script ● Ui Layout in Data ● Pro: Ui Code separate from GP Code ● Pro: Lots of flexibility ● Con: Buggy Tools ● Con: No Experts ● Con: Iteration Times - UI Code written scripting language ○ Idea: “easy enough” for the Ui designer ○ Separate Tool to define the ‘Layout’ of Ui Screens - Pro ○ Both Layout and Ui-Code should be written by the same person (In Theory) ○ Clear separation between Ui Code and gameplay code ○ Standalone tool to test / develop the Ui Requests without the actual game (input/output was defined) - Con ○ Buggy Tools (Debugger for the Scripting language came too late) ○ Most of the Ui-Code was written by someone else in the end anyway (wasn’t easy enough) - Goal was clear separation between UI Code and gameplay code (don’t mix - KScript: simple functional language .. debugger (late) - ‘Request’ style ui: the gameplay code started async ui requests and interacted with them (send data in / out) - Each request corresponded to one ui script
- 11. Olympus Rising Platforms: iOS, Android, UWP - Mobile F2P Title (still running) - “Game as a Service” -> Looong lifetimes - Lots of changes over the Lifetime of the title (especially in Ui) - The whole “meta-game” required for F2P mobile titles is mostly implemented through Ui
- 12. ● Ui Code in C++ ● Layout + Logic in Code ● Mockups by Ui Designer ● Pro: Simple Solution ● Pro: Shared Code ● Con: Ui Designers had to wait ● Con: Coder “Art” ● Con: Iteration times Olympus Rising Platforms: iOS, Android, UWP - Ui Code written in C++ by gameplay coders - Layout + Logic defined in Code - Ui Designer created ‘mockups’ in Photoshop and Unity for new dialogs - ‘Retained’ mode Ui style (MVC) - Pro ○ ‘Simpler’ Solution .. only one place+language for everything (no converter/exporter) ○ Shared Code -> the Ui Code was just like ‘normal’ GP Code (Code reviews, coding standards, …) - Cons ○ Ui Designers had to wait for coders (for everything!) ○ Ui was pretty static / fixed (partially due to the restrictions of the mockup tools) ○ Iteration times were not great.. Build+Restart the whole game for every change
- 13. Portal Knights Platforms: PC/Steam, PS4, XB1, Switch, iOS, Android - Early Access title -> very tight schedule until first shipped version - PC only in the beginning - Big push to switch the ui system after Sacred 3 - Everyone else is doing it…
- 14. Portal Knights Platforms: PC/Steam, PS4, XB1, Switch, iOS, Android Hopes: - Ui Designers can do everything - Ui decoupled from Gameplay code - Stable / familiar tools - Easier to hire new UI Designers - “Visual” workflow would lead to better UI Designs - The ‘real’ programmers don’t need to deal with UI Code anymore
- 15. Scaleform - The good ● We shipped ● Porting to XB1/PS4 was (pretty) easy ● Happy Ui Designer ● Ui Code isolated - We actually managed to ship the first version of Portal Knights into Early Access using Scaleform -> it worked! - We managed to port the scaleform UI to PS4+XB1 with manageable effort - The UI Designer was pretty happy in general - The UI Code was completely isolated from the gameplay code and all data was moved through a narrow interface
- 16. Scaleform - The bad ● Performance ● Bugs ● Complexity ● Tools ● Iteration Times ● Hiring - The Performance turned out to be pretty bad ○ Especially when porting to Consoles and Mobile later (PC was less of a problem) ■ We had to run Scaleform in a separate thread in parallel to the client on console (took more time than the game) ○ Mostly hidden performance costs in value-copies, cache invalidation ○ Pretty easy to hit a ‘slow path’ ○ Surprising cost for some render effects (scissor using stencil buffer was pretty slow on mobile) ○ No (easy) way for us to optimize ■ Problems were detected rather late.. - Bugs: ○ Several memory leaks ○ The Console backend of Scaleform had a couple interesting bugs ○ At least we had the source code .. so it was possible to fix these - Complexity ○ Huge C++ Codebase with no experts around.. Wasn’t really fun.
- 17. ■ Around 800k LOC ○ Lots of dependencies to third party libraries (pcre, curl, …) we didn’t want / need but where used in several places ○ Version conflicts with some of the third party libraries - Tools ○ No debugger for the Ui Code - Iteration Times weren’t great as well.. - Hiring ○ Wasn’t really easier to find Ui Designers with Flash + Game background
- 18. Scaleform - The ugly ● No Switch support ● Performance on Android + iOS was *really* bad Then after the PS4/XB1 port was done we got asked if we could do a Switch version of the game.. - No Switch Support in Scaleform…. And while we are at it: What about Mobile? - Around 50-60 ms on Mid Level Android Devices
- 19. Scaleform - The ugly ● No Switch support ● Performance on Android + iOS was *really* bad Then Autodesk killed Scaleform And finally: Scaleform gets canceled - The biggest problem with third party middleware: No control.. - Usually you want to wrap it to be safe.. - Not possible in this case (The interface is basically the whole Ui Code)
- 20. Scaleform - What now? ● Option 1: Fix it ● Option 2: Split it ● Option 3: Get rid of it - Option 1: Write our own scaleform backend + optimize performance ○ We had the source code .. so at least we had this option - Option 2: Don’t use scaleform on switch+mobile ○ Would be pretty painful to have multiple implementations of the Ui - Option 3: Don’t use scaleform ○ Buy something else? ■ Nothing on the market (Iggy1 has no Android support) ○ So there is only one Way: get rid of Scaleform
- 21. Scaleform - The way out: Ui2 Rewrite all Ui from scratch: ● Step 1: Hide Scaleform ● Step 2: Implement new Backend ● Step 3: ??? ● Step 4: Profit!! In 2017 we decided to abandon Scaleform and switch over to a new system Rewrite all UI from scratch: - Step 1: Abstract all Scaleform code from the rest of the codebase - Step 2: Implement new ui2 backend on the side (because we could still ship with scaleform on other platforms) - Step 3: Switch over to the Ui2 Backend on all platforms - Step 4: profit!! (Delete all Scaleform code) It wasn’t really clear how long it would take - 62k LOC Actionscript3 code + 15k LOC of C++ Code
- 22. Decision Time: What language for Ui2? ● Option 1: Declarative language ● Option 2: Scripting language ● Option 3: Use C++ ? We still had to decide what the alternative (Ui2) would look like: - Option 1: Use a (custom) declarative language ○ Not flexible enough.. Would stil need some kind of ‘real’ programming language - Option 2: Use a scripting language ○ We tried that before on Sacred3 + TNT Racers -> “Friction Costs tend to be high” - Option 3: Use C++ for everything ○ Only used C++ for the ‘Code’ part before ○ Unclear how to define the layout/look efficiently in C++ code
- 23. C++ as Ui Language - Pros ● Support ● Performance ● Tools ● ‘Native’ data model ● All Platforms ● We can do everything ● Compatibility - Better support from our team (all of them are C++ experts, none of them knows Actionscript/Javascript/…) - Better baseline performance - An actual Debugger, Profiler and we can use all the in-house tools we developed for C++ - ‘Native’ data model binding -> no data model transfer overhead - Support for all platforms - We can do everything / unbounded flexibility ○ Not restricted by a Scripting Language API / third-party SDK / - Compatibility with the rest of the code base
- 24. C++ as Ui Language - (Potential) Cons ● C++ is ‘hard’ ● Memory management ● Iteration times ● Productivity ● Crashbugs We had to find solutions to some potential problems to make this work C++ has an image of a ‘hard’ language.. This is mostly due to the insane complexity of the language combined with - Manual memory management is really hard - Iteration times are bad (rebuild the game for each change?) - Rewriting all the code will be a nightmare / take too long - Ui Code Bugs will crash the whole Game So we tried to tackle these issues:
- 25. C++ as Ui Language: Make it simpler + faster ● Make it simpler: ○ Immediate UI ○ No explicit memory management ○ Lots of helpers ● Faster: ○ Hot reloading ○ Fast build times ● Handle Crashbugs - Simpler: ○ Use Immediate UI Method to simplify the code ○ Remove all explicit memory management needed in UI Code ○ Remove need for pointer management in UI Code ○ Add helper / wrapper for common things - Faster iteration times: ○ Hot reloading ○ Fast build times (Separate .DLL project with minimal dependencies + restructuring of the physical layout to optimize build times) - Sensible handling of Crashbugs in UI Code
- 26. Immediate Ui - Super Short Intro vs. - Short introduction into immediate mode ui concept - On the fly creation of widgets - No explicit create/destroy - No callbacks for event handling - Term coined by Casey Muratori (public video presentation in 2005) - DearImGui/Nuklear are popular imgui implementations - Allegedly used in Unity3d (I wouldn’t know;)
- 27. Immediate Ui - Advantages ● Dynamic ● Simpler ● Hot reloading is easy ● Some experience ● Composable - Much more ‘dynamic’ binding to game state - Layout is implicit / code driven -> so we can do lots of dynamic tweaks depending on the state / platform / timing - if( platform == ps4 ) { if( button( “ps4 specific thing” ) { doPs4Thing() ) }; - No callbacks - Memory Management is a lot easier (and faster!) - State is built up each frame .. so it’s pretty easy to support hot-reloading - Our programmers used this for debug ui for a couple years already - Composable - doComplexButton() { … doButton() … };
- 28. Ui2: Multi Pass Immediate Ui ● Linear List of UiFrames ● Additional Stack for Hierarchy ● Three passes: ○ Layout ○ Render ○ Input ● Same code executed each time ● Different state Three Passes - Layout - collects layout information from the widgets - Input - reacts on input events - Render - records render commands - Potential problem: the ui code can pretty easily break when it behaves differently in the different passes
- 29. ● Collect all UiFrames with a unique id ● Store layout parameters ● Layout Frames ● Flexbox and Grid layout ● Optional scrolling of content ● Store Result Ui2: Layout - The first pass is just collecting all the Ui Frames and assigning them a unique id - During this pass we store all the input parameters for the layout algorithm (min/max/preferred size, layout of child elements, padding, …) - After that pass is done we run over the frame hierarchy bottom up to propagate the size constraints upwards and after that we traverse from the top down to actually place the frames in their final position - We support a simple ‘flexbox’ style and a grid layout method for child frames - Optional Scrolling is implemented through a simple ‘content offset’ in each frame and a scissor rect during rendering - The Final result of the layout pass (basically a rectangle for each frame) is stored for the later passes
- 30. Ui2: Rendering ● Simple API ● Use layout data ● Collect Draw Commands ● Scissor Rects ● Transforms ● Layers ● Compositing - Simple drawing API for lines/rects/(textured)quads/shapes - Sits on top of our cross-platform Graphics Api abstraction - The Ui code will call drawXXX functions which create UiDrawCommands - All draw commands are gathered and sorted according to sort key - Support for Scissor Rects (Push/Pop Style) - Support for Transforms (Scale/Offset only) - Support for overlapped ‘layers’ within the same Window - Simple ‘compositing’ API
- 31. Ui2: Composite Rendering Renders to dynamic offscreen targets Support for a couple blend modes Easily extensible when required Used a lot for transition animations (fading/scaling/moving/rotation of whole windows / hierarchies)
- 32. Ui2: Input Handling ● Input Events ● Use Layout Data ● Focus handling ● Local Multiplayer (Window based) ● Text Input - Each input event coming in will create a Ui Pass ○ We merge adjacent mouse move events - Player id attached to Input event (for local multiplayer) - Input Device Id attached to Input event - Frames can filter input based on player id / device type - Full support for keyboard/mouse/gamepad/touch - Each frame can mark itself as ‘focusable’ - On each level of the tree a focus frame is maintained - The focus frame gets the keyboard / gamepad input - Support for virtual and real keyboard text input
- 33. Ui2: Wrapper: Windows ● Special Frames ● Virtual Size ● Scope Style API Wrapper for Window - Frames - Windows can overlap / move - Windows have a screen transform (aligned/scaled on the screen) - Each contains a tree of UiFrames - Constructor ‘pushes’ the window and destructor pops it again
- 34. Ui2: Wrapper: Layouts ● Wrapper for Layout Frames ● Scope API Another wrapper for ‘layout’ frames - Internally just push_frame / pop_frame - Can’t forget the pop_frame
- 35. Ui2: Wrapper: Widgets ● doXXX calls for widgets ● Wrap frame hierarchy ● Complete input and output as parameters ● Composable (>100 Widget Functions) Wrappers for widgets (mostly functions.. Some classes) - Each doFloatValue() creates the UiFrames necessary for the Widget and adds them as child frames to the currently open Layout frame - Pretty minimal information - The hasMouse is a good example for the simplicity of dynamic / platform-specific UI adjustments
- 36. Frame “State” Memory ● ‘Persistent’ State API State memory is bound to the frame .. will be removed automatically - Use explicit “States” for data that should persist across frame boundaries: - Uses a simple hashmap using the frame-id as key - Two lifetimes: “Temporary” and “Persistent” - Temporary State will be removed when the Host-Frame is not present anymore (so when during a ui pass you don’t call newState<> on a specific frame-id - this is mostly useful for animation-states (like the example) and other widget-local states - Persistent states will live throughout the whole game -> they are never removed - Useful to remember the last focus element in a window so on reopen it will focus the same element again
- 37. Ui2: Results ● Performance ● Amount of Code ● Iteration times ● Support from Other Coders ● Simple C++ ● Ui Designer (pretty) Happy ● Lots of ‘Magic’ Numbers - Performance ○ Android: > 50 ms -> 4ms / Frame - Amount of Code ○ 46k LOC C++ Code vs 62k LOC As3 Code + 14k C++ - Iteration times ○ 1-3 seconds after any change of the code the results are visible ○ In-place live editing of values through tweakable variable interface - Gameplay programmers can (and do) help with UI Tasks / Bugs - Engine coders can (and do) help with performance / new features - The concrete Ui Code contains lots of ‘magic numbers’ for sizes / positions.. But that is fine it’s the ‘design’ of the ui and the whole idea is to have it inline in the code
- 38. ● No silver bullet ● Work with your team ● Ui Code is Code ● DIY! ● Iteration times! ● Wrap your middleware! Ui2 Conclusion - Very happy with the results -> but no silver bullet.. ○ Depends on your team structure and the game - Our way forward for the next projects - You should use whatever works for you - We still could improve the density of the UI code and try to share more between projects - We plan to experiment with a serialized / data representation of the Frame Data to allow a more traditional “Ui Editor” for simple cases - We actually decided to replace QT with the immediate Ui Framework for our (new) Tools
- 40. Idea: Convert the ActionScript to C++ ● Parse Actionscript ● Convert to C++ ● Compile C++ ● Profit ? - The idea was to parse the ActionScript code and create equivalent C++ code. Our thought was that having an actual optimizing compiler + profiling tools would be enough to get the performance up. - We actually implemented an ActionScript 3 parser and were able to create the AST of the Ui Code (62k Lines of code) - Lots of memory allocations (Action Script uses a GC -> no such thing in C++) - Big Flash interfaces we would have to reimplement - Actionscript was not the main problem.. It was the concepts / the used API was too abstract / disconnected
- 41. Bonus: Hot reloading of C++ code - It’s awesome! - Pretty easy - 1-3s - Can change everything - Lots of different ways - We decided to just export the main entry point (doPkUi) from a .dll and have a separate .dll project - The game executable still contains the complete ui code as well as a fallback / for shipping - Build times are pretty good.. Around 1-3 seconds. - Instantly replaces *all* ui code -> no restriction on which part can be hot-reloaded - No state kept.. All Frame-States are ‘new’ - Structured exception handling around the Ui code -> fallback to the .exe code
- 42. Bonus: Text Rendering ● Trade-Offs ● SDF Glyph Renderer ● Parse .TTF / .OTF at runtime ● Emojis ● Simple Layout - A *very* complex topic - We implemented a Signed distance field (SDF) font renderer - Directly parse .TTF files, extract the outlines and create the SDF at runtime - We support .TTF and .OTF files and can use the system fonts on Android and iOS directly (as a fallback) (mostly to reduce size) - Partial support for Emojis - Text Layout is pretty simple / incomplete but enough for our games - We don’t write Text Editors - Thought about buying a third-party library.. But decided against it ○ We want to control the code of critical components ○ We have to support mobile ○ We wanted to support the system fonts to reduce download size) ○ We want to make the trade-offs ourselves
- 43. Bonus: Stepping through the Code A (very) shallow step through the main Ui code.. The main entry point for the Ui code is called doPkUi() PkUiContext contains references to all data we can reference This function is called for each Ui pass (for each input event, layout+render)
- 44. Ui2: Entry Point: multiple players Split into ‘player ui’ which is player specific (we support local-coop) and ‘system’ ui which is global
- 45. Ui2: Entry Point: multiple players Very easy to read at a top-level .. simple to understand / modify / extend
- 46. Ui2: Widget implementation Just a simple real world code snippet for a (Portal Knights specific) widget
- 48. Bonus: Debug Visualization of Frames We implemented a simple debug ui to visualize the Ui frame Tree at runtime and highlight the frames when hovering over them This is pretty nice already but we are thinking about even better tooling to inspect / browse the dynamic part of the ui state