Hearthstone helper using optical character recognition techniques for cards detection
- 1. Autor Conducător științific Universitatea Politehnica București Facultatea de Automatică și Calculatoare Catedra de Calculatoare Hearthstone Helper - Using Optical Character Recognition Techniques For Cards Detection Costin-Gabriel CHIRU, Florin Oprea costin.chiru@cs.pub.ro
- 2. Introduction • Purpose: capture, process and analyze images from the video stream of the Hearthstone game in order to obtain relevant information on the conduct of parties in this game • Constraint: results should be offered in real- time on a regular computer / tablet • Methodology: apply different techniques for extracting information from the images and then use this information to identify and log the cards that have been played. 09.09.2016 AIMSA 2016 1
- 3. Heartstone (Heroes of Warcraft) • Free-to-play online game developed by Blizzard (2014) • Turn-based game involving 2 players in a 1vs1 match • Each player has his/her own deck of cards with 30 cards • Each player start with 30 life points and the purpose of the game is to play their cards so that to destroy the opponent before they are themselves defeated • The game features between 814 and 1003 different cards used to cast spells, summon minions to help in the battle or equip weapons – They are classified in 5 categories based on their rarity and usefulness and in 10 different classes according to their availability to all the heroes, or to a specific one. 09.09.2016 AIMSA 2016 2
- 4. Application Usefulness • By monitoring the game, the application is able to memorize the cards that have been played and thus: – the player knows what remaining cards he/she has and can compute the likelihood of extracting a particular card – he/she knows which cards will no longer be available in the opponent deck (the decks may contain at most two cards of the same type, or only one if it is a legendary card) – the player can predict the most probable deck of the opponent based on the cards played so far, previous history, opponent's rating, game style and current meta-game. 09.09.2016 AIMSA 2016 3
- 5. Theoretical Background (1) • To determine real time in-game events, methods that are fast, efficient and accurate are needed both for image capturing and for cards recognition. • Image capturing: acquire sufficient game snapshots per seconds to detect changes – Using API provided by Windows that allows direct access to the opened windows from the operating system – up to 15 frames per second without overwhelming the CPU 09.09.2016 AIMSA 2016 4
- 6. Theoretical Background (2) • Cards recognition: needed to – detect when a new card appeared (to avoid continuous processing of video stream) • Using Background Subtraction – recognize different pictures based on the elements from them. Can be done by: • comparing their histograms (colors distribution) • key points matching: choose specific points of interest and then find the association between them. • using key points in combination with decision trees (to accelerate the execution) • using perceptual hashes • using OCR on the text appearing in the pictures and comparing the text 09.09.2016 AIMSA 2016 5
- 7. Implementation Details (1) • First of all, the Windows API was used to extract the snapshots extract information from the snapshots • Resolution might be large involving a large processing time only analyze specific regions in the images to determine the start and end of a game (analyzed areas are about 10 times smaller than the original image) • Next step: identify the moments of time when these areas contain images of cards using Background Subtraction and analyze the frames sufficiently highlighted 09.09.2016 AIMSA 2016 6
- 8. Implementation Details (2) 09.09.2016 AIMSA 2016 7
- 9. Implementation Details (3) • Next, we investigated the methods for image recognition: – Histograms: bad results due to the fact that all the images were created from the same template – Key points matching - invariant to rotation, scaling or brightness: SURF (Speeded up robust features) and SIFT (Scale-invariant feature transform) in combination with Flann (Fast Library for approximate nearest neighbor) for matching the key points • Can be done offline for the cards from the DB, but the matching algorithm might take minutes (~ 1000 cards * tenths of a second for each card) can’t be used for real- time recognition • For heroes recognition, SURF and Flann takes ~ 3 seconds (only 9 possible heroes) acceptable for this part used 09.09.2016 AIMSA 2016 8
- 10. Implementation Details (4) 09.09.2016 AIMSA 2016 9
- 11. Implementation Details (5) – Apply OCR techniques: used Tesseract and specified specific text areas that help determining the playing card (the card name and its attributes - cost, attack and life points) • Binarized the picture better results for OCR and smaller execution times • Wavy text problems for the recognition engine had to post-process the OCR output: – compute the Levenshtein distance between the obtained text and the cards name and – use the attribute values returned by the OCR to filter out cards • Very fast: took between 0.1 and 1.5 s • Using OCR, the whole process (from capturing the window image to the card classification) < 2 s 09.09.2016 AIMSA 2016 10
- 12. Application Output • The GUI is to inform the player about relevant changes in the application: – the status of the application, – the cards that have been played, – the remaining cards 09.09.2016 AIMSA 2016 10 • The information is updated in real-time as the cards are played
- 13. Cards Identification (1) • Stormwind Knight: – Cards from the DB with their scores: • "Silver Hand Knight": 2, • "Stormwind Knight": 10, • "Blood Knight": 10 • "The Black Knight": 2 – Use Levenshtein Distance: • "Stormwind Knight": 9, 11, 11, 4, 8, 5 • "Blood Knight": 11, 12, 15, 8, 12, 10 – and cards’ attributes • "Stormwind Knight": 4, 2, 5 • "Blood Knight": 3, 3, 3 • The card is correctly identified 09.09.2016 AIMSA 2016 12
- 14. • Kor'kron Elite : – Do not contain any words from the DB: – Use cards’ attributes (4, 4, 3): • "Kor’kron Elite" • "Houndmaster” • "Spellbreaker” – And then Levenshtein Distance: • "Kor’kron Elite": 4, 6, 2, 5, 3, 4 • "Houndmaster": 11, 13, 11, 10, 11, 12 • "Spellbreaker": 12, 14, 13, 11, 12 • The card is correctly identified 09.09.2016 AIMSA 2016 12 Cards Identification (2)
- 15. Runtime and Errors • Results obtained using an Intel(R) Core(TM) i7-4500U CPU @ 2.40 GHz • Time analysis: – Detection of a new game (SURF & FLANN): avg. 0.104s – of the heroes of the two players (SURF & FLANN): avg. 0.346s – Cards’ detection (OCR): • Accuracy: system average accuracy was 99.1%. – Errors due to partial occlusion of the card that needs to be detected. – GUI offers the possibility of manually correcting the errors. 09.09.2016 AIMSA 2016 12 Type of cards OCR attributes OCR names Total Cards extracted by the user 0.0146 0.7305 1.0159 Cards played by the opponent 0.0086 0.8234 0.9361 Type of cards Identified cards Errors Cards extracted by the user 209 1 Cards played by the opponent 227 3 Total 436 4
- 16. Conclusions • We presented an application whose purpose was to monitor in-game events that occur in the Hearthstone game • Main requirement was to be quick enough to provide answers in real time • For the recognition of the analyzed images, we have tried two categories of methods: – One based on the picture recognition: using histograms or using algorithms for key point matching time consuming – one based on OCR use only the text from the pictures fast and accurate • The application helps players in making the right decisions or in taking calculated risks, thus improving their performance and gaming experience 09.09.2016 AIMSA 2016 12
- 17. Questions 09.09.2016 AIMSA 2016 12 Thank you very much!