Migrating Katalon Studio Tests to Playwright with Model Driven Engineering
- 1. Migrating Katalon Studio Tests to Playwright with Model Driven Engineering 2nd of July 2025 Nicolas Hlad Benoit Verhaeghe IWST 2025 — Gdańsk
- 2. 2 About this talk Introduction What this presentation is about • An industrial use case of Pharo Moose • Migration of test scripts using Model Driven Engineering (MDE). • Our shared experience of • what worked • what didn't (kind of)
- 3. ② ③ ④ ⑤ 3 Migrating Katalon Studio Tests to Playwright with Model Driven Engineering — IWST 2025 Introduction ① Industrial context and problems De fi ning a migration strategy Designing our migration tool Early results Conclusion
- 4. 4 Industrial context and Problems Problem de fi nition — 1 Migrating in time 1. Four years ago, some Business Units (BUs) adopted Katalon Studio to implement their functional behavior tests 2. In December 2024, BUs were instructed to change for Playwright as a global solution for functional tests. 3. They had to migrate all tests by June 2025 within 6 months
- 5. 5 Industrial context and Problems Problem de fi nition — 2 Katalon vs playwright Katalon Studio is an EDI for testing web, mobile, API and desktop applications. Playwright is a scripting framework for testing web, mobile, API and desktop applications.
- 6. 6 Industrial context and Problems Problem de fi ntion — 3 From Katalon to Groovy Fig —Katalon Test script can be exported into source code (Groovy)
- 7. 7 Industrial context and Problems Problem de fi ntion — 4 From Groovy… to Playwright ? Fig —A Katalon that was re-written manually into a Playwitght test script (Typescript) Initialy, we were told that it would take "hours" to rewrite a test by hand And we counted 1412 test cases (a) Katalon (b) Playwitght
- 8. 8 Industrial context and Problems Industrial Context De fi ning concepts of Katalon's Tests
- 9. 9 Industrial context and Problems Katalon studio De fi ning concepts of Catalan's Tests The script of a speci fi c test An instruction in a Test case The action executed during a speci fi c step (can have parameters) The receiver of an action set by a keyword
- 10. 10 Industrial context and Problems Katalon speci fi cities Additional challenges • Testers used Groovy to declare their own custom keyword • They wrote some tests using only Groovy, with full Groovy syntax (classes, loops, conditions, switch, etc). • Some repositories were full of emojis, accents, and errors such as misaligned test cases between the Katalon Studio and the Groovy representation (save problems).
- 11. 11 De fi ning a migration strategy Our research/engineering questions Planning the migration • How to organise this migration between stakeholders ? • How to migrate: • The language ? from Groovy to Typescript • The framework actions ? Katalon's Keywords to Playwright's instructions • How to automate the migration ?
- 12. 12 How to organise the migration ? Migration Collaborating on the migration Manual Migration Supervisor BU (testers) Tooled Migration Supervisor Share repository Validate possible migration Run KLT2PLW Request review of the playwright export Integrate playwright tests in pipelines Finalize export and share with BU Contains problems Yes No Migration fails No Yes Identify why and ask for corrections Update their repository
- 13. 13 How to migrate Language and Framework ? Migration Migrating Language and Framework (a) Katalon Groovy and Typescript are di ff erent languages but share some similarities in their syntax. However, framework's actions are mostly di ff erent. (b) typical Playwright
- 14. 14 How to migrate Language and Framework ? Migration Migrating Language and Framework (a) Katalon Simple Keyword Complexe Keyword Groovy and Typescript are di ff erent languages but share some similarities in their syntax. However, framework's actions are completely di ff erent. (b) typical Playwright
- 15. 15 How to migrate Language and Framework ? Migration Simple keywords strategy (a) Katalon (b) normal Playwright (c) our Playwright We implemented 97 Keywords in the WebUI.ts
- 16. 16 How to migrate Language and Framework ? Migration Complexe keywords strategy (a) Katalon 1. Locating the objet de fi nition fi le (in Katalon Repository)
- 17. 17 How to migrate Language and Framework ? Migration Complexe keywords strategy (a) Katalon 2. Create a Login TS object 2.1. Matching folder name 2.2. Exporting each Katalon Objects
- 18. 18 How to migrate Language and Framework ? Migration Complexe keywords strategy (a) Katalon (b) our Playwright Other cases like findTestCase our custom keyword are handle similarly
- 19. 19 How to automate the migration ? Migration tools in Pharo/Moose Katalon2Playwright (KTL2PLW) Fig — C4 Model of KTL2PLW usage
- 20. • Goal: to fi x issues that cause Error when parsing the fi les • Remove accents from code and fi le names • Transform Katalon “descriptions” into Groovy comments 20 How to automate the migration ? — Cleaning Migration tools in Pharo/Moose Cleaning repository Fig — A Katalon comment inside the groovy fi le representing a test case
- 21. 21 How to automate the migration ? — Parsing Migration tools in Pharo/Moose Parsing with Tree-Sitter Tree-Sitter Groovy Parser (in C) Pharo Image (smalltalk) Tree-Sitter is a C framework to build parser. It has a incremental construction of the AST. It bene fi ts from a large library of grammar from the community. Providing an EBF grammar, it generates a parser as a lib of all major OS platforms. Tree-Sitter-Pharo (GitHub)
- 22. 22 How to automate the migration ? — Modeling Migration tools in Pharo/Moose Modeling Katalon in Moose Fig — simplify Metamodel Katalon
- 23. 23 How to automate the migration ? — Modeling Migration tools in Pharo/Moose Modeling Katalon in Moose Fig — initial Metamodel Katalon However, it lacks source declaration of custom keyword or the use of Groovy statement in Test Steps
- 24. 24 How to automate the migration ? — Modeling Migration tools in Pharo/Moose KTLModel Fig — Full Metamodel Katalon In insight, integrating Groovy into Katalon model was a mistake
- 25. 25 Results Results Running on a medium size project Note: • 829 object fi les vs. 828 object declarations (i.e. one empty fi le) • 10 Custom Keywords fi les for 118 Custom Keyword (i.e. multiple keywords per fi le) • 513 MB project migrated in ~10sec (i.e. not all fi les interested) Fig — Firmadoc project description Fig — KTL model's instance for Firmadoc
- 26. 26 Results Results Early results We computed an estimated 8512 hours of manual migration for all test cases only. With Ktl2Plw, we migrated all projects in 218920 msecs.
- 28. 28 Petit titre Lessons learned Take away messages • Migrating code involves communication • Involve a lot of feedback loops • For MDE, divide the concepts into sub-models • separate Model Framework from Model language (Katalon / Groovy) • Model can be connected together later on (moose connector) • Migrate step by step • Deliver and iterate to add more features
- 30. 30
- 31. 31 Petit titre Titre de section Texte du titre Fig — Retaining fi le structure while migrating scripts
- 32. 32
- 33. 33 Petit titre Industrial context Migrating complexe keyword (a) Katalon