Quality Assurance of FME Scripts
0 likes180 views
Gitter, a Swedish IT consulting company, presented a testing method for quality assurance of FME scripts at the 2022 FME User Conference, focusing on the automation of testing to avoid errors during model updates. The solution involves generating baseline data and comparing it with data after updates to identify any discrepancies, along with implementing test automation within Azure DevOps. The presentation highlighted the necessity for established testing methodologies in the FME community to improve quality assurance practices.
Quality Assurance of FME Scripts
- 1. Quality assurance of FME scripts
- 2. FME User Conference 20 22 Gitter is a Swedish IT consulting company focused on providing GIS solutions mainly in the areas of energy, infrastructure, public environments and forestry.
- 3. 20 22 FME User Conference Feel Safe How to test your models and Changing a model or updating software?
- 4. 20 22 FME User Conference Agenda We will present a test method for quality assurance of FME models (scripts or custom transformers) that is not based on defining test cases but instead works by comparing features at different measure-points in a model to assure that data has not changed in an unwanted way after updating a model or component. We will also talk about how to automate tests in Azure DevOps. In this presentation: ● The challenge ● A real case ● Our solution ● Test automation ● Conclusions
- 5. 20 22 FME User Conference The challenge of maintaining complex FME models
- 6. 20 22 FME User Conference Challenge FME makes it possible to solve complex problems, however that often leads to complex models which are: - difficult and time consuming to verify Typical scenarios when tests are needed: - developing a new model - updating an existing model - updating FME version, transformer versions or dependent components (for example ArcGIS)
- 7. 20 22 FME User Conference What we updated • FME Server: FME 2017.1 to FME 2020.2 • ArcGIS 10.3.1 to ArcGIS 10.6.1 • The models were not updated What happened FME DateTime functions returned a <null> value in the model WHEN: • arcpy was used in a PythonCaller AND • the DateTime functions were used after the PythonCaller Total number of models: > 100 Number of models affected by the unexpected behaviour: <5% Reason: The combination FME 2020.2 and ArcGIS 10.6.1 resulted in this unexpected behaviour. When the unexpected happens A real case
- 8. 20 22 FME User Conference Sequence that did not work Result: _timestamp = <null> Sequence that worked Result: _timestamp = 20220611102232 When the unexpected happens A real case
- 9. 20 22 FME User Conference ▪ All scripts were published to a FME Server in a test environment ▪ The scripts were scheduled and ran during a test period ▪ The scripts were updating more than 700 tables and just 20 were affected by the error ▪ Most of the tables were used for background information in map services ▪ We never found the errors during the test period ▪ The errors were found after publishing to FME Server production environment Data verification: Only manual inspections were performed When the unexpected happens A real case
- 10. 20 22 FME User Conference With appropriate test methods we should have been able to find this unwanted behaviour Could we have found this behaviour at an earlier stage?
- 12. 20 22 FME User Conference Our solution To be able to find the unwanted behaviour described earlier we need a test that: ▪ is easy to implement in existing models ▪ can identify where information differs from the expected for any given point of measure in a model Idea • Save FFS files at different measure points in the models that can be used to compare the results, before and after the updates
- 13. 20 22 FME User Conference Step1 - Generate a baseline (before any updates) TestTool Transformers TestTool Settings: - Generate Baseline - START - UniqueName - 1000 (no of objects to save) Settings: - Generate Baseline - STOP - UniqueName - 1000 The tool is used before and after the transformers in the model that should be tested
- 14. 20 22 FME User Conference Step 2 – Collect data (after any updates) TestTool TestTool Settings: - Collect data for test - START - UniqueName - 1000 objects Settings: - Collect data for test - STOP - UniqueName - 1000 objects BASELINE_START data is read back into the flow (pipeline) to assure that same data enters the transformers. Transformers
- 15. 20 22 FME User Conference Step 3 - Analysis A separate model analyses the FFS files generated during the baseline- and collect runs at the stop measure points. Any differences between the files are identified and logged. Data at the STOP measure points are analysed
- 17. 20 22 FME User Conference Summary The TestTool is a custom transformer that is easy to add to existing models and is used to save FFS files at different points in a model. The TestTool can be used in a passthrough mode. The analysis model is used to compare FFS files at STOP points and reports any differences.
- 18. 20 22 FME User Conference What do we need to be able to run the tests systematically? Test Automation!
- 20. 20 22 FME User Conference Benefits of test automation - Tests runs automatically without human intervention when models are pushed into a repository - Untested models are not deployed into production - Manual testing is time consuming - The test results are transparent and instantly visible for all developers - Continuous testing minimizes the risk that any contribution introduces errors in the model As software developers, test automation is a natural part in the software developing process.
- 21. 20 22 FME User Conference Test automation in Azure DevOps Gitter and most of our customers are using Azure DevOps in the software developing process. DevOps pipelines are a proven way to support test automation. Place your screen capture here
- 22. 20 22 FME User Conference Implementation of test automation for FME models in Azure DevOps Azure DevOps On-prem server Local development environment triggers pipeline push code and tests run tests run tests r e a d q u e u e a n d d o w n l o a d c o d e w r i t e r e s u l t self hosted agent
- 23. 20 22 FME User Conference Example - Test result
- 25. 20 22 FME User Conference Summary FME and FME Server has become a central and important software and integration platform and deploying untested models may cause severe consequences. Our customers asks for quality assurance Today we presented one test method that verified a specific problem. However other methods are needed to achieve full test coverage. We are aware of previous work in this field within the fme community, e.g. rTest We feel that established test methodologies and test frameworks for FME are missing
- 26. 20 22 FME User Conference Call to Action We would like to propose: - Safe together with partners and users work for establishing recognized test methodologies, frameworks and best practices.
- 29. 20 22 FME User Conference Please Share Your Session Feedback Log in & navigate to the agenda. Select the session you attended Scroll down to “Session Feedback” Download EventMobi app Event code: fmeuc Click “Launch App” Fill out the survey and submit