Furore devdays2017 tdd-1-intro
- 1. FHIR® is the registered trademark of HL7 and is used with the permission of HL7. The Flame Design mark is the registered trademark of HL7 and is used with the permission of HL7. Amsterdam, 15-17 November | @fhir_furore | #fhirdevdays17 | www.fhirdevdays.com Test Driven Development I - Intro Richard Ettema, AEGIS.net, Inc.
- 2. Presented by • Name: Richard Ettema • Position: • Lead Consultant, AEGIS.net, Inc. • FHIR® Certified Implementer • Background: • 34+ years IT industry experience • 14+ years leading HIT development/implementation efforts • 4+ years contributing to the HL7® FHIR® specification (focus on testing) • Sr. Architect / Lead Developer for the Touchstone Project • Author of the AEGIS WildFHIR public test server and client
- 3. What would happen if airlines practiced “one and done” testing? How important is on-going or continuous testing?
- 4. Question… How can implementations based on Industry based Standards like FHIR® ensure they are truly Interoperable with each other? • Choice: Expensive 1-time-before-release conformance testing • Will most likely uncover previously unknown conformance issues • Introduce unneeded RISK and additional cost (budget and schedule) Or, Moderate expense with continuous testing (Test Driven Development) • Expense decreases over time as testing becomes infused into the development life cycle • Developers code to the specification as it becomes embedded within each coding cycle • Budgets and schedule are less risky around conformance
- 5. Test Driven Development Philosophy • Software development technique that implements short development cycles • Building on the Agile technique of coding to User Stories; Test Driven Development asks developers to build functionality to pass very specific Test Cases • Each short development cycle requires that the software pass the new Test Cases
- 6. How to integrate TDD into the development lifecycle
- 7. Test Driven Development with FHIR • To ensure interoperability between applications claiming conformance to the specification, a testing framework has been established within the FHIR specification itself https://www.hl7.org/fhir/STU3/testing.html • This framework defines a Test Engine for processing a TestScript resource as a natural language, computable format of a test case • The TestScript resource is an implementation-agnostic description of tests that allows test engines to evaluate if a FHIR implementation conforms with the FHIR specification https://www.hl7.org/fhir/STU3/testscript.html
- 8. FHIR Testing Framework – Test Engine Workflow http://hl7.org/fhir/STU3/testing.html#execution • Pre-Processing • Determine whether or not the TestScript interactions match the capabilities of the system under test. If supported, execution proceeds; if not, the test engine may halt execution. • Setup Execution • Optional operations that prepare the system under test for subsequent test execution. • Test Execution • Execute and record each test’s operations and assertions. • Teardown Execution • Optional operations that revert the system under test to its pre-test state. • Post-Processing • Cleanup of test execution data and collection of test results.
- 9. Pre-processing – Capability based testing • The test engine will use the CapabilityStatement of the system under test to determine how to process a TestScript. • If the TestScript defines the testing of capabilities that the system under test does support, the test engine will continue the workflow process. • If the TestScript defines the testing of capabilities that the system under test does not support, the test engine should allow the TestScript to be skipped.
- 10. Test Execution – Operations and Asserts • Test execution consists of two action types: • operation: The FHIR RESTful API interaction to be executed. • assert: The rule that is evaluated against the results of the operation. If true, the assert passes. • All FHIR operations including the extended operations framework are supported • Key concept: The operation is always executed regardless of whether or not an assert follows.
- 11. A little more about asserts… • An assert acts on the current executed operation context • It states a condition, rule or expression that will be evaluated against that executed operation context Asserts provide support for: • Evaluation of both the inbound request or outbound response • Complex and simple evaluation of HTTP headers and response codes • Operators for equality, relational comparison, containment, etc. • Expression language support for XPath, JSONPath and FHIRPath • Payload comparison to defined fixtures (static or dynamic) • Payload validation against defined FHIR profile(s) • Plus a whole lot more…
- 12. Introducing Touchstone • Available as a publicly accessible cloud-based testing platform (TaaS) • Provides automated, internet-based interoperability testing of the HL7® FHIR® specification • Tests the capabilities of and interoperability between both FHIR Server and Client implementations • A blend between Test-Driven-Development (TDD) methodologies and Natural Language Processing (NLP) TestScripts
- 14. Getting started with Touchstone • Anyone can access the Touchstone landing page and view the publicly available test definitions (TestScripts) • Registration and acceptance of the user agreement must be completed before any other features become accessible including test execution • Every user must belong to an organization • Organizations may have 1 or more registered users in Touchstone (based on subscription level)
- 15. Getting started – user registration (Hands on exercise) • If you are not yet registered in Touchstone, select the Register menu item in the upper- right navigation bar • Enter your • Touchstone user name • Valid email address (for notifications) • Password (twice for confirmation) • Select “I’m not a robot” • Click “Register”
- 16. Getting started – create a test system (Hands on exercise) • A test system must be available for test executions • You can use any existing, publicly accessible test system, or • Create your own test system: • Select the “New Test System” menu item • Fill out the required fields • Change the privacy settings if desired
- 17. Getting started – Test Setup (Hands on exercise) • Test executions are based on Test Setups • Steps: • Select one or more Test Definitions (TestScripts) • Click the Create Test Setup link • The Test Setup UI will be displayed • Select the Test System(s) • Fill in any other required fields • Click Execute…
- 18. Getting started – Test Execution (Hands on exercise)
- 19. Getting started – TestScript Execution (Hands on exercise)
- 20. Getting started – TestScript Execution – Results (Hands on exercise)
- 21. Analytics are built into Touchstone Conformance
- 22. Analytics are built into Touchstone Test Execution History
- 23. Test Driven Development with Touchstone • Has this happened to you? Your development team has been working hard to deliver their implementation. Your QA team is ready to go and you ask the developers – “is the app ready?” – meaning, did you finish your testing? Answer: Yes, sir. Good to go. Reality: The QA team starts in and finds many issues. Problem: Anyone can run a test and say a system passed or failed. How do you know?
- 24. Evidence-Based Quality Assurance • Continuous Testing against a defined standard ensures ‘no surprises’ when publishing new releases • Continuous Testing generates data used in analytics which can • show patterns and predict trouble areas • pinpoint issues in development • help show continuous interoperability to your customers, highlighting your competence and reliability • provide metrics that can tell the story about problem areas or where there is little volatility
- 25. Benefits of using Touchstone – A Real world example • The following graphs show the progression of testing for an organization using the Touchstone production environment. 100% Passing *Data used with organization’s permission 100% Passing
- 26. So, how can your organization use Touchstone? • First and foremost, Touchstone can help your organization with FHIR conformance testing Touchstone • maintains test suites for multiple FHIR versions • includes basic, advanced, and various Connectathon scenarios • tracks and maintains test results allowing organizations to view and share those results as they wish • provides the opportunity to validate conformance based on the test system’s CapabilityStatement their server actually issues; you test for what you assert your system is capable of doing
- 27. So, how can your organization use Touchstone? • Second, organizations can utilize the Touchstone test suites during software build cycles Touchstone • provides thorough logging of requests and responses allowing developers and testers greater insight into why a test may have failed • provides embedded, version specific links to the specification within the test execution results to help narrow focus • APIs are available to help automate testing (more about that in the advanced session)
- 28. So, how can your organization use Touchstone? • And, organizations have the opportunity to write and run their own TestScripts within Touchstone • Build your own scripts: conformance? regression? build verification and validation? • Can be uploaded to Touchstone by AEGIS upon approval or based on your subscription level • Coming soon! Touchstone IDE / TestScript Editor • Governing organizations can build Implementation Guide test suites and house them in Touchstone to facilitate conformance testing • AEGIS offers Multi-day tutorials and training on Touchstone, FHIR Testing and TestScript Authoring - http://www.aegis.net
- 29. Hands on Exercises • Getting Started – registration and test system setup • Test Setup and Execution - FHIR3-0-1-DevDays17 • Individual basic operations: read, search, create, update, delete • Workflow: basic operations in a single TestScript illustrating a simple workflow • Invalid Resource: attempts to create an invalid resource showing use of integrated validation engine • Connectathon Test Track • We will review and select one of the test tracks for the next HL7 FHIR Connectathon 17 event and begin the development of TestScripts
- 30. Discussion (Q & A)
- 31. FHIR® is the registered trademark of HL7 and is used with the permission of HL7. The Flame Design mark is the registered trademark of HL7 and is used with the permission of HL7. Amsterdam, 15-17 November | @fhir_furore | #fhirdevdays17 | www.fhirdevdays.com Thank you!
Editor's Notes
- #21: Developers don’t need to go back to their server logs