The Myth of “Make the product Right” and “Make the Right Product”

If you’ve been around for any length of time in medical device development, then you know the shorthand for Design Verification and Design Validation: “did I make the product right?” and “did I make the right product?” respectively.

Are they true, or just clever games of word re-arrangement? While both are good starting points to illustrate that Design Verification and Design Validation are indeed different, these social definitions still raise plenty of room for confusion and misinterpretation. Here’s my shot at new phrases and a moderately nuanced discussion of what both processes mean.

What the Standard and Regulation Say

ISO 13485:2016 and 21CFR 820 are fairly well-aligned on the definitions of Design Verification and Design Validation:

In addition to Design Validation, 21CFR section 820.3(z)(1) also offers a definition of Process Validation:

Process validation means establishing by objective evidence that a process consistently produces a result or product meeting its predetermined specifications.

This is notable because Process Validation is actually related to making the product. Neither Design Verification nor Design Validation are about making the product, they relate to the design and development the product. Pedantic semantics? Adding to the confusion? Maybe, but read on.

A product doesn’t get to market without the successful interaction of multiple processes, so it doesn’t make sense to appropriate terms from other processes and give them different meanings. In med-tech, “making” should be a word for manufacturing, not design & development.

Design Verification

“Did I make the product right” is really the question for the Production Team, not Design & Development! Obviously we’re all smart enough not to confuse the two but we can do better, starting with a more selective review of how FDA and ISO describe Design Verification.

FDA’s old, but not obsolete, Design Control Guidance for Medical Device Manufacturers is quite clear that Design Verification is done on the design specification (the Design Output), not necessarily a finished product, meaning Design Verification is not “did I make the product right”. Design Verification may very well start before you’ve actually built any products. Two separate sentences illustrate the point:

As the design effort progresses, verification activities become progressively more comprehensive.

Verification activities are conducted at all stages and levels of device design.

The guidance authors’ choice of a building design & construction analogy provides some elaboration. It describes the building requirements (Design Inputs) as being followed by the specifications to meet those requirements (Design Outputs). Then you could verify that the air-conditioning system, for example, meets the written requirements by verifying the system specification, energy efficiency, Btu/h etc. Subsequently you would need to build the building, put people in it and turn on the AC to validate it. To the point, we didn’t need to build the building to verify this AC requirement. It’s from this example that we can see that verification is not exclusively testing, and verification is not always done on a finished assembly.

Of course, for the most part, you would build a medical device to do Design Verification. But let’s say you are buying a valve, for example, to use in a larger assembly. It would be acceptable, almost preferable, to verify the valve during the design process so that you could justify its inclusion in the final device. You could also wait until the whole thing is designed and built before doing a single minute of Design Verification– both options work but one carries more risk of disrupting your commercialisation plans.

Especially in devices with hardware & software, subsystems or sub-assemblies, it may be preferable or even necessary to conduct Design Verification testing before you “make” the whole device. If your sub-assembly is inaccessible in the finished device, you may not be able to verify that it meets all the inputs – simply measuring a hidden component becomes impossible.

To take the point a little further, Fault tree analysis of a process or design and Failure modes and effects analysis are two “verification methods and activities” listed in the FDA guidance. While both are more commonly associated with risk management, as part of Design Verification you wouldn’t wait to conduct these activities until the device is finished and built.

Similarly, the ISO 13485 Practical Guide is explicit:

For larger projects, the design and development process is often broken into stages and the verification can be carried out on a stage-by-stage basis.   

In terms of Design Outputs meeting Design Inputs, we want to know if our Outputs (specifications, drawings, BOMs, training materials, programs, QC procedures, records etc.) satisfy the written, measurable requirements we wrote.

Design Verification is not “did I make the product right”, Design Verification is “did I design the product I said I would?”.

So What “Product” do I Verify?

While there is no rule about what qualifies something for Design Verification, the ISO 13485 Practical Guide gives a clue:

Product used for verification is to be representative of the final product. Requirements should be established to demonstrate the design and development output is met in a consistent manner.

Design Verification can be done on prototypes, as far as the prototypes represent the final specification. A stereo-lithography print, for example, could be used to verify the volume of a storage chamber, but not to verify the robustness of a bonded connection (due to the non-specification materials). Alternatively, a mathematical analysis based on dimensions, or an in-silico (computer-aided) analysis could verify the volume.

More commonly, we would do Design Verification on the real thing, so that we take all manner of doubt out of the frame. However these devices are probably built by technicians in the development team, or engineers intimately familiar with the device. There is no requirement in 21CFR 820 or ISO 13485 for Design Verification parts to be production equivalent, however you do need to be able to show why the parts were valid to demonstrate conformance of the Design Outputs with the Design Inputs (ala the text from the Practical Guide above).

It seems that the authors of the FDA guidance also anticipated potential misinterpretation of Design Verification and provide this critical text:

Some manufacturers erroneously equate production testing with verification. Whereas verification testing establishes conformance of design output with design input, the aim of production testing is to determine whether the unit under test has been correctly manufactured.

While it may seem obvious not to mistake product testing during development for production testing, there is one good reason why this disambiguation is needed. 21CFR itself indirectly refers to “production testing” as “verification”:

§820.75(a) Where the results of a process cannot be fully verified by subsequent inspection and test, the process shall be validated.

The message here is that testing in production is not Design Verification, and that Design Verification is not about making the product, Production is.

Design Validation

“Did I make the right product”, taken literally, is a question more appropriate for the Operations Team: did I make the product on the work order? More likely, it’s a Top Management question: does this product support my organisation’s mission and financial objectives?

The sentiment of “did I make the right product” is that the “right product” is one that people will buy.

Layers within “right” include function, cost, quality, leadtime, service, ease of adoption, integration, reimbursement and other factors that a customer will consider before buying. The absence of any discussion of cost considerations in ISO 13485 and 21CFR 820 reminds us that Design Validation doesn’t reach that far, which demands a more precise shorthand than “did I make the right product”.

Design Validation is our attempt “to ensure that the resulting product is capable of meeting the requirements for the specified application or intended use.” A compliant Design Validation is not the same as ensuring people want to buy the product or even use it. That objective is a combination of the Intended Use, User Needs, business case and marketing plan. It is possible to have a fully compliant Design & Development process, including successful Design Verification, Design Validation (including Usability Validation) and market authorisation, but still not sell a single device – in which case you’ve failed your organisation.

Design Validation is actually not “did I make the right product?”, it’s “did I design a product that does what people want it to do?”.

“What people want it to do” is defined in your User Needs and Intended Use, which closes the circle to the standard definitions and regulations. However successful Design Validation will not guarantee that people will buy your product or even use it.

User Needs vs Design Inputs vs “what the user needs”

In the past, I have personally made the mistake of overlapping or even confusing User Needs with Design Inputs. User Needs are what the user would tell you they need, and Design Inputs are measurable specifications. This is important in terms of understanding the objectives of Design Verification vs Design Validation.

A user probably won’t tell you they need a >=12.0N detachment force or SAL 10-6 - those are Design Inputs and it’s the job of the product development engineer to translate them from User Needs. The corresponding User Needs would be something like “the tubing connections must not leak or break when the blood bag is full and the system is hanging from the cart”, and “the device must not cause contamination to the sample”. While we aim to translate all User Needs to verifiable Design Inputs, we also need something to close the loop on whether the Design Input actually meets the User Need. That’s the job of Design Validation and it’s why we are not checking if we “made the product right”, we are checking if we designed a product that does what people want it to do (User Needs).

Another hard-learned lesson was User Needs vs what the user needs. The latter should drive the former, rather than setting an arbitrary or aspirational bar that, if not met, could scuttle a whole project or, at best, cause costly delays. A few years ago I participated in development of a quantitative lateral flow test. Our User Need for the test-read time was <5 minutes which already challenged the performance limit of lateral flow technology. What the user needed was actually far less than that (they weren’t interested in buying a test that took 5 minutes) and our written User Need undershot what the user actually needed. It’s too late at Design Validation to learn if you got your User Needs right and thankfully we shelved that project before reaching Design Validation.

A project involving development of a different sampling device wasn’t played so well. A User Need regarding the device’s flow rate was based on some user interviews. During development we struggled to achieve consistent device performance, leading to months of expensive and time-consuming fine-tuning and testing. Yet upon presenting the device to users for simulated-use testing labs, they were fine with the device and said “yes it does the job, it meets our safety and effectiveness requirements and we’d use it”! In this case the User Need overshot what the user actually needed. If we had at first been able to identify how much performance actually met the threshold for “added value” vs what was ideal, we could have put a device on the market much sooner (and moved on to new things sooner too!).

Often User Needs also suffer from willful ignorance of how much a customer is willing to pay vs the value it delivers. Cost is not a required User Need as far as ISO 13485 or 21CFR 820 are concerned, however without including it as a quasi-User-Need, you are missing the whole point of your organisation’s existence. You can’t deliver improved healthcare outcomes unless someone actually buys your product. You can design a great device that saves lives, successfully conduct Design Verification & Design Validation and get market authorization all over the world, but if no one buys it because you designed it at the wrong cost point… enough said.

What “Products” to I (Design) Validate?

In contrast to Design Verification, Design Validation must be done on “production equivalent” products, which rules out prototypes or devices assembled without traceability or control:

21CFR §820.30(g) Design validation.

Design validation shall be performed under defined operating conditions on initial production units, lots, or batches, or their equivalents.

ISO 13485 7.3.7

Design validation shall be conducted on representative product. Representative product includes initial production units, batches or their equivalents. The rationale for the choice of product used for validation shall be recorded.

I do note ISO 13485’s use of “representative” for Design Validation at 7.3.7, and “representative of the final product” for Design Verification in the Practical Guide. I can’t vouch for TC 210’s decision on this wording however, the point is moot given that 7.3.7 immediately elaborates and names initial production units, batches or their equivalents. Clearly prototypes would not be adequately equivalent.

You do need to make the product for Design Validation, but Design Validation isn’t testing if you made it right, or if you made the right one. It’s testing if your product meets the User Needs and Intended Use. If your Intended Use and User Needs (what the user said they need) are correctly calibrated with the needs of the user (the performance threshold that adds value, and the cost they will pay to get that value) you’re on track for success.

It’s not uncommon for organisations to conduct Design Transfer prior to Design Validation and, in some cases, before Design Verification. The value of this for Design Verification is debatable, but for Design Validation a product that is transferred to production represents the perfect “representative product”. Referring to FDA’s definition, one would be challenged to answer how one could use “initial production units” without doing Design Transfer. Thankfully, both the standard and the regulation allow us to use “equivalents”, noting that you are still strongly advised to use devices that are simply a few signatures away from actual Design Transfer. In any case, traceability to raw materials, processes, operators and dates is essential.

In absence of full or partial Design Transfer, at the very least Design Validation needs final materials including the adhesives, insulators, lubricants, packaging etc. that you intend to use in the commercial device. Ideally you would also have completed Process Validation (PV) for assembly, packaging and sterilisation so that your Design Validation uses the same quality of device that your user and/or patient will experience in commercial production. ISO 13485’s requirement that the rationale for the choice of product used for validation shall be recorded is easily answered if you’ve done PV, but anything less (not recommended) needs a rock-solid rationale. A reminder that Usability Validation falls under Design Validation, so the same applies to Summative Usability Testing.

I haven’t discussed the difference between simulated and real-use testing for Design Validation since it depends very much on your device.

Conclusion

While they are clever condensations of complex topics, “did I make the product right” and “did I make the right product” don’t guide us to exactly what medical device Design Verification and Design Validation truly are. For those willing to dig deeper, the official written guidances on meeting the regulation and standard have not evolved much over the last 20 or more years, so medical device developers continuously face the same uncertainty and ambiguity, especially start-ups and fresh entrants. Improved shorthand or social definitions might improve overall understanding within the industry and lead organisations, both old and new, to success faster.

I hope these suggestions either clarify or raise some impactful discussion:

Design Verification: “did I design the product I said I would?”

Design Validation: “did I design a product that does what people want it to do?”

And as a bonus, the following suggestions might help fill out more of the picture:

• User Needs: what the user will tell you they need the device to do, and what you write in your file.
• What the user needs: the thing that adds value for them, and the cost bearable to get that value.
• Usability Validation: “did I design a product that people can use the way I want them to?”
• Process Verification: “did I develop a process that makes the product that I designed?” and “is every single product coming off this line meeting the acceptance criteria?”
• Process Validation: “did I develop a process that can be repeated without having to check every single product?”
• Design Transfer: “did I create the right specifications and instructions to manufacture the product I designed, every time?”

Chris Whelan has 20+yrs’ medical device design, development and commercialisation experience in Class 1 and Class 2 sterile devices. He likes to ask questions and explore topics that would have saved years of fumbling earlier in his career. His work at ITL BioMedical is in concept-to-commercialisation of single-use sterile devices. Chris has also had the pleasure of working on the development of IVDs with software and hardware.