Efficient Testing with VUnit Part 2 – Optimizing Long Tests

This is the second of a series of blogs describing recent additions to VUnit that help improve your test efficiency.

In the first blog we described the importance of testing your code early and often in a continuous stream of short code/test iterations. Code developers must engage in this work since relying on system verification alone becomes a very long feedback loop. In order to support a highly iterative way of working, the test framework must be tailored for that mode of operation and we described a number of old and new features that VUnit provides. It basically comes down to a few principles:

• Test selection, or the art of maximizing the work not done: For example, the --𝚌𝚑𝚊𝚗𝚐𝚎 option runs only the tests affected by code changes.
• Efficient use of compute resources: VUnit's support for multi-threaded simulations and active load-balancing of threads are examples of this.
• Run tests in order of likelihood to fail: Prioritizing the most likely failures provides faster feedback and allows issues to be addressed while the remaining tests complete.

Dealing with Long Tests

In the first blog, we also showed an example of a perfectly load-balanced test run.

However, if we increase the number of threads, the result changes. We still achieve an overall speed-up, but VUnit reaches the lower load-balancing bound, which is determined by the longest test, t2.1, as shown in Figure 2.

In this case, all tests are fairly short, but in practice, the longest test may be significantly longer, as noted in a comment on the first blog:

"I often end up in situations where I have multiple short unit tests and one long top-level test that takes significantly more time than the others."

There are several ways to address long tests:

1. Ensure that the test verifies only what is necessary. It can be tempting to verify multiple features in a single test case when they share a common long setup, which is often the case with top-level tests. However, if the execution time is limited by the lower load-balancing bound, it is better to verify each feature in a dedicated test. Splitting the test also allows one feature to fail without affecting the others and provides more detailed feedback from the test run.
2. Where possible, parameterize the design to process smaller datasets. For example, in image processing, width and height generics can be introduced to allow a smaller image to be used during testing. These generics can be propagated to the top-level testbench and controlled from the run script. This makes it possible to create two configurations of the long test: one with a smaller image size and one with the full image size. Attributes can be assigned to the configurations, such as .𝚕𝚘𝚗𝚐_𝚝𝚎𝚜𝚝 for the full-size configuration and .𝚜𝚖𝚘𝚔𝚎_𝚝𝚎𝚜𝚝 for the smaller version. The run script can then be executed with --𝚠𝚒𝚝𝚑𝚘𝚞𝚝-𝚊𝚝𝚝𝚛𝚒𝚋𝚞𝚝𝚎 .𝚕𝚘𝚗𝚐_𝚝𝚎𝚜𝚝 for a faster test run, or with --𝚠𝚒𝚝𝚑𝚘𝚞𝚝-𝚊𝚝𝚝𝚛𝚒𝚋𝚞𝚝𝚎 .𝚜𝚖𝚘𝚔𝚎_𝚝𝚎𝚜𝚝 for a full test run, typically used during nightly regression tests.

Randomized Tests

Another example of long tests is randomized tests which tend to run for as long as we can afford waiting for them, as longer execution reduces the risk of missing defects. The same approach described above can be applied here: using a generic to control the duration of the randomized test and configurations with attributes to define and control the appropriate level of effort.

For randomized tests, there is also another strategy. Rather than aiming to achieve full coverage in a single run, coverage can be accumulated over time while reducing the effort required for each individual invocation. This approach is particularly effective in an iterative development process, where each test is executed many times throughout the project.

Regardless of the randomization solution used, there will be a method for setting the seed. A typical approach when using OSVVM is shown in Figure 3.

By using the random variable’s instance name as input to 𝙸𝚗𝚒𝚝𝚂𝚎𝚎𝚍, each random variable in the testbench will have a different initial seed and produce a distinct randomized sequence. However, these sequences remain the same for every invocation of the test, and additional test runs do not improve test coverage. To address this limitation, VUnit provides a set of 𝚐𝚎𝚝_𝚜𝚎𝚎𝚍 subprograms. Each simulation receives a unique base seed from 𝚛𝚞𝚗𝚗𝚎𝚛_𝚌𝚏𝚐, and this base seed can be combined with a salt value, such as the random variable 𝚒𝚗𝚜𝚝𝚊𝚗𝚌𝚎_𝚗𝚊𝚖𝚎, to generate several unique seeds within the simulation. This is shown in Figure 4.

Suppose our long test, t2.1, is a randomized test and we are already using 𝚐𝚎𝚝_𝚜𝚎𝚎𝚍 to obtain unique seeds for each invocation. To improve efficiency further, we can reduce the run time by lowering the coverage effort. We then create multiple configurations of the test, each of which will be assigned a unique seed. The combined coverage of these configurations matches that of the original test and because each configuration has a shorter execution time, they can be scheduled more efficiently. Figure 5 illustrates how two configurations of t2.1 can be created, and Figure 6 shows how this reduces the total execution time from 13 seconds to 9.5 seconds.

📌Note: This is an example where the simplified heuristic used for load-balancing does not produce the optimal solution, which is 8.5 seconds. A more effective heuristic that remains efficient is available and will achieve the optimum but has not yet been released.

📌Note: When a test is split into multiple configurations, there is a penalty due to the additional start-up times for the simulator invocations. In these examples, the start-up time is approximately one second.

Reproducibility

The seed generated for a test is logged in the test output file and, in the event of a test failure, it is also displayed on the console, as shown in Figure 7.

To reproduce the failing test setup and verify a bug fix, the failing seed can be specified using the --𝚜𝚎𝚎𝚍 option, as shown in Figure 8.

The --𝚜𝚎𝚎𝚍 option can also be set to 𝚛𝚎𝚙𝚎𝚊𝚝, in which case the seed values from the previous test run are retrieved from the test history and applied to each test in the new test run.

Summary

In this blog we have discussed how to manage test efficiency in the presence of long tests, particularly long randomized tests. In the next blog, we will show how VUnit can be integrated with external systems such as your version control system in order to decide what tests to run.