![Using a Formal Property Checker for Simulation Coverage Closure
Tim Blackmore, David Halliwell and Philip Barker Kerstin Eder and Naresh Ramaram
Infineon Technologies, Infineon House Computer Science Department
Great Western Court, Hunts Ground Road University of Bristol
Bristol BS34 8HP, UK Bristol BS8 1UB, UK
firstname.surname@infineon.com Kerstin.Eder@bristol.ac.uk
1 Introduction indicating why our method is both more efficient and effec-
We present a methodology that uses a formal property tive. Finally we report the results of using this methodology
checker to analyse coverage holes left by module-level sim- on the development of the Infineon TriCore 1.6 microcon-
ulation in order to achieve early coverage closure. Cover- troller.
age [3] is used to assess completeness of simulation-based Although the methodology includes formal verification,
verification. Coverage metrics are either based on the code the development of the properties does not require formal ver-
being verified, ie code coverage, or on a more abstract view ification expertise beyond what is described in Section 2, The
of the functionality of the design, ie functional coverage. Methodology. In addition, interaction with the formal tool
The most common code coverage metrics are statement and is encapsulated within the process thus automating away any
branch coverage, and 100% are normally required during ver- direct interaction with this tool.
ification.
Coverage targets need to be carefully defined and justified
since it is rarely possible to exercise all statements or take all 2 The Methodology
branches. This is for several reasons, such as code structure The core principle of this methodology is based on the notion
or configurability of a design. Thus, for example, it may only of temporal induction. A behaviour of a design will vary with
be possible to take 95% of the branches in the code for the time as the state of the design changes. Temporal induction
configuration being verified, perhaps leaving several hundred states that to show B is true for any state of the design it is
branches uncovered. These several hundred branches have to sufficient to prove the following two properties:
be analysed to decide whether they can in fact be covered,
1. B holds at reset and
if so then how, and if not, justification given as to why not,
2. B ⇒ next(B) holds
potentially identifying a coding error.
There are two major challenges in practice. Firstly, anal- Intuitively temporal induction works as follows. The first
ysis of the missing coverage can consume a large amount of property shows B holds at reset, then the second property
valuable engineering time. Secondly, recording of code that shows that B holds at cycle reset+1, another application of
cannot be covered is typically based on line number or pat- the second property shows that B holds at cycle reset+2, and
tern recognition, both of which are susceptible to changes to so on.
the code; indeed changes to the code may mean that code In particular, if there is a branch in the code such as
that previously could not be covered can now be covered. To
avoid this, coverage closure is typically left to near the end if X then ...
of the project, after sufficient code stability is achieved. This
it can be shown that the branch is never entered by substitut-
makes schedules more difficult to predict and means that bugs
ing B above with not(X), ie showing that
in the design and test bench may only be found late, both in-
creasing the likelihood of schedule slips. 1. not(X) at reset and
Our methodology uses a formal property checker to over- 2. not(X) ⇒ next(not(X))
come these two challenges by automating the analysis of cov-
erage holes. To illustrate the fundamental idea, let us consider both hold. These properties are simple in that they can be
an uncovered branch. We observe that, if the condition is not easily handled by any property checker in seconds and they
satisfied then the branch will not be entered during simula- can be derived from a coverage report via scripting.
tion. Now, if it is possible to formally prove that this condi- The properties are conservative in that if they pass then the
tion is never satisfied, then clearly this branch is not cover- branch (and any statements within the branch) definitely can-
able and can be discounted wrt branch coverage. If, however, not be covered, but if they fail it may or may not be possible
the proof fails, then a counter example is produced by the to cover the branch.
property checker. This counter example provides an indica- The properties can be strengthened while still remaining
tion (rather than proof) of whether the branch really can be simple by analysing code structure. Thus for priority coding
covered and if so how to do it. of branches eg an else if,
Some formal tools have dead code checks built in, eg [1, 2],
and we compare our methodology with such built-in checks if X then A
else if Y ...](https://image.slidesharecdn.com/2010bristolq1formal-property-checkers-130417160854-phpapp02/85/Using-a-Formal-Property-Checker-for-Simulation-Coverage-Closure-1-320.jpg)
![it can be proven that the else if branch is never entered Formal tools with built-in dead code analysis provide the
by substituting B above with not(X) and not(Y ). above advantages to some extent. However, there are some
Similarly for nested branches: major benefits to using this methodology in terms of perfor-
mance and effectiveness. The main reason that performance
if X then will be significantly better is that properties are only gener-
if Y then ... ated for uncovered code. This means that, even very early in
it can be proven that the nested branch is never entered by a project, more than 90% of the code will not be considered
substituting B above with X and not(Y ). which makes the difference between an overnight run and a
With understanding of the HDL used, these rules can be run taking several weeks. A built-in solution will only be
easily generalised to branches with any level of priority cod- able to take advantage of the coverage information if the for-
ing or nesting to give the highest possible chance of proving a mal tool is integrated with the coverage tool and specifically
branch (and its corresponding statements) cannot be covered. designed to do this.
The methodology, applied to close branch coverage, starts The fact that the user has control over the properties
from analysing coverage reports from simulation and then greatly improves the effectiveness of the proposed method-
generates a set of properties for each uncovered branch. If ology compared to built-in tools. Thus, for example, includ-
these properties pass then the branch can be filtered from the ing priority coding or nesting of branches, is only possible
coverage report. By using a naming convention, eg one that because the user is writing the scripts that extract the proper-
contains the line number of the code, this filtering can easily ties. This alone, in our experience, already identified a greater
be scripted. Thus the full process of reading the coverage re- amount of unreachable code than a built-in solution. For
port, generating and proving the properties, and filtering code the TriCore 1.6 microcontroller, this methodology found 93%
that cannot be covered becomes push button. of uncovered statements were indeed not coverable, while a
In practice, some code will not be covered because the test built-in tool only found 55%. Also, the addition of extra as-
bench is not intended to cover it. This information can be en- sumptions to reflect test bench scope and input behaviour may
capsulated in the properties in the form of extra assumptions. or may not be available when using a built-in solution.
For example, if an input is intentionally never driven high by We have found that built-in solutions target a limited num-
the test bench, or if inputs never violate a bus protocol, then ber of coverage metrics, often only statement and branch cov-
this can easily be added to the properties as extra assump- erage. Our scripted solution can be tailored for any metric,
tions. Assumptions can also specify values of internal sig- including focused-expression coverage and even functional
nals, eg configuration registers. Of course, all of these extra coverage, provided that the functional coverage is specified in
assumptions should be checked as assertions during simula- a language understood by formal tools such as SVA or PSL.
tion to ensure that they correctly reflect test bench behaviour.
Going one step further, for someone with formal expertise,
unreachable state information about the design can be proved
4 Results
and added as an assumption to all of these properties. In this This methodology was applied at Infineon Technologies dur-
way, it is possible to formally prove that all uncovered code ing the TriCore 1.6 microcontroller verification. Statement,
cannot be covered, although the effort for this may be consid- branch and focused-expression coverage (FEC) metrics were
ered too great (unless required for eg safety accreditation). considered, all in the context of branch prioritisation and nest-
ing. For statement coverage, 331 of 41074 statements were
not covered during simulation and of these, 309 were proved
3 Contributions not coverable. For branch coverage, 353 of 12341 branches
Firstly, because the entire process, once scripted, is fully au- were not covered during simulation and of these, 334 were
tomatic, the methodology saves considerable engineering ef- proved not coverable. For FEC, 1581 of 27230 FEC points
fort. In addition, the scripts are re-usable between projects. were not covered during simulation and of these, 1080 were
Secondly, this methodology can be applied early in the proved not coverable. Using this methodology meant that
project since it does not rely on code stability. Code changes statement and branch coverage were achieved well before
are automatically reflected in the generated properties. Engi- RTL tape-out, and the number of FEC points left for consid-
neering effort can be invested into adding assumptions to the eration could be prioritised. This directly led to the discovery
properties as described above in order to increase the num- of several bugs in the RTL, test bench and random constraints.
ber of holding properties. Hence, code that is not covered but These bugs may not have been found otherwise.
can be covered can be identified much earlier, allowing tests
or constraints to be written, regressions improved and bugs References
found earlier avoiding significant late code changes.
A third advantage is that it has been formally shown that [1] M. Andrews. Tightening the Loop in Coverage Closure. Mentor
any code excluded from coverage in this manner cannot be Graphics, December 2008. EDA Tech Forum.
covered. This contrasts with the standard approach, often [2] OneSpin Solutions GmbH. User Documentation: OneSpin
based on informal arguments. In the context of safety-critical 360TM, August 2006. Version 4.0.
applications, or when re-verifying legacy code with new cov- [3] A. Piziali. Functional Verification Coverage Measurement and
erage metrics, this is particularly significant. Analysis. Springer, 2004.
2](https://image.slidesharecdn.com/2010bristolq1formal-property-checkers-130417160854-phpapp02/85/Using-a-Formal-Property-Checker-for-Simulation-Coverage-Closure-2-320.jpg)
Using a Formal Property Checker for Simulation Coverage Closure
- 1. Using a Formal Property Checker for Simulation Coverage Closure Tim Blackmore, David Halliwell and Philip Barker Kerstin Eder and Naresh Ramaram Infineon Technologies, Infineon House Computer Science Department Great Western Court, Hunts Ground Road University of Bristol Bristol BS34 8HP, UK Bristol BS8 1UB, UK firstname.surname@infineon.com Kerstin.Eder@bristol.ac.uk 1 Introduction indicating why our method is both more efficient and effec- We present a methodology that uses a formal property tive. Finally we report the results of using this methodology checker to analyse coverage holes left by module-level sim- on the development of the Infineon TriCore 1.6 microcon- ulation in order to achieve early coverage closure. Cover- troller. age [3] is used to assess completeness of simulation-based Although the methodology includes formal verification, verification. Coverage metrics are either based on the code the development of the properties does not require formal ver- being verified, ie code coverage, or on a more abstract view ification expertise beyond what is described in Section 2, The of the functionality of the design, ie functional coverage. Methodology. In addition, interaction with the formal tool The most common code coverage metrics are statement and is encapsulated within the process thus automating away any branch coverage, and 100% are normally required during ver- direct interaction with this tool. ification. Coverage targets need to be carefully defined and justified since it is rarely possible to exercise all statements or take all 2 The Methodology branches. This is for several reasons, such as code structure The core principle of this methodology is based on the notion or configurability of a design. Thus, for example, it may only of temporal induction. A behaviour of a design will vary with be possible to take 95% of the branches in the code for the time as the state of the design changes. Temporal induction configuration being verified, perhaps leaving several hundred states that to show B is true for any state of the design it is branches uncovered. These several hundred branches have to sufficient to prove the following two properties: be analysed to decide whether they can in fact be covered, 1. B holds at reset and if so then how, and if not, justification given as to why not, 2. B ⇒ next(B) holds potentially identifying a coding error. There are two major challenges in practice. Firstly, anal- Intuitively temporal induction works as follows. The first ysis of the missing coverage can consume a large amount of property shows B holds at reset, then the second property valuable engineering time. Secondly, recording of code that shows that B holds at cycle reset+1, another application of cannot be covered is typically based on line number or pat- the second property shows that B holds at cycle reset+2, and tern recognition, both of which are susceptible to changes to so on. the code; indeed changes to the code may mean that code In particular, if there is a branch in the code such as that previously could not be covered can now be covered. To avoid this, coverage closure is typically left to near the end if X then ... of the project, after sufficient code stability is achieved. This it can be shown that the branch is never entered by substitut- makes schedules more difficult to predict and means that bugs ing B above with not(X), ie showing that in the design and test bench may only be found late, both in- creasing the likelihood of schedule slips. 1. not(X) at reset and Our methodology uses a formal property checker to over- 2. not(X) ⇒ next(not(X)) come these two challenges by automating the analysis of cov- erage holes. To illustrate the fundamental idea, let us consider both hold. These properties are simple in that they can be an uncovered branch. We observe that, if the condition is not easily handled by any property checker in seconds and they satisfied then the branch will not be entered during simula- can be derived from a coverage report via scripting. tion. Now, if it is possible to formally prove that this condi- The properties are conservative in that if they pass then the tion is never satisfied, then clearly this branch is not cover- branch (and any statements within the branch) definitely can- able and can be discounted wrt branch coverage. If, however, not be covered, but if they fail it may or may not be possible the proof fails, then a counter example is produced by the to cover the branch. property checker. This counter example provides an indica- The properties can be strengthened while still remaining tion (rather than proof) of whether the branch really can be simple by analysing code structure. Thus for priority coding covered and if so how to do it. of branches eg an else if, Some formal tools have dead code checks built in, eg [1, 2], and we compare our methodology with such built-in checks if X then A else if Y ...
- 2. it can be proven that the else if branch is never entered Formal tools with built-in dead code analysis provide the by substituting B above with not(X) and not(Y ). above advantages to some extent. However, there are some Similarly for nested branches: major benefits to using this methodology in terms of perfor- mance and effectiveness. The main reason that performance if X then will be significantly better is that properties are only gener- if Y then ... ated for uncovered code. This means that, even very early in it can be proven that the nested branch is never entered by a project, more than 90% of the code will not be considered substituting B above with X and not(Y ). which makes the difference between an overnight run and a With understanding of the HDL used, these rules can be run taking several weeks. A built-in solution will only be easily generalised to branches with any level of priority cod- able to take advantage of the coverage information if the for- ing or nesting to give the highest possible chance of proving a mal tool is integrated with the coverage tool and specifically branch (and its corresponding statements) cannot be covered. designed to do this. The methodology, applied to close branch coverage, starts The fact that the user has control over the properties from analysing coverage reports from simulation and then greatly improves the effectiveness of the proposed method- generates a set of properties for each uncovered branch. If ology compared to built-in tools. Thus, for example, includ- these properties pass then the branch can be filtered from the ing priority coding or nesting of branches, is only possible coverage report. By using a naming convention, eg one that because the user is writing the scripts that extract the proper- contains the line number of the code, this filtering can easily ties. This alone, in our experience, already identified a greater be scripted. Thus the full process of reading the coverage re- amount of unreachable code than a built-in solution. For port, generating and proving the properties, and filtering code the TriCore 1.6 microcontroller, this methodology found 93% that cannot be covered becomes push button. of uncovered statements were indeed not coverable, while a In practice, some code will not be covered because the test built-in tool only found 55%. Also, the addition of extra as- bench is not intended to cover it. This information can be en- sumptions to reflect test bench scope and input behaviour may capsulated in the properties in the form of extra assumptions. or may not be available when using a built-in solution. For example, if an input is intentionally never driven high by We have found that built-in solutions target a limited num- the test bench, or if inputs never violate a bus protocol, then ber of coverage metrics, often only statement and branch cov- this can easily be added to the properties as extra assump- erage. Our scripted solution can be tailored for any metric, tions. Assumptions can also specify values of internal sig- including focused-expression coverage and even functional nals, eg configuration registers. Of course, all of these extra coverage, provided that the functional coverage is specified in assumptions should be checked as assertions during simula- a language understood by formal tools such as SVA or PSL. tion to ensure that they correctly reflect test bench behaviour. Going one step further, for someone with formal expertise, unreachable state information about the design can be proved 4 Results and added as an assumption to all of these properties. In this This methodology was applied at Infineon Technologies dur- way, it is possible to formally prove that all uncovered code ing the TriCore 1.6 microcontroller verification. Statement, cannot be covered, although the effort for this may be consid- branch and focused-expression coverage (FEC) metrics were ered too great (unless required for eg safety accreditation). considered, all in the context of branch prioritisation and nest- ing. For statement coverage, 331 of 41074 statements were not covered during simulation and of these, 309 were proved 3 Contributions not coverable. For branch coverage, 353 of 12341 branches Firstly, because the entire process, once scripted, is fully au- were not covered during simulation and of these, 334 were tomatic, the methodology saves considerable engineering ef- proved not coverable. For FEC, 1581 of 27230 FEC points fort. In addition, the scripts are re-usable between projects. were not covered during simulation and of these, 1080 were Secondly, this methodology can be applied early in the proved not coverable. Using this methodology meant that project since it does not rely on code stability. Code changes statement and branch coverage were achieved well before are automatically reflected in the generated properties. Engi- RTL tape-out, and the number of FEC points left for consid- neering effort can be invested into adding assumptions to the eration could be prioritised. This directly led to the discovery properties as described above in order to increase the num- of several bugs in the RTL, test bench and random constraints. ber of holding properties. Hence, code that is not covered but These bugs may not have been found otherwise. can be covered can be identified much earlier, allowing tests or constraints to be written, regressions improved and bugs References found earlier avoiding significant late code changes. A third advantage is that it has been formally shown that [1] M. Andrews. Tightening the Loop in Coverage Closure. Mentor any code excluded from coverage in this manner cannot be Graphics, December 2008. EDA Tech Forum. covered. This contrasts with the standard approach, often [2] OneSpin Solutions GmbH. User Documentation: OneSpin based on informal arguments. In the context of safety-critical 360TM, August 2006. Version 4.0. applications, or when re-verifying legacy code with new cov- [3] A. Piziali. Functional Verification Coverage Measurement and erage metrics, this is particularly significant. Analysis. Springer, 2004. 2
- 3. Appendix 343 if ( ready && sel mast ) 344 begin 345 if ( burst <= 3) 346 begin 347 case(Mast num) 348 1003 Mast’h1 : master = 1; 349 ** 0 ** Mast’h2 : master = 3; 350 5000 default : master = 0; 351 endcase 352 end 353 else if ( burst > 4 ) 354 begin 355 case(Mast num) 356 507 Mast’h0 : master = 2; 357 432 Mast’h3 : master = 4; 358 872 default : master = 0; 359 endcase 360 end 361 end Fig. 1: Small fragment of a coverage report consisting of three columns. The first column refers to the line number in the RTL file. The second column indicates the number of times the statement has been executed during simulation. The third column refers to the actual RTL code. Note that for line number 349 there are zero hits, ie there is a coverage hole at this line and coverage closure will focus on this line. Label Control Flow Conditions branch path 343 ready && sel mast branch path 348 ready && sel mast && burst <=3 && Mast num == 1 branch path 349 ready && sel mast && burst <=3 && Mast num == 2 branch path 350 ready && sel mast && burst <=3 && Mast num != 1 && Mast num != 2 branch path 356 ready && sel mast && burst > 4 && Mast num == 0 branch path 357 ready && sel mast && burst > 4 && Mast num == 3 branch path 358 ready && sel mast && burst > 4 && Mast num != 0 && Mast num != 3 branch path n343 !( ready && sel mast ) Fig. 2: Control flow conditions extracted from the code fragment depicted in Figure 1. Note that the expression on the last line, the one labeled branch path n343, is generated from the implicit else branch that complements line 343. 3
- 4. macro branch path 349 ( ready && sel mast && burst <=3 && Mast num == 2 ) endmacro; property branch not covered 349 = always ( !(branch path 349) ) @ ( posedge clk ); endproperty; assert branch not covered 349; Fig. 3: Translation of the control flow expression for line 349 in Figure 2 to the formal property. The property is defined in three steps, namely macro generation, claiming of property and asserting the property. The macro generation encapsulates the control flow condition in formal syntax and assigns a name to the macro. The property definition then refers to this macro and claims that the specified condition is not reachable, hence the negation. The final step is asserting the property so that it is executed by the formal property checker. Fig. 4: Waveform of a counter example provided by the formal property checker in case the property fails. Observe that the signal Mast num has assumed a value of 1 at time ’t’. The formal tool inductively tries to prove that it holds the same value in the next time interval ’t+1’. However, in this case the property fails and the property checker provides a counter example in form of a waveform. This waveform shows that the signal Mast value can take a value of 2, which indicates that the code is reachable. 4