Coverage-Driven Functional Verification Using Coverage to Speed Verification and Ensure Completeness

Verisity Design, Inc.
September 2001

Introduction

After a couple of weeks of using a testbench automation tool to create additional tests and using functional coverage for feedback, the engineer had achieved one hundred percent of the defined functional coverage.

Functional coverage is defined as explicit functional requirements derived from the device and test plan specifications. Most verification engineers see the value of code coverage and functional coverage, however, most do not make substantial use of coverage for various reasons, mainly because of time pressure.

This document suggests a complementary approach of both functional and code coverage, starting from early stages in the verification process. Coverage is used as a goal and as a metric, by which the testbench and test suite will be measured throughout this verification process. The approach saves significant human and machine resources, shortens time-to-market and eventually contributes to a mature and timely tape-out decision and a high-quality product.

This paper discusses the need for objective metrics, studies the approaches used before functional coverage evolved and defines functional coverage and the complementary technology of code coverage. It then demonstrates the proposed approaches using Specman Elite's functional coverage engine and SureCov's code coverage capabilities. At last, it suggests a flow for maximizing testbench effectiveness.

Motivation

The magnitude and complexity of recent designs introduce new challenges. Two such major challenges are how to reduce the time it takes to do verification to a reasonable size, and how to ensure complete verification.

Effective Simulation
As exhaustive tests are practically impossible, the verification engineer should spend his time carefully. In order to avoid unnecessary repetitions, the engineer needs to be able to answer questions like: Were all possible stimuli variations injected? Were all possible results achieved? Were all Device Under Test (DUT) states visited? Did all the internal transitions take place? Did all the interesting events occur? By answering such questions we can re-set our simulation goals and dramatically expedite the verification process.

Ensuring Completeness
The second challenge is to ensure completeness - how can we tell that verification is done? This decision is usually accompanied with tension and the emotional stress of trying to organize the abundant information gathered throughout the recent simulation sessions. Taping out an immature design can have an impact at both the personal and corporate levels. At the personal level, your job and professional reputation are at stake. It can be extremely embarrassing when a chip has to be re-spun due to a common use of the IC that was never verified. At the corporate level, small start-ups may literally collapse due to product quality, and even well established companies may find it hard to recover from the financial repercussions.

What about unnecessary delays in tape out? Tight schedules and narrow market windows characterize most of the industry today. Some companies quantify a one-day delay in the millions.

In order to exercise different aspects of the design, different inputs or tests need to be fed into the system. For most systems, creating these tests has become very challenging; it involves creating extremely sophisticated data, with parallel stimulus from various input streams, etc. To simplify test creation, automatic test generation was developed to substantially reduce the time it takes to create an effective test. This automated approach allows users to express high-level intentions, which allows the testbench automation tool to create various stimuli that manifest those intentions. By reviewing the input that was created, the user can then direct the generation towards areas that are yet to be exercised to ensure completeness.

Historically, a few coverage metrics have been used to evaluate the verification process, but all fail to address the aforementioned needs. A more suitable solution for these hard problems is functional coverage measurement. But let's first examine the historical coverage measurements and their limitations.

Current Coverage Approaches

Toggle Coverage
One of the oldest coverage measurements, toggle coverage, was historically used for manufacturing tests. Toggle coverage monitors the bits of logic that have toggled during simulation. If a bit doesn't toggle from 0 to 1, or from 1 to 0, it hasn't been adequately verified. Toggle coverage does not ensure completeness. It cannot assure that a specific bit toggle sequence that represents high-level functionality has occurred. Toggle coverage does not shorten the verification process. It may even prolong it as engineers trying to toggle a bit, which cannot toggle according to the specifications. Toggle coverage is also very low level and it may be cumbersome to relate a specific bit to a test plan item.

Code Coverage
The most popular coverage metric is code coverage. The basic assumption of code coverage is that unexercised code potentially bears bugs - the code is guilty until proven innocent. One needs to be adventurous to synthesize and tape-out un-exercised logic. That makes code coverage analysis a necessity, but does it solve the challenges mentioned above? No. It is often said that code coverage checks how well your RTL code was exercised, rather than how well your design functionality was exercised. Therefore, code coverage is a necessity, but not a complete metric. Code coverage should be employed as part of a more comprehensive coverage strategy (see section 'Functional Coverage and Code Coverage').

Other Approaches
In addition to the more technical approaches, some management-oriented techniques are being used to indicate the verification progress. Examples of such techniques include measuring the elapsed time between bugs found or between RTL changes. Not finding a bug in a week may indicate the design is clean, but it can also indicate test deficiencies. It is critical to distinguish between the two and this is where coverage tools start justifying their investment.

Some engineers would advocate stressing the simulator for a few extra cycles to ensure completeness instead of using coverage metrics. Employing more simulators, or using an emulator or an accelerator, for peace of mind could achieve this. Is that enough? While 5000x improvement over simulation speed looks very impressive, the clock is still slow compared to real clock speed. A more intelligent bug search is a better solution than this "brute force" method. The intelligent search has proven to have a faster coverage results on a per week basis and is significantly more effective in discovering the real hard-to-find bugs.

What about formal verification tools? They mathematically prove correctness, isn't that our ultimate goal? Who needs coverage or simulation when we have an exhaustive proof? Unfortunately, formal techniques suffer from capacity issues; even the most recent tools are struggling to verify single modules. Currently the design complexity is increasing at a rate faster than the improvement in formal techniques. Using such tools may be appropriate on a single module, but this does not solve the overall problem.

The approaches mentioned above prove to be in sufficient in fulfilling our needs. Most of them have very limited capabilities to indicate completeness, and almost no capabilities to shorten the verification process.

The following chapters describe a set of requirements and verification methodologies that meet our needs. Later on, a functional coverage tool fulfilling these requirements will be discussed.

Coverage Requirements

First, let's examine the requirements for coverage. We can categorize the requirements into two groups: demands for the data gathering and analysis engine, and requirements for the surrounding testbench that will allow efficient usage of the accumulated information.

Coverage engine requirements

Expressiveness of Queries
Clearly, expressiveness to define any query is a major requirement. It should be easy to specify a query, and get the needed information. It should also be easy to filter out the relevant details. Specifically, a functional coverage tool should handle high-level queries such as: have I exercised all types of packets while handling an interrupt?

Informative Reports
Getting coverage results should be readable and intuitive. Both a Text User Interface (TUI) and a Graphic User Interface (GUI) should be provided. Usually, engineers use the GUI since it provides an easy means to review, query and print the coverage database. The TUI is useful when trying to forward the results to other automatic tools or manipulate the data into custom reports.

Beneficial Coverage Feedback
When coverage results are less than satisfying, it should be easy to deduce the appropriate adjustments and generate tests to improve the coverage results.

Testbase Ranking
The ability to accumulate and analyze coverage reports from multiple simulation runs is crucial. Test suites today comprise of large number of tests. This ability to analyze cumulative coverage allows you to:

1. Get an overall picture of the entire verification environment, and measure your recent progress. Using these two measurements, you can objectively predict the tape-out date,
2. Avoid test redundancy. By measuring the amount of coverage added by each test, redundant tests can be identified and removed.

Timing of Analysis
The coverage tool should allow the engineer to analyze the coverage information both between simulations and during a test run. The first approach requires the ability to save the collected information to be reviewed later on. The second approach requires a run-time interface to the coverage database that allows it to be used during simulation¹.

Grading
Various coverage holes may be prioritized and the overall progress can be better represented by setting individual goals for each concern (how many times must I cover each scenario?), and by setting weights to distinguish the relative importance of different coverage scenarios.

Optimizing the testsuite
Ranking capabilities should allow you to create a subset of tests that verify the DUT to a significant degree, with a minimum amount of resources. Running this subset of tests instead of your entire test suite, drastically reduces the total number of cycles needed for verification.

Open Environment
Intellectual property reuse, and design complexity, have turned our verification environment into a mix of design representations and verification languages. Having the flexibility to use the same methodology on all types of designs is critical.

Methodology Requirements
In addition to coverage tool requirements, some verification environments allow extensive use of the collected data while others enable merely partial utilization of the coverage results. For a more comprehensive coverage environment, the methodology should allow:

Self Checking
A self-checking verification environment notifies you once a malfunction has occurred.

Test Generation Control
Being able to easily target coverage holes using test generation adds great efficiency to the use of coverage metrics.

Repeatability
It is important that as the design and the testbench evolve, that the current test suite with the current design still covers all of the functional coverage specified. Test generation engines that have "random stability" minimize the disruption of small changes to the functional coverage.

The following chapter defines functional coverage and differentiates it from code coverage.

Functional Coverage and Code Coverage

Code Coverage
As mentioned above, code coverage reflects how thorough the HDL code was exercised. A code coverage tool traces the code execution, usually by instrumenting or modifying the HDL code.

The set of features provided by code coverage tools usually includes line/block coverage, arc coverage for state machines, expression coverage, event coverage and toggle coverage. More recent code coverage tools also include capabilities that automatically extract finite state machines from the design and simplify the task of ensuring complete states and arc coverage (see 'SureCov's Code Coverage' section).

Code coverage is a necessity. Most people would agree it is unacceptable to synthesize code that is either dead or unverified. Nevertheless, code coverage is not enough. Most functional coverage requirements cannot be mapped into lines of code. For example, code coverage cannot indicate whether we've been through all the legal combination of states in two orthogonal state machines. Another example might be whether we have tried all the possible inputs while the DUT was in all the different internal states. Also, code coverage does not look at sequences of events, such as what else happened before, during, or after I executed this line of code. Thus, code coverage does not ensure completeness and does not fulfill most of the requirements that allow expediting the verification task.

In general, three inherent limitations of code coverage tools are: overlooking non-implemented features, the inability to measure the interaction between multiple modules and the ability to measure simultaneous events or sequences of events (see 'Functional Coverage' section).

Functional Coverage
Functional coverage perceives the design from a user's or a system point of view. Have you covered all of your typical scenarios? Error cases? Corner cases? Protocols? Functional coverage also allows relationships, "OK, I've covered every state in my state machine, but did I ever have an interrupt at the same time? When the input buffer was full, did I have all types of packets injected? Did I ever inject two erroneous packets in a row?"

The bulk of low-level details may be hidden from the report reviewer. Functional coverage elevates the discussion to specific transactions or bursts without overwhelming the verification engineer with bit vectors and signal names.

This level of abstraction enables natural translation from coverage results to test plan items. Specman Elite's functional coverage engine is an example of a powerful functional coverage tool (see 'Specman Elite Functional Coverage' section).

As functional and code coverage are complementary in nature, a tool or methodology which combines both approaches is extremely beneficial. As mentioned earlier, this combined methodology will provide a complete overview of the verification progress and a clearer correlation between the functional coverage definitions and the actual design implementation. In the following sections, we introduce two specific implementations of code coverage and functional coverage tools: SureCov's code coverage and Specman Elite's functional coverage.

SureCov's Code Coverage

Figure A -- SureCov's Gui Figure B -- Bubble Diagram for FSM

No changes to Source
SureCov automatically extracts FSMs from Verilog code without requiring any code modifications.

Automatic FSM Analysis
By automatically recognizing common Finite State Machine types from the Verilog RTL design description, SureCov eliminates the manual pre-processing required in some tools. The Verilog HDL can be visualized as a bubble diagram without any user intervention. SureCov paints the states and transition arcs in colors indicating verified, unverified, and partially verified. The user can define paths through FSMs and query the SureCov database to check coverage for the FSM paths. In addition, the FSM analysis capability allows you to find unreachable state machines and is complemented by an easy to use filtering mechanism (see figure B).

Intuitive Block, Arc & Toggle Coverage
An intuitive display of a text-oriented design is often in the original code. SureCov displays block, arc, and toggle coverage highlighted right in your code. SureCov colors the lines entirely verified green, and those unverified in red. Code that has been executed, but not from all possible combinations, is yellow.

Unique Expression Coverage
In multi-term expressions, SureCov allows you to determine which terms contribute to a true value for the expression's 'On Terms', and which terms contribute to a false value for the expression's 'Off Terms'. A minimized sum of products, and product of sum analysis, is provided for each expression. SureCov displays expression and event coverage right in the code, or by clicking on the source it takes you to the expression detail of the terms.

Regression Analysis Utilities
SureCov has many utilities for test analysis and optimization. This includes a coverage API that enables reactive test generation.

Low Simulation Overhead
SureCov has a very low simulation overhead. By analyzing the source code without burdening the simulator, typical overhead is 4x less than most other code coverage tools.

Specman Elite's Functional Coverage

Example
Using Verisity's verification language, e, one can easily capture both the topology and input specification.

type PacketType :[ EVII, E802_3, ESNAP ];
type PacketCorrectness: [good, crc_error, size_error];

// packet definition
struct packet {

// service fields kind :PacketType; size :uint; keep soft size in [64..1518]; delay :uint[1..10]; start_time :uint; correctness :PacketCorrectness; // physical fields %header :packet_header; %payload[size] :list of byte; %crc :uint; event ready; // emitted when generated event done; // emitted when depart the switch

};
struct packet_header {
... // header fields
};

Figure A - Illustrates an Ethernet Packet Definition

Notice that in addition to the fields that hold the actual contents of the packet (header, payload and crc), several virtual fields (e.g., kind, size, delay, etc.) were added to facilitate the generation, coverage and checking of the packet. The packet description above is all you need to specify to have Specman Elite generate any number of Ethernet packets.

extend packet { // extend packet's basic definition⁽²⁾

	cover ready is { // coverage group
		item kind; // coverage item item delay; item correctness; item dut_state: uint(bits:2) = 'top.dut.state'; item size using ranges = {
			range([64..500], "small"); range([501..1000], "medium"); range([1001..1518], "large");
		};
	}; cover done is {
		item latency :uint = sys.time - start_time using ranges = {
			range([0..4], "short"); range([5..9], "medium"); range([10..14], "long");
		};
	};

};

Figure C - shows the definition of functional coverage for the packet.

Figure D -- Coverage Report Example	The coverage is collected in groups. Each coverage group has a trigger event which specifies when to sample, and includes a list of items that specifies what to sample. During the simulation, the coverage information is accumulated. Figure D demonstrates a coverage report collected from the code example above. In this snapshot, we see the values sampled for the kind field, indicating the kind of packets generated by Specman Elite. Another major kind of query is transition coverage. Have we ever had a CRC error right after size error and vice versa?
Figure E -- Transaction Coverage	extend packet {   cover ready is also {     transition correctness;   };   };     A simple addition as illustrated in figure E can be layered on top of the existing environment. The results are illustrated as well. The report indicates (note the grade of 0) that we have never tried an erroneous size packet after a good one, nor consecutive CRC errors and size errors.
Figure F -- Cross Coverage	extend packet {   cover ready is also {     cross kind, correctness;   };   };     Yet another important query kind is a combinational query. Although we have full coverage of packet kinds as well as a full coverage of error kinds, have we covered all the combinations of the two? The report suggests that we have never sent as an input E802.3 packet that is a good packet, or the same packet kind with erroneous size.

How about covering time related information, e.g., scenarios over time? This is facilitated by Specman Elite's temporal language. For example, have I exercised all possible packet types while handling an interrupt? What was the DUT status at the time of the interrupt?

extend packet {

	event intr_during_exec is
		{@send; [..] * not @done; @interrupt} @clk;
	cover intr_during_exec is {
		item kind; item correctness; item dut_fifo_size: uint = 'top.my_switch.fifo.size'; item intr_kind;
		cross kind, intr_kind;
	};

};

The event intr_during_exec is capturing a simple scenario, consisting of three stages: the testbench started sending a packet (@send event), a few clock cycles have passed but the packet was still ongoing([..] * not @done), and then an interrupt occurred (@interrupt).

Once this scenario occurs, we will capture all the interesting information as coverage items. Figure G demonstrates the cross-coverage report of packet kind and interrupt kind.

Figure G -- The figure illustrates several coverage holes, for example we never had an external interrupt during an EVII packet processing.

Coverage Requirements Review
So how does Specman Elite's functional coverage fulfill the requirements we have listed earlier?

Expressiveness of Queries
The coverage directives syntax has evolved through the years to express the most complicated queries, and at the same time be user friendly and intuitive. Effective coverage of the inputs may take just a few minutes to create. In addition, Specman Elite's access to 100% of the HDL signals allows functional coverage of events and signals in the design itself. The tight collaboration of Specman Elite's engines provides unique expressiveness. Temporal expressions allow capturing complicated simulation scenarios, which spans more than one cycle, and using them as a trigger for coverage sampling. For example, how can I define a query that ensures that an interrupt was received while processing each and every packet kind? Defining an event that captures an interrupt-during-packet scenario, and using it as a trigger for a coverage group sampling will achieve that, as well as multi-cycle scenarios.

Specman Elite's GUI allows easy querying, sorting and filtering. The coverage reports show a total test grade, a coverage group's grade, an items grade and a range grade. The grade functions are automatic, but highly configurable by modifying the coverage goals or items' weights. Graphic reports using bar charts make the report readable both to the novice engineer and management, which is interested in the overall picture rather than low-level details.

Beneficial Coverage Feedback
The translation from coverage reports to generation directives (constraints) to the test generator in Specman Elite, is usually straightforward³. Such simple translation is obtained by the fact that the coverage reports refer to the same fields that specify the data structures to be generated. For example, the delay, size and kind items are struct-members of packet. Identifying a hole in the coverage results can be naturally translated into additional constraints on those fields.

Grading
Specman Elite's grading mechanism is easy to use and at the same time highly configurable. Coverage goals and weight default can be reset to adjust to any need. A few suggested formulas to grade a test are suggested. The formula definition is also configurable. User perception of coverage, and the specific needs of the verification environment, may incline you toward a specific formula.

Testbase Ranking
Advanced test ranking capabilities allow effective highly configurable test base analysis. You can easily combine additional tests, classify tests families, and review various aspects of your test suite.

Timing of Analysis
Specman Elite's GUI allows post-simulation coverage review. In addition, the Coverage Application Programming Interface (API) allows run-time access to the accumulated coverage from this test as well as from previous executions.

Optimizing the testsuite
Specman Elite allows you to create an optimal list of tests on the basis of efficiency (amount of coverage per cost) or just grade (amount of coverage).

Open Environment
Specman Elite is an open environment. Coverage groups can be comprised of information from various tools and languages that make up the verification environment. A verification environment could be comprised of Verilog and VHDL modules, C++ reference module and a verification language such as e. Specman Elite is able to treat such heterogeneous environments as one, allowing sampling and analyzing information from all these sources all together.

Methodology Requirements
Tight integration between Specman Elite's engines allows further utilization of each engine's capabilities.

Self Checking
In addition to data checking⁴, Specman Elite allows temporal checking or protocol checking. In specific it allows:

• Natural translation from the design specification to assertions, which will be validated throughout simulation.
• Immediate simulation stop. When a temporal rule is violated, the simulation stops immediately and the message associated with that rule is printed. This preserves the design's state for further inspection.
• Informative error messages that make it easy to understand the cause of the buggy scenario.

Generation Control
Test generation is one of the most important parts of simulation as it is the only way to improve coverage results. Specman Elite's unique language expressiveness and constraint solver allow powerful test generation to automate the process of generating tests for your entire test plan.

Repeatability
Specman Elite addresses the repeatability issue by introducing the concept of "random stability". This feature ensures tests will maintain the same pseudo-random values over multiple runs for a given seed, even if some modifications are done to the test, the testbench or thread ordering. This is a built-in capability, that requires no special action or attention from the user. This ensures that functional coverage will not be "lost" from a test suite due to changes in the design or in the testbench.

Recommended Approach to Coverage

Phase One: Test Plan
A good test plan should list all the interesting test cases to verify the design. In specific it should include all configuration attributes, all variations of every data item, interesting sequences for every DUT input port, all corner cases to be tested, all error conditions to be created and all erroneous inputs to be injected.

An encompassing test plan is a good start to ensure complete verification. Experience and creativity can be used to identify areas that are prone to bugs. It is advisable to consult people with extensive verification background and the designer to get internal insights into the design.

Phase Two: Functional Coverage Specification
Define what should be covered. Decide on the interesting data fields/registers. Define separate buckets for legal values, illegal values, and boundary values, i.e. corner cases. Examine both interfaces and internal states. Choose the state registers and state transitions of important state machines. Identify interesting interactions between some of the above states or data, e.g., the state of one state machine relative to another, or to a value of a signal.

Phase Three: Build the Testbench
Build your environment parameterized in a way that each test can direct it to a specific area of concern. This enables you to use the coverage results and translate coverage holes into new tests. At this point your e functional coverage code should be written. Make sure that the verification strategy you have chosen is suitable for the entire verification needs.

Phase Four: Writing Tests and Simulation
Write tests and run them. Try to enhance your test suite by using the iterative process of analyzing coverage reports and adding additional tests to fill the uncovered areas. From time to time, update and optimize your regression suite using the ranking capabilities. There is no need to frequently run tests that have only marginal contribution to the verification process.

Phase Five: Code Coverage Integration
Once your RTL code is mature enough, use code coverage. Start with block coverage. Unreachable code should be carefully analyzed; it may save time to ask the implementer to identify the code's functionality. Dead code should be removed. In cases of reachable non-exercised logic, identify the untested scenario and write tests or constrain the test generator to fill coverage holes. At the same time identify the untested functionality. Once identified, review your test plan and make sure it is not an overlooked area in the test plan. Update the test plan and functional coverage code as necessary.

Phase Six: Thorough Coverage
At this point you have hopefully achieved one hundred percent block coverage. Use other coverage metrics (expression, event, state machine, arc and toggle coverage) to achieve as high coverage as possible within your allotted time.

Following these guidelines provides a more complete metric and methodology that can be examined through the various phases, while steering the verification process towards a rapid completion.

Conclusion

Consistent coverage review makes the verification process well proportionate and more predictable. Engineers can reset their simulation goals and management can monitor the verification progress to avoid last minute surprises and identification of more bugs. Future technologies may further automate the verification process by intelligently enhancing the testsuite based on coverage results.

Functional coverage and code coverage are the only objective measurements available today. Verisity's SureCov code coverage tool and Specman Elite's functional coverage engine are pioneers in this recent evolving technology.

¹ Extra care should be taken when making use of the run-time feedback capabilities. An appropriate mixed measure of repetition and uniqueness prove to be more efficient in discovering bugs. Mostly, analyzing accumulated results of few simulations and intelligently determining the next step preserves realistic simulation, as well as ensures completenses.

² The extend feature in e allows the engineer to expand a single type definition in several files. It is commonly used in verification environments due to the partitioning into basic testbench architecture and tests layered on top of it. This layered approach provides flexibility and enhances the verification reuse. Any tests can enhance the type definition to address different aspects of the design.

³ Some of the corner cases, especially Dut state coverage, are harder to archieve. The DUT complexity may involve complicated inspection of the design behavior. It may be easier to raise the probability that a state will be visited rather than providing a deterministic input scenario using a testbench automation tool.

⁴ data checking is based on comparing the output with the input. Data checking is usually done using scoreboarding technique or reference models (see Verification Advisor).

Bibliography

1. Bergeron, Janick. "Writing Testbenches, Fuctional Verification of HDL Models." Kluwer Academic Publishers, 2000.

2. Geist, D.; Brian, G.; Arons, T.; Slavkin, M.; Nustov, Y.; Farkas, M.; Holtz, K.; Long, A.; King, D.; Barret., S. "A Methodology for the Verification of a eSystem on Chip," Design Automation Conference, New Orleans, LA, June 1999.

3. Hanoch, C. "High-Level Verification Automation: A New Mehodology for Functional Verification of Systems/ASICs". DesignCon, Santa Clara, CA February 1998.

4. "Spec-Based Verification," http://www.verisity.com/resources/whitepaper/specbase.html

5. "Functional Verification Automation for IP, Bridging the Gap Between IP Developers and IP Integrators," http://oahu.verisity.com/resources/whitepaper/ipwhitepaper.html

Verisity Design, Inc.  •  2041 Landings Drive  •  Mountain View, CA  94043  •  phone: (650) 934-6800  •  Fax: (650) 934-6801  •  www.verisity.com/

Specman Elite,  SureCov, Verisity, and the Verisity logo are trademarks of Verisity Design, Inc. All other trademarks are the exclusive property of their respective holders. ©2001 Verisity Design, Inc.