Spec-Based Verification A New Methodology for Functional Verification of Systems/ASICs

Abstract

This paper presents a new methodology for functionally verifying systems and ASICs—spec-based verification—an automated and measurable approach to verification that enables more effective verification methodologies while cutting the overall resource investment in half.

Introduction

With design complexities rapidly advancing toward one million gates, functional verification has become the main bottleneck of the design process. Faster simulators are only part of the solution. Enabling the execution of more cycles provides some benefit, but often many of these cycles are wasted because they add no additional design coverage.

The first step toward improving the efficiency of functional verification for large, complex designs is to raise the level of abstraction of the verification environment to the specification level. The common specifications driving the verification process are the design spec, the interface spec and the functional test plan. The verification team can develop and implement a comprehensive verification strategy only when the rules defined in these specifications can be captured in an executable form. However, raising the level of abstraction is insufficient, in and of itself. Once these rules have been captured, the task of generating tests and checking results must be automated for the team to have any hope of attaining full functional coverage within reasonable time and resource budgets.

Spec-based verification addresses the functional verification bottleneck in a manner similar to the way the introduction of logic synthesis tools addressed the design challenges posed by increasing design complexity. First came the hardware description languages (HDLs) such as Verilog and VHDL, which raised the level of abstraction from gate-level designs to register-transfer-level (RTL) designs, making large designs much more manageable. However, not until the introduction of logic synthesis automated the translation of RTL designs to gates did the real breakthrough occur in time and resource reduction.

Similarly, the real value of spec-based verification is created by a composite of features that deliver automation—a spec-based verification environment, automatic generation of high-quality tests, data and temporal checkers, and accurate measurement and analysis of functional coverage. With an automated functional verification approach at the center of the verification methodology, the designer gains the three essential elements needed to overcome the verification bottleneck: consistency, efficiency, and quality. Verification methodologies that incorporate spec-based verification produce reusable verification environments, shorten the verification cycle, and reduce the risk of costly silicon re-spins.

The Verification Problem

Unfortunately, neither the schedule nor the available engineering resources offer much in the way of flexibility. More importantly, the cost of missing a time-to-market schedule can force design teams to prematurely terminate the verification effort. This leads to incomplete and inadequate functional coverage, creating the potential for a pattern of debug cycles, redesigns and re-spins that erode the profitability and deliverability of the end-product. Thus, ASIC design quality becomes a function of the verification schedule, rather than the verification metrics.

Today's Verification Methodologies: Components

Beginning with an initial design specification, the design team partitions the design into functional blocks, which are then assigned to specific design team members. Then the interface specs and functional test plan are written. The functional test plan describes what functionality should be tested and how, including normal operating behavior as well as important corner cases to be tested.

The next step in the verification strategy is to assemble the test environment, comprising pieces of code (often written in "C" ) called "stubs", which model the components surrounding the device under test (DUT) in the real system. These stubs interact with the device, injecting stimuli into the device and receiving its outputs according to the protocol defined in the interface specification. Sometimes, the test environment also contains monitors used to check the correctness of the functionality.

Depending on the types and quantity of tests required, the designer may choose to write the tests manually, or create a tool that generates tests according to specific directives or parameters. When using an automated test generation strategy, the designer must decide whether to generate the complete test before simulation, or to generate the test on-the-fly as the simulation progresses, reacting to the state of the device.

The designer has numerous checking strategies at his or her disposal for checking the results of a test. Initially, the designer must decide whether to use white, black, or gray-box testing. White-box testing, most commonly used for module and algorithm testing, enables the verification engineer to both drive and sense internal signals. Gray-box testing allows only sensing of internal signals, and is the method most often used with high-complexity designs. Black-box testing offers no access to internal signals at all.

Similar to the test generation strategy, the designer must decide whether to have the tests checked manually or automatically, on-the-fly or after the simulation is done.

In addition, one of the most difficult and critical challenges facing the designer is to establish adequate metrics to track the progress of the verification effort and measure the coverage of the functional test plan. Effective coverage metrics are essential to avoid redundant or unnecessary testing, as well as to determine when the verification is complete.

Today's Verification Methodologies: Problems

Another problem related to the size issue is the difficulty experienced by engineers attempting to track assumptions made by fellow engineers working on different pieces of the design. Lacking a centrally accessible and unambiguous means of communicating and tracking design intent at the specification level, the complexity of today's designs can often lead to architectural-level bugs that require enormous effort to locate and remove.

Locating bugs is always a problem, especially when they occur in unpredictable places. Even the most comprehensive functional test plan can completely miss bugs generated by obscure functional combinations or ambiguous spec interpretations. This is why so many bugs are found in emulation, or after first silicon is produced. Without the ability to make the spec itself executable, there's really no way to ensure comprehensive functional coverage for the entire design intent.

The relative inefficiency with which today's verification environments accommodate midstream spec changes also poses a serious problem. Since most verification environments are an ad hoc collection of HDL code, C code, a variety of legacy software, and newly acquired point tools, a single change in the design can force a ripple of required changes throughout the environment, eating up time and adding substantial risk.

Perhaps the most frustrating problem facing design and verification engineers is the lack of effective metrics to measure the progress of verification. Indirect metrics, such as toggle testing or code coverage, indicate if all flip-flops toggled or all lines of code were executed, but they do not give any indication of what functionality was verified. For example, they do not indicate if a processor executed all possible combinations of consecutive instructions. There is simply no correspondence between any of these metrics and coverage of the functional test plan. As a result, the verification engineer is never really sure whether a sufficient amount of verification has been performed.

Test Methodologies: Deterministic

Although deterministic testing offers the verification engineer precise control, providing accessibility to hard-to-reach corner cases, it has several drawbacks. Generating deterministic tests is a time-consuming, manual programming effort. Although simple tests can be written in minutes, the more complex ones can take days to write and debug. For example, to test a corner case requiring that two asynchronous data streams reach a specific point at exactly the same time, the verification engineer might have to resort to trial-and-error methods, running the test, seeing how far off it is, correcting it, and trying again. Moreover, midstream changes to the design's temporal behavior can cause the engineer to go through this process repeatedly. And when this test is completed, the corner case is tested through only one possible path.

An average project normally develops many hundreds of deterministic tests, which can easily consume several man-months to create. Checking deterministic tests also consumes considerable time and resources, whether it is performed manually or written into the test.

Test Methodologies: Pre-run generation

The generator normally outputs the test into a file, which is then read by the simulator and stored in memory. The simulator reads the next entry whenever it's prepared to inject the next set of inputs. Although pre-run generation provides much higher throughput than deterministic testing, it is difficult to control. The parameters are static and do not provide much flexibility. Also, most generators of this type don't allow for interdependencies between data streams ñ each data stream is generated independently. This can cause the generator to generate unlikely or even illegal tests.

Reaching corner cases using pre-run generation is nearly impossible. The engineer has very little control over the sequences generated. This makes it difficult to force the occurrence of specific combinations. As such, pre-run generation makes a suitable complement to deterministic testing, but cannot replace it.

Another problem with pre-run generation is that it is hard to maintain. As the verification process progresses, new parameters are often needed. This normally requires modifying the program, sometimes affecting delicate interdependencies between different parts of the generator.

Maintenance problems can also occur when updating the program after a bug is found in the RTL design. To temporarily avoid generating the "buggy" test sequence again, the engineer must patch the code until the bug is fixed in the RTL design. When the bug is fixed, the code must be "unpatched". Several such patches often coexist in the code. The patching/unpatching process sometimes introduces bugs into the generator, which may not be noticed until several hours or days of simulation have transpired.

The cost of developing and maintaining such a generator requires a minimum of several man-months per project, and increases significantly as the generation becomes more complex.

A side-effect of this methodology is that the full test is usually very large, since it is generated in advance. It is commonly loaded into a simulation memory at the beginning of the test and run from there. This significantly increases the memory requirements for simulation, often causing the simulator to swap memory. This can slow the simulation down by orders of magnitude.

Test Methodologies: Checking Strategies

Reference models are most common for processor-like designs where the correct result can be predicted relatively easily. Designers usually develop the reference model in C or C++ . Stimuli are injected into the reference model as well as the device, and their outputs are compared. In gray- and white-box methodologies, the comparisons also include the state of internal registers and nodes.

Rule-based approaches are more common in communication devices for networking applications, where there can be several legal outputs for the same input, or where it is not easy to predict the correct result. In this case, the engineer often uses specialized techniques to check data integrity and protocols, such as scoreboarding, which tracks information about cells or packets without worrying about the order in which they appear on the output ports.

Engineers perform these checks either on-the-fly or post-run. Simple checks and protocol checks can be performed on-the-fly by the stubs and monitors using an HDL. Post-run checks are often performed using a C/C++ or PERL/AWK program. The outputs of the test are either saved in a simulator memory and then dumped into a file, or written into the file directly. The program reads the inputs and outputs and checks the correctness of the results. Often, these methodologies still require some amount of manual checking, usually achieved by viewing actual waveforms or data dumps.

The problem with these checking strategies stems from the way they are most commonly implemented today. Post-run checking wastes cycles. If a test runs for 500,000 cycles, but a bug occurred after cycle 2,000, then 498,000 cycles were wasted. In addition, since the post-run checking cannot detect a problem real-time, the designer does not have access to the values of the registers and memories of the device at the time the problem happened. In general, debugging these problems requires rerunning the simulation to the appropriate point.

On-the-fly checking is more powerful. However, on-the-fly checks are most often implemented in Verilog or VHDL. These languages do not have a powerful temporal language to simplify protocol checks. They are low level and lack features like dynamic memory, which simplifies the process of writing the stubs/monitors and increases performance.

In addition, reference model checking is often hard to implement on-the-fly, since intermediate results are not always available. On-the-fly reference models also require a direct interface to the simulator (through PLI or FLI) which is not easy to write and maintain.

Test Methodologies: Coverage Metrics

• Toggle testing verifies that over a series of tests, all nodes toggled at least once from 1 to 0 and back;
• Code coverage demonstrates that, over a series of tests, all the source lines were exercised. In many cases there is also an indication as to whether branches in conditional code were executed. Sometimes an indication of state-machine transitions is also available;
• Possibly the most common metric used to measure progress is to track how many bugs are found each week. After a period of a few weeks with very low or zero bugs found, the designer assumes that the verification process has reached a point of diminishing returns.

Unfortunately, none of the metrics described above has any direct relation to the functionality of the device, nor is there any correlation to common user applications. For example, neither toggle testing nor code coverage can indicate if all the types of cells in a communication chip (with and without CRC errors) have entered on all ports. Neither can these metrics determine if all possible sequences of three instructions in a row were tested in a processor.

As a result, coverage is still measured mainly by the gut feeling of the verification manager, and eventually the decision to tape out is made by management without the support of concrete qualitative data.

Not knowing the real state of the verification progress causes verification engineers to perform many more simulations than necessary, trading off CPU cycles for "confidence". This usually results in redundant tests that provide no additional coverage or assurance that the verification is complete. The real risk is that the design will be sent to production with bugs in it, resulting in another round of silicon. The cost of re-spinning silicon includes non-recoverable engineering (NRE) costs to do the additional production process, the cost of extending the team's work on the project, and the major cost of reaching the market a few weeks late. For most designs this amounts to many millions of dollars.

Spec-Based Verification

• Automates the verification process, reducing by as much as four times the amount of manual work needed to develop the verification environment and tests;
• Increases product quality by focusing the verification effort to areas of new functional coverage and by enabling the discovery of bugs not anticipated in the functional test plan;
• Provides functional coverage analysis capabilities to help measure the progress and completeness of the verification effort;
• Raises the level of abstraction used to describe the environment and tests from the RTL level to the specification level, capturing the rules defined in the specs in a declarative form and automatically ensuring conformance to these rules.

Verisity's Specman Elite encompasses the spec-based verification flow.

Enabling Technologies

Functional Coverage
Functional coverage analysis is a key enabling technology for spec-based verification methodologies. This technology allows the verification engineer to define exactly what functionality of the device should be monitored and reported. To accomplish this, the functional test plan is translated into executable directives fed directly to the functional coverage analyzer. This makes the entire functional test plan executable, including complex multicycle scenarios, state machine transitions and temporal sequences.

Cross coverage window showing the registers accessed for each CPU opcode.

The coverage information can also be crossed, providing information on interrelated functionality. This "querying" ability answers questions about the simultaneous occurrence of certain functions. For example, in a processor design, the verification engineer can cross information about instructions, addressing modes and machine state to see exactly how many times each instruction was executed using the different addressing modes for every machine state.

Using functional coverage analysis, the engineer finally has a clear indication which functionality has been exercised, and more importantly, which functionality has not. This focuses the verification effort and eliminates superfluous or redundant tests, ensuring that every test adds coverage. In so doing, the availability of functional coverage analysis has opened the door to major advancements in verification methodologies.

The ability to generate clear reports showing which parts of the functional test plan were tested has enabled a new approach to verification. Beginning with a base of automated tests, coverage reports are generated to indicate which parts of the functional test plan were exercised. This allows the engineer to close quickly on significant functional coverage of the device with minimal effort. Afterward, the only additional effort required is to generate tests that are focused on the functionality not yet covered. The coverage reports after each set of tests direct the verification engineer to the types of tests to focus on next. Deterministic tests are needed only for the corner cases that were not reached over the course of automated testing. This minimizes the time spent writing deterministic tests.

Functional coverage reports are also an essential tool for verification managers. They help monitor the progress of the verification, providing a clear indication of the state of the verification, and ultimately, the moment the device is ready to be fabricated.

Functional coverage reports also enable the verification engineer to verify the effectiveness of the tests. Once a certain functionality has been tested enough, according the metrics set by the engineer, other functionality can be targeted and tests that add no new coverage can be discarded. There is no need to continue testing "just in case."

The impact to the verification schedule when using this coverage approach is significant. It eliminates much of the manual-intensive work of developing the deterministic tests, saving many man-months of frustrating work.

The risk of sending the device to production while part of its functionality has not been properly tested, causing silicon re-spins, is also significantly reduced. This can result in savings of millions of dollars.

Constraint-Driven Generation
The one feature spec-based verification offers that most significantly contributes to reducing the verification schedule is automated test generation. Armed with a constraint-driven test generator, a spec-based verification methodology can slash weeks, or even months off of the verification schedule, while placing enormous power in the hands of the engineer.

Inputs to the constraint solver—the core technology enabling constraint-driven generation.

Constraint-driven generators offer some key advantages over the more common, parameterized generators. Constraint-driven generators give the verification engineer full control over the generation process. By using simple constraints, they can generate tests that are completely random, completely deterministic or anywhere in between. These constraints can be either spec constraints, used to define the legal parameters that the generator must always hold to, or test constraints which target the generator to a specific test from the functional test plan. Instead of randomizing only what the user specifically instructs the system to randomize, constraint-driven generators take the infinity-minus approach. They randomize everything except what the engineer specifically constrains the generator not to randomize. This allows the generator to approach any given test scenario from multiple paths and find bugs that were not identified in the functional test plan.

This capability to capture the "same" test scenario from different paths is critical. Most post-silicon functional bugs are not due to an inadequacy in the functional test plan, but due to unanticipated usage by the end user or by ambiguities in the interface specifications. It is impossible to "think" of all the possible bugs when writing the functional test plan and so it is critical that all tests be run from multiple, different paths.

Besides offering support for the common pre-run generation methodology, constraint-driven generators can also support the more advanced "on-the-fly" generation methodology. Using the on-the-fly approach, the generator interacts with the DUT, and reacts to the state of the device in realtime. In this manner, even hard-to-reach corner cases can be captured because the generation constraints are state dependent.

On-the-fly generation also eliminates the need for big memories to hold the entire test. Instead of generating the test all at one time, the inputs are generated as required. This reduces the required memory and swapping by the simulator, significantly increasing simulation performance.

Constraint-driven generators are also generic, or design-independent. They receive a description of the data elements to be generated, i.e., instructions, packets or polygons, and generate them accordingly. If the designer introduces changes to the architecture, the generator easily adapts, making the tests easy to reuse.

One of the main challenges facing a verification engineer is to test corner cases in the functionality of the device. These corner cases are usually reached after a long sequence of inputs, and are often the result of several independent streams of input reaching a special combination at the same time. The constraint-driven generator's ability to generate massive amounts of tests on the fly allows the generator to constantly monitor the state of the device and generate the required inputs at exactly the right time in order to reach the desired corner case. The result is high throughput of effective tests which achieve high functional coverage. Thus, when used in combination with functional coverage analysis, constraint-driven generation provides easy confirmation that a corner case was reached, and under what conditions.

A powerful feature often missing in home-grown pre-run generators is the interface to the simulator. In the generic constraint-driven generators, this simulator interface does not have to be specified for every project, but is included as part of the generic generator. Manually created generators require that a custom interface be created that is specific not only to the simulator, but also to a particular version of the simulator, creating a maintenance nightmare.

All of these capabilities supplied by on-the-fly, constraint-driven generators result in a major reduction in the verification cycle while significantly increasing the verification quality.

On-the-fly Temporal Checks
Today's highly integrated designs contain many components. Often, these components are developed by different engineers, resulting in different interpretations of the protocols between components. This can cause the component interfaces to become the weakest link in the design. On-the-fly temporal checks verify the temporal behavior and protocols associated with the design or system. They constantly monitor the design, sensing triggers (or events) which signal the beginning of a sequence. They then follow the sequence, verifying that it conforms to the temporal rules specified by the verification engineer.

There are several types of rules. In some cases, the protocol can be specified explicitly, defining exactly on which cycle each part of the protocol should be performed. In other cases, the exact cycle of a response can't be defined—it is known only that it must happen eventually, or that it will occur with a certain interval of time. All of these checks—explicit, eventual and interval—must be easy to specify and must run on-the-fly in order to allow for protocol verification and efficient debugging of interface problems.

Although temporal checks can be done using the current HDLs, their implementation is not always simple. New temporal constructs allow defining these rules in a simple and straightforward manner and are key to the spec-based verification methodology.

On-the-fly checking, particularly in conjunction with on-the-fly generation, is a memory-efficient approach to verification. Since the checks are performed on-the-fly, only relevant information is stored, and, as soon as it is checked, it can be removed. On-the-fly checking also enhances the debug capability. When a check fails, it is possible to access the full state of the device at that point and generate a detailed report. Moreover, the test can then be aborted, since the following cycles will not be useful and merely waste simulation cycles.

High-level temporal constructs also enable the engineer to define the rules independently. This allows the engineer to reuse checks developed for a lower-level block when verifying higher levels of integration that incorporate them.

In summary, on-the-fly temporal checks provide an effective means to capture the interface specification and verify the protocol of these interfaces, efficiently debug them, and eliminate the ineffective simulation cycles that follow the bugs occurrence. The powerful temporal constructs used by these checkers can also minimize the size of complex checkers, reducing by a factor of four the time it takes to write these checkers.

The Specman Methodology

Spec-based verification flow with Specman.

The Specman Elite verification flow is very similar to the traditional flow, with the addition of the enabling technologies described above. As always, the input to the functional verification process is the design spec, which defines the device's architecture, functionality and protocols, and the interface spec, which defines the target system and the protocols between the device under test (DUT) and the rest of the system. These specs are used to develop the functional test plan, which defines the verification strategy, the test environment and the functionality and user scenarios that are to be tested. These tests should include typical tests, stress tests, error tests and corner case tests.

Once the test plan is defined, the test environment is developed. Existing code from previous projects written in Verilog, VHDL or C can be used and linked into the system. The test environment includes structs, which define the data to be generated, and spec constraints (from the interface specification - see figure 1). Spec constraints constrain the generator to generate only legal values. The verification environment also includes the data and temporal checks (see figure 2) which define how to check protocols and monitor the simulations for correct behavior.

At this point, a few deterministic tests can be written and run, to verify that the device is getting out of reset and performing the basic operations correctly. Once this is done, instead of writing many deterministic tests and checking them, constraint-driven tests can now be generated. Specman Elite can generate many tests, drive them into the device and check that the device responded according to the spec. But more important, Specman Elite collects the functional coverage information, showing what functionality was actually excersized. With this information, it is now easy to know what functionality has not been covered and write more specific test constraints (see figure 3) which target the generator to focus on the untested functionality. This process ensures an efficient verification cycle, guaranteeing that each test adds new coverage.

Conclusion

Spec-based verification provides an effective composite of enabling technologies that automate resource-intensive manual processes and establish qualitative metrics to ensure sufficient functional coverage. Using these technologies increases the quality of the design while reducing the resources needed to implement a powerful verification methodology. Verification engineers report experiencing a two- to four-fold reduction in the verification schedule, while achieving significantly higher verification quality.

Verisity Design, Inc.  •  2041 Landings Drive  •  Mountain View, CA  94043  •  phone: (650) 934-6800  •  Fax: (650) 934-6801  •  www.verisity.com/

Specman Elite,  SureCov, Verisity, and the Verisity logo are trademarks of Verisity Design, Inc. All other trademarks are the exclusive property of their respective holders. ©1999 Verisity Design, Inc.