Verification Automation Management

1. vManager in context

• create and track extensive verification plans,
• architect and build multiple complex test benches,
• compile and execute thousands of simulation runs,
• collate and analyze large lists of failures,
• identify and analyze diverse gaps in verification progress (coverage holes) and
• integrate and summarize large chunks of verification progress (coverage) data.

Against this backdrop of unprecedented effort, the strain on current verification processes is evident. Progress and quality metrics are unreliable and schedule slips and bug escapes are the norm.

vManager in summary
vManager leverages automation and reuse to enable verification teams to deploy a scalable, metric-driven process that dramatically improves both schedule predictability and device quality. It touches all the steps of the process, from upfront planning to tapeout decision, and brings value to all members of the verification team, from engineering to management.

Commonly realized benefits when applying vManager in production environments:

• Over 5x improvement in predictability of schedule from planning to verification closure
• Up to 3x increase in project-end verification closure rate by disqualifying redundant simulations
• Over 10x decrease in effort to regularly generate and distribute status reports
• More than 2x reduction in respins through introduction of metric-based planning and tracking
• Over 10x increase in management effectiveness using metrics that cannot be manipulated

At the engineering level, vManager enables verifiers to focus their skills on the most important problems by automating many of the mechanical tasks in the verification process. vManager also offers engineers new analysis tools, including failure analysis, coverage analysis, and planning analysis engines that accelerate functional verification closure.

At the management level, vManager offers project leadership multiple project-specific perspectives on verification plan progress that facilitate better and more timely resource allocation. vManager also offers project leadership a number of reporting capabilities to facilitate communication of verification status to all the stakeholders and decision makers.

This paper discusses vManager in detail and supplies the background and motivations for its feature set. Here are some highlights:

vManager and massive parallel verification
The verification engineer's productivity can be crushed under the sheer weight of the repetitive and mechanical work required just to administer thousands of simulation runs. vManager greatly reduces this effort by first integrating with your simulation runner through an interface that includes session input and output formats, a verification attribute definition interface, and the runner gasket itself. Then vManager enables the verification engineer to easily specify, execute, and track from hundreds to thousands of parallel simulation runs.

vManager and the annotated verification plan
In the typical verification process, the verification team creates a verification plan document that sets out coverage and checking goals for the verification cycle With vManager, the verification team can drive the verification process with the vPlan, a reusable, machine-readable verification plan. The vPlan is easily specified and maintained throughout the project, and identifies the checking and coverage goals at all levels of verification hierarchy, for each phase of the project from module verification to system integration, for each target customer, and more. As simulation runs are completed, their results are automatically annotated into vManager's vPlan view and made available for interactive analysis and reporting.

vManager and coverage analysis
The verification team must repeatedly analyze the available coverage information to guide the verification process to closure. vManager leverages Specman's coverage and assertion interface (CAI) to integrate and summarize multiple sources of coverage information to enhance Specman's native functional coverage measurement. All of these sources of coverage information are available to vManager's coverage analysis engines which facilitate rapid coverage hole filling and easy optimization of regression sessions.

vManager and failure analysis
When there are thousands of simulation runs and hundreds of failures, failure analysis can overwhelm the verification effort. vManager reduces the overall failure analysis effort and shortens failure debug time. It rapidly separates design failures from tool failures, sorting and grouping these failures for easy selection and action. It quickly identifies the least costly simulation to exhibit failure and the optimal case for repeated debugging.

2. The metric-driven verification process

Figure 1 shows a high-level, simplified view of this process. This process diagram shows deliverables along the horizontal task progress lines, with diamonds and vertical arrows showing the dependencies between the artifacts. Now let's look at this process in detail, summarizing not only what is accomplished during each task of the process, but also citing some of the burdens and roadblocks along the way.

Verification plan

Tasks
At the beginning of the verification cycle, the team develops a plan consisting of an initial set of requirements for the verification efforts that follow. Verification requirements detail the checking and coverage requirements for each of the subsystems of the design, and for the integrated system. The requirements also define other, related goals for checking and coverage, along dimensions such as project milestones and customer-related feature sets. The team captures all these requirements in a document that it uses to guide later efforts.

Burdens and roadblocks
One main problem the team faces is creating the connection between the text-based verification requirements, specified as coverage of varying kinds, and the actual data collected during the verification cycle. This is a problem during the coverage reporting tasks described below, but its source is here. The coverage data specified in the plan may come from many tools and sources, and could include functional coverage, code coverage, and assertion coverage, further complicating the efforts required to report the actuals with respect to the plan. Typically, verification teams must create custom scripts to obtain the coverage reports they need.

Another of the problems the team faces with planning is that as the verification cycle progresses, the requirements may change, especially as the verification team's understanding of the DUT improves. The plan documentation may be burdensome to maintain, and as a result, it stagnates and can no longer inform interpretation of coverage data, as the measurement models have evolved beyond what's documented. Further, as the plan changes, any custom scripts that collect, analyze, or report on the data will need to be updated to include the new data as it is collected.

Verification environment

Tasks
Next, the team architects and builds verification environments for various levels and configurations of the design hierarchy. The generation, checking, and coverage measurement capabilities of these environments needs to support fulfilling the verification requirements outlined earlier. These environments are comprised of the RTL models of the DUT, the HVL models of the DUT's operating environment, and the compile and execute infrastructure that integrates the models and the tools needed to execute them. Since the team is using eRM, their module-level environments are constructed from interface e verification components (eVCs), obtained either from previous projects or commercial offerings. Then, their system-level environments are constructed by composing the components of module environments.

Burdens and roadblocks
One problem with current practice is that the verification plan is an important part of the overall verification component IP, but it is not designed for reuse. There needs to be a way to encapsulate and reuse the plan data, i.e. a modular and composable verification plan component, that could be packaged with any verification component.

Coverage targeting

Tasks
Once the verification environments are in place, and based on the requirements outlined in the verification plan, the team creates and iterates a framework for running simulations that will achieve maximal coverage throughput (coverage per simulation cycle). This framework has two levels:

• Verification Environment-level (VE-level) controls
• Session-level controls

At the VE level, the framework consists of constraint-set files that divide the behavioral space of the verification environment into functional subspaces that will be easier to conquer. These constraint-set files typically take advantage of specific stimuli sequences out of the full set of stimuli sequence libraries that is defined in the verification environment (for information on stimuli sequences, please see the documentation of Verisity eRM mentioned above). At the session level, the framework includes data about random seeds, number of runs to execute, length of each run, and more.

Burdens and roadblocks
The iteration of coverage targeting takes a substantial portion of the overall verification effort, and there are many difficulties associated with its administration. Chief among these is a lack of standard ways to express the session-level and run-level control inputs. Further, the ways that do exist are often not modular and efficient, and as new environments, tools, or DUT representations are added, there are heavy maintenance burdens.

Simulation

Tasks
The sets of runs that were specified during the coverage targeting phase are executed on the compute grid, producing per-run failure and coverage data.

Burdens and roadblocks
The lack of standard ways to manage large numbers of simulations and their data at run time lead to ad hoc solutions that vary in their effectiveness and in their maintainability.

Coverage analysis and reporting

Tasks
The coverage results from the simulation runs are collated and analyzed to understand what has and has not been verified. The results of the analyses support refinement of verification targeting to aim at coverage goals not yet achieved, as well as the progress reports discussed below.

Burdens and roadblocks
Creating useful coverage reports that actually achieve their ends is a major time sink in the metric-driven verification process. Teams typically address reporting with home-grown scripts and tools that need frequent updating and maintenance. Moreover, without specific tool assistance, understanding and prioritizing the coverage that should be targeted next is a labor-intensive analytical exercise. One especially difficult problem is understanding the commonalities in holes in large cross coverage matrices. These matrices identify new coverage requirements as the cross-product of previously identified coverage items, and can become large very quickly, numbering into the thousands of buckets. As coverage data is reviewed, the holes in these crosses can appear, without analytical assistance, to be atomized and unrelated, making it difficult to efficiently target the holes in subsequent sessions.

Failure analysis and reporting

Tasks
The results of the simulation runs are reviewed and analyzed to understand the failures that have occurred, and to decide which ones to concentrate on debugging. Some of these failures may be related to the verification execution itself, such as compile errors, and should be referred to the tool czars. Others may be known bugs and should be set aside.

Burdens and roadblocks
Sifting through all the failure data from a session consisting of a hundred or more simulations is a time-consuming task, but proceeding to debug without thorough analysis of the big picture leads to inefficiencies as a non-optimum set of failures is debugged. Verification teams spend more time and create non-optimal ad hoc scripts to help sort and collate to get around this problem, creating additional maintenance burden.

Bug reports and tracking

Tasks
Verification and design engineers work together on the verification failures that have been selected for debug, tracking down the source of the problem. Once diagnosed, the failures are filed as bug reports and then sent to the design team for repair.

Burdens and roadblocks
One of the problems here is facilitating the easy recreation of all the verification runs that produced the failure being debugged. Once a failure has been selected for debug, all of the inputs necessary to recreate it must be captured from the runs' various simulation input controls and sent to the engineer responsible for debug.

Verification Progress reports

Tasks
Periodically, the verification team will generate reports about progress towards the coverage and checking goals outlined in the verification requirements. Project leadership reviews these reports to help estimate the time remaining to verification closure, and to support resource allocation decisions.

Burdens and roadblocks
Typically, the production of verification progress reports based on the coverage data being collected during simulation is another labor-intensive activity. There is a large effort involved in massaging the data to connect it to the various dimensions of project goals and requirements. For instance, the same data set may need to be viewed and normalized to multiple sets of goals reflecting different phases of the project or different customers for the DUT.

3. vManager in the metric-driven verification process

Verification plan with vManager
At the beginning of the process, the verification team using vManager captures their initial verification requirements using vManager's vPlan human-readable and executable format. vManager can read the vPlan, automatically back-annotate its sections with the latest coverage data from the simulation runs, and display the results in an interactive, hierarchical format.

By capturing the requirements in the vPlan format, the team overcomes many of the problems outlined in the original metric-driven process. The vPlan leverages Specman's CAI to bring together manifold sources of coverage, and adds easy specification of multiple management dimensions such as project milestones and customer feature sets. Since the vPlan is machine-readable, when the plan changes, there are no reporting scripts to be updated, as the reports are updated by construction (see the vPlan chapter below for more details).

Verification environment with vManager
Because the vPlan format is modular and composable, the team can build a project-level vPlan by assembling vPlans from lower-level modules, even reusing vPlans for module interfaces from previous projects. Therefore, the addition of the vPlan to the verification process adds another component that can be reused. As environment construction proceeds, vPlan components are packaged along with the other parts of the reusable eVCs (see the vPlan chapter below for more details).

Coverage targeting with vManager
Remember that, at a higher-level, coverage targeting inputs include specifications of sets of runs including data about random seeds, number of runs to execute, and the particular verification environments to compile and execute. vManager makes it easy to specify runs and to compose higher-order run specifications from lower-order specifications with standard formats and a standard simulation runner interface (see the massive parallel verification chapter below for more details).

Simulation with vManager
vManager makes it easy to track all the runs executing on the compute grid with its session tracking capabilities (see the massive parallel verification chapter below for more details).

Coverage reports with vManager
With vManager, the coverage results from the simulation runs are automatically collated and back-annotated onto the vPlan-based verification plan. Then, simply by opening a vPlan window, the verification engineer can instantly begin to understand what has and has not been tested. Moreover, easy navigation from the vPlan window to coverage hole analysis engines makes identifying holes and how to target them in the next verification runs direct and fast (see the vPlan chapter below for more details), thereby improving the efficiency of subsequent sessions.

Failure reports with vManager
With vManager, all of the results from each of the simulation runs in a session are collated and ready for analysis as soon as a session is loaded. vManager presents a spreadsheet of run data, where the rows are the runs themselves, and the columns show any user-selected run attributes, from run time to coverage score to, of course, failure messages. The verification engineer uses a complete set of analysis tools, including filtering, grouping, and sorting, to divide the runs into categories for handoff to debug (see the failure analysis chapter below for more details).

Bug reports with vManager
With vManager, all of the data necessary to recreate a failed run is available in the runs database, and vManager offers automatic creation of simulation inputs for one or more runs to speed the path to completing debug.

Progress reports with vManager
With vManager, progress reports are quick and easy. Since the vPlan is automatically back-annotated with progress data each time a session is loaded, sending a report to management is a matter of selecting a predefined vPlan perspective and a level of detail, and then clicking a button to create an HTML report. In addition, vManager can graph progress in a number of ways (see the vPlan chapter below for more details).

4. vManager and massive parallel verification

Massive parallel verification, i.e. running sessions consisting of thousands of directed-random simulation runs on a verification farm, has become a pre-requisite for effective ASIC verification. Thus, integrating with simulation runners is central to vManager. vManager provides a number of standard interfaces and file formats to make it easy to accommodate any kind of simulation runner. The screen shot below shows vManager's Session view controlling a vManager compliant simulation runner. The verification engineer has initiated two sessions, and one has completed with no failures.

Simulation runner integration
Figure 3 shows a typical vManager / simulation runner integration. The verification engineer has written a session input file specifying the simulation runs she would like to run. This session input file can specify multiple simulation runs for multiple combinations of , among others

• device under test (DUT),
• simulation and verification environment (SVE),
• coverage targeting constraint-set input (often called test files),
• random or specific seed.

The vManager reads in the session input file and passes this information to the simulation runner. The simulation runner has responsibility for initiating simulation runs and keeping track of their results by writing the session output file. There are hooks in vManager to facilitate automating session execution activities, including managing file relocation as the runs progress, as well as merging coverage data on the fly and compressing the results databases as needed. When the simulation runs have completed, vManager reads the session output file to access the results for analysis. Let's dive a little deeper to understand these components in more detail.

Session input file
The session input file specifies the simulation runs that the user wants initiated by the simulation runner. Figure 4 shows a simple example. Session input files use a standard format, called .esif. (As we'll see below, the .esif has a companion format, the .esof, used in the session output file) The format consists of simple containers that hold attribute-value pairs. vManager has the capability to read the .esif format and make the data available for the user's integration with any simulation runner (see below).

As you can see in the figure, one container is the session container. The session container holds attribute values that pertain to all the simulation runs the runner will initiate, such as the path to a directory for all session work.

Another container is the test container that specifies the one or more simulation runs. Test containers typically specify some files to load on top of the baseline verification executable, and the either the number of runs to executed with random seeds or the random seed to use for a single run.

The attribute-value pairs in containers include predefined attributes, which vManager understands out-of-the-box, as well as user-defined attributes.

vManager includes more than 30 predefined attributes that can be used as needed to communicate with any type of simulation runner. Some basic predefined attributes are shown below:

User-defined attribute interface
While vManager tries to accommodate any simulation runner with its predefined attributes, there can be a need for attributes not included with vManager. For instance, a particular simulation runner may need a watchdog timeout value for each run that it uses to stop the verification if a user-specified time has passed. vManager's attribute interface makes it easy to declare new attributes like timeout that vManager can then interpret when it reads the session input file.

User-defined attributes are captured in e and loaded on top of vManager. This can be as simple as instantiating a newly named attribute from the rich set of attribute classes provided with vManager. This attribute will then be automatically recognized by the .esif or .esof reader. Or if the application demands that the attribute receive special handling in the .esif or .esof reader, the integrator can have her attribute inherit from the one of the attribute classes and modify its behavior as needed. Making it easy to declare new attributes and read them from the session input and output files is one way vManager streamlines integration with other tools in the verification workflow.

The simulation runner
A simulation runner is a tool that, given a request of what to run, causes multiple simulation runs to happen. A simulation run is an activation of Specman Elite, a simulator, and any other runtime tools. Simulation runners are typically sets of shell or perl scripts that set up verification directories, build executables, and run simulation runs. They range in complexity from simple serial runners that execute simulation runs one after the other on a single machine to integrations with the compute grid's load balancing system that manage multiple simulation runs in parallel.

Communicating with the runner
Just what's going on along that path from the session input file through vManager to the simulation runner? As mentioned earlier, vManager includes a .esif reader that makes the session input file's contents available to the test runner integration code. Specifically, vManager includes an empty e method, start_session(), that the tool czar will use to implement the communication between vManager and the simulation runner. start_session() first reads the session input file using the built-in read_esif() method, and then is free to implement, in e, any type of communication with the runner.

For instance, many simulation runners that predate vManager will have their own text file input for specifying what to run. In this case, the start_session() method may simply translate the information from the session input file into the format required by the legacy runner, write this information to a file, and launch the runner. This would complete the front end of the simulation runner integration.

To accommodate the backend simulation runner integration, the tool czar would modify the runner itself to write the session output file. vManager includes a number of perl scripts to make it easy to accomplish this backend integration, including writing the session output file in its standard .esof format.

Session Output File
The simulation runner is responsible for writing entries to the session output file at the end of each run.
As you can see in Figure 5, session output files are similar to session input files, and are written in the human-readable standard format called .esof. Session output files have three containers, session, run, and failure. As in the session input file, the session container holds attribute-value pairs that pertain to all the simulation runs in the session. The session output file contains one run container for each verification initiated during the session. Run containers hold attributes that contain all the information needed to reproduce the run, that point at all the log and coverage results files from the run, and a failure container for each failure logged. The simulation runner is responsible for writing entries to the session output file at the end of each run.

5. vManager and the annotated verification plan

What is the annotated verification plan?
The annotated verification plan, or vPlan, is a standard, human readable format for describing the verification plan for a DUT or a group of related DUTs. The primary goal for the vPlan is to be the top-level framework for organizing and managing the entire verification process, including hardware and software, from high-level modeling to RTL to post-silicon, and from module to system. To accomplish this, it provides

• a hierarchical description of the DUT features, corresponding to sections in the plan, that should be verified,
• a description of the desired checking and coverage goals to be achieved at each hierarchical section,
• and a description of how the sections and goals relate to the various phases of the project and to various project deliverables.

The vPlan format
The vPlan format describes a forest of sections. Each such section typically describes one functional part of the verification plan. Figure 6 shows part of an SoC vPlan that we'll use to describe the vPlan format in this section:

From this first example, you can see:

• the vPlan consists of a hierarchy of sections,
• the "description" attribute contains an informal text description of the section which is ignored during vManager processing,
• the "coverage" attribute specifies a set of coverage items relevant for that section,
• the "checks" attribute specifies a set of checks relevant for the section in a format that use string matching to identify the checks by their text messages and
• other vPlans can be declared and instantiated.

Here is how this vPlan might look in vManager:

Once the section hierarchy of the vPlan is displayed, the annotated results can be viewed to any level of detail simply by pointing, clicking and expanding (or contracting) the scope of the tree view. The user also accesses the coverage analysis engines through this view (see below).

Perspectives
Given a project vPlan, project leadership may need to define different perspectives on the plan, where each perspective has the opportunity to modify the checking and coverage goals of each section.

Here are some typical dimensions of the vPlan perspective:

• The "project phase" dimension, which has phases such as "Ready for integration", "Ready for prototype", and so on,
• The "DUT representation" dimension. Typical representations are "HLM" (High Level Model), "RTL", "Post silicon", and so on,
• The "DUT subsystem" dimension. For instance, the team may be verifying separately block by block, or a set of blocks, or the full SoC without this SW, and so on. Typical names in this dimension would be "DSP", "Controller", "GPIO", "USB", "I/O Subsystem" (containing the previous two), "SoC HW", "SoC HW and SW drivers", "Full system with SW", and so on.

Other dimensions are also possible. For instance, the project may have an important customer X who is only interested in a subset of the device's features. Project leadership can create a "Customer X" perspective which gives very high weights to those features only, so management can evaluate verification progress from vantage of customer X sales.

Perspectives and verification goals
Each vPlan perspective can change any or all of the goals, weights, and at_least number of the coverage items that comprise some or all of the sections.

The goal is simply a number between 0 and 100. Essentially, this is the expected grade in percentage for that section, or the word NA for "Not Applicable". When the perspective is selected, the coverage results will be presented relative to this goal, rather than the default 100%.

The weight is a non-negative number that describes the relative importance of the section. It gives a weight to the section relative to other sections. The default weight is 1.

The at_least is a non-negative number that specifies the minimum number of times each coverage item in the section needs to appear. It influences all coverage under the current section.

Representing perspectives in the vPlan format
Perspectives are represented in the vPlan format using the perspective container and the section extension. Figure 7 shows a simple excerpt from a perspective definition for the previous SoC example.

Loading these perspective definitions on top of the previously defined section hierarchy allows project leadership to selectively view progress normalized to two different goals. In this way, the coverage holes that are important to the project from different perspectives can be highlighted and targeted.

Coverage progress reporting
vManager not only helps management understand the current situation, it also helps analyze the project trends. By loading the results from any selection of session data, project leadership can graph the progress from any perspective. vManager can graph the progress towards vPlan goals with the data grouped by dated session and with all the runs coverage data summarized as the verification status for that project date. vManager can also graph the progress towards vPlan goals with the data grouped in many other ways, including by individual run as shown below.

In addition to its own graphing and reporting capabilities, vManager can export data in the industry standard .csv format for import into existing management reporting applications.

Coverage hole analysis
As mentioned above, viewing the coverage data through the lens of the vManager vPlan sections and perspectives help the verification engineer identify and prioritize unachieved coverage. However, large coverage cross-products with widely-scattered holes still present a problem for coverage targeting. Typically, cross-product coverage models include hundreds to thousands of buckets to fill, and large numbers are unfilled. vManager's vPlan view makes sessions more efficient by analyzing and grouping cross holes to facilitate more direct targeting. The screen shot below shows the results of such an analysis. This example shows that a 32 bucket cross coverage space with 18 atomized holes can be reduced to just two holes for the purposes of coverage targeting.

6. vManager and failure analysis

Sorting
The data in the rows can be sorted interactively by one or more columns simply by clicking on the column header. Sorting is useful, for example, in identifying the failed run from a set of failed runs with identical errors that exhibits the failure the earliest during the run.

Grouping
One of the common requirements when analyzing sessions is finding similarities between runs. Those similarities can be very simple, such as all runs that were completed successfully or all runs that failed. The similarities can also be very complicated, such as all runs whose coverage grade on a specific item is more than zero, had no warnings, and took less than 1000 cycles to run. Finding such commonalities, either simple or complex, is enabled by the grouping feature.

Filtering and context
In many cases, analysis needs to be performed only a subset of the runs in a session or sessions. For example, a verification engineer may want to filter out any failed runs that he is not responsible for debugging. To enable selecting and using a subset, vManager provides a filtering tool that lets you drop runs from the display.

The filtering tool can filter based on the values of one or more attributes. Each filtering action can be applied so that all matching elements are either included or excluded.

Filtering establishes a context for other analyses. The runs that remain after a filtering action are the only runs that subsequent analysis engines, such as coverage analysis, will operate on.

7. Custom process automation

Configuration
The grouping, filtering, and sorting rules, as well as the current run attribute set, is called the vManager configuration. Configurations can be named and saved, and then recalled instantly. During analysis, the user can switch between alternative configurations as needed, and then continue analysis on top of the selected configuration.

For instance, a verification engineer could define the my_failures configuration that combines the filtering rule of "include only runs associated with my name", the grouping rule "first error" and a set of attributes relevant for failures. Then this engineer would load the my_failures configuration each time she entered vManager for failure analysis.

GUI-based batch macros
Any action that can be accomplished through the vManager GUI can be easily scripted in e and played back at any time. In this way, a user's common analyses can be scripted and replayed without having to use the mouse.

Database API
Deeper access is available through the vManager Database API. This API makes available all the data stored in the current sessions to user e-code. By using the API, the user can create any kind of analysis engine on top of vManager.

8. Conclusion