Using
Code Coverage with Axis' RCC Technology
Table of Contents
INTRODUCTION
Simulation acceleration and emulation products are seeing a dramatic
upsurge in popularity because designs are getting so large that
it takes days, weeks and even months to run the tests that are needed
to verify the design. Engineers need to be able to run tests and
get results overnight and are using simulation acceleration and
emulation to achieve this goal.
However, once they move to acceleration and
emulation to get higher performance how does an engineer know what
he is verifying and what areas are unverified? To help answer these
questions, design and verification engineers have been using coverage
with software simulators. By using coverage, they have been able
to measure how much of their design has been simulated. They are
able to use this coverage information to help improve their verification
productivity.
This paper will first describe some typical
applications where simulation acceleration and emulation are used.
It will then introduce TransEDA's coverage solution for Axis' verification
systems. Finally, it will walk through some examples of how users
can apply coverage to more efficiently use their Axis products to
further increase verification productivity.
SIMULATION
ACCELERATION AND EMULATION APPLICATIONS
To keep up with the increasing complexity of
large multi-million gate SoC designs, verification teams have embraced
simulation acceleration and emulation technologies to improve the
performance of their verification effort. These systems provide
a dramatic reduction in time-to-market by speeding functional verification
at the system-level.
Applications such as wireless, networking,
multimedia and microprocessor are typical targets for acceleration
and emulation. Here, focused automatic test generation must often
be employed to adequately represent the complexity of real-world
usage models. Those models might not represent real-world conditions,
and the resulting test-sets are so large, that a traditional software
simulator would require run-times of weeks, perhaps months. Simulation
acceleration and emulation make it possible to complete testing
in an acceptable amount of time. Figure 1 shows the time required
to run ¼ second of real networking traffic using software simulation,
simulation acceleration and emulation.
High-performance verification systems can be
used for verification of both the chip and its operational or system
diagnostic software. This hardware software co-verification can
either be done in simulation acceleration mode, where the testbench
is executed on the workstation, or in emulation mode using a target
connection to external hardware to provide the stimulus. Both methods
provide higher run-time performance compared to software simulation
tools, and allow designers to test the embedded software with the
RTL hardware design much earlier in the project.
Historically, simulation acceleration and emulation
systems have provided highperformance, but have lacked the flexibility
and ease of integration with valuable simulation tools such as the
ability to perform coverage analysis. Coverage is commonly used
with software simulators to provide an objective measure of test
coverage and thus identify unverified parts of a design that may
contain bugs. Axis has overcome this drawback by providing a verification
platform that runs and feels just like a software simulator. This
new class of acceleration and emulation systems provides the necessary
flexibility and ease of integration to run all of the commonly used
simulation applications on a high-performance hardware platform.
APPLYING
COVERAGE TO SIMULATION ACCELERATION AND EMULATION
Coverage is commonly accepted in the electronics
industry today as part of the logic simulation verification flow.
Companies are using coverage to:
- Provide an objective measure of verification
completeness and quality.
- Assist in the creation of test cases by focusing
on uncovered areas of the design.
- Prioritize tests according to their effectiveness,
enabling the construction of efficient regression suites.
As can be readily appreciated, many of the
benefits of coverage analysis apply equally to simulation acceleration
and emulation. However, the traditional techniques used for coverage
collection with software simulators do not easily leverage the performance
provided by hardwarebased verification tools. For example, many
coverage tools use the PLI (Verilog) and FLI (VHDL) interfaces during
the simulation of a design. Even though the Axis supports PLI, using
it often during simulation decreases overall execution speed. The
coverage solution should limit the use of PLI to the start and end
of simulation to take full advantage of the hardware-assisted performance.
In software simulation, coverage tools insert
counters and signals into the design for coverage. These constructs
increase the simulation's memory footprint, but between real and
virtual memory the simulation is able to easily manage the memory
usage and has little impact on the user of coverage. For the software
simulator, coverage's impact on performance is a critical factor.
For acceleration and emulation, the registers and signals added
by coverage have little or no impact on the run-time performance
of the design, but do have an impact on capacity. If coverage adds
too much logic and the design cannot fit into the hardware resources
then coverage cannot be performed and valuable data cannot be obtained.
TransEDA addresses these special needs by shifting
the optimization focus from simulation speed to hardware capacity.
The result is a common tool with a common database format that provides
optimized coverage for both software simulation and simulation acceleration
and emulation systems.
TRANSEDA'S VN-CoverT
EMULATOR
TransEDA is helping its customers to meet the challenges of verification
by enhancing the VNCover T Coverage Analysis product to provide
coverage on acceleration and emulation platforms. This industry-first
solution provides integration with current hardware-based and software-based
workflows and debug environments. The result is that users can improve
the effectiveness of high-performance verification tools and ensure
the completeness of the verification process by quickly generating
the coverage data that they need.
Supported coverage metrics
VN-Cover supports a number of coverage metrics
for simulation acceleration and emulation systems. They can be divided
into the following types:
- Code based metrics such as statement and
branch that report on specific HDL language constructs
- FSM-based metrics such as FSM State and Arc
that report on design behavior
Statement Coverage
Statement coverage identifies which statements
contained within conditional blocks were executed during simulation.
Statement coverage provides similar information to branch coverage,
but the importance of a statement coverage is that it provides a
measure of the importance of a module. For example, a module containing
ten statements has twice the weight and hence the importance of
a module with five statements.
Statement coverage results are annotated onto
design source files so that unexecuted lines of HDL can be easily
identified. The user needs to verify that he has tested all of the
lines of code. If he has not covered some code, there is a possibility
that the code is wrong. If the test suite exercises all of the functionality
then statements that are not covered indicates code that can potentially
be removed from the design saving area.
Branch Coverage
Branch coverage identifies which conditional
blocks, in if-else and case statements, were executed during emulation/acceleration.
In hardware terms, branch coverage shows whether or not a conditional
logic block was enabled during testing.
Branch coverage provides more information than
statement coverage. For example branch coverage indicates whether
the controlling if expression evaluated true and whether it evaluated
false. If there is no statement associated with the false branch,
statement coverage alone cannot provide this information. Branch
analysis is also performed on conditional and selected signal assignments
(VHDL) and conditional expressions (Verilog). Branch equivalent
control flow is identified in these statements.
Finite State Machine (FSM) Coverage
FSM Coverage provides a more understandable
view of the behavior represented by the lower level code statements.
This makes it easier for the user to identify untested behavior
and to create new tests to cover that behavior.
VN-Cover is able to automatically identify
and extract FSMs from the design. It then identifies the FSM's states
and arcs and can collect coverage information on them.
- FSM State coverage - Analyzes all of the
states that a finite state machine has and indicates if the state
machine enters each state. FSM State coverage highlights problems
such as unreachable states
- FSM Arc coverage - Analyzes all of the Arcs
that a finite state machine has and indicates if the state machine
has taken each state-to-state arc. FSM Arc coverage highlights
problems such as terminal states
COVERAGE IN THE HIGH-PERFORMANCE
VERIFICATION FLOW
The following figure illustrates the flow and major components that
make up the VN-Cover Emulator solution for Axis Systems Xcite®,
Xtreme® and Xtreme II® verification systems. A brief description
of each major step follows.
Instrumentation
The first step in the coverage process is to
instrument the design. During the instrumentation step the user
selects the design files and specifies the desired coverage criteria.
VN-Cover is used to instrument the design generating
a new set of HDL files that contain the "probes" that will be used
during simulation. These probes will record information about the
design's coverage. Instrumentation does not make any changes to
the behavior of the user's original HDL source files.
Instrumentation also produces a set of control
files that are using during the results phase. When instrumentation
is complete, the user is now ready to select a test bench and perform
simulation.
Simulation Acceleration and Emulation
When the user has successfully instrumented
and compiled the design, they are ready to run the simulation. The
Xcite and Xtreme Verification Systems are unique because they support
the use of the Verilog PLI at anytime during the simulation using
the Instantaneous Simulation Swap feature. This enables VN-Cover
to use PLI at the start of simulation and at the end of simulation
to perform initialization and results gathering. VN-Cover provides
the highest possible runtime speed by not using any PLI during simulation.
This unique feature from Axis allows for easy integration of VN-Cover
for both simulation acceleration and emulation.
Simulation begins by running the simulation
executable (vlg) in software simulation mode. After coverage initialization
is done at time 0 the entire RTL design can be swapped into the
hardware using the $rcc(run) command for simulation acceleration
and the $rcc(autorun) command for emulation. When simulation is
complete, the user then swaps the simulation from hardware back
into software using the $rcc(stop) command and VN-Cover uses the
PLI to gather the coverage results before the user exits the simulator.
View Results and Test Suite Analysis
VN-Cover is then be used to load and analyze
the history file containing the results. The contents of the results-file
can be treated in the same way as results obtained from the simulator.
This means that the complete suite of results analysis tools, including
results filtering, available within VN-Cover are available to the
user.
If the user has the results from individual
tests they could be loaded and analyzed using VNOptimize. VN-Optimize
provides the user with the ability to sort, analyze and compare
the coverage results from different runs to develop regression suites
and/or identify which tests cover which parts of the design.
TYPICAL APPLICATIONS
This section described two typical applications
where users of hardware-assisted verification systems can improve
their verification process by using coverage.
Focused Test Generation
Some users have adopted a methodology of using
focused test generation to verify their system-level design. A weakness
of this methodology is that they continue to generate and run tests
without knowing if they have ever verified the entire design. To
address this weakness many companies have been adopting coverage
to help them identify and fill holes resulting from this methodology.
These users first establish the base-line coverage
for the entire verification suite. They then identify verification
holes by looking for parts of the design that have not been emulated.
VNCover's metrics view aids them in this analysis by providing a
list of instances sorted from worse to best covered.
Using their knowledge of the specification,
users analyze the coverage holes and identify if the hole is the
result of an error in the RTL code, a missing test case, or a specification
problem. If it is a missing test case they then change the settings
of their focused test generator to target and fill in these coverage
holes. If it is too difficult to focus the test generation on these
holes then users will mix in a directed test to fill the hole. The
goal here is to ensure that the entire design has been covered before
spending too much time looking for system-level interactions.
Companies that have adopt this methodology
with their logic simulation-based test have generally found the
following:
- There are verification holes that are as
much as 25% to 30% of the design.
- There is a significant drop in the number
and rate of bugs found as they approach 100% statement and branch
coverage.
- There are critical bugs found that they believe
that they would not have found without using coverage.
Hardware/Software Co-Verification
HW/SW co-verification is another valuable application
that is used with simulation acceleration and emulation. Co-verification
allows embedded system diagnostics and firmware to be verified by
running at very high speeds. Coverage is helpful in all co-verification
applications, but it provides the greatest benefit when it is used
to help verify system diagnostic software. Today's SoC projects
typically develop a comprehensive suite of diagnostic tests that
attempts to identify all hardware design and specification errors.
When developing and verifying system diagnostics
the goal is to be able to verify that specific areas of the design
are operational. Companies are finding coverage can be an integral
part of the system diagnostic verification process because it helps
the users to know if they tested what they expected to test, and
if there are any holes in their system diagnostics. Coverage metrics
are key in establishing confidence in the correctness and completeness
of the system diagnostic software. The combination of VNCover with
Xpert for ARM Microprocessors provides an ideal combination to develop,
debug, and measure the quality of diagnostics.
When companies use coverage with operational
software verification they may find that:
- By seeing what parts of the design the software
used they are able to reduce debug time when complex hardware
software interactions occur.
- There is unused or "extra" hardware that
may contain bugs.
Summary
This paper has described how TransEDA's VNCover can be used with
Axis verification systems.
- To measure simulation or verification completeness.
- To assist in creation of acceleration and
emulation tests.
- To prioritize tests for regression testing.
This is becoming even more critical as companies
use acceleration and emulation systems to speed their verification
process.
The methodologies that are typically used with
accelerators and emulators, focused test generation and hardware/software
co-verification do not have any feedback mechanism to help improve
productivity. Coverage provides this feedback, enabling the user
to increase their verification performance and productivity.
|
