A promising approach to overcome the verification gap of modern SoC designs

Abstract

NEC Electronics (Europe) has introduced the Verisity Specman Elite test bench automation tool in a successful move to achieve the maximum level of functional quality both for this controller and for future designs, which are likely to require ever-increasing design complexity and exponentially growing verification effort. The main advantages of this new verification methodology are:

• Automated constraint-driven test generation
• Metric for the completeness of verification efforts using code and functional coverage
• Establish a reusable, easily maintainable verification environment

The entire feature set, including memory access arbitration and pipelined bus transactions with error detection and handling, were verified using advanced scoreboarding techniques and compact protocol checkers. Strict separation of all functional aspects within the environment, such as stimulus generation and bus functional models and checks, enabled the project team to achieve a high degree of reusability for subsequent projects.

Introduction

NEC Electronics (Europe) decided to use the Verisity’s test bench automation tool Specman Elite and the verification language e together with lint and code coverage tools to establish such an environment. Specman Elite has powerful capabilities that significantly boost verification efficiency in order to achieve high quality products in a shorter time. Other essential parts of the verification environment are a revision control system, a script-based simulation environment and bug tracing.

The system-under-test provides bridging functions, among various bus interfaces, and heterogeneous memory configurations, including SDRAM, FLASH and even peripheral devices and full DMA capabilities. The main features of the system controller are:

• 133 MHz SDRAM memory controller
• FLASH/peripheral memory controller
• 2-channel DMA controller
• 266 MHz CPU bus interface
• 66 MHz system bus interface
• Memory access arbitration

Motivation

Very complex protocols can be integrated and tied together to build even more complex systems such as W-CDMA, network controllers and set-top boxes implementing up to 10 to 20 million gates in one ASIC or ASSP.

Very complex performance-optimized data, and the control flows for such complex SoC designs with a huge amount of multi-dimensional modes and settings, require a rapidly increasing number of test cases to be run. Since the number of combinations to be verified grows exponentially with the number of specified features, most complex designs can no longer be tested to saturation.

Another major deficiency of traditional HDL test bench coding is not knowing when ‘enough’ verification has been done. Classic code coverage analysis can only be judged as a necessary criterion, not as a sufficient one. It cannot determine whether all specified features are properly implemented; it can only determine that all implemented code has been sufficiently verified. What is missing is a specification-driven functional coverage.

Figure 2-1 Verification GAP

Unless the traditional HDL test bench verification approach is sharply improved, there will be a serious verification gap (see Figure 2-1 Verification Gap). Verification effort, including development cost and time, required engineering resources and the probability of undetected bugs, all increase non-linearly as design complexity increases. In a complex SoC design development, functional verification becomes the dominant part of overall development effort and cost.

However, market demand usually limits development time to 6 to 12 months, whereby development costs should not increase, nor development quality decrease. And, as a general rule, engineering resources are usually also limited.

How can we escape this verification gap?
There are several ways of reducing the verification gap. One is to design for verification. One golden rule is to make designs as simple as possible. This does not necessarily mean reducing complexity; more important is efficient partitioning of the design to reduce multi-dimensionality by orthogonal features/modes and to remember that verification effort is mainly determined by the specification. Another approach is to improve verification efficiency by increasing the abstraction level for writing test scenarios, by making use of random tests to catch multi-dimensional bugs and by making use of test bench automation tools. Both of these approaches were applied to the described project. However, improving verification efficiency is the only subject of this publication.

System-Under-Test

Figure 3-1 Block Diagram

The macro is designed in such a way that the various bus interfaces can easily be replaced by a different interface type, or even extended to multiple bus interfaces. Key features of this system controller are:

• High-performance memory bandwidth
  - Simultaneous memory access from CPU and system bus interface
• Data rate CPU bus: I-Fetch 1.7 Gbps
• D-Read 1.2 Gbps
• D-Write 3.3 Gbps
• Data rate system bus: D-Read 1.7 Gbps
• D-Write 1.9 Gbps
• Standard SDRAM interface
  - 100 MHz, 133 MHz SDRAM
  - Controlled precharge with command look-ahead
  - First-come, first-serve arbitration scheme
  - Bank interleaving
  - Prefetch buffer
• Standard FLASH/peripheral interface
  - Up to 64 Mbytes
  - 8/16/32-bit bus width
  - Intel/Motorola mode
  - Detailed timing set-up
  - Boot capable
  - Endian conversion
• Standard DMA controller
  - Block transfers
  - Handshake transfers
  - Unaligned addresses
  - Different burst length on source and destination
  - 24-bit transfer count

Verification Plan

The verification plan is written based on a detailed analysis of the system controller and critical scenarios are carefully examined. Before design implementation was started, several brainstorming meetings were held, also with engineers not involved in the project. The outcome was an almost complete initial verification plan that provided the basis for the test bench development.

Besides tests and test scenarios executed in the project-related simulation environment, the verification plan also includes formal tests, e.g., tests to be executed during/after synthesis. These different classes of tests are explained below in more detail. Passing formal tests and sanity tests is a requirement for further integration. Random tests and directed tests complete the functional verification of the device-under-test. Regression tests attempt to stress the device as much as possible.

Formal Tests
The verification plan also specifies tests that are not executed by the simulation environment. For example, a common synchronization circuit was used to synchronize the signals between the various clock domains. For the correct behaviour of this circuit, it is essential that the output driving the synchronization circuit has to be registered. Lint, coding and naming-rule checks also have to be applied.

Sanity Tests
Sanity tests are tests that check basic settings and functions of the device-under-test to ensure smooth integration to a higher level of the design hierarchy. These tests include reset values of all registers, read/write checks, memory map, basic functions of all interfaces and general initialization of the device-under-test. As a general rule these tests take the form of directed tests to ensure easy controllability and reproducibility.

Random-Based Tests
Random-based tests are reproducible constraint-driven test scenarios covering complete verification bandwidth. They are used to achieve 100% code and functional coverage. These tests are based on bus transactions minimizing the engineering effort and they allow a huge amount of parallel automation.

Regression Tests
Regression tests are a set of sanity and random-based tests for the various interfaces and controllers. The focus is to stress the device-under-test as much as possible.

Directed Tests
Directed tests are used to check exceptions not covered by random-based tests.

Verification Approach

The test bench development was driven only by the design specification. No details of the internal implementation, e.g., clock domain synchronization latencies, were needed to set up the verification environment. The project team was able to develop the verification environment in parallel to the RTL design process and therefore different engineering resources were assigned for design and verification tasks. Because of the more abstract nature of the verification environment described in the e language, and therefore easier modelling of the specified functionality, the project team was already able to use the test bench during the early RTL design process. This reduced the impact of verification on the overall development schedule. Inconsistencies in the functional specification of the designunder-test could also be detected very early in development.

Figure 5-1 Testbench Block Diagram

Easy usage and reusability for further projects were high priority in test bench development. The result was a modular concept incorporating as many well-defined, implemented and documented e Verification Components ‘eVCs’ as possible. Figure 5-1 Test bench Block Diagram is the block diagram of the test bench.

Self-checking capabilities have been implemented on all functional aspects. Cycle-accurate protocol checkers and monitors on all interfaces, and transaction-based scoreboards check all the data paths of the system controller. To reduce the test generation effort, checks are made only by the verification environment and not by the test case itself.

The major advantage of using a test bench automation tool is automated constraint-driven test case generation. This enables the project team to generate huge numbers of test scenarios in parallel. The only limits were the available CPU time and number of tool licenses. This automated constraint-driven test case generation is also the basis for a wide distribution of test scenarios over most functional aspects covering areas that were not directly specified by the functional verification plan. Functional coverage provides clear feedback on whether a target scenario was achieved or not. Specman Elite’s functional coverage database gave the project team an easy way of assessing the degree to which the verification target was reached.

Experience

Figure 6-1 Verification Efficiency

It can be seen that running test cases increase dramatically as a proportion of the whole. This results in a higher degree of quality assurance given for a fixed development time. Both traditional HDL test benches, and the test bench automation approach have resource

limitations. The number of engineering resources is the limiting factor for traditional HDL test benches. By contrast, when with Specman Elite, the achievable quality is determined to a very great degree by available CPU time and the number of available tool licenses. There are several reasons for the increased verification efficiency. Unlike a traditional HDL test bench, a test bench automation tool developed for verification supports:

• Constraint-driven stimulus generation
• Self-checking test bench like scoreboarding techniques
• Temporal assertion
• Functional coverage definition
• Verification measurement

In addition, e Verification Components ‘eVCs’ ensure a high level of reuse. These are preverified, reusable pieces of verification code consisting of three integrated functional units:

• Stimulus generator for injecting and generating traffic
• Monitors and checkers for viewing outputs and protocol checking rules
• Coverage report showing functional coverage of scenarios on the interface as well as the design

Being able to access Verilog and to extend structures of the verification environment are additional benefits of test bench automation tool-based verification. This completely hides the complexity of the verification environment and test-case-specific modifications at no point corrupt the verification environment.

Our successful verification was due in part to the excellent consulting support the project team received from Verisity. Verisity’s trained consulting staff spent more than three weeks in our office to oversee the structure of our e code. Verisity’s excellent support and practical advice throughput the project were an important factor in its successful conclusion within the allocated time.

Conclusion