OSVVM’s Intelligent Coverage is 5X or More Faster than SystemVerilog’s Constrained Random

If the measure of test case generation was a large number of well randomized test cases, SystemVerilog and UVM would be on par with VHDL’s OSVVM.  However the true measure of test case generation is functional coverage closure – all test cases identified in the test plan are done.  Functional coverage closure is a big challenge for constrained random approaches to verification as used in SystemVerilog or ‘e’.  On the other hand, functional coverage closure is the focus of OSVVM’s Intelligent Coverage™.   This article takes a look at why Constrained Random has challenges and how Intelligent Coverage solves it.

1. Constrained Random Repeats Test Cases

In my post, Functional Coverage Made Easy with VHDL’s OSVVM, we used randomization with a uniform distribution (shown below) to select the register pairs for the ALU. Constrained random at its best produces a uniform distribution. As a result, this example is a best case model of constrained random tests.

Src1 := RV.RandInt(0, 7) ; -- Uniform Randomization
Src2 := RV.RandInt(0, 7) ;

The problem with constrained random testbenches is that they repeat some test cases before generating all test cases. In general to generate N cases, it takes “N * log N” randomizations. The “log N” represents repeated test cases and significantly adds to simulation run times. Ideally we would like to run only N test cases.

Running the previous ALU testbench, we get the following coverage matrix when the code completes. Note that some case were generated 10 time before all were done at least 1 time. It took 315 randomizations to generate all 64 unique pairs. This is slightly less than 5X more iterations than the 64 in the ideal case. This correlates well with theory as 315 is approximately 64 * log(64). By changing the seed value, the exact number of randomizations may increase or decrease but this would be a silly way to try to reduce the number of iterations a test runs.


2. Intelligent Coverage

“Intelligent Coverage” is a coverage driven randomization approach that randomly selects a hole in the functional coverage and passes it to the stimulus generation process. Using “Intelligent Coverage” allows the stimulus generation to focus on missing coverage and reduces the number of test cases generated to approach the ideal of N randomizations to generate N test cases.

Lets return to the ALU example. The Intelligent Coverage methodology starts by writing functional coverage. We did this in the previous example too. Next preliminary stimulus is generated by randomizing using the functional coverage model. In this example, we will replace the call to RandInt (uniform randomization) with a call to RandCovPoint (one of the Intelligent Coverage randomization methods). This is shown below. In this case, Src1 and Src2 are used directly in the test, so we are done.

architecture Test3 of tb is
  shared variable ACov : CovPType ;  -- Declare 
  TestProc : process 
    variable RV : RandomPType ;
    variable Src1, Src2 : integer ;
    -- create coverage model
    ACov.AddCross( GenBin(0,7), GenBin(0,7) );  -- Model

    while not ACov.IsCovered loop    -- Done?
      (Src1, Src2) := ACov.RandCovPoint ; -- Intelligent Coverage Randomization

      DoAluOp(TRec, Src1, Src2) ;    -- Transaction
      ACov.ICover( (Src1, Src2) ) ;  -- Accumulate
    end loop ;

    ACov.WriteBin ;  -- Report 
    EndStatus(. . . ) ;   
  end process ;

When randomizing across a cross coverage model, the output of RandCovPoint is an integer_vector. Instead of using the separate integers, Src1 and Src2, it is also possible to use an integer_vector as shown below.

variable Src : integer_vector(1 to 2) ;
. . . 
Src := ACov.RandCovPoint ;      -- Intelligent Coverage Randomization

The process is not always this easy. Sometimes the value out of RandCovPoint will need to be further shaped by the stimulus generation process.  We do this in our VHDL Testbenches and Verification class.

The Intelligent Coverage methodology works now and works with your current testbench approach. You can adopt this methodology incrementally. Add functional coverage today to make sure you are executing all of your test plan. For the tests that need help, use the Intelligent Coverage.

3. Intelligent Testbenches in SystemVerilog

With SystemVerilog you can certainly buy a simulator that implements Intelligent Testbenches. However, this is a signification upgrade, so it will cost. In addition using an intelligent testbench tool tends to require vendor specific coding – so you are locked into a particular vendor.

On the other hand, with VHDL’s OSVVM, the Intelligent Testbench capability is built into the functional coverage modeling.  It is free.  All of the customizations to the randomization are done by writing VHDL code and initiating transactions.

4. References

Here are a couple of articles on Intelligent Testbench approaches that also remove or reduce the repetition of test cases. However, these solutions are not free like OSVVM.

Wally Rhines. From Volume to Velocity. DVCon Keynote March 2011

Mark Olen. Intelligent Testbench Automation Delivers 10X to 100X Faster Functional Verification. Verification Horizons Blog June 2011

Brian Bailey. Enough of the sideshows – it’s time for some real advancement in functional verification! EDA DesignLine Blog May 2012