Random Stability in Testbench part-1 (Based on SV)

By -

Introduction 

     Over the past few years, random verification has proven itself to be very effective in rapidly uncovering or unexplored design bugs. So the quality of the random stimulus or testbench is very important in order to achieve a high degree of confidence that a design has been well tested and bug-free. But using random constrain verification there is potential that the test case may become reproducible due to certain changes in testbench during the development cycle. In this article, I try to cover why random stability is required for any testbench and its consciousness and significance. Also cover what are the solutions to achieve random stability for SystemVerilog and UVM.

Required understanding of Random Number Generator (RNG)

    Before understanding random stability we need to understand How to declare or initiated variables to get a random value. All EDA developers have their own RNG. Therefore when the same code run in a different simulator with the same seed then also random variable gets different values. here we do not understand how RNG works but try to understand when RNG calls and what happens when an input changes. 

    Any RNG has input as a seed and output as a random number according to applying the constraint. So the bottom line of any RNG was a giving seed may change the output of RNG as results random value changed. 

    In above figure show which kind of seeds use when to apply to a command line. First checked the "DON'T" part. 

1) Try to avoid the use of zero seed as input. 

2) Try to avoid datatype bit, byte etc. It has a very limited range of random sequences as a result of random number repeats. e.g. byte datatype having only 0 to 255 different combinations. 

3) Most common way to seed any stimuli was using the Time and date format. But here the problem was appear to be a unique number and very little variation in back to back seeds. e.g. Four digits of the year change once in a year. the same thing happens with the date and month. Apart from that minute and second only has 0 to 59 changes. it is also limit the range.

4) Easy way to generate a random seed for regression was using base shells built-in $random variable. it's using the time of day in second and microsecond plus process ID. generally, process Id lies between 0 to 2^15 - 1 so it has a 15-bit number only.

    Look at "DO" part. Try to make sure the used or applied seed comes from a random source and is independent of RNG where we apply that seed. Applied seed at least 32 bit so it avoids repetition of seeds. Try to used physical randomness sources like system clock jitter, white noise from the audio port, etc.  It can be accessed via the below command:

% head -4 /dev/urandom | od -N 4 -D -A n | awk '{print $1}' 

What is Random Stability?

    A specific type of small modification in a testbench leads to a big simulation difference in terms of random variable values that usually user won't expect. After modification of testbench user-run testbench again with same giving seed then usually user expects all random declared variables to get the same values as previous one but that not happened. Eventually, it turns into a randomly unstable testbench. But random stable testbench adding code in the middle of stimulus would make simulator generate identical output up to the point where new code was inserted. The only difference that users would observe in the random stable testbench. Previously used or declare random variable won't affect for giving same seed. 

    Random Stability has its own significance and it provides reproducibility towards testbench. 

  1. Consistent Results: After modification of DUT or Testbench code, stable Testbench are more likely to give the same results as compared to the previous one for the same given seed. But with unstable testbench generated scenario might change radically that user will doubt their earlier observation and it falls in inconsistent results. As a result, quite a possible bug disappear.
  2. Replicating Bugs:  Testbench must replicate bugs again and again with the same seed value. Due to some reason configuration of the testbench for that particular seed changed. As a result respective excepted results also change and may possible bugs disappear while running 2nd time. 
  3. Testing Bug fixes: it's quite similar to 2nd one. After a change in TB and fixing bugs and running with the same seed again, randomization results should not change. then and then users ensure the bug was fixed. therefore TB must run a test under similar conditions in which the bug has been exposed.
    In nutshell were ensure that during the development of TB, it must be randomly stable. 

Affecting factors for Random Stability

    Let's Look at what are the actual reason or factors to affect the random stability.
  1. Thread Locality.
  2. Hierarchical seeding.

    Before going to that first see how and when RNG calls in any code.

    The above diagram has 2 parts 1) structural RNG and 2) Process RNG. Structural RNG contains modules, program blocks packages and interface blocks. 2nd one is process RNG which contains all rest of procedural blocks(initial, always, etc blocks), object instances. Here "S" denotes the seed associated with the thread. In the Above dig. S0 seed gives to the simulator by using. That S0 seed propagates to two different modules and both get an S0 seed. Here simulator treated both modules as a thread or process and the SV standard suggested every thread inside structural RNG gets the same value of seed that's why both get S0. In the above figure, the upper module has 2 more procedure threads "initial" and "always" block both getting S1 and S2 seed which randomize by module (upper one) thread with S0 seed. So here module call's RNG with S0 seed input and generate S1 and S2. Now "initial" has two more blocks/thread so based on "initial block" RNG it generates 2 more random number S2 and S3. The same process follows for other threads as well. 

    It is quite possible "always block" thread has S2 and the first box of  "initial block" also has S2 side due to limiting range of seed value which I mention above section. Here one more observation is in the first module has an S0 seed and based on that it generates S1 and S2 for respective threads. So in 2nd module same thing happened. Both first "initial" thread gets (1st module and 2nd module) S1 seed. That represents a deterministic property of RNG.

  1. Thread locality: Thread locality refers to fact that call to RNG in each thread are independent of all other calls in other threads. In simple words Two threads call RNG then both are independent of each other. In the above example, 1st module has two initial blocks both are independent of each other.
  2. Hierarchical seeding: What if I want to change the order of "initial" and "always" block or between them add a new  "initial" thread. So would you get the same seed as before or changed? It's changed due to Hierarchical changes occurring.
  3. Exception: $random and $dist_uniform($dist_*) are non-Hierarchical seeds.

Reasons for Random Instability

  1. A change in the order of random calls with a  process.
  2. Insertion of new processes before previously defined ones.
  3. A change in the order of creation of forked processes.
  4. A change in the order of object creation.
Below are all examples ready to run format.
If anyone wants to see how seeds are allocated to each thread then please refer below the presentation slides. 
Case 1: 

module test;
  class pkt;
    rand bit [7:0] data;
  endclass: pkt
  initial begin
      pkt p;
      int unsigned var1;
      for (int i=0; i<5; i++) begin
        var1 = $urandom();
        $display("urandom before calling object new var1: %0h", var1);
      end
      p = new(); // Object new after for-loop
  end
endmodule

module test1;
  class pkt;
    rand bit [7:0] data;
  endclass: pkt
  initial begin
    pkt p;
    int unsigned var1;
    p = new(); // Object new before for-loop
    for (int i=0; i<5; i++) begin
      var1 = $urandom();
      $display("urandom after calling object new var1: %0h", var1);
    end
  end
endmodule



 
  Case 2: 

class pkt;
    rand bit [7:0] data;
endclass: pkt
  module test2;
      initial begin
          pkt p = new();
          int unsigned var1,var2;
          p.randomize();
          $display("p->data %0h ", p.data);
          fork begin
            for (int i=0; i<3; i++) begin
                var1 = $urandom();
                $display("thread-block var1: %0h", var1);                        
              end
          end join
          for (int i=0; i<3; i++) begin
            var2 = $urandom();
            $display("module-block var2: %0h", var2);
          end
      end
  endmodule: test2
  module test3;
      initial begin
          pkt p = new();
          int unsigned var1,var2;
          byte var3;
          #1; $display("----------TEST3---------");
          p.randomize();
          $display("p->data %0h ", p.data);
           for (int i=0; i<3; i++) begin
                    var3 = $urandom();
                    $display("thread-block var3: %0h", var3);
            end
           fork
               begin
                  for (int i=0; i<3; i++) begin
                      var1= $urandom();
                      $display("thread-block var1: %0h", var1);
                  end
             end
          join
         for (int i=0; i<3; i++) begin
            var2 = $urandom();
            $display("module-block var2: %0h", var2);
         end
      end
  endmodule: test3




Case 3:

class test1;
    //….
endclass: test1
module ex1;
  test1 t1,t2,t3,t4;
  initial begin
   #1; $display("module ex1");
      begin: b1
        t1 = new("t1");
        t1.randomize();
        $display("t1.a = %0d",t1.a);
      end: b1
      begin: b2
        t2 = new("t2");
        t2.randomize();
        $display("t2.a = %0d",t2.a);
      end: b2
      begin: b3
        t3 = new("t3");
        t3.randomize();
        $display("t3.a = %0d",t3.a);
      end: b3
      begin: b4
        t4 = new("t4");
        t4.randomize();
        $display("t4.a = %0d",t4.a);
      end: b4
  end
endmodule: ex1
module ex2;
  test1 t1,t2,t3,t4;
  initial begin
    #1; $display("module ex2");
      begin: b1
        t2 = new("t2");
        t2.randomize();
        $display("t2.a = %0d",t2.a);
      end: b1
      begin: b2
        t3 = new("t3");
        t3.randomize();
        $display("t3.a = %0d",t3.a);
      end: b2
      begin: b4
        t4 = new("t4");
        t4.randomize();
        $display("t4.a = %0d",t4.a);
      end: b4
      begin: b3
        t1 = new("t1");
        t1.randomize();
        $display("t1.a = %0d",t1.a);
      end: b3
  end
endmodule: ex2


Case 4:

class pkt;
    rand bit [7:0] data;
endclass: pkt
  module test1;
    initial begin
      pkt p = new();
      int unsigned var1,var2,var3;
      p.randomize();
      $display("initial block p->data %0h ", p.data);
      fork begin
        for (int i=0; i<3; i++) begin
          var1 = $urandom();
          $display("thread-block var1: %0h", var1);
        end
      end
      begin
        for (int i=0; i<3; i++) begin
          var2 = $urandom();
          $display("thread-block var2: %0h", var2);
        end
       end
      join
      var3 = $urandom();
      $display("initial block var3: %0h",var3);
    end
  endmodule: test1
  module test2;
    initial begin
      pkt p = new();
      int unsigned var1,var2,var3;
      #1 $display("\tTest2\t");
      p.randomize();
      #1;$display("initial block p->data %0h ", p.data);
      fork begin: th1
        for (int i=0; i<3; i++) begin
          var2 = $urandom();
          $display("thread-block var2: %0h", var2);
        end
      end: th1
      begin: th2
        for (int i=0; i<3; i++) begin
          var1 = $urandom();
          $display("thread-block var1: %0h", var1);
        end
      end : th2
      join
      var3 = $urandom();
      $display("initial block var3: %0h",var3);
    end
  endmodule: test2


How to Achieve Hierarchical Seeding Stability?

  1. Controlling the Creation of Hierarchical elements. Try to add new elements/threads bottom of previously existing code.
  2. Manually Seeding using srandom() method. Used srandom() method before randomize any variable and making sure the srandom() argument is each and every time unique number. 
  3. Using a template stimulus generator (Not recommended).
Case 1:

class test1;
  rand bit [7:0]a;
  function new(string s="t1");
    $display("class %0s created",s);
  endfunction: new
endclass: test1

module ex;
  test1 t1,t2,t3;
  initial begin
    t1 = new("t1");
    t2 = new("t2");
    t3 = new("t3");
    //add new instance here
    fork
      begin
        t1.randomize();
        $display("t1.a = %0d",t1.a);
      end
      begin
         t2.randomize();
        $display("t2.a = %0d",t2.a);
      end
      begin
        t3.randomize();
        $display("t3.a = %0d",t3.a);
      end
    join
  end
endmodule: ex


Case 2:

module ex;
  test1 t1,t2,t3;
  initial begin
    fork
      begin
        t1 = new("t1");
        t1.srandom(12);
        t1.randomize();
        $display("t1.a = %0d",t1.a);
      end
      begin
        t2 = new("t2");
        t2.srandom(5);
        t2.randomize();
        $display("t2.a = %0d",t2.a);
      end
      begin
        t3 = new("t3");
        t3.srandom(17);
        t3.randomize();
        $display("t3.a = %0d",t3.a);
      end
    join
  end
endmodule: ex


Case 3:

Case 3 is based on this fig. approach.

Refer below link for UVM based random stability.

Reference

[1].  "IEEE Standard for SystemVerilog - Unified Hardware Design, Specification, and Verification Language," IEEE Std 1800-2009, 2009.
[2].  Smith, D., 2013. Random Stability in SystemVerilog. In: D. Smith, ed., 1st ed. [online] Austin, Texas, USA: Doulos. Available at: https://www.doulos.com/media/1293/snug2013_sv_random_ stability_paper.pdf [Accessed 15 February 2022].


Verification, UVM, SystemVerilog, ASIC

Comments

Post a Comment

All Posts

All About UVM Reporting

By -
Image
     What we have in Verilog and SystemVerilog for Reporting.           Verilog and SystemVerilog have some system tasks for reporting and displaying messages. We all used $display, $error, $info, $monitor, $write, $fatal, $strobe and whatnot for different purpose like display data or messages, display errors, formatted messages and etc. Without going into detail, quickly recap these system tasks. $display:   Printing a formatted and non-formatted message in the console. Automatically add a new line at end of the message. $info:   Indicated assertion fail or not. It prints messages with additional information like time, path, line no and etc. $warning:  It provides a run-time warning and is suppressed by a tool. $error:  It provides a run-time error message. $fatal:  It provides run-time fatal error and terminated simulation when it encounters.    $monitor:  Keep tracing variables and printing them...
Read more

Random Stability in Testbench part-2 (Based on UVM)

By -
Image
  Introduction I already posted what is random stability and why it is required in the testbench. If you want to go through please click here  (Random Stability in Testbench part-1).  UVM itself provides a random stability mechanism. Therefore users need to follow guidelines and make sure provided mechanism don't override. In this blog, I try to cover how UVM random stability mechanism works? what are the drawbacks and how to overcome them? Let's deep dive into UVM based random stability. Understanding of UVM Random Stability Mechanism. In typical testbench architecture has two areas where most of the randomization activity is done or the users are generally accepted. 1) uvm_component and 2) uvm_sequence, uvm_sequence_item and uvm_transaction. Both areas have their own mechanism to handle random stability. But both, component and sequence used hash algorithm. So first we understand how the hash algorithm is used fully in terms of achieving stability. Wherever any componen...
Read more

Interrupt Generation and headlining Through UVM PART-1

By -
Image
What is interrupt?  The interrupts are signaled through which some unusual or special events acknowledge to users.  Interrupts are either hardware or software or both generated. Sometimes system generates interrupts or the user also can generate interrupts, it totally depends on system design and its use case. Software interrupts usually used register to acknowledge and hardware interrupts used direct signals to acknowledge. In nutshell, interrupts are events that can be triggered parallel threads of processes. Importance of Interrupts For any SoC level or system-level where multiple blocks work together, prepare complex design, in that case, all components work together and are compatible with each other that's why to introduce some exceptions and acknowledge of it, its call interrupts. Sometimes SoC Design used a separate Interrupt controller to handle such requests. So interrupt controller helps to manage system or subsystem level interrupts via enabling and disabling inter...
Read more