[dv] V2S Coverage Implementation #1895

ctopal · 2022-11-02T11:08:00Z

This commit adds coverpoints and crosses for security countermeasures implemented in the design.

Resolves #1762

Signed-off-by: Canberk Topal ctopal@lowrisc.org

andreaskurth

Thanks @ctopal! This LGTM modulo one code typo and one question.

dv/uvm/core_ibex/fcov/core_ibex_fcov_if.sv

doc/03_reference/coverage_plan.rst

ctopal · 2022-11-02T15:41:10Z

Thanks @andreaskurth for the swift review, sorry about the silly typos!

andreaskurth

No worries, all good now 👍

andreaskurth · 2022-11-03T15:33:34Z

ctopal · 2022-11-03T15:43:15Z

For cp_data_ind_timing_instr and cp_dummy_instr we might need to add +add_csr_write=MSTATUS,MEPC,MCAUSE,MTVAL,0x7c0,0x7c1 to some other tests like riscv_debug_single_step_test(this would probably provide hits for InstrCategoryDRet + InstrCategoryWFI + InstrCategoryCompressedIllegal + InstrCategoryUncompressedIllegal) and riscv_mem_error_test for InstrCategoryFetchError

andreaskurth · 2022-11-03T17:01:23Z

cp_rf_b_ecc_err not hitting 1 was a bug in the test (the b port had a much smaller chance of getting glitched due to an architectural subtlety) that I'll fix.
cp_rf_we_glitch_err does not hit 1 because we don't have a test glitching the write enable. That's covered by FPV, see [dv] Verify register file integrity checking and write enable glitch detection #1756 (comment). I suggest we remove that CP.

GregAC · 2022-11-04T12:20:30Z

doc/03_reference/coverage_plan.rst

+
+* ``cp_dummy_instr_type`` - Type of the injected dummy instruction.
+
+* ``cp_dummy_instr`` - Executing each instruction category while dummy instruction insertion feature is enabled.


We should add coverage for interrupt and debug requests whilst a dummy instruction is being executed (different points for IF, ID/EX and WB stages).

I've just added two commits that should add this coverage

ctopal · 2022-11-04T17:56:03Z

Latest commit adds random writes to custom CSRs, enables us to see more diverse instruction categories while enabling the security features. We might need to end up increasing the iteration numbers for these tests but with 20 seeds of debug_single_step_test we close every hole except InstrCategoryFetchError (which should be closed by mem_error_test).

andreaskurth · 2022-11-07T09:44:09Z

I just tried riscv_debug_single_step_test with 40 seeds (starting at 26462); pass rate was 85% (26467, 26476, 26477, 26478, 26486, and 26493 failed due to cosim mismatches) and cp_data_ind_timing_instr was covered 81.82% (InstrCategoryEBreakDbg, InstrCategoryFence, InstrCategoryFetchError, and InstrCategoryCSRIllegal missing). The coverage rate could be improved but I think it's not critical (because functional coverage overall must be >=90%, and other tests should hit this CP too). I think the pass rate <90% is critical, though.

riscv_mem_error_test with 15 seeds (starting at 18092) had a pass rate of only 60%; details below. It also failed in CI. It did not hit InstrCategoryFetchError for cp_data_ind_timing_instr. I think fixing this should be prioritized.

Details of failing tests

riscv_mem_error_test.18092
--------------------------
binary:          test.bin
rtl_log:         rtl_sim.log
rtl_trace:       trace_core_00000000.log
iss_cosim_trace: spike_cosim_trace_core_00000000.log

[FAILED]: error seen in 'rtl_sim.log'
---------------*LOG-EXTRACT*----------------
    119: 107839: Illegal instruction (hart 0) at PC 0x80002f06: 0xf14e6373
    120: 107859: Illegal instruction (hart 0) at PC 0x80002f06: 0xf14e6373
    121: 107879: Illegal instruction (hart 0) at PC 0x80002f06: 0xf14e6373
    122: 107899: Illegal instruction (hart 0) at PC 0x80002f06: 0xf14e6373
    123: UVM_INFO /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_test_lib.sv(1240) @ 115529: uvm_test_top [uvm_test_top] latched_imem_err: 0x1
[E] 124: UVM_FATAL /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_base_test.sv(407) @ 119209: reporter [uvm_test_top] Check failed signature_data == core_status (10 [0xa] vs 9 [0x9]) Core did not register correct memory fault type
    125:
    126: --- RISC-V UVM TEST FAILED ---
    127:
    128: UVM_INFO /nas/lowrisc/tools/cadence/xcelium/21.09-s006/tools/methodology/UVM/CDNS-1.2/sv/src/base/uvm_report_catcher.svh(705) @ 119209: reporter [UVM/REPORT/CATCHER]
--------------------------------------------



riscv_mem_error_test.18095
--------------------------
binary:          test.bin
rtl_log:         rtl_sim.log
rtl_trace:       trace_core_00000000.log
iss_cosim_trace: spike_cosim_trace_core_00000000.log

[FAILED]: error seen in 'rtl_sim.log'
---------------*LOG-EXTRACT*----------------
    117: UVM_INFO /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_test_lib.sv(1182) @ 73092: uvm_test_top [uvm_test_top] Injected dmem error
    118: UVM_INFO /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_test_lib.sv(317) @ 94232: uvm_test_top [uvm_test_top] mcause: 0x7
    119: UVM_INFO /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_test_lib.sv(1223) @ 100912: uvm_test_top [uvm_test_top] exiting mem fault checker
    120: UVM_INFO /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_test_lib.sv(1200) @ 103992: uvm_test_top [uvm_test_top] Injecting imem fault
    121: UVM_INFO /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_test_lib.sv(1240) @ 103992: uvm_test_top [uvm_test_top] latched_imem_err: 0x1
[E] 122: UVM_FATAL /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_base_test.sv(407) @ 103992: reporter [uvm_test_top] Check failed signature_data == core_status (12 [0xc] vs 9 [0x9]) Core did not register correct memory fault type
    123:
    124: --- RISC-V UVM TEST FAILED ---
    125:
    126: UVM_INFO /nas/lowrisc/tools/cadence/xcelium/21.09-s006/tools/methodology/UVM/CDNS-1.2/sv/src/base/uvm_report_catcher.svh(705) @ 103992: reporter [UVM/REPORT/CATCHER]
--------------------------------------------



riscv_mem_error_test.18096
--------------------------
binary:          test.bin
rtl_log:         rtl_sim.log
rtl_trace:       trace_core_00000000.log
iss_cosim_trace: spike_cosim_trace_core_00000000.log

[FAILED]: error seen in 'rtl_sim.log'
---------------*LOG-EXTRACT*----------------
    131: 222579: Illegal instruction (hart 0) at PC 0x80003010: 0x7b003af3
    132: UVM_INFO /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_test_lib.sv(317) @ 246009: uvm_test_top [uvm_test_top] mcause: 0x7
    133: UVM_INFO /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_test_lib.sv(1223) @ 255049: uvm_test_top [uvm_test_top] exiting mem fault checker
    134: UVM_INFO /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_test_lib.sv(1200) @ 256429: uvm_test_top [uvm_test_top] Injecting imem fault
    135: UVM_INFO /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_test_lib.sv(1240) @ 263409: uvm_test_top [uvm_test_top] latched_imem_err: 0x1
[E] 136: UVM_FATAL /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_base_test.sv(407) @ 264029: reporter [uvm_test_top] Check failed signature_data == core_status (12 [0xc] vs 9 [0x9]) Core did not register correct memory fault type
    137:
    138: --- RISC-V UVM TEST FAILED ---
    139:
    140: UVM_INFO /nas/lowrisc/tools/cadence/xcelium/21.09-s006/tools/methodology/UVM/CDNS-1.2/sv/src/base/uvm_report_catcher.svh(705) @ 264029: reporter [UVM/REPORT/CATCHER]
--------------------------------------------



riscv_mem_error_test.18101
--------------------------
binary:          test.bin
rtl_log:         rtl_sim.log
rtl_trace:       trace_core_00000000.log
iss_cosim_trace: spike_cosim_trace_core_00000000.log

[FAILED]: error seen in 'rtl_sim.log'
---------------*LOG-EXTRACT*----------------
    135: UVM_INFO /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_test_lib.sv(1259) @ 246567: uvm_test_top [uvm_test_top] exiting mem fault checker
    136: 248897: Illegal instruction (hart 0) at PC 0x80002e7e: 0xf11899f3
    137: 248937: Illegal instruction (hart 0) at PC 0x80002e7e: 0xf11899f3
    138: UVM_INFO /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_test_lib.sv(1182) @ 256247: uvm_test_top [uvm_test_top] Injected dmem error
    139: UVM_INFO /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_test_lib.sv(317) @ 266527: uvm_test_top [uvm_test_top] mcause: 0x7
[E] 140: UVM_FATAL /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_test_lib.sv(321) @ 266527: reporter [uvm_test_top] Check failed mcause[ibex_mem_intf_agent_pkg::DATA_WIDTH-2:0] == cause (7 [0x7] vs 0 [0x0]) mcause.exception_code is encoding the wrong exception type
    141:
    142: --- RISC-V UVM TEST FAILED ---
    143:
    144: UVM_INFO /nas/lowrisc/tools/cadence/xcelium/21.09-s006/tools/methodology/UVM/CDNS-1.2/sv/src/base/uvm_report_catcher.svh(705) @ 266527: reporter [UVM/REPORT/CATCHER]
--------------------------------------------



riscv_mem_error_test.18104
--------------------------
binary:          test.bin
rtl_log:         rtl_sim.log
rtl_trace:       trace_core_00000000.log
iss_cosim_trace: spike_cosim_trace_core_00000000.log

[FAILED]: error seen in 'rtl_sim.log'
---------------*LOG-EXTRACT*----------------
    153: 228601: Illegal instruction (hart 0) at PC 0x800064da: 0xf1412df3
    154: 228601: Illegal instruction (hart 0) at PC 0x800064da: 0xf1412df3
    155: 228621: Illegal instruction (hart 0) at PC 0x800064da: 0xf1412df3
    156: 228641: Illegal instruction (hart 0) at PC 0x800064da: 0xf1412df3
    157: UVM_INFO /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_test_lib.sv(1240) @ 235491: uvm_test_top [uvm_test_top] latched_imem_err: 0x1
[E] 158: UVM_FATAL /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_base_test.sv(407) @ 241051: reporter [uvm_test_top] Check failed signature_data == core_status (10 [0xa] vs 9 [0x9]) Core did not register correct memory fault type
    159:
    160: --- RISC-V UVM TEST FAILED ---
    161:
    162: UVM_INFO /nas/lowrisc/tools/cadence/xcelium/21.09-s006/tools/methodology/UVM/CDNS-1.2/sv/src/base/uvm_report_catcher.svh(705) @ 241051: reporter [UVM/REPORT/CATCHER]
--------------------------------------------



riscv_mem_error_test.18105
--------------------------
binary:          test.bin
rtl_log:         rtl_sim.log
rtl_trace:       trace_core_00000000.log
iss_cosim_trace: spike_cosim_trace_core_00000000.log

[FAILED]: error seen in 'rtl_sim.log'
---------------*LOG-EXTRACT*----------------
    138: UVM_INFO /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_test_lib.sv(1223) @ 289288: uvm_test_top [uvm_test_top] exiting mem fault checker
    139: UVM_INFO /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_test_lib.sv(1200) @ 293128: uvm_test_top [uvm_test_top] Injecting imem fault
    140: 345138: Illegal instruction (hart 0) at PC 0x80003514: 0xf128ecf3
    141: 345178: Illegal instruction (hart 0) at PC 0x80003514: 0xf128ecf3
    142: UVM_INFO /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_test_lib.sv(1240) @ 363228: uvm_test_top [uvm_test_top] latched_imem_err: 0x1
[E] 143: UVM_FATAL /home/dev/src/dv/uvm/core_ibex/tests/core_ibex_base_test.sv(407) @ 367348: reporter [uvm_test_top] Check failed signature_data == core_status (10 [0xa] vs 9 [0x9]) Core did not register correct memory fault type
    144:
    145: --- RISC-V UVM TEST FAILED ---
    146:
    147: UVM_INFO /nas/lowrisc/tools/cadence/xcelium/21.09-s006/tools/methodology/UVM/CDNS-1.2/sv/src/base/uvm_report_catcher.svh(705) @ 367348: reporter [UVM/REPORT/CATCHER]
--------------------------------------------

ctopal · 2022-11-07T12:43:13Z

We shouldn't see failures after #1907 gets merged so let's wait for it first to see. Locally I've seen 92.5% pass rate with that rebased.

andreaskurth · 2022-11-07T14:22:15Z

With this PR rebased on top of #1907, I got:

riscv_mem_error_test 10/15 iterations passing (starting at seed 20607), with InstrCategoryFetchError and InstrCategoryCSRIllegal getting hit (thus positive for coverage, but still too low pass rate).
riscv_debug_single_step_test 32/40 iterations passing (starting at seed 18220) with all bins of cp_data_ind_timing_instr hit except InstrCategoryEBreakDbg and InstrCategoryFetchError) (thus very positive for coverage but still too low pass rate).

Overall we should hit all bins of cp_data_ind_timing_instr and cp_dummy_instr except InstrCategoryEBreakDbg, and for cp_dummy_instr_mask we should hit all bins except 5. So coverage LGTM, but what are we missing to get pass rate back to 90%?

This commit adds coverpoints and crosses for security countermeasures implemented in the design. Signed-off-by: Canberk Topal <ctopal@lowrisc.org>

Signed-off-by: Andreas Kurth <adk@lowrisc.org>

… dummy instruction Signed-off-by: Andreas Kurth <adk@lowrisc.org>

This commit adds random custom CSR writes to debug_single_step_test and riscv_mem_error_test. Signed-off-by: Canberk Topal <ctopal@lowrisc.org>

ctopal · 2022-11-07T16:42:32Z

rebased

ctopal · 2022-11-07T18:40:07Z

Errors in riscv_mem_error_test came from a DV error with processing iside faults. I'll open up a new PR for that fix as it's not going to be related with V2S coverage work.

ctopal added Type:Enhancement Feature requests, enhancements Component:DV Design verification (DV) or testing issue labels Nov 2, 2022

ctopal requested review from GregAC and andreaskurth November 2, 2022 11:08

andreaskurth requested changes Nov 2, 2022

View reviewed changes

ctopal force-pushed the v2s_fcov branch from 09d3d61 to 6c8730e Compare November 2, 2022 15:40

ctopal requested a review from andreaskurth November 2, 2022 15:41

andreaskurth approved these changes Nov 3, 2022

View reviewed changes

andreaskurth added this to the V2S milestone Nov 3, 2022

GregAC approved these changes Nov 4, 2022

View reviewed changes

andreaskurth force-pushed the v2s_fcov branch from d9cc4ee to a9a5259 Compare November 4, 2022 16:40

ctopal and others added 4 commits November 7, 2022 16:41

[dv] V2S Coverage Implementation

8f9d8de

This commit adds coverpoints and crosses for security countermeasures implemented in the design. Signed-off-by: Canberk Topal <ctopal@lowrisc.org>

[dv] Add coverpoints for dummy instructions in each stage

eb9bcb4

Signed-off-by: Andreas Kurth <adk@lowrisc.org>

[dv] Add coverage for debug requests and interrupts while executing a…

0dd4cd1

… dummy instruction Signed-off-by: Andreas Kurth <adk@lowrisc.org>

[dv] Generate random writes in custom CSRs

4e6715e

This commit adds random custom CSR writes to debug_single_step_test and riscv_mem_error_test. Signed-off-by: Canberk Topal <ctopal@lowrisc.org>

ctopal force-pushed the v2s_fcov branch from 151b040 to 4e6715e Compare November 7, 2022 16:41

ctopal merged commit 56268c6 into lowRISC:master Nov 7, 2022

andreaskurth mentioned this pull request Nov 8, 2022

[dv] Fix how we catch iside_err in cosim #1909

Closed


		* ``cp_dummy_instr_type`` - Type of the injected dummy instruction.

		* ``cp_dummy_instr`` - Executing each instruction category while dummy instruction insertion feature is enabled.

[dv] V2S Coverage Implementation #1895

[dv] V2S Coverage Implementation #1895

Uh oh!

Conversation

ctopal commented Nov 2, 2022

Uh oh!

andreaskurth left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ctopal commented Nov 2, 2022

Uh oh!

andreaskurth left a comment

Choose a reason for hiding this comment

Uh oh!

andreaskurth commented Nov 3, 2022 • edited by ctopal Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ctopal commented Nov 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andreaskurth commented Nov 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GregAC Nov 4, 2022

Choose a reason for hiding this comment

Uh oh!

andreaskurth Nov 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ctopal commented Nov 4, 2022

Uh oh!

andreaskurth commented Nov 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ctopal commented Nov 7, 2022

Uh oh!

andreaskurth commented Nov 7, 2022

Uh oh!

ctopal commented Nov 7, 2022

Uh oh!

ctopal commented Nov 7, 2022

Uh oh!

Uh oh!

andreaskurth commented Nov 3, 2022 •

edited by ctopal

Loading

ctopal commented Nov 3, 2022 •

edited

Loading

andreaskurth commented Nov 3, 2022 •

edited

Loading

andreaskurth Nov 4, 2022 •

edited

Loading

andreaskurth commented Nov 7, 2022 •

edited

Loading