Aivora
AI Daily中文
arXivAI ResearchAdvanced

Testing AI Hardware Verification: EquivSVA Dataset Reveals LLM Robustness Across Equivalent RTLs

硬體驗證 AI 的試金石：EquivSVA 形式驗證資料集，揪出 LLM 程式碼生成的虛實

2 min read
中文English
Testing AI Hardware Verification: EquivSVA Dataset Reveals LLM Robustness Across Equivalent RTLs
The 30-second version

While LLMs are increasingly used to generate SystemVerilog Assertions (SVA), evaluating whether they capture true external behavior or merely memorize implementation details remains challenging. EquivSVA addresses this with 120 'behavior families' spanning 12 categories. Each family contains 4 structurally distinct but functionally equivalent RTL implementations, shared gold properties, and controlled mutants, all verified through a strict 17-job suite. Evaluating Qwen2.5-Coder-7B-Instruct revealed that the number of sound generated assertions varies across equivalent RTLs for 14 of 24 test families, highlighting critical robustness gaps.

Key points

01

Behavior Family Structure

Each family contains 4 equivalent yet structurally distinct RTL implementations to test if models are biased by superficial implementation details.

02

Rigorous Formal Verification

All entries passed a 17-job validation suite covering RTL equivalence, property reachability, and mutant distinguishability to guarantee absolute accuracy.

03

Exposing LLM Robustness Gaps

Evaluations revealed that for 14 of 24 test families, the number of sound properties generated by the model varied solely based on how the RTL was written.

04

Standard Splits and Open Source

Includes family-safe train/dev/test splits, and fully open-sources the dataset, generators, validation scripts, and baseline artifacts.

How it works

EquivSVA Behavior Family Structure & Validation Flow
containsdefinesgeneratescross-verifyprove propertiesdistinguishabilityyieldsBehavior Family (120total)4x Equivalent RTLsShared Gold Properties3x Controlled Mutants17-Job Validation SuiteVerified Dataset Splits

Why it matters

Evaluating AI-generated hardware assertions often overlooks how sensitive LLMs are to implementation-level variations. EquivSVA fills this gap with zero-compromise, formally verified data. It proves that even state-of-the-art coding models struggle to remain robust when the underlying RTL structure changes without affecting behavior. This is crucial for building high-assurance, automated chip verification pipelines where AI cannot afford to hallucinate based on syntactic style.

Who it affects

  • AI Developer
  • AI Researcher
  • Enterprise Leader

How to use it

  1. 1Benchmarking LLMs on SVA generation and robustness against syntactic changes.
  2. 2Fine-tuning and aligning AI models specialized in semiconductor verification.
  3. 3Mutation-based automated spec analysis and hardware vulnerability detection.

Limitations & caveats

  • The initial evaluation only benchmarked Qwen2.5-Coder-7B-Instruct; performance on other proprietary or open-source models remains to be explored.
  • The dataset focuses strictly on SystemVerilog Assertions (SVA) and may not directly generalize to other hardware verification languages.

Related