How PPABench grades a chip
A submission is a prompt → a core hardened into a fixed, manufacturable chassis. PPABench grades it on both halves of a real chip — physical implementation and functional behavior — deterministically, with no LLM judge anywhere in the loop.
1 · The chassis
Every submission targets one frozen wrapper, user_project_wrapper, on the Efabless
OpenFrame / chipignite shuttle (sky130 HD). The pinout is locked: a Wishbone control port at
0x30000000, a QSPI flash interface on io[24:29], a UART on io[30:31],
a byte-stream interface, and done/error status pins. Fixing the chassis makes
every submission directly comparable and guarantees the result is a real, manufacturable layout — not an
abstract netlist. The reference SoC is picosoc (picorv32 RV32IMC + QSPI XIP + UART).
2 · Harden — the physical half
Hardening turns RTL into a manufacturable layout and is the only way to know a design's real area, speed, and power. PPABench runs a deterministic, LLM-free flow:
It emits the routed area, fmax (from worst-slack timing closure), power split into dynamic and leakage, a DRC-clean GDSII, and an LVS match. These are the P-P-A the benchmark ranks on.
3 · Hidden testbench — the functional half
A cocotb/Verilator harness boots the submission's firmware from a QSPI-flash model, releases it from reset
by writing CTRL.start, and runs it to done. Two principles make the result trustworthy:
- Cycles are counted externally. The graded cycle count is the harness counting
wb_clk_iedges betweenstartanddone— never a register the design reports. A core cannot lie about the clock edges the harness sees. - Inputs are random per run. Keys, plaintext, or test images are drawn fresh each run and injected into the flash image, so an output cannot be precomputed or hardcoded; the harness re-derives the golden in software and requires K trials to all match.
4 · Anti-cheat / verifiability
The threat model is concrete: a submission could hardcode a known answer, fake a cycle counter, or drop in a fixed-function block that isn't really programmable. PPABench defeats each:
| Attack | Defense |
|---|---|
| Hardcode the expected output | Random input per run → output is unpredictable |
| Fake the cycle count | Cycles counted externally at the wrapper boundary |
| Precompute in parallel | Sequential workloads (e.g. AES-CBC chaining) |
| Overfit to the public task | Held-out challenge variants, kept private |
| Emit plausible garbage | Result verified against a software golden / reference decoder |
5 · The scorecard — pass/fail vs ranking
Pass/fail (hard gates): PPA caps (area / power / gates / macros) · functional_ok ·
timing_met (WNS ≥ 0) · DRC-clean · LVS-match · pinout/contract. Everyone who passes is a real,
manufacturable chip.
Ranking (reported, not gated): the energy / area / performance figures of merit. The leaderboard sorts passing designs by efficiency.
6 · Tracked metrics
Energy is computed at the operating clock (the rate at which cycles elapse and power is drawn), not fmax.
| Metric | Definition |
|---|---|
energy_per_cycle_pj | P[mW] / f[MHz] × 1000 |
total_energy_nj | P[mW] × cycles / f[MHz] |
energy_per_instruction_pj / _per_byte_pj / _per_pixel_pj | total energy ÷ workload units |
leakage_power_mw, dynamic_power_mw, leakage_fraction | from OpenROAD report_power |
area_um2, util_pct, gate_count, area_per_kgate_um2 | routed physical area |
fmax_mhz, timing_met | from worst-slack closure (WNS ≥ 0) |
ipc / cpi / mips | retired ÷ cycles (fixed-ISA tracks) |
cycles_per_byte, throughput_bytes_per_s, pixels_per_cycle | workload rate |
edp_nj_us | energy × delay |
edap_nj_us_um2 | energy × delay × area |
See the paper for the full methodology and reference results, or submit a design →