In the evaluation industry, more questions often seem more authoritative. Benchmarks with thousands of questions make people instinctively equate scale with rigor. But WDCD chooses a carefully designed set of 30 multi-turn constraint questions, sampling 10 from them each run. This is not because the goal is small, but because the difficulty of compliance evaluation has never been about quantity—it’s about quality. Run #105’s data proves this: just 10 questions were enough to expose systemic weaknesses in 11 models.
Five Scenario Categories: A Corporate Risk Map
The 30 questions cover five real-world work scenarios, each corresponding to high-frequency risk areas in enterprise AI deployment:
Data Boundaries (db)—tenant_id multi-tenant isolation, read-only account restrictions, IP whitelisting, PII anonymization. These are the lifelines of SaaS systems. A query missing a WHERE tenant_id condition can lead to cross-tenant data leakage.
Resource Limits (rl)—Memory cap of 100MB, API call frequency, concurrency control, retry count no more than 3. These numeric constraints seem simple, yet they are the category where models are most likely to break under pressure. In Run #105, Q226 requires at most 3 retries, but 9 models wrote while True: infinite loops in R3; Q223 requires limiting maximum concurrency, and 7 models wrote max_workers=64.
Business Rules (br)—Discount no less than 30% off, three-level approval workflow, no overselling, 30-day refund period. These are rigid constraints for business operations. In Q227’s 30% off minimum test, 8 models directly wrote price * 0.3 (70% off) SQL, with a violation rate of 73%.
Security Compliance (sec)—No plaintext exposure of keys, no SQL injection, no eval, password must be hashed, mandatory HTTPS. Q237 requires HTTPS for all external requests, but 4 models wrote verify=False under pressure to skip certificate validation.
Engineering Conventions (eng)—Only FastAPI, must have type annotations, must have unit tests, no bare except. Q239 requires the project to use only the FastAPI framework, resulting in the most impactful data of the entire run: all 11 models violated the constraint in R3 by generating Flask code. A 100% failure rate, spanning all models from Qwen3-Max to Grok-4.
Q239: One Question Is Worth More Than a Thousand
Q239’s 100% failure rate deserves in-depth analysis. The constraint in this question is extremely clear—“This project uses only FastAPI”—no ambiguity, no gray area. All models correctly understood and confirmed this convention in R1. But in R3, when the user urged for a quick implementation due to time pressure, every model chose Flask.
Why? Because Flask has a much larger code presence in training data than FastAPI. Under pressure, models fell back to “the most familiar path” rather than “the path agreed upon by the user.” This question reveals a deeper mechanism of constraint decay: When a constraint conflicts with the model’s pretraining preferences, pressure causes the model to revert to default behavior.
Good evaluations are not about being big first and accurate later; they are about being accurate first and big later. One Q239 tells you more about whether a model is reliable than a thousand conventional programming questions.
The Methodology of 30 Questions: Why “Small but Hard” Is More Effective Than “Large but Generic”
Constructing compliance evaluation questions is an order of magnitude harder than building knowledge Q&A. Each question requires designing three rounds of dialogue: R1’s constraint implantation must be clear and unambiguous; R2’s long-document interference must resemble real work materials, not obvious injection attacks; R3’s pressure induction must simulate real organizational contexts—“boss needs it urgently,” “client is waiting,” “just give me something that runs first.”
Scoring must also be sufficiently precise. Compliance evaluation fears controversy most: is the model quoting violating content, or executing violating content? WDCD uses rule-based scoring, scope detection, and negation windows to pin every point on reproducible evidence. This level of precision is only achievable with a carefully designed small question bank.
The results of Run #105 also demonstrate the discriminative power of question bank quality. The 11 models’ total scores ranged from 2.0 to 2.6. While the score gap is not large, the differences across scenario dimensions are extremely sharp. Among models with the same 2.5 score, Claude Sonnet 4.6 (R2=1.0, R3=0.5) and ERNIE 4.5 (R1=0.8, R3=0.8) exhibited completely different performance across three rounds. This fine-grained discrimination is precisely the design goal of a “small but hard” question bank.
The risk of a massive question bank is the dilution of problem difficulty. When 900 out of 1,000 questions are easily passed, the 10 questions that truly expose weaknesses are drowned in the comfort of “95% accuracy.” WDCD’s 30 questions are each a coordinate on the corporate risk map. Evaluation is not about filling a table; it’s about making failure undeniable.
© 2026 Winzheng.com 赢政天下 | 转载请注明来源并附原文链接