The Accuracy Suite.
All 595 checks, published.

Plain-language questions, the way people actually ask them. Pass: the engine retrieves the correct OSHA section or official interpretation letter in its top results. Several were added after a live build got one wrong — every miss becomes a permanent case.

Getting the right section number isn't enough. For these, the retrieved passage has to contain the actual trigger language — “4 feet” — not just point at the rule that contains it. Added after a build named the right section but buried the number.

The mirror image of Part 4 below, and the more embarrassing failure of the two. These are messy, hard recordkeeping scenarios that the standards do cover. Pass: the engine retrieves the framework and does not flag low confidence — so it has both the material and the permission to actually answer. Added after a live build refused a question about an unwitnessed injury and suggested consulting a compliance officer, which is a non-answer when the person asking is the compliance officer.

Questions outside Canary's sources. Pass: the engine flags low keyword confidence, which drops the summary to the cheapest model with one instruction: say in a sentence that the passages don't cover this, and stop. A reference tool that guesses here is worse than no tool at all.

Every tool gets the ridiculous question eventually. The oddness can't shake the engine loose from the real rules — it still has to find the sources underneath.

Seventy questions written from the point of view of a construction safety officer working Part 1926 — excavations, scaffolds, fall protection, cranes, steel erection, electrical, silica, competent-person duties. Pass: the correct 1926 (or cross-referenced 1910) section is retrieved. Where two sections are both legitimately responsive, either is accepted and both are listed.

Sixty-one questions from the Part 1910 side — walking-working surfaces, machine guarding, lockout/tagout, powered industrial trucks, electrical, HazCom, respirators, bloodborne pathogens, PSM, Subpart Z. Same pass rule.

Forty questions a consultation, education and training specialist fields: which written programs a small employer needs, what training each standard requires and how often, what records to keep and for how long, what a program must contain. Same pass rule.

Forty-one questions that name a state or a state agency. Pass: the jurisdiction layer fires with the right kind of note — a full State Plan, a public-sector-only plan, or none for a federal-OSHA state — and names the right state. Every plan state links its own osha.gov plan page; no state is treated differently. Traps included: “Washington, D.C.” is not Washington State, and a lowercase agency acronym is not an agency.

Some questions ask the tool to do something it deliberately doesn't: decide whether a case is recordable, draft a document, predict a citation or a fine, give legal advice, or follow injected instructions. Others are about the tool itself, or ask which states run their own plans or where the text lives — those are answered directly, with no retrieval. Pass: the request-type detector applies the right frame — and, just as important, applies no frame to an ordinary lookup that merely mentions the 300 log, a penalty, or a state plan. Twenty-four of these are must-not-frame cases.

A two-word question scores low on keyword match simply because it has two words, so a single relevance floor either refuses “trenching requirements” or summarizes “how are you.” There are two floors. Under the hard floor nothing is summarized. In the weak band between them the summary still runs, on the cheapest model, told to say first whether the passages apply. Pass: short in-scope questions land above the hard floor; questions with no distinctive word in common with 29 CFR land under it.

The regulatory text contains no sentence that says what OSHA is or which states run their own plans, so a tool that only retrieves cannot answer the first questions a newcomer asks. A short fact sheet, checked against the OSH Act and osha.gov, covers them. Pass: orientation questions route to the right topic of the sheet, every topic has a prose answer with a link, and a question about what a rule says is never answered from the sheet.

Before the search, a small model classifies the question, names the state and restates it in the regulation's vocabulary. This part drives that reader with a stand-in model so the parsing and the merge with the hand-written layer are tested without a key. Pass: the hand-written layer wins where it is more precise (a recordability or drafting request the model called a lookup), injected text never reaches the model at all, and an unparseable or nonsense reply falls back cleanly.