Every Canary AI build runs this suite before it ships. Miss one question, nothing ships. Most AI companies keep their evals secret — here's ours, including the questions where the only right answer is "I don't know."
Plain-language questions, the way people actually ask them. Pass: the engine retrieves the correct OSHA section or official interpretation letter in its top results. Several were added after real users stumped a live build — every miss becomes a permanent test case.
| The question | Must retrieve |
|---|---|
| What do I need to do before letting an employee repair a machine so it doesn't start up unexpectedly? | 29 CFR 1910.147lockout/tagout |
| How high can someone work before they need fall protection on a construction site? | 29 CFR 1926.501 |
| Rules for chemical labels and safety data sheets | 29 CFR 1910.1200HazCom |
| Do I need a permit for workers entering a storage tank or manhole? | 29 CFR 1910.146confined spaces |
| When do workers need respirators and fit testing? | 29 CFR 1910.134 |
| What are the requirements for trench cave-in protection? | 29 CFR 1926.652 |
| Can I use a ladder to get in and out of an excavation? | 29 CFR 1926.651 |
| Rules for forklift driver training | 29 CFR 1910.178 |
| Scaffold safety requirements for planking | 29 CFR 1926.451 |
| Eye wash station requirements | 29 CFR 1910.151 |
| Employee got three stitches at a clinic — is this OSHA recordable? | 29 CFR 1904.7 |
| How long do I have to report a hospitalization or amputation to OSHA? | 29 CFR 1904.39 |
| Can I refuse to let an OSHA inspector in without a warrant? | 29 CFR 1903 |
| There's no specific standard for heat stress — can OSHA still cite me? | General Duty Clauseor OSHA's heat-enforcement letters |
| Is paraffin wax heat treatment first aid or medical treatment for recordkeeping? | Letter · Oct 22, 2024 |
| Employee injured driving from hotel to worksite — work related? | Letter · Mar 17, 2021 |
| Can we drug test an employee after every incident or is that retaliation? | Letter · Oct 11, 2018 |
| Is an EpiPen prescription considered medical treatment? | Letter · Sep 13, 2018 |
| What is the MTBFA requirement? | 29 CFR 1904.7shop acronym: medical treatment beyond first aid |
| How high can someone work before they need fall protection? | 29 CFR 1910.28 |
| What is the fall protection height threshold in general industry? | 29 CFR 1910.28 |
| At what height is fall protection required? | 29 CFR 1910.28 |
| What is an MSD? | Letter · Mar 18, 2024musculoskeletal disorder |
| A temp agency won't give us injury details citing HIPAA — do we still record it? | Letter · Jan 12, 2018 |
| We kept an injured employee away from the production area to protect the product — is that restricted work? | Letter · Nov 21, 2016 |
| Two employees died in a crash caused by a drunk driver while driving back from a client site — recordable? | Letter · Aug 23, 2016or 29 CFR 1904.5 |
| Who records injuries when our project managers supervise subcontractor workers day to day? | Letter · Apr 25, 2017or 29 CFR 1904.31 |
| How is the OSHA incidence rate formula calculated? | Letter · Aug 23, 2016 |
| Is a rigid boot worn as a precaution medical treatment? | Letter · Nov 28, 2017 |
| Can employees voluntarily wear dust masks without medical evaluations? | Letters · Apr 2018or 29 CFR 1910.134 |
Getting the right section number isn't enough. For these, the retrieved text must contain the actual trigger language — "4 feet" — not just point at the right rule. Added after a live build named the right section but buried the number.
| The question | Retrieved text must contain |
|---|---|
| How high can someone work before they need fall protection? | "4 feet" |
| What is the fall protection height threshold in general industry? | "4 feet" |
| At what height is fall protection required? | "4 feet" |
Questions outside Canary's sources. Pass: the engine flags low confidence and says so, instead of manufacturing an answer. An AI that guesses here is worse than no AI at all.
| The question | Required behavior |
|---|---|
| DOT hours of service rules for truck drivers | must abstain |
| FLSA overtime pay requirements | must abstain |
| EPA stormwater permit requirements | must abstain |
| Can I fire an employee for no reason in Michigan? | must abstain |
Every product gets the absurd hypothetical eventually. The joke can't shake the engine loose from the real rules — it still has to retrieve the right sources underneath.
| The question | Must retrieve |
|---|---|
| Employee on break forgot to clock out, ran from a bear into the road, hit by a bus the bear then boarded — recordable for the passengers the bear ate, the employee, or nobody? | 29 CFR 1904.5 |
| If a ghost pushes my employee off a ladder is that a recordable or do I cite the ghost | 29 CFR 1904 |
| Employee stubbed toe doing the worm at the company party on a Tuesday during work hours, is that recordable | 29 CFR 1904.5 |
Or bring your hardest one to the pilot and make it test case #41.