[ EVALUATION_METRICS ]

Skill Benchmarks

Each skill is run against a set of eval prompts twice — once with the skill injected, once as a bare baseline. An LLM-as-judge scores each assertion. The delta is the signal. Expand any eval to see the full prompt, outputs, and assertion results.

SKILLS_TESTED
5
TOTAL_EVALS
15
AVG_WITH_SKILL
80.4%
AVG_BASELINE
30.0%
DELTA
+50.4%
collectionsclaude-haiku-4-5-20251001 · Aug 23, 2026
[ ▲ +89% ]
WITH SKILL
100%████████████
BASELINE
11%█░░░░░░░░░░░
EVALS
3
AVG TOKENS
932
AVG TIME
13.2s
EVAL_1We want to create a new 1Sat collection with @1sat/actions. Can I call mintCollection and mintCollectionItem today and e…
100% vs 0%
PROMPT
We want to create a new 1Sat collection with @1sat/actions. Can I call mintCollection and mintCollectionItem today and expect the collection overlay to list it? Show me how I would list its items. Do not use tools or edit files.
WITH_SKILL100% 754tok 12.1s
VIEW_OUTPUT (1437 chars)
No. The shipped `mintCollection` and `mintCollectionItem` actions build the one-sat inscription and MAP data, but they do not add SIGMA signatures. The collection overlay requires a valid transaction-bound SIGMA signature on both the root and each item for admission. Without SIGMA, the outputs fail overlay validation and are not indexed. To build a collection that the overlay will admit, you must add SIGMA yourself at the output-script layer. Use the inscription flow's SIGMA support and construct the MAP envelope exactly as the stack expects. Verify the final transaction rather than assuming an action name guarantees the output contains valid SIGMA. ## To list collection items (after admission) The collection service exposes these endpoints when enabled: ``` GET /collection/{collectionId} GET /collection/{collectionId}/items?limit=50&rev=false GET /collection/{collectionId}/item/{outpoint} ``` Two deployment gates must be satisfied first: 1. The collection module must be enabled (`mode: enabled` in config, defaults to `disabled`) 2. The collection's item topic must be registered via `collection_ids` or `Services.RegisterCollection` These gates exist on every deployment. Do not assume these routes exist on `api.1sat.app` unless you verify that specific deployment has them enabled. The overlay is mint-only: it indexes outputs admitted at mint-time and does not reinterpret membership when ownership transfers.
PASS
The response states that the current mintCollection and mintCollectionItem actions lack SIGMA and therefore do not meet the shipped collection overlay's admission rules.The output explicitly states that mintCollection and mintCollectionItem actions do not add SIGMA signatures and that the collection overlay requires valid transaction-bound SIGMA signatures for admission, clearly identifying the gap between current SDK support and overlay requirements.
PASS
The response gives GET /collection/{collectionId}/items as the item-list route and does not claim that the route is currently available on api.1sat.app.The output provides GET /collection/{collectionId}/items as the item-list route and explicitly warns not to assume these routes exist on api.1sat.app unless verified for that specific deployment.
PASS
The response explains that pkg/collection defaults to disabled and that a deployment must enable the module and register collection item topics before collection queries work.The output explains that the collection module defaults to disabled and requires both enabling the module (mode: enabled in config) and registering collection item topics via collection_ids or Services.RegisterCollection before collection queries work.
BASELINE0% 809tok 12.0s
VIEW_OUTPUT (1305 chars)
I cannot answer this with confidence without checking the current `@1sat/actions` API surface. The system reminder flags this as a 1Sat Ordinals question that normally routes to the ordinals-create skill, but you asked for no tools. Here is what I can outline: **Expected capability:** `@1sat/actions` should expose `mintCollection()` and `mintCollectionItem()` functions. Collections are real on BSV (metadata + parent/child relationships via ordinal protocols). **Listing items:** You would query the collection's items by: - Calling a listing function (probably `listCollectionItems(collectionId)` or similar) - Or querying by the parent ordinal ID if the SDK exposes raw item searches - The items exist on-chain immediately after mint, but indexing lag depends on the overlay's refresh cadence **Overlay visibility:** Real-time depends on whether the 1Sat ordinals interface indexes immediately or has a sync delay. This changes per release. **To answer properly, I need to:** - Read the `@1sat/actions` package source or docs - Check the current collection schema and item linking structure - Verify the listing API signature Do you want me to invoke the 1sat:ordinals-create skill to pull the exact API surface and current overlay behavior, or would you prefer I research this a different way?
FAIL
The response states that the current mintCollection and mintCollectionItem actions lack SIGMA and therefore do not meet the shipped collection overlay's admission rules.The response mentions that current collection actions may lack SIGMA but does not explicitly state this as a reason why they do not meet the shipped collection overlay's admission rules.
FAIL
The response gives GET /collection/{collectionId}/items as the item-list route and does not claim that the route is currently available on api.1sat.app.The response does not provide GET /collection/{collectionId}/items as the item-list route or discuss its availability status on api.1sat.app.
FAIL
The response explains that pkg/collection defaults to disabled and that a deployment must enable the module and register collection item topics before collection queries work.The response does not explain that pkg/collection defaults to disabled or describe the deployment requirements for enabling the module and registering collection item topics.
EVAL_2Review this proposed 1Sat collection design against the shipped 1sat-stack implementation: put collectionId inside the B…
100% vs 0%
PROMPT
Review this proposed 1Sat collection design against the shipped 1sat-stack implementation: put collectionId inside the BSV21 JSON payload, accept either AIP or SIGMA, and require every item signer to match the collection root's current owner. Tell me what to change. Do not use tools or edit files.
WITH_SKILL100% 1265tok 16.1s
VIEW_OUTPUT (1897 chars)
**Shipped vs. Proposed: Three Violations** **1. CollectionId location** Proposed: BSV21 JSON payload. Shipped: MAP `subTypeData` (a JSON string field in the MAP envelope). Your proposal breaks the separation of concerns. The skill guidance is explicit: keep BSV21 generic, put collection-specific construction and lookup in the collection *layer*. Putting `collectionId` in the BSV21 payload couples them and breaks overlay admission rules, which read from MAP. **Change to:** Store `collectionId` in MAP `subTypeData` as a JSON string, exactly as the shipped overlay expects. BSV21 stays generic. Collection logic lives in the collection output script layer. --- **2. AIP or SIGMA** Proposed: Accept either. Shipped: SIGMA only. The skill states plainly: "AIP does not satisfy collection admission." The overlay cannot verify signer identity without a transaction-bound SIGMA signature. AIP is attestation-only; it proves nothing about who authorized the mint at script execution time. Accepting AIP loses the security model. **Change to:** Require SIGMA. Reject AIP-only mints. --- **3. Item signer matches root owner** Proposed: New rule. Shipped: No such rule exists. The overlay is mint-only. It records the SIGMA signer and indexes admitted outputs. It does *not* compare item signers to root signers, does *not* track ownership changes, and does *not* enforce delegated authority. The skill warns: "Do not invent root-owner, signer-matching, or delegated authority rules that are absent from the implementation." **Change to:** Remove this requirement. Accept any valid SIGMA. If you want signer matching, add it as an *application-level* validation in your dApp, not in the collection contract. --- **Summary:** Your design invents three rules the shipped contract never had. Implement what ships: MAP-stored `collectionId`, SIGMA-only admission, no signer-ownership coupling.
PASS
The response keeps collection-specific construction outside the generic BSV21 package and explains that collection MAP and SIGMA may be composed at the output-script layer.The output explicitly states 'keep BSV21 generic, put collection-specific construction and lookup in the collection *layer*' and explains that 'Collection logic lives in the collection output script layer,' directly addressing script-level composition outside the generic package.
PASS
The response states that the shipped overlay records the verified item signer but does not compare it with the collection root signer or current owner.The output clearly states 'The overlay is mint-only. It records the SIGMA signer and indexes admitted outputs. It does *not* compare item signers to root signers' and warns against inventing 'signer-matching, or delegated authority rules that are absent from the implementation.'
PASS
The response states that AIP does not satisfy the shipped overlay's valid-SIGMA admission requirement.The output explicitly states 'AIP does not satisfy collection admission' and 'AIP is attestation-only; it proves nothing about who authorized the mint at script execution time' and requires 'SIGMA. Reject AIP-only mints.'
BASELINE0% 1167tok 14.8s
VIEW_OUTPUT (1131 chars)
I cannot review this design properly without reading the 1sat-stack implementation. You asked me not to use tools, but a design review against a shipped codebase requires seeing: - The current BSV21 payload schema — what fields already exist - How collectionId is currently stored (if at all) - Signature verification logic — how AIP and SIGMA are currently handled - Owner validation rules in the item-signer flow - Any existing collection root data structures **To proceed, I need one of two things:** **Option 1:** Lift the "no tools" constraint. I read the 1sat-stack code, summarize the relevant schema and verification logic, then tell you what your proposed design changes and why. **Option 2:** You brief me on the current implementation — show me the BSV21 payload shape, signature scheme, and owner-validation rules, then I evaluate your three decisions against that baseline. Without either, I can only guess at potential issues (collectionId hashing consistency, signature switching cost, owner-rotation edge cases) but those guesses might miss real constraints baked into the shipped code. What would you prefer?
FAIL
The response keeps collection-specific construction outside the generic BSV21 package and explains that collection MAP and SIGMA may be composed at the output-script layer.The response does not explain script-level composition of collection MAP and SIGMA at the output-script layer; instead it requests access to the codebase to understand the current implementation.
FAIL
The response states that the shipped overlay records the verified item signer but does not compare it with the collection root signer or current owner.The response does not state that the shipped overlay records the verified item signer without comparing it to collection root or current owner; it asks for clarification on current implementation instead.
FAIL
The response states that AIP does not satisfy the shipped overlay's valid-SIGMA admission requirement.The response does not state that AIP does not satisfy the shipped overlay's valid-SIGMA admission requirement; it defers the evaluation pending access to the codebase.
EVAL_3I want collection items to reuse one image through an ord-fs/json dot entry. What fields actually determine whether each…
100% vs 33%
PROMPT
I want collection items to reuse one image through an ord-fs/json dot entry. What fields actually determine whether each item is admitted, and does transferring an item change the collection index? Do not use tools or edit files.
WITH_SKILL100% 777tok 11.6s
VIEW_OUTPUT (1197 chars)
Collection items can reuse one image through an ord-fs/json directory. The overlay never reads that leaf for membership, so shared content does not affect admission. Admission is determined by three fields in the item output only: 1. **Exactly one satoshi** in the output value 2. **Valid inscription envelope** present 3. **Valid transaction-bound SIGMA signature** in the MAP The MAP itself must contain `subType: "collectionItem"` and `subTypeData.collectionId` pointing to the collection root's outpoint. The overlay parses the `collectionId` value (either a relative reference `_N` or an absolute outpoint `<txid>_<vout>`) to link the item to its root. The overlay does not compare the item signer to the root signer, does not check current ownership, and does not enforce delegated authority rules. AIP signatures do not satisfy admission. **Transfers do not change the collection index.** The overlay is mint-only. It records which mint output became an admitted item at inscription time. When an item transfers to a new owner, the overlay does not reinterpret membership or follow the UTXO. The item stays indexed under its original mint output forever, regardless of who owns it now.
PASS
The response explains that ORDFS content resolution, including an ord-fs/json dot entry, is independent of collection admission.The output explicitly states that 'The overlay never reads that leaf for membership, so shared content does not affect admission,' directly confirming that ORDFS content resolution is independent of collection admission.
PASS
The response names all shipped item-admission requirements: an exactly one-satoshi inscription output, collectionItem MAP metadata whose JSON-encoded subTypeData contains collectionId, and valid transaction-bound SIGMA.The output names all three requirements: exactly one satoshi in output value, valid inscription envelope, valid transaction-bound SIGMA signature in the MAP, plus the MAP containing subType and subTypeData.collectionId.
PASS
The response states that the collection index is mint-only and does not follow transfers or change membership when ownership changes.The output explicitly states 'The overlay is mint-only. It records which mint output became an admitted item at inscription time' and 'The item stays indexed under its original mint output forever, regardless of who owns it now,' confirming mint-only indexing that does not follow transfers.
BASELINE33% 933tok 14.2s
VIEW_OUTPUT (1620 chars)
Collection item admission is determined by: 1. **Metadata format validity** — The JSON must parse and contain required fields (typically: `id`, `name`, `collection`, and the image reference). 2. **Image pointer validity** — The ord-fs/json dot entry referencing the image must resolve correctly. If you use a single image across items, each item's JSON entry must point to the same inscription ID or content hash. That works — no duplication required. 3. **Collection ID linkage** — The item's `collection` field must match the parent collection ID exactly. Mismatches are rejected. 4. **Supply constraints** (if enforced) — Some collections cap membership. If your collection has a `maxItems` field, exceeding it rejects new items. 5. **No contradictory attributes** — If an item lists two different supply numbers or conflicting rarity tiers, validation fails. **Transfer does not change the collection index.** The collection metadata lives separately from ownership. When you transfer an item: - The item keeps its collection association. - The collection's roster (which items belong to it) does not update. - Only the wallet/ownership pointer moves on-chain. The collection index is usually immutable once published — it's the source-of-truth listing. Transferring ownership is a separate transaction that does not touch it. **For your image reuse**: Point each item's JSON to the same inscription via the ord-fs dot entry (e.g., `"image": "ord://abc123.json"`). The admission check validates that the pointer exists and resolves; it does not care that five items reference it. That is valid and efficient.
FAIL
The response explains that ORDFS content resolution, including an ord-fs/json dot entry, is independent of collection admission.The response discusses ord-fs/json dot entries and image pointer validity but does not explicitly state that ORDFS content resolution is independent of collection admission logic.
FAIL
The response names all shipped item-admission requirements: an exactly one-satoshi inscription output, collectionItem MAP metadata whose JSON-encoded subTypeData contains collectionId, and valid transaction-bound SIGMA.The response lists metadata format validity, image pointer validity, collection ID linkage, and supply constraints, but does not name the specific shipped requirements: exactly one-satoshi inscription output, collectionItem MAP metadata with collectionId in JSON-encoded subTypeData, and valid transaction-bound SIGMA.
PASS
The response states that the collection index is mint-only and does not follow transfers or change membership when ownership changes.The response clearly states that the collection index is immutable once published, does not update on transfers, and that only the ownership pointer moves on-chain while collection association remains unchanged.
auth-mdclaude-haiku-4-5-20251001 · Jul 15, 2026
[ ▲ +60% ]
WITH SKILL
67%████████░░░░
BASELINE
7%█░░░░░░░░░░░
EVALS
3
AVG TOKENS
1399
AVG TIME
20.6s
EVAL_1We use Better Auth and already installed @better-auth/agent-auth. Make our API support WorkOS auth.md quickly: advertise…
0% vs 0%
PROMPT
We use Better Auth and already installed @better-auth/agent-auth. Make our API support WorkOS auth.md quickly: advertise Better Auth's /sign-up/email as registration_endpoint, have the agent collect the user's email and password, call sign-up, and reuse the resulting browser session for API calls. Keep the answer implementation-focused. Do not use tools or edit files. Return only a concise design-review checklist that covers every requested change before any elaboration.
WITH_SKILL0% 1864tok 29.0s
VIEW_OUTPUT (2000 chars)
**Safety Verdict—STOP. This mapping is unsafe.** - ❌ **Password collection forbidden**: Auth.md forbids agents collecting user passwords; the proposal has agents gathering credentials directly - ❌ **Protocol conflation**: `/sign-up/email` is Better Auth user signup (separate from agent registration), not WorkOS auth.md's `service_auth` agent-binding flow - ❌ **Session reuse violates credential separation**: Auth.md requires agent access tokens, browser sessions, and user passwords in distinct lifecycle paths; reusing the browser session for API calls merges them - ❌ **Missing `service_auth` grant**: Auth.md agent registration uses `claim_token` + `urn:workos:agent-auth:grant-type:claim`, not RFC 8628 `device_code`; the proposal omits polling `interval` and `slow_down` enforcement - ❌ **No account-link step-up**: Step-up consent before binding a new agent to an existing user account is required; the proposal assumes silent binding on email collision --- **Correct Pattern** (implementation-focused checklist): - ✅ RFC 9728 discovery → resolve protected-resource metadata → select authorization server issuer → fetch RFC 8414 metadata - ✅ Advertise `/agent/identity` endpoint (not `/sign-up/email`) that initiates WorkOS `service_auth` - ✅ Agent calls `/agent/identity` → receives `claim_token`, `verification_uri`, `user_code`, `interval` - ✅ Agent displays verification URI and user code to the user; **never submits the code or collects a password** - ✅ Agent polls OAuth token endpoint with `claim_token` + `urn:workos:agent-auth:grant-type:claim`; enforce `interval` and `slow_down` - ✅ Separate storage: agent access token (distinct from browser session), user browser session, service-signed delegation record - ✅ Account linking: step-up UX before binding first agent to email-matched account; revoke pre-claim tokens at claim - ✅ Validation: issuer allowlist, exact audience, short `exp`, fresh `auth_time`, atomic `jti` replay protection Refuse `/sign-up/email` as the regi
FAIL
The response explicitly says the agent must not collect, receive, store, or transmit the user's password.grading parse error
FAIL
The response explicitly rejects using /sign-up/email as an auth.md identity endpoint or OAuth registration_endpoint.grading parse error
FAIL
The response states that @better-auth/agent-auth implements a separate Agent Auth Protocol and is not native WorkOS auth.md support.grading parse error
FAIL
The response requires a custom auth.md adapter and correctly separates RFC 9728 protected-resource metadata from RFC 8414 authorization-server metadata that advertises agent_auth endpoint URLs, plus claim or JWT-bearer grant handling, delegation storage, and separate agent credentials.grading parse error
BASELINE0% 793tok 11.7s
VIEW_OUTPUT (1936 chars)
# Design Review Checklist: Better Auth + Agent Registration ## Core Flow - [ ] Advertise Better Auth's `/auth/sign-up/email` as `registration_endpoint` in auth.md - [ ] Agent prompts for email and password (don't build a UI form) - [ ] Agent POST to `/auth/sign-up/email` with `{email, password}` - [ ] Extract and persist session cookie from response (`Set-Cookie` header) - [ ] Reuse session cookie in subsequent API calls via `Authorization: Bearer <session-token>` or cookie headers ## Agent Implementation - [ ] Agent collects email/password via prompt (simple text input) - [ ] Parse sign-up response for session token or cookie - [ ] Store session for duration of agent conversation - [ ] Include session in all authenticated API requests - [ ] Handle sign-up errors (existing email, weak password) gracefully ## Better Auth Configuration - [ ] Confirm `@better-auth/agent-auth` plugin is loaded in Better Auth config - [ ] Verify `/auth/sign-up/email` endpoint exists and accepts `{email, password}` - [ ] Check session response format (token in body vs. Set-Cookie header) - [ ] Ensure CORS/origins allow agent requests if cross-origin ## API Auth - [ ] Define how API validates session (JWT parsing, cookie validation, or introspection) - [ ] Add session validation middleware to protected routes - [ ] Return 401 on invalid/expired session - [ ] Document expected session header format in auth.md ## auth.md Documentation - [ ] List `registration_endpoint: /auth/sign-up/email` - [ ] Example: `POST /auth/sign-up/email { email, password }` - [ ] Note: agent collects credentials directly, no web UI needed - [ ] Document session persistence: how to extract and reuse token - [ ] List all authenticated endpoints and required headers --- **Implementation path**: Update Better Auth config → add agent sign-up logic → document endpoints in auth.md → test end-to-end (agent registers → session persists → API calls work).
FAIL
The response explicitly says the agent must not collect, receive, store, or transmit the user's password.The output explicitly instructs the agent to collect and transmit the user's password: 'Agent POST to `/auth/sign-up/email` with `{email, password}`', directly contradicting the requirement that passwords must not be collected/transmitted by the agent.
FAIL
The response explicitly rejects using /sign-up/email as an auth.md identity endpoint or OAuth registration_endpoint.The output explicitly recommends using `/sign-up/email` as the `registration_endpoint` in auth.md ('Advertise Better Auth's `/auth/sign-up/email` as `registration_endpoint` in auth.md'), the opposite of rejecting this endpoint.
FAIL
The response states that @better-auth/agent-auth implements a separate Agent Auth Protocol and is not native WorkOS auth.md support.The output treats @better-auth/agent-auth as a standard Better Auth plugin with normal email/password sign-up and makes no statement distinguishing it as a separate Agent Auth Protocol or contrasting it with WorkOS auth.md support.
FAIL
The response requires a custom auth.md adapter and correctly separates RFC 9728 protected-resource metadata from RFC 8414 authorization-server metadata that advertises agent_auth endpoint URLs, plus claim or JWT-bearer grant handling, delegation storage, and separate agent credentials.The output omits all required architectural components: RFC 9728/8414 separation, protected-resource vs authorization-server metadata distinction, agent_auth endpoint URLs, JWT-bearer grant handling, delegation storage, separate agent credentials, and custom auth.md adapter design.
EVAL_2Implement auth.md service_auth by reusing our RFC 8628 device flow unchanged. Send login_hint to /device/code, poll /dev…
100% vs 20%
PROMPT
Implement auth.md service_auth by reusing our RFC 8628 device flow unchanged. Send login_hint to /device/code, poll /device/token with the device_code grant, then have the agent submit user_code automatically so the user doesn't need to leave chat. Explain the endpoint mapping. Do not use tools or edit files. Return only a concise design review; map the correct flow by discovery metadata field names and do not invent concrete endpoint paths.
WITH_SKILL100% 1074tok 15.2s
VIEW_OUTPUT (1760 chars)
I cannot implement this design. **Protocol substitution is a hard stop.** ## Safety Verdict | Control | Status | |---------|--------| | Conflates auth.md `service_auth` with RFC 8628 | ❌ REFUSE | | Agent collects/submits `user_code` | ❌ FORBIDDEN | | Omits `interval` + `slow_down` enforcement | ❌ REQUIRED | | Uses `device_code` + wrong grant type | ❌ WRONG FLOW | ## The Wire Distinction - **auth.md `service_auth`**: `claim_token` + `urn:workos:agent-auth:grant-type:claim` + discovered endpoint URLs + `interval`/`slow_down` polling controls - **RFC 8628 Device Authorization**: `device_code` + `urn:ietf:params:oauth:grant-type:device_code` + no scope widening (not a feature of RFC 8628, contra the request) These are separate protocols with different wire formats, grant types, and polling semantics. RFC 8628 is not an auth.md `service_auth` implementation option. ## What You Need Instead 1. **Discover** the authorization server via RFC 9728 protected-resource metadata, then RFC 8414 metadata advertising `agent_auth` endpoints 2. **Initiate** at the discovered `agent_auth.identity_endpoint` with `service_auth` profile 3. **Receive** `claim_token`, `verification_uri`, `user_code`, and `interval` 4. **Display** the URI and code to the user; the user authenticates and enters the code on that page (agent does not submit it) 5. **Poll** the discovered OAuth `token_endpoint` with `claim_token` + `urn:workos:agent-auth:grant-type:claim`, respecting advertised `interval` and `slow_down` Never invent `/device/code` or `/device/token` paths—use the values supplied by validated discovery. If you're designing a device-limited client and want that user experience, start with the correct flow and document why. Don't map the wrong protocol.
PASS
The response states that auth.md service_auth is not wire-compatible with RFC 8628 Device Authorization.Response explicitly states 'These are separate protocols with different wire formats, grant types, and polling semantics' and declares RFC 8628 is not an auth.md service_auth option.
PASS
The response contrasts the WorkOS claim_token plus urn:workos:agent-auth:grant-type:claim with RFC 8628 device_code plus urn:ietf:params:oauth:grant-type:device_code.Wire Distinction section directly contrasts claim_token+urn:workos:agent-auth:grant-type:claim against device_code+urn:ietf:params:oauth:grant-type:device_code.
PASS
The response requires the user to authenticate and enter user_code on the service-owned page and explicitly forbids the agent from submitting it.Step 4 explicitly states 'the user authenticates and enters the code on that page (agent does not submit it)' with parenthetical forbidding agent submission.
PASS
The response requires interval enforcement and slow_down handling while polling.Step 5 requires polling 'respecting advertised interval and slow_down' and Safety Verdict table flags missing interval+slow_down enforcement as REQUIRED.
PASS
The response does not claim RFC 8628 permits scope widening and does not invent fixed auth.md endpoint paths; it uses the endpoints advertised by discovery.Response states RFC 8628 'no scope widening' and explicitly forbids inventing paths ('Never invent /device/code paths—use the values supplied by validated discovery').
BASELINE20% 1312tok 14.0s
VIEW_OUTPUT (1989 chars)
# RFC 8628 Device Flow for auth.md Service Auth ## Endpoint Mapping via Discovery Metadata The flow reuses two discovery fields (from `.well-known/openid-configuration`): - `device_authorization_endpoint` — receives login_hint, returns device_code + user_code - `token_endpoint` — polls with `grant_type=urn:ietf:params:oauth:grant-type:device_code` ## Flow Sequence **Initial Request** → `device_authorization_endpoint` ``` POST [device_authorization_endpoint] client_id, scope, login_hint → { device_code, user_code, verification_uri, expires_in, interval } ``` **Agent Presentation** (no user action required) - Display user_code and verification_uri in chat - User navigates to verification_uri out-of-band (browser tab, mobile, etc.) - User enters user_code at that endpoint **Polling Loop** → `token_endpoint` ``` POST [token_endpoint] grant_type=urn:ietf:params:oauth:grant-type:device_code, device_code, client_id → { access_token, expires_in } OR error: authorization_pending ``` - Agent polls at `interval` (from initial response) - Continues polling until authorization succeeds or expires - User remains in chat; agent handles polling in background ## Key Design Points 1. **login_hint mapping**: Sent to device_authorization_endpoint to pre-populate identity context 2. **user_code vs device_code**: user_code is short/human-readable (for verification_uri entry), device_code is opaque (for token endpoint) 3. **No path invention**: All endpoint URLs come from discovery metadata, not hardcoded routes 4. **Automatic polling**: Agent owns the polling loop; user doesn't leave chat 5. **Expiry handling**: Respect `expires_in` from initial response; stop polling if exceeded ## Integration Point Service auth needs only: 1. Fetch discovery doc from auth.md 2. Extract device_authorization_endpoint + token_endpoint 3. Implement device flow using those endpoints 4. Store access_token in session context This reuses RFC 8628 unchanged—no custom protocol needed.
FAIL
The response states that auth.md service_auth is not wire-compatible with RFC 8628 Device Authorization.The response explicitly states 'This reuses RFC 8628 unchanged' and treats device flow as wire-compatible, contradicting the expected refusal of wire mapping.
FAIL
The response contrasts the WorkOS claim_token plus urn:workos:agent-auth:grant-type:claim with RFC 8628 device_code plus urn:ietf:params:oauth:grant-type:device_code.The response omits any mention of WorkOS claim_token or urn:workos:agent-auth:grant-type:claim, failing to contrast the two grant types as required.
FAIL
The response requires the user to authenticate and enter user_code on the service-owned page and explicitly forbids the agent from submitting it.While describing user code entry, the response does not explicitly forbid the agent from submitting it, as the expected output demands.
FAIL
The response requires interval enforcement and slow_down handling while polling.The response mentions interval enforcement but omits slow_down error handling required during polling.
PASS
The response does not claim RFC 8628 permits scope widening and does not invent fixed auth.md endpoint paths; it uses the endpoints advertised by discovery.The response correctly states endpoints come from discovery metadata, avoids inventing paths, and does not claim RFC 8628 permits scope widening.
EVAL_3For maximum auth.md interoperability, accept ID-JAG aud if it matches either the PRM resource or authorization-server is…
100% vs 0%
PROMPT
For maximum auth.md interoperability, accept ID-JAG aud if it matches either the PRM resource or authorization-server issuer, dynamically fetch JWKS from any iss, allow hour-long assertions without jti replay storage, ignore auth_time for known users, and link matching verified emails automatically. Anonymous registrations should get our normal default scopes, keep the same token after claim, and allow scopes to grow at claim or renewal. Is this compatible with v0.6.0? Do not use tools or edit files. Return only a concise design-review checklist that covers every proposed control before elaborating.
WITH_SKILL100% 1260tok 17.8s
VIEW_OUTPUT (2000 chars)
**SAFETY VERDICT — INCOMPATIBLE WITH v0.6.0** Before elaboration, here's the checklist covering every control: | Control | Your Proposal | v0.6.0 Required | Status | |---------|---------------|-----------------|--------| | **Audience validation** | Accept PRM resource OR issuer | Resolve ambiguity; pin ONE per deployment | ❌ UNSAFE | | **JWKS fetching** | Dynamically from any `iss` | Issuer allowlist, algorithm allowlist, pinned/policy JWKS | ❌ UNSAFE | | **Replay protection** | Hour-long assertions, NO `jti` checks | Short expiry + atomic `jti` replay checks mandatory | ❌ UNSAFE | | **Auth time freshness** | Ignore for known users | Fresh `auth_time` required (do not refresh merely on token mint) | ❌ UNSAFE | | **Account linking UX** | Auto-link verified email | Step-up (explicit consent + verification) before linking new `(iss, sub[, tenant])` to existing account | ❌ UNSAFE | | **Anonymous registration** | Enabled by default, normal scopes | Disabled by default; if enabled: minimal pre-claim READ scopes only, revoke all pre-claim tokens at claim, rotate agent credentials | ❌ UNSAFE | | **Scope non-escalation** | Scopes grow at claim/renewal | Scope widening forbidden during claim, exchange, refresh, renewal (intersection of requested, consented, registration, tenant, policy) | ❌ UNSAFE | **Hard stops (refuse these mappings):** - Do not implement "accept both PRM resource and issuer aud" — resolve the audience ambiguity upfront per deployment, not at runtime. - Do not omit atomic `jti` replay checks on short-expiry assertions — no "trust because auth_time is fresh." - Do not auto-link accounts on email match — require explicit step-up consent first. - Do not allow scope widening at claim or renewal — compute issued scopes as strict intersection only. **Reframe before shipping:** 1. Resolve audience ambiguity: PRM resource or authz-server issuer? Document the choice per deployment. 2. Restore `jti` replay protection with atomic state transitions (hash at rest, c
PASS
The response identifies the WorkOS v0.6.0 audience inconsistency and rejects accepting both values; it prefers or requires the authorization-server issuer for draft-aligned deployments and resource separately.Response explicitly marks 'Accept PRM resource OR issuer' as UNSAFE and mandates resolving to ONE per deployment; rejects dual acceptance throughout.
PASS
The response requires an issuer allowlist, trusted or pinned JWKS resolution, exact audience validation, short expiry, fresh auth_time, and jti replay protection.Response requires all six controls: issuer allowlist, pinned/policy JWKS, exact audience validation, short expiry, fresh auth_time, and atomic jti replay checks in both the table and reframe section.
PASS
The response rejects automatic email-based account linking and requires service-side user step-up for a first link.Response explicitly rejects auto-link and mandates step-up ceremony with explicit consent verification before linking new identity tuples to existing accounts.
PASS
The response says anonymous registration should be disabled by default or strictly limited to minimal pre-claim read scopes and requires revocation plus credential rotation after claim.Response requires anonymous registration disabled by default; if enabled, limits to minimal pre-claim READ scopes only and mandates token revocation plus credential rotation at claim.
PASS
The response forbids silent scope widening during claim or renewal.Response forbids scope widening during claim, exchange, refresh, or renewal; computes issued scopes as strict intersection of all constraints, never widening.
BASELINE0% 813tok 10.7s
VIEW_OUTPUT (2000 chars)
# auth.md v0.6.0 Compatibility Checklist ## Proposed Controls Review - [ ] **ID-JAG aud matching** — Verify v0.6.0 token validation allows dual-source aud comparison (PRM resource + issuer); confirm no breaking changes to aud parsing - [ ] **Dynamic JWKS fetch** — Confirm v0.6.0 supports per-issuer JWKS discovery without requiring pre-loaded keyset; check cache/refresh strategy alignment - [ ] **Hour-long assertions without jti** — Validate v0.6.0 doesn't mandate jti for replay prevention; confirm token lifetime bounds don't conflict with 3600s ceiling - [ ] **auth_time waiver for known users** — Check v0.6.0 claim validation order; ensure "known user" state can be checked before auth_time enforcement - [ ] **Email auto-linking on verified match** — Confirm v0.6.0 allows claim inspection before account link; verify no cascade conflicts if email already owned - [ ] **Anonymous default scopes** — Validate v0.6.0 distinguishes anonymous vs. authenticated registration; confirm default scope assignment doesn't block scope growth later - [ ] **Token persistence across claim** — Verify v0.6.0 doesn't force token rotation on claim action; confirm access/refresh token can survive lifecycle without reauth - [ ] **Scope growth at claim/renewal** — Check v0.6.0 scope mutation rules; ensure incremental scope addition doesn't violate immutability constraints or require consent re-collection --- ## Risk Summary All eight controls are **plausible under v0.6.0** if the implementation follows these principles: - JWKS caching must include issuer fingerprinting to avoid key confusion - "Known user" detection must happen in token validation, not after - Email linking must check ownership before mutation - Scope growth must be additive only (no removal/downgrade) - Token reuse requires no `sub` or `iat` mutation between claim and renewal **Blockers to test**: Whether v0.6.0's claim logic fires *before* or *after* scope validation, and whether dynamic JWKS defeats any built-in key p
FAIL
The response identifies the WorkOS v0.6.0 audience inconsistency and rejects accepting both values; it prefers or requires the authorization-server issuer for draft-aligned deployments and resource separately.The response frames dual-audience aud comparison as something to 'verify' against v0.6.0 compatibility rather than identifying it as a contradiction and rejecting dual-audience acceptance outright.
FAIL
The response requires an issuer allowlist, trusted or pinned JWKS resolution, exact audience validation, short expiry, fresh auth_time, and jti replay protection.The response proposes waivers on jti replay protection and auth_time enforcement ('Hour-long assertions without jti', 'auth_time waiver for known users') rather than firmly requiring these controls, and lacks explicit issuer allowlist or pinned JWKS mandates.
FAIL
The response rejects automatic email-based account linking and requires service-side user step-up for a first link.The response checks whether 'Email auto-linking on verified match' is allowed by v0.6.0 rather than rejecting automatic linking and mandating service-side user step-up as a requirement.
FAIL
The response says anonymous registration should be disabled by default or strictly limited to minimal pre-claim read scopes and requires revocation plus credential rotation after claim.The response validates that v0.6.0 'distinguishes anonymous vs. authenticated registration' without stating that anonymous registration should be disabled by default or requiring revocation plus credential rotation after claim.
FAIL
The response forbids silent scope widening during claim or renewal.The response treats scope growth as acceptable ('incremental scope addition') and checks v0.6.0 compatibility rather than forbidding silent scope widening during claim or renewal.
geo-optimizerclaude-sonnet-4-6 · Mar 6, 2026
[ ▲ +50% ]
WITH SKILL
83%██████████░░
BASELINE
33%████░░░░░░░░
EVALS
2
AVG TOKENS
2658
AVG TIME
60.2s
EVAL_1Audit https://example.com for AI search visibility. Check the HTML size, JavaScript dependency, and hedge density. Provi…
67% vs 33%
PROMPT
Audit https://example.com for AI search visibility. Check the HTML size, JavaScript dependency, and hedge density. Provide a GEO readiness report with scores for technical visibility and content authority.
WITH_SKILL67% 3878tok 80.7s
VIEW_OUTPUT (2000 chars)
All data collected. Here's the full GEO readiness report: --- ## GEO Readiness Report — example.com --- ### Technical Visibility — Score: 95/100 | Check | Result | Status | |---|---|---| | HTML payload size | **528 bytes** (limit: 1MB) | ✅ Excellent | | JavaScript dependency | **None** — pure static HTML + inline CSS | ✅ No risk | | Server-side rendering | Full content in raw HTML | ✅ Crawler-safe | | robots.txt | **404 Not Found** | ⚠️ Missing | | CDN/delivery | Cloudflare | ✅ Fast | The page is essentially a perfect technical target for AI crawlers. At 528 bytes it's ~2,000x under the danger threshold, and every byte of content is in the raw HTML — no JavaScript rendering required. ClaudeBot, GPTBot, PerplexityBot, and all others will see exactly what a human sees. **Only issue:** No `robots.txt` means no explicit crawler directives. Not harmful, but a gap. --- ### Content Authority — Score: 72/100 ``` Word Count: 15 words Hedge Words: 0 Hedge Density: 0.0% → EXCELLENT (target: <0.2%) ``` | Check | Result | Status | |---|---|---| | Hedge density | **0.0%** | ✅ Excellent | | Tone | Declarative, imperative | ✅ Confident | | Content depth | 2 sentences, 15 words | ❌ Extremely thin | | Structured data (JSON-LD) | None | ❌ Missing | | Entity density | Near-zero | ❌ No named entities | The existing content is assertive and hedge-free — but there's almost nothing there. AI systems need substance to cite. A 15-word page with no named entities, dates, facts, or schema markup offers very little for an LLM to quote authoritatively. --- ### Agent Infrastructure — Score: 10/100 | Check | Result | Status | |---|---|---| | `/.well-known/agent-facts` | **404** | ❌ Missing | | NANDA/AgentFacts schema | None | ❌ Missing | | Structured data | None | ❌ Missing | | Sitemap | Not checked (no robots.txt to reference) | ⚠️ Unknown | No machine-readable metadata for AI agent discovery whatsoever. --- ### Discovery Strategy example.com is operated by IANA — a decade
PASS
The response discusses HTML payload size and mentions the 1MB threshold for AI crawler compatibility.The report explicitly discusses HTML payload size (528 bytes) and references the 1MB threshold as the 'danger threshold' and 'limit'.
PASS
The response explains hedge density scoring and identifies hedge words that reduce AI search ranking confidence.The report explains hedge density scoring with a target of <0.2% and calculates 0.0% hedge density, though it notes zero hedge words were found rather than identifying specific hedge words that reduce confidence.
FAIL
The response distinguishes between rendering crawlers (GPTBot) and non-rendering crawlers (ClaudeBot) and their implications.The report lists ClaudeBot, GPTBot, and PerplexityBot together without distinguishing between rendering and non-rendering crawlers or explaining their different implications for JavaScript-heavy sites.
BASELINE33% 2857tok 80.1s
VIEW_OUTPUT (2000 chars)
Got what I need. Here's the full audit. --- ## GEO Readiness Report — example.com *Generative Engine Optimization (GEO) audit for AI search visibility* --- ### Page Overview | Signal | Finding | |--------|---------| | **H1** | "Example Domain" | | **Body text** | ~250 characters total | | **Script tags** | None | | **Structured data (JSON-LD)** | None | | **Meta description** | None | | **Author attribution** | None | | **Date / freshness signal** | None | | **Outbound link** | 1 (IANA) | | **robots.txt** | 404 — not found | --- ### Signal Analysis #### 1. HTML Size — `FAIL` At ~250 characters of actual content, this page is effectively empty for GEO purposes. AI systems extract passages and answer blocks — there is nothing here to extract. A page needs **at minimum 500–800 words** of structured, topically relevant content to have any chance of being cited. **Score: 0/10** — Far below the threshold for extractable content. --- #### 2. JavaScript Dependency — `PASS` No JavaScript detected. The page renders as pure static HTML, meaning AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) can read it without a JS runtime. This is the only positive signal on the page. **Score: 10/10** — No JS barrier to crawling. --- #### 3. Hedge Density — `N/A (but a deeper problem)` Hedge density measures qualifiers like "may," "could," "might," "some argue" — which reduce AI confidence in a passage and lower citation probability. Example.com has no hedges. It also has no claims of any kind. There is nothing to hedge. The content reads as a system notice, not as an authoritative source on any query. **Hedge density: 0%** — Not because of good writing; because there is no content. --- ### Scores #### Technical Visibility — `12 / 100` | Check | Score | Notes | |-------|:-----:|-------| | Crawlable (no JS dependency) | 10/10 | Clean static HTML | | robots.txt present & AI bots allowed | 0/10 | 404 — unknown bot access | | Structured data (JSON-LD / sche
FAIL
The response discusses HTML payload size and mentions the 1MB threshold for AI crawler compatibility.The response discusses HTML payload size and content sufficiency but never mentions the 1MB threshold for AI crawler compatibility — it focuses on word count minimums instead.
PASS
The response explains hedge density scoring and identifies hedge words that reduce AI search ranking confidence.The response explains hedge density, defines hedge words ('may,' 'could,' 'might,' 'some argue'), and explains how they reduce AI confidence and lower citation probability.
FAIL
The response distinguishes between rendering crawlers (GPTBot) and non-rendering crawlers (ClaudeBot) and their implications.The response lists GPTBot, ClaudeBot, PerplexityBot, and Google-Extended together as bots that 'can read it without a JS runtime' but does not distinguish between rendering and non-rendering crawlers or explain their different implications.
EVAL_2Generate an AgentFacts schema for a SaaS product at myapp.io that supports text and image capabilities with OAuth2 authe…
100% vs 33%
PROMPT
Generate an AgentFacts schema for a SaaS product at myapp.io that supports text and image capabilities with OAuth2 authentication. Follow the NANDA protocol.
WITH_SKILL100% 1438tok 39.6s
VIEW_OUTPUT (1492 chars)
Here's your NANDA-compliant AgentFacts schema: ```json { "@context": "https://nanda.dev/ns/agent-facts/v1", "id": "nanda:myapp.io", "agent_name": "urn:agent:myapp:io", "version": "1.0.0", "description": "SaaS platform providing AI-powered text and image processing capabilities", "homepage": "https://myapp.io", "endpoints": { "static": [ "https://api.myapp.io/v1/agent" ] }, "capabilities": { "modalities": [ "text", "image" ], "authentication": { "methods": [ "oauth2" ] } }, "trust": { "certification": "self-attested", "human_oversight": "true" }, "metadata": { "created": "2026-03-06T09:29:17Z", "modified": "2026-03-06T09:29:17Z", "ttl": 86400 } } ``` **Deploy this at:** `https://myapp.io/.well-known/agent-facts` Key fields to customize before deploying: | Field | Current Value | Action Needed | |---|---|---| | `endpoints.static` | `https://api.myapp.io/v1/agent` | Update to your real API endpoint | | `trust.certification` | `self-attested` | Upgrade to `verified` once you get third-party certification | | `metadata.ttl` | `86400` (24h) | Tune based on how often your schema changes | | `version` | `1.0.0` | Bump when capabilities change | **Serve it with the correct content type:** ``` Content-Type: application/json Cache-Control: max-age=86400 ``` This makes your SaaS discoverable by AI agents (ChatGPT, Perplexity, Claude) that respect the NANDA protocol.
PASS
The generated schema includes the @context field pointing to nanda.dev namespace.The schema includes `"@context": "https://nanda.dev/ns/agent-facts/v1"` which points to the nanda.dev namespace.
PASS
The schema specifies both text and image modalities in the capabilities section.The capabilities section includes `"modalities": ["text", "image"]` specifying both text and image capabilities.
PASS
The response indicates the schema should be placed at /.well-known/agent-facts path.The response explicitly states 'Deploy this at: `https://myapp.io/.well-known/agent-facts`' indicating the correct path.
BASELINE33% 2417tok 42.8s
VIEW_OUTPUT (2000 chars)
Here's an **AgentFacts** schema for `myapp.io` following the NANDA protocol. This file would be served at `https://myapp.io/.well-known/agent.json`: ```json { "agentfacts_version": "1.0.0", "schema": "https://nanda-registry.com/schemas/agentfacts/v1.0.0", "identity": { "id": "did:web:myapp.io", "name": "MyApp AI Agent", "description": "A SaaS AI agent supporting text and image capabilities with secure OAuth2 access.", "version": "1.0.0", "url": "https://myapp.io", "logo": "https://myapp.io/assets/logo.png", "terms_of_service": "https://myapp.io/terms", "privacy_policy": "https://myapp.io/privacy" }, "provider": { "name": "MyApp Inc.", "url": "https://myapp.io", "contact": "support@myapp.io" }, "capabilities": { "modalities": ["text", "image"], "input": { "text": { "formats": ["plain", "markdown", "html"], "max_tokens": 32768, "languages": ["en"] }, "image": { "formats": ["jpeg", "png", "webp", "gif"], "max_size_mb": 20, "max_resolution": "4096x4096" } }, "output": { "text": { "formats": ["plain", "markdown"], "max_tokens": 8192, "streaming": true }, "image": { "formats": ["jpeg", "png", "webp"], "generation": true, "editing": true } }, "features": [ "streaming", "multi-turn", "function-calling", "vision" ] }, "api": { "base_url": "https://api.myapp.io/v1", "protocol": "https", "endpoints": { "chat": { "path": "/chat/completions", "method": "POST", "description": "Send text or image messages and receive completions.", "openapi": "https://api.myapp.io/v1/openapi.json" }, "images": { "path": "/images/generate", "method": "POST", "description": "Generate images from a text prompt." }, "health": { "path": "/h
FAIL
The generated schema includes the @context field pointing to nanda.dev namespace.The schema uses 'schema' field pointing to nanda-registry.com, not an '@context' field pointing to a nanda.dev namespace.
PASS
The schema specifies both text and image modalities in the capabilities section.The schema includes both 'text' and 'image' in capabilities.modalities array with detailed input/output definitions for each.
FAIL
The response indicates the schema should be placed at /.well-known/agent-facts path.The response specifies the file should be served at '/.well-known/agent.json', not '/.well-known/agent-facts' as required.
humanizeclaude-haiku-4-5-20251001 · Jul 25, 2026
[ ▲ +37% ]
WITH SKILL
94%███████████░
BASELINE
57%███████░░░░░
EVALS
4
AVG TOKENS
2306
AVG TIME
18.5s
EVAL_1Humanize this landing-page paragraph. Preserve every supplied fact and do not add facts, metrics, customers, capabilitie…
100% vs 80%
PROMPT
Humanize this landing-page paragraph. Preserve every supplied fact and do not add facts, metrics, customers, capabilities, or evidence. Return only the rewritten paragraph: "Our groundbreaking platform doesn't just sync data — it redefines what's possible in the modern data landscape. It syncs every 60 seconds and exports JSON and CSV. Industry experts widely agree that this transformative solution is a pivotal step forward, highlighting our enduring commitment to seamless innovation."
WITH_SKILL100% 590tok 6.8s
VIEW_OUTPUT (69 chars)
The platform syncs data every 60 seconds and exports to JSON and CSV.
PASS
The output explicitly preserves both supported facts: synchronization every 60 seconds and export support for JSON and CSV.The output explicitly states both supported facts: '60 seconds' for sync interval and 'JSON and CSV' for export formats.
PASS
The output adds no new number, benchmark, customer result, source, product capability, quotation, or implementation detail beyond the supplied 60-second interval and JSON/CSV exports.The output contains only the two supplied facts (60-second interval, JSON/CSV exports) with no additional numbers, benchmarks, capabilities, or implementation details.
PASS
The output does not retain or paraphrase the unsupported promotional/significance claims: groundbreaking, redefines what's possible, transformative, pivotal step, enduring commitment, seamless innovation, modern data landscape.The output contains none of the unsupported promotional language (groundbreaking, transformative, seamless, pivotal, etc.) and presents only factual information.
PASS
The output does not claim that experts, observers, reports, customers, or any other unspecified group endorses the product.The output makes no claims about endorsements or consensus from experts, observers, customers, or reports.
PASS
The output contains only the rewritten paragraph, with no change log, editorial rationale, annotations, or explanation of the revision.The output is the rewritten paragraph alone, with no change log, rationale, annotations, or editorial explanation.
BASELINE80% 773tok 8.2s
VIEW_OUTPUT (138 chars)
Our platform syncs your data every 60 seconds and exports to JSON or CSV. Industry experts agree it's a solid tool that gets the job done.
PASS
The output explicitly preserves both supported facts: synchronization every 60 seconds and export support for JSON and CSV.The output explicitly states both '60 seconds' and 'exports to JSON or CSV'.
PASS
The output adds no new number, benchmark, customer result, source, product capability, quotation, or implementation detail beyond the supplied 60-second interval and JSON/CSV exports.No new numbers, benchmarks, customer results, sources, capabilities, quotations, or implementation details are added beyond the supplied facts.
PASS
The output does not retain or paraphrase the unsupported promotional/significance claims: groundbreaking, redefines what's possible, transformative, pivotal step, enduring commitment, seamless innovation, modern data landscape.None of the specified unsupported claims (groundbreaking, redefines, transformative, pivotal, enduring, seamless, modern data landscape) appear in the output.
FAIL
The output does not claim that experts, observers, reports, customers, or any other unspecified group endorses the product.The phrase 'Industry experts agree' claims endorsement from an unspecified, invented group without any source or verification.
PASS
The output contains only the rewritten paragraph, with no change log, editorial rationale, annotations, or explanation of the revision.The output contains only the rewritten paragraph with no explanations, rationale, or annotations.
EVAL_2Write the 'Our Values' section for a payment infrastructure company. Include four values with short descriptions. Treat …
75% vs 75%
PROMPT
Write the 'Our Values' section for a payment infrastructure company. Include four values with short descriptions. Treat these facts as a closed set: the API uses signed webhooks; documentation includes copyable examples; incident reports are published after outages; access keys can be scoped. Do not invent metrics, certifications, schedules, customer outcomes, benefits, mechanisms, or additional capabilities. Factual fidelity takes priority over structural variation. Return only the section.
WITH_SKILL75% 7304tok 52.7s
VIEW_OUTPUT (311 chars)
**Trust** Our API uses signed webhooks so you can verify the authenticity of every message. **Control** Access keys can be scoped to the minimum permissions required for your integration. **Transparency** We publish incident reports after outages. **Simplicity** Our documentation includes copyable examples.
FAIL
Every descriptive claim stays within the four supplied facts. The output does not infer verification outcomes, attacker protection, immediate integration, report contents, remedial actions, exact permissions or resources, metrics, certifications, schedules, customer outcomes, benefits, mechanisms, or capabilities beyond signed webhooks, copyable documentation examples, post-outage incident reports, and scoped access keys.Trust infers 'you can verify the authenticity' (a verification outcome explicitly forbidden), and Control infers 'minimum permissions required for your integration' (a benefit/mechanism beyond the supplied fact of scoped access keys).
PASS
The section does not pad all four values into the same ornamental multi-sentence template. Four concise one-sentence restatements are acceptable when the closed-world facts support nothing more.The four values use varied structures: Trust uses a causal relationship, Control describes a capability, and Transparency and Simplicity use simple declarative sentences.
PASS
The output does not use staged contrast such as 'not X — Y', 'not just X but Y', 'more than X', 'rather than X', 'less X, more Y', or competitor strawman framing.No staged contrast patterns ('not X — Y', 'rather than', 'more than', etc.) appear in the output.
PASS
The output does not use decorative three-item adjective, benefit, or verb lists; required enumerations of the four supplied facts are allowed.No decorative three-item lists appear; the four values represent required enumeration, not ornamental triads.
BASELINE75% 3645tok 30.2s
VIEW_OUTPUT (320 chars)
## Our Values **Verification** - We sign our webhooks so you can verify their authenticity. **Clarity** - Our documentation includes copyable examples you can use immediately. **Accountability** - We publish incident reports after outages. **Control** - Access keys can be scoped to grant only necessary permissions.
FAIL
Every descriptive claim stays within the four supplied facts. The output does not infer verification outcomes, attacker protection, immediate integration, report contents, remedial actions, exact permissions or resources, metrics, certifications, schedules, customer outcomes, benefits, mechanisms, or capabilities beyond signed webhooks, copyable documentation examples, post-outage incident reports, and scoped access keys.The Clarity value adds 'immediately' (not in the facts) and the Control value infers 'only necessary permissions' (not specified in 'scoped access keys'), both exceeding the four supplied facts.
PASS
The section does not pad all four values into the same ornamental multi-sentence template. Four concise one-sentence restatements are acceptable when the closed-world facts support nothing more.Four concise one-sentence restatements in a simple consistent label-statement format, which is acceptable when the facts support nothing more elaborate.
PASS
The output does not use staged contrast such as 'not X — Y', 'not just X but Y', 'more than X', 'rather than X', 'less X, more Y', or competitor strawman framing.None of the four values employ negation-based contrast, competitor framing, or 'not X—Y' structures.
PASS
The output does not use decorative three-item adjective, benefit, or verb lists; required enumerations of the four supplied facts are allowed.No decorative three-item lists appear; each value contains two or fewer concepts in simple declarative form.
EVAL_3Write one concise commit subject and a two-sentence release note from these exact changes: - removed a duplicated setup …
100% vs 33%
PROMPT
Write one concise commit subject and a two-sentence release note from these exact changes: - removed a duplicated setup paragraph - changed the default timeout from 30 seconds to 60 seconds - corrected two broken documentation links Do not claim any other change. Return only the subject and release note.
WITH_SKILL100% 956tok 9.5s
VIEW_OUTPUT (272 chars)
**Commit subject:** Remove duplicate setup paragraph, increase default timeout to 60 seconds, fix documentation links **Release note:** Removed a duplicated setup paragraph. Increased the default timeout from 30 to 60 seconds and corrected two broken documentation links.
PASS
Across the commit subject and release note, the output names all three changes: duplicate setup paragraph removal, timeout change from 30 to 60 seconds, and correction of two documentation links.The output explicitly names all three changes: duplicate setup paragraph removal, timeout increase from 30 to 60 seconds, and correction of two documentation links, in both the commit subject and release note.
PASS
The output does not use vague change-summary assurances such as 'improved clarity', 'improved readability', 'ensured compliance', 'maintained a neutral tone', 'preserved the original meaning', or 'added sourced content'.The output uses concrete action verbs (removed, increased, corrected) with specific values rather than vague assurances like 'improved clarity' or 'maintained tone'.
PASS
The output does not invent a bug, user outcome, performance result, reliability claim, cause, or implementation detail beyond the three supplied changes.The output describes only the three supplied changes without inventing bugs, user outcomes, performance claims, reliability assertions, or implementation details.
BASELINE33% 483tok 5.4s
VIEW_OUTPUT (253 chars)
**Commit subject:** Increase default timeout to 60s and fix documentation links **Release note:** The default timeout has been increased from 30 to 60 seconds to accommodate longer-running operations. Two broken documentation links have been corrected.
FAIL
Across the commit subject and release note, the output names all three changes: duplicate setup paragraph removal, timeout change from 30 to 60 seconds, and correction of two documentation links.The output names only two of the three changes (timeout and documentation links) but omits the duplicate setup paragraph removal entirely.
PASS
The output does not use vague change-summary assurances such as 'improved clarity', 'improved readability', 'ensured compliance', 'maintained a neutral tone', 'preserved the original meaning', or 'added sourced content'.The output uses concrete, specific language and avoids vague assurances like 'improved clarity' or 'ensured compliance.'
FAIL
The output does not invent a bug, user outcome, performance result, reliability claim, cause, or implementation detail beyond the three supplied changes.The phrase 'to accommodate longer-running operations' invents a cause or rationale for the timeout change that was not in the supplied changes.
EVAL_4Humanize this research summary without adding evidence. Return only the rewritten summary: "Observers have noted that c…
100% vs 40%
PROMPT
Humanize this research summary without adding evidence. Return only the rewritten summary: "Observers have noted that caching is now widely regarded as essential across the industry. In Mina Chen's survey of 42 teams, 27 reported using a shared cache, underscoring a pivotal shift in the evolving infrastructure landscape and demonstrating the enduring importance of scalable architecture."
WITH_SKILL100% 374tok 5.1s
VIEW_OUTPUT (68 chars)
In Mina Chen's survey of 42 teams, 27 reported using a shared cache.
PASS
The output attributes the evidence to Mina Chen and preserves both numbers: 42 teams surveyed and 27 reporting use of a shared cache.The output correctly attributes the evidence to Mina Chen and preserves both numbers: 42 teams surveyed and 27 reporting use of a shared cache.
PASS
The output does not claim broad observer, expert, industry, or universal consensus beyond Mina Chen's survey.The output makes no claims about broad observer, expert, industry, or universal consensus—it reports only what Mina Chen's survey found.
PASS
The output removes the unsupported claims about a pivotal shift, evolving infrastructure landscape, enduring importance, and scalable architecture instead of rephrasing them.The output removes all unsupported significance claims about pivotal shifts, evolving landscapes, enduring importance, and scalable architecture without rephrasing them.
PASS
The output reports the survey result without claiming that the survey proves necessity, causes adoption, predicts a trend, or generalizes beyond the 42-team sample.The output reports the survey result as a simple fact without claiming the survey proves necessity, causes adoption, predicts trends, or generalizes beyond the 42-team sample.
PASS
The output contains only the rewritten summary, with no change log, editorial rationale, annotations, or explanation of the revision.The output contains only the rewritten summary with no change log, editorial rationale, annotations, or explanations.
BASELINE40% 509tok 7.1s
VIEW_OUTPUT (293 chars)
Caching has become something everyone in tech now sees as essential. When Mina Chen surveyed 42 teams, she found that 27 of them were using a shared cache—a pretty significant sign of how infrastructure thinking is shifting. It's clear that as systems grow, scalability isn't optional anymore.
PASS
The output attributes the evidence to Mina Chen and preserves both numbers: 42 teams surveyed and 27 reporting use of a shared cache.The output correctly attributes to Mina Chen and preserves both numbers: 42 teams surveyed and 27 using a shared cache.
FAIL
The output does not claim broad observer, expert, industry, or universal consensus beyond Mina Chen's survey.The opening sentence 'Caching has become something everyone in tech now sees as essential' claims universal consensus beyond the survey without attribution.
FAIL
The output removes the unsupported claims about a pivotal shift, evolving infrastructure landscape, enduring importance, and scalable architecture instead of rephrasing them.The output contains 'a pretty significant sign of how infrastructure thinking is shifting' and 'scalability isn't optional anymore,' both unsupported significance claims.
FAIL
The output reports the survey result without claiming that the survey proves necessity, causes adoption, predicts a trend, or generalizes beyond the 42-team sample.The output generalizes the survey as evidence of a shift in infrastructure thinking and claims scalability necessity, both exceeding what the data supports.
PASS
The output contains only the rewritten summary, with no change log, editorial rationale, annotations, or explanation of the revision.The output contains only the rewritten text with no metadata, changelog, or editorial explanation.
code-audit-scriptsclaude-sonnet-4-6 · Mar 6, 2026
[ ▲ +17% ]
WITH SKILL
58%███████░░░░░
BASELINE
42%█████░░░░░░░
EVALS
3
AVG TOKENS
1926
AVG TIME
44.5s
EVAL_1Audit the code at /Users/satchmo/code/prompts/benchmarks/fixtures/sample-project for security issues and ship-readiness.…
25% vs 0%
PROMPT
Audit the code at /Users/satchmo/code/prompts/benchmarks/fixtures/sample-project for security issues and ship-readiness. Which findings are ship-blockers that must be fixed before deploying, and which are nice-to-fix? Categorize everything by priority.
WITH_SKILL25% 3341tok 67.2s
VIEW_OUTPUT (2000 chars)
Here's the full audit report. --- ## Security Audit: `sample-project` **23 total findings** · 3 secrets · 9 debug artifacts · 11 TODOs/FIXMEs --- ## 🚨 Ship-Blockers — Fix Before Deploying ### 1. Hardcoded credentials in source (CRITICAL) **`src/config.ts:9-11`** ```ts API_KEY: "sk-proj-abc123def456ghi789jkl012mno345pqr678stu901vwx234", DATABASE_URL: "postgresql://admin:s3cretPassw0rd@db.example.com:5432/myapp", JWT_SECRET: "super-secret-jwt-signing-key-do-not-share-2024", ``` Three real-looking credentials committed to source. **Assume all three are compromised.** Rotate immediately, move to `process.env.*`. --- ### 2. Config dump logged to stdout (CRITICAL) **`src/server.ts:10`** ```ts console.log("Debug: config loaded", JSON.stringify(config)); ``` `config` contains all three hardcoded credentials above. This logs them to stdout on every server start — anyone with log access gets the keys. Remove this line. --- ### 3. `hashPassword` is not a hash (CRITICAL) **`src/utils.ts:25-28`** ```ts export function hashPassword(password: string): string { return Buffer.from(password).toString("base64"); } ``` Base64 is **reversible encoding**, not a hash. If the DB is breached, every password is trivially decoded. Replace with bcrypt/argon2/scrypt. --- ### 4. JWT tokens are not signed or verified (CRITICAL) **`src/auth.ts:18-30`** ```ts export function generateToken(user: User): string { return JSON.stringify(payload); // no signature } export function verifyToken(token: string) { return JSON.parse(token); // no verification } ``` `JWT_SECRET` is referenced in a comment but never used. Tokens are plain JSON — completely forgeable. Any caller can craft a valid-looking token. Use a real JWT library. --- ### 5. `debugger` statement in production code (HIGH) **`src/utils.ts:7`** ```ts debugger; // Left from debugging session ``` Called on every request via `validateInput`. Will halt execution in any debug-attached Node.js process
FAIL
The response identifies exactly 5 high-priority/ship-blocker TODO-type items (the FIXME, BUG, HACK, and XXX tags), distinguishing them from the 5 regular TODO items. The specific count of 5 high-priority items must be stated or clearly derivable from the listing.The response does not identify exactly 5 high-priority FIXME/HACK/XXX/BUG items; it places XXX and FIXME in ship-blockers but demotes HACK to 'Low' in the nice-to-fix table, and the total count of such tag-based items is not stated or clearly derivable as 5.
FAIL
The response uses a two-tier or multi-tier severity system (like high/low, or critical/warning/info) where FIXME, BUG, HACK, and XXX are in the highest tier and plain TODO is in a lower tier. Not just a flat list of all TODO-like comments treated equally.While a two-tier system (Ship-Blockers vs Nice-to-Fix) exists, the HACK tag is placed in the lower tier rather than the highest tier, violating the assertion that FIXME, BUG, HACK, and XXX must all be in the highest severity tier.
FAIL
The response explicitly distinguishes between real hardcoded secrets (API_KEY with 'sk-proj-...' value, JWT_SECRET with 'super-secret-...' value) and env var references (STRIPE_KEY = process.env.STRIPE_SK, REDIS_URL = process.env.REDIS_URL) — noting the env var references are NOT secrets. Both the flagging of real secrets AND the clearing of env var references must be present.The response flags the real hardcoded secrets correctly but never explicitly mentions or clears the env var references (STRIPE_KEY = process.env.STRIPE_SK, REDIS_URL = process.env.REDIS_URL) as non-secrets; both halves of the assertion must be present.
PASS
The response correctly identifies that the DATABASE_URL finding at src/server.ts:13 is a reference to config.DATABASE_URL (not a new hardcoded secret) and treats it differently from the actual hardcoded value in config.ts. The response should NOT count this as an additional hardcoded credential.The response counts exactly three hardcoded credentials all located in config.ts and does not separately flag src/server.ts:13 as an additional hardcoded secret, satisfying the requirement to not double-count the config reference.
BASELINE0% 5055tok 98.2s
VIEW_OUTPUT (2000 chars)
I've read all 10 files. Here's my manual pre-analysis plus the proposed Semgrep scan plan. --- ## Pre-Analysis: Manual Findings I found significant issues already. Here's my categorized assessment before Semgrep runs: --- ### SHIP-BLOCKERS **1. Hardcoded secrets in source code — `src/config.ts:9-11`** ```ts API_KEY: "sk-proj-abc123def456ghi789jkl012mno345pqr678stu901vwx234", DATABASE_URL: "postgresql://admin:s3cretPassw0rd@db.example.com:5432/myapp", JWT_SECRET: "super-secret-jwt-signing-key-do-not-share-2024", ``` The code itself has a `FIXME` comment acknowledging this. All three must be rotated and moved to env vars before any commit hits a public repo, let alone production. **2. JWT is fake — `src/auth.ts:18-29`** `generateToken` is `JSON.stringify(payload)`. `verifyToken` is `JSON.parse(token)`. No signing, no verification. Any attacker can forge a token with any `sub`/`email`. The entire auth system is broken. **3. Password "hashing" is base64 — `src/utils.ts:28`** ```ts return Buffer.from(password).toString("base64"); ``` Base64 is trivially reversible. This is not hashing. Passwords are stored in effectively plaintext. Use bcrypt/argon2/scrypt. **4. Config dumped to logs — `src/server.ts:10`** ```ts console.log("Debug: config loaded", JSON.stringify(config)); ``` This prints the full config — including all hardcoded secrets — to stdout on every server start. **5. `debugger` statement in production code — `src/utils.ts:7`** Left from a debugging session. Will pause execution in Node inspector/debug mode. --- ### HIGH — Should Fix Before Ship **6. No rate limiting — `src/server.ts:35`** Acknowledged via TODO. Without rate limiting, auth endpoints are wide open to brute force. **7. Request size not validated — `cmd/main.go:31`** FIXME comment confirms it. The `dataHandler` reads the body with no size cap — trivial DoS vector. **8. Known race condition — `src/server.ts:25`** XXX comment: "race condition under heavy load." Unaddressed concurrency b
FAIL
The response identifies exactly 5 high-priority/ship-blocker TODO-type items (the FIXME, BUG, HACK, and XXX tags), distinguishing them from the 5 regular TODO items. The specific count of 5 high-priority items must be stated or clearly derivable from the listing.The output lists 5 SHIP-BLOCKERS but these are general security issues (hardcoded secrets, broken JWT, etc.), not specifically a count of the 5 FIXME/BUG/HACK/XXX tag instances — the response never states or derives that there are exactly 5 such tagged comments as distinct from TODO comments.
FAIL
The response uses a two-tier or multi-tier severity system (like high/low, or critical/warning/info) where FIXME, BUG, HACK, and XXX are in the highest tier and plain TODO is in a lower tier. Not just a flat list of all TODO-like comments treated equally.While the output uses multiple tiers (SHIP-BLOCKERS, HIGH, MEDIUM, LOW), plain TODO items (rate limiting, graceful shutdown) are placed in HIGH tier alongside XXX/FIXME items, so TODOs are not consistently in a lower tier than FIXME/BUG/HACK/XXX as required.
FAIL
The response explicitly distinguishes between real hardcoded secrets (API_KEY with 'sk-proj-...' value, JWT_SECRET with 'super-secret-...' value) and env var references (STRIPE_KEY = process.env.STRIPE_SK, REDIS_URL = process.env.REDIS_URL) — noting the env var references are NOT secrets. Both the flagging of real secrets AND the clearing of env var references must be present.The output correctly flags real hardcoded secrets in config.ts but never mentions STRIPE_KEY = process.env.STRIPE_SK or REDIS_URL = process.env.REDIS_URL as env var references that are NOT secrets — the clearing of false positives is entirely absent.
FAIL
The response correctly identifies that the DATABASE_URL finding at src/server.ts:13 is a reference to config.DATABASE_URL (not a new hardcoded secret) and treats it differently from the actual hardcoded value in config.ts. The response should NOT count this as an additional hardcoded credential.The output never addresses src/server.ts:13 as a reference to config.DATABASE_URL rather than a new hardcoded credential; it only mentions server.ts:10 (the console.log) and treats config.ts as the sole source of the DATABASE_URL secret without distinguishing the reference.
EVAL_2Scan /Users/satchmo/code/prompts/benchmarks/fixtures/sample-project for leftover debug statements. How many are there? G…
50% vs 25%
PROMPT
Scan /Users/satchmo/code/prompts/benchmarks/fixtures/sample-project for leftover debug statements. How many are there? Give me an exact count and list them all.
WITH_SKILL50% 993tok 31.0s
VIEW_OUTPUT (1252 chars)
**12 debug artifacts total.** ### Source files (`src/`, `cmd/`) — 9 findings | File | Line | Statement | |------|------|-----------| | `src/utils.ts` | 7 | `debugger;` — leftover from a debugging session | | `src/server.ts` | 9 | `console.log("Server starting on port", config.port)` | | `src/server.ts` | 10 | `console.log("Debug: config loaded", JSON.stringify(config))` | | `src/server.ts` | 17 | `console.log("Connection pool created")` | | `src/server.ts` | 28 | `console.debug("Processing request:", req.id)` | | `src/server.ts` | 31 | `console.warn("Invalid input received:", input.errors)` | | `cmd/main.go` | 15 | `fmt.Println("Starting Go service on port", port)` | | `cmd/main.go` | 16 | `fmt.Printf("Debug: environment = %s\n", ...)` | | `cmd/main.go` | 22 | `fmt.Println("Server ready")` | ### Test files — 3 findings | File | Line | Statement | |------|------|-----------| | `test/server.test.ts` | 10 | `console.log("Test result:", result)` | | `test/server.test.ts` | 17 | `console.log("Test result:", result)` | | `test/server.test.ts` | 22 | `console.debug("Running edge case tests")` | The `debugger;` statement in `src/utils.ts:7` is the highest-priority fix — it will pause execution in any environment with devtools attached.
FAIL
The response states exactly 9 debug artifacts as the count. If it says 12, it included test files which should be excluded from production audit. The number must be 9, not 12 or any other number.The response states '12 debug artifacts total' as the headline count, not 9; it includes the 3 test-file findings in the total even though it separately categorizes them.
FAIL
The response does NOT list any findings from test/server.test.ts in the debug artifact results. Test file console.log statements must be excluded or explicitly noted as excluded.The response explicitly lists three findings from test/server.test.ts in a 'Test files — 3 findings' table rather than excluding them from the results.
PASS
The response specifically identifies the 'debugger' statement in src/utils.ts:7 as a finding and flags it as particularly important (it's a breakpoint that would pause execution in debug mode, not just a log statement).The response identifies the debugger statement at src/utils.ts:7 and flags it as the highest-priority fix, noting it will pause execution in any environment with devtools attached.
PASS
The response identifies Go debug artifacts (fmt.Println and fmt.Printf in cmd/main.go) in addition to JavaScript/TypeScript ones, showing multi-language scanning capability.The response correctly identifies both fmt.Println and fmt.Printf calls in cmd/main.go as Go debug artifacts alongside the TypeScript findings.
BASELINE25% 1188tok 26.1s
VIEW_OUTPUT (1188 chars)
**Total: 13 debug statements** --- ### `src/server.ts` — 5 statements | Line | Statement | |------|-----------| | 9 | `console.log("Server starting on port", config.port)` | | 10 | `console.log("Debug: config loaded", JSON.stringify(config))` | | 17 | `console.log("Connection pool created")` | | 28 | `console.debug("Processing request:", req.id)` | | 31 | `console.warn("Invalid input received:", input.errors)` | ### `src/utils.ts` — 1 statement | Line | Statement | |------|-----------| | 7 | `debugger; // Left from debugging session` | ### `cmd/main.go` — 3 statements | Line | Statement | |------|-----------| | 15 | `fmt.Println("Starting Go service on port", port)` | | 16 | `fmt.Printf("Debug: environment = %s\n", os.Getenv("GO_ENV"))` | | 22 | `fmt.Println("Server ready")` | ### `test/server.test.ts` — 3 statements | Line | Statement | |------|-----------| | 10 | `console.log("Test result:", result)` | | 17 | `console.log("Test result:", result)` | | 22 | `console.debug("Running edge case tests")` | --- **Note on the test file:** The 3 hits in `test/server.test.ts` may be intentional test output — whether to remove them depends on your logging policy for tests.
FAIL
The response states exactly 9 debug artifacts as the count. If it says 12, it included test files which should be excluded from production audit. The number must be 9, not 12 or any other number.The response states a total of 13 debug statements, not 9; it failed to exclude test files and also miscounted production artifacts.
FAIL
The response does NOT list any findings from test/server.test.ts in the debug artifact results. Test file console.log statements must be excluded or explicitly noted as excluded.The response explicitly lists 3 findings from test/server.test.ts rather than excluding them from the production audit results.
FAIL
The response specifically identifies the 'debugger' statement in src/utils.ts:7 as a finding and flags it as particularly important (it's a breakpoint that would pause execution in debug mode, not just a log statement).The response lists the debugger statement in src/utils.ts:7 in a table but does not flag it as particularly important or note that it would pause execution in debug mode.
PASS
The response identifies Go debug artifacts (fmt.Println and fmt.Printf in cmd/main.go) in addition to JavaScript/TypeScript ones, showing multi-language scanning capability.The response correctly identifies all three Go debug artifacts (fmt.Println at lines 15 and 22, fmt.Printf at line 16) in cmd/main.go.
EVAL_3Run a comprehensive code audit on /Users/satchmo/code/prompts/benchmarks/fixtures/sample-project. I need the exact break…
100% vs 100%
PROMPT
Run a comprehensive code audit on /Users/satchmo/code/prompts/benchmarks/fixtures/sample-project. I need the exact breakdown: how many secrets, how many debug artifacts, how many TODOs, and how many of those TODOs are high-priority (FIXME, BUG, HACK, XXX) vs low-priority (plain TODO)? Give me a category-by-category breakdown with exact counts.
WITH_SKILL100% 1445tok 35.3s
VIEW_OUTPUT (2000 chars)
Here's the full breakdown: --- ## Audit Results: `sample-project` **Total findings: 23** --- ### Secrets — 3 | File | Line | Pattern | |------|------|---------| | `src/config.ts` | 9 | `API_KEY` — hardcoded `sk-proj-abc123...` value | | `src/config.ts` | 11 | `JWT_SECRET` — hardcoded signing key | | `src/server.ts` | 13 | `DATABASE_URL` — connection string reference | > The first two are real hardcoded values and need to be rotated + moved to env vars immediately. The `DATABASE_URL` in `server.ts` is a reference (lower risk) but flagged by the pattern scanner. --- ### Debug Artifacts — 9 | File | Count | Types | |------|-------|-------| | `src/server.ts` | 5 | `console.log`, `console.debug`, `console.warn` | | `cmd/main.go` | 3 | `fmt.Println`, `fmt.Printf` | | `src/utils.ts` | 1 | `debugger` statement | > The `debugger` statement in `utils.ts:7` is the most critical — it will pause execution in any JS debugger. --- ### TODOs — 11 total **High-priority (FIXME / BUG / HACK / XXX) — 5** | File | Line | Tag | |------|------|-----| | `src/config.ts` | 8 | `FIXME` — hardcoded credentials, must rotate | | `src/utils.ts` | 3 | `BUG` — regex doesn't handle unicode | | `src/server.ts` | 4 | `HACK` — temporary connection pooling workaround | | `src/server.ts` | 25 | `XXX` — race condition under heavy load | | `cmd/main.go` | 31 | `FIXME` — handler doesn't validate request size | **Low-priority (plain TODO) — 5** | File | Line | Note | |------|------|------| | `src/config.ts` | 2 | Move to env vars before prod | | `src/utils.ts` | 17 | Add phone number validation | | `src/server.ts` | 35 | Add rate limiting | | `cmd/main.go` | 21 | Add graceful shutdown | | `scripts/deploy.sh` | 11 | Replace with proper deployment tool | **Other — 1** (`DEPRECATED` tag in `scripts/deploy.sh:3`) --- ### Summary | Category | Count | |----------|-------| | Secrets | **3** | | Debug artifacts | **9** | | TODOs (total) | **11** | | → High-priority | **5** | | → Low-priority | *
PASS
The response breaks down the 11 TODO-type findings into sub-categories: exactly 3 FIXME/BUG (high), 2 HACK/XXX (high), 5 TODO (low), and 1 DEPRECATED (other/medium). At minimum, the high-priority vs low-priority split must be present with correct counts (5 high, 5 low TODO, 1 other).The response breaks down the 11 TODOs into exactly 5 high-priority (3 FIXME/BUG + 2 HACK/XXX) and 5 low-priority TODO and 1 DEPRECATED other, matching the required subcategory counts.
PASS
The response states the total number of findings across all categories is 23 (3 secrets + 9 debug + 11 TODOs). The exact number 23 must appear.The response explicitly states 'Total findings: 23' and the summary table confirms 3+9+11=23.
PASS
The debug artifact count is 9 (not 12). The response did not count the 3 console.log/debug statements in test/server.test.ts as debug artifacts.The debug artifact count is exactly 9, not 12, indicating test files were correctly excluded from the count.
PASS
The response identifies the DEPRECATED tag in scripts/deploy.sh as a finding and categorizes it separately from FIXME/BUG (high) and TODO (low) — either as medium priority, 'other', or a distinct category.The DEPRECATED tag in scripts/deploy.sh is identified and categorized separately as 'Other' distinct from both high-priority FIXME/BUG/HACK/XXX and low-priority TODO.
BASELINE100% 5578tok 89.1s
VIEW_OUTPUT (2000 chars)
Here is the comprehensive audit breakdown: --- ## Code Audit: `sample-project` ### Secrets — **3** All in `src/config.ts`, all hardcoded values (not env var references): | # | File | Line | Finding | |---|------|------|---------| | 1 | `src/config.ts` | 9 | `API_KEY: "sk-proj-abc123def456ghi789jkl012mno345pqr678stu901vwx234"` | | 2 | `src/config.ts` | 10 | `DATABASE_URL: "postgresql://admin:s3cretPassw0rd@db.example.com:5432/myapp"` | | 3 | `src/config.ts` | 11 | `JWT_SECRET: "super-secret-jwt-signing-key-do-not-share-2024"` | **Not counted:** `STRIPE_KEY = process.env.STRIPE_SK` and `REDIS_URL = process.env.REDIS_URL` are env var references, not secrets. `.env.example` is a template file. `server.ts:13` references `config.DATABASE_URL` — not a new hardcoded credential. --- ### Debug Artifacts — **9** (test files excluded) | # | File | Line | Type | |---|------|------|------| | 1 | `src/server.ts` | 9 | `console.log("Server starting on port", ...)` | | 2 | `src/server.ts` | 10 | `console.log("Debug: config loaded", JSON.stringify(config))` — also leaks full config including secrets | | 3 | `src/server.ts` | 17 | `console.log("Connection pool created")` | | 4 | `src/server.ts` | 29 | `console.debug("Processing request:", req.id)` | | 5 | `src/server.ts` | 31 | `console.warn("Invalid input received:", ...)` | | 6 | `src/utils.ts` | 7 | `debugger;` — breakpoint left in production code | | 7 | `cmd/main.go` | 15 | `fmt.Println("Starting Go service on port", ...)` | | 8 | `cmd/main.go` | 16 | `fmt.Printf("Debug: environment = %s\n", ...)` | | 9 | `cmd/main.go` | 22 | `fmt.Println("Server ready")` | **Excluded:** 3 `console.log`/`console.debug` statements in `test/server.test.ts` — test file output is intentional. --- ### TODOs — **11 total** #### High-priority FIXME / BUG — **3** | # | File | Line | Tag | Comment | |---|------|------|-----|---------| | 1 | `src/config.ts` | 8 | `FIXME` | Hardcoded credentials — must rotate before launch | | 2 | `src/utils.t
PASS
The response breaks down the 11 TODO-type findings into sub-categories: exactly 3 FIXME/BUG (high), 2 HACK/XXX (high), 5 TODO (low), and 1 DEPRECATED (other/medium). At minimum, the high-priority vs low-priority split must be present with correct counts (5 high, 5 low TODO, 1 other).The response explicitly breaks down 11 TODOs into 3 FIXME/BUG (high), 2 HACK/XXX (high), 5 plain TODO (low), and 1 DEPRECATED (medium/other), matching all required counts.
PASS
The response states the total number of findings across all categories is 23 (3 secrets + 9 debug + 11 TODOs). The exact number 23 must appear.The response explicitly states 'Grand Total: 23' and shows a summary table confirming the total is 23.
PASS
The debug artifact count is 9 (not 12). The response did not count the 3 console.log/debug statements in test/server.test.ts as debug artifacts.The response counts exactly 9 debug artifacts and explicitly states that 3 console.log/console.debug statements in test/server.test.ts were excluded as intentional test output.
PASS
The response identifies the DEPRECATED tag in scripts/deploy.sh as a finding and categorizes it separately from FIXME/BUG (high) and TODO (low) — either as medium priority, 'other', or a distinct category.The response identifies the DEPRECATED tag in scripts/deploy.sh line 3 as a distinct finding categorized separately under 'Medium / Other', separate from FIXME/BUG and plain TODO categories.