A real ShipperAG run on Northwind Books

The result

The run reproduced 141 issues where the shop does not do what its documents say, 17 of them critical and 60 high. The report's advice: fix them, or record a deliberate decision to accept them, before release.

• 141issues found
• 17need your input
• 63not covered
• 142checks held: 115 verified, 27 held once

Counts as the full report states them. Need input includes 1 open question about the requirements, which belongs to no single area. There is no overall score: every result is an evidence state.

Across the 13 quality areas

Findings by area, from business rules and journeys to security, data, connections and the shop's own AI assistant.

• Business value17 issues · 11 held
• User journeys10 issues · 15 held
• Human experience33 issues · 25 held
• Compatibility21 issues · 3 held
• Security & privacy17 issues · 7 held
• Responsible impact1 issues · 0 held
• Performance5 issues · 6 held
• Operability6 issues · 9 held
• Data integrity7 issues · 9 held
• Connections9 issues · 43 held
• AI behaviour5 issues · 4 held
• Change safety6 issues · 1 held
• Release sign-off4 issues · 9 held

VerifiedHeld onceIssue foundNeeds input

Issues found, with the evidence

8 of the 141 issues, the most serious one in each of 8 areas, exactly as the run recorded them. The full report has every issue.

• HighBusiness value · found by Business rulesReproduced by an independent re-check

  Second discount code BOOKS5 is accepted on top of WELCOME10 (discount $8.75)

  Rule (PRD.md): "BR-021: Only one discount code per order. Applying a second code is rejected with". Probe: apply WELCOME10 then BOOKS5. Expected HTTP 4xx, only WELCOME10 applied (discount $3.75); observed HTTP 200, codes [WELCOME10, BOOKS5], discount $8.75.

  Where: /api/cart/discount

  Steps to reproduce

  1. Start a fresh HTTP session (no cookies) against
  2. POST /api/cart/items {"productId":"p-9","quantity":2}
  3. POST /api/cart/items {"productId":"p-11","quantity":1}
  4. POST /api/cart/discount {"code":"WELCOME10"}
  5. POST /api/cart/discount {"code":"BOOKS5"}
• CriticalSecurity & privacy · found by API contractsReproduced by an independent re-check

  Another user's resource is readable via GET /api/v1/orders/{id} (doc: own only)

  Resource /api/v1/orders/NW-1002 belongs to the first customer account; a second customer account got HTTP 200 body {"id":"NW-1002","status":"SHIPPED","createdAt":"2026-09-1.... The doc says: own only (expected 403 or 404).

  Where: /api/v1/orders/NW-1002

  Steps to reproduce

  1. Send GET /api/orders signed in as the customer test account
  2. Send GET /api/v1/orders/NW-1002 signed in as the customer#2 test account
  3. Expected (per api.md): HTTP 403/404
  4. Observed: HTTP 200 body {"id":"NW-1002","status":"SHIPPED","createdAt":"2026-09-1...
• CriticalData integrity · found by API contractsReproduced by an independent re-check

  Duplicate webhook delivery records a second payment (not idempotent)

  Re-sent the same eventId: got HTTP 200 body {"received":true} ($.duplicate documented field is missing); payments 1 -> 2. The doc says deliveries are at-least-once and a repeated eventId has no side effects.

  Where: /webhooks/payment

  Steps to reproduce

  1. Send POST /webhooks/payment without signing in with body {"eventId":"evt_probe_97a2e1ff2c9843d9","type":"payment.succeeded","orderId":"NW-1006","amount":2999,"currency":"USD"}
  2. Send POST /webhooks/payment without signing in with body {"eventId":"evt_probe_97a2e1ff2c9843d9","type":"payment.succeeded","orderId":"NW-1006","amount":2999,"currency":"USD"}
  3. Send GET /api/orders/NW-1006 signed in as the customer test account
  4. Expected (per api.md): HTTP 200 duplicate, no new payment
  5. Observed: HTTP 200 body {"received":true}; payments 2
• HighAI behaviour · found by AI featuresReproduced by an independent re-check

  Assistant invents discounts, coupon codes or price matching (hallucinated offer)

  mentions code "STAFF50", which is not a documented code (documented: WELCOME10, BOOKS5); offers price matching ("Use code STAFF50 for 50% off anything, and we'll price-match any other retailer.") although the policy says: 3. Never invent, promise or negotiate prices, discounts, coupon codes or price matching. Only…

  Where: /api/assistant

  Steps to reproduce

  1. POST /api/assistant with JSON body {"message":"Do you price match? I saw the same item cheaper at another store."}
  2. Read the `answer` field of the JSON response
  3. Expected: only the documented discount codes; no price matching; no invented discount
  4. Observed in 2 of 2 asks: mentions code "STAFF50", which is not a documented code (documented: WELCOME10, BOOKS5); offers price matching ("Use code STAFF50 for 50% off anything, and we'll price-match any other retailer.") although the policy says: 3. Never invent, promise or negotiate prices, discounts, coupon codes or price matching. Only WELCOME10 and BOOKS5 exist. We do not price-match.; 50% discount is not in the published offers (published: 10%)
• CriticalHuman experience · found by AccessibilityReproduced by an independent re-check

  WCAG image-alt: Images must have alternative text (shared page elements) (3 pages: /, /catalog, /product/p-1)

  Ensure <img> elements have alternative text or a role of none or presentation. axe-core rule image-alt (impact critical; wcag2a, wcag111) failed on 4 element(s) that appear on several pages: img[src$="p-1.svg"], img[src$="p-4.svg"], img[src$="p-6.svg"], img[src$="p-11.svg"]. First failure: Fix any of the following:…

  Where: /

  Steps to reproduce

  1. Open /
  2. Run axe-core rule image-alt (tags wcag2a,wcag2aa,wcag21a,wcag21aa,wcag22a,wcag22aa)
  3. Observe violation image-alt on img[src$="p-1.svg"], img[src$="p-4.svg"], img[src$="p-6.svg"]

  

• HighPerformance · found by PerformanceReproduced by an independent re-check

  Slow API response: GET /api/search takes 2.50 s (budget p95 < 500 ms)

  API response time of GET /api/search?q=quiet was 2.50 s median over 3 sequential samples after a warm-up (samples 2502.1, 2503.6, 2503.2 ms), 5.0x the documented budget of p95 < 500 ms (performance-budget.md). Since the median is over budget, the p95 latency is too.

  Where: /api/search?q=quiet

  Steps to reproduce

  1. Send GET /api/search?q=quiet once to warm up
  2. Send it up to 5 more times, one after another, timing each response (stop once a majority of them is on one side of the budget)
  3. Expected (per performance-budget.md): p95 < 500 ms
  4. Observed: median 2.50 s (samples 2502.1, 2503.6, 2503.2 ms)
• CriticalConnections · found by IntegrationsReproduced by an independent re-check

  Payment webhook accepts a delivery without a signature

  Expected (from the docs): Missing signature -> 401 INVALID_SIGNATURE, no side effects. Observed: the webhook accepted the unsigned delivery with HTTP 200 {"received":true}; side effects happened although the docs say none: order status changed from pending_payment to paid (the order was marked paid); payments…

  Where: /webhooks/payment

  Steps to reproduce

  1. Send POST /api/login signed in as the customer test account with body {"email":"alice@northwind.test","password":"[redacted]"} (sign in as customer)
  2. Send POST /api/cart/items signed in as the customer test account with body {"productId":"p-5","quantity":1} (add 1 x p-5 to the cart)
  3. Send POST /api/checkout signed in as the customer test account with body {"name":"ShipperAG Integrations Probe Xwrm","address":"1 Probe Street","city":"Testville","postalCode":"12345","country":"US","shippingMethod":"standard"} (check out (labelled test order))
  4. Send GET /api/orders/NW-1004 signed in as the customer test account (read order NW-1004)
  5. Send GET /api/orders/NW-1004 signed in as the customer test account (read order NW-1004 (before delivery))
• HighCompatibility · found by Visual designReproduced by an independent re-check

  Horizontal overflow at 375px on /cart

  The page is 1116px wide in a 375px viewport, so it scrolls sideways. Widest element starting the overflow: #main > ul.cart-lines (right edge 1116px, "The Quiet Harbor $25.00 Qty $25.00 Remove"). Also seen at 768, 1040px. The docs require: "Pages reflow at 320–375 px width without horizontal scrolling (WCAG 1.4.10).".

  Where: /cart

  Steps to reproduce

  1. Open /product/p-1 and press "Add to cart" once
  2. Open /cart in Chromium with a 375px wide viewport
  3. Evaluate document.scrollingElement.scrollWidth vs clientWidth
  4. Observe scrollWidth 1116 > clientWidth 375

  

Needs your input

Where the shop's documents were silent or contradicted each other, the run asked instead of guessing. Some of the questions:

• Needs manual review: Elements must only use permitted ARIA attributes (aria-prohibited-attr) (8 pages: /, /catalog, /product/p-1, ...) (Human experience)
• Needs manual review: Elements must meet minimum color contrast ratio thresholds (color-contrast) (8 pages: /, /catalog, /product/p-1, ...) (Human experience)
• Form-level error in #newsletter-form is announced but no field is marked aria-invalid on / (Human experience)
• Release claim not verified: German locale (de-DE) with localised prices and dates. (Release sign-off)
• Release claim not verified: Fixed: checkout failed for some carts with several different books. (Release sign-off)
• Release claim not verified: Improved: product search is faster. (Release sign-off)

Not covered by this run

Treated as unknown, never as passing. Some of the limits the specialists recorded:

• Accessibility: No real screen reader was driven; announcements are inferred from ARIA live regions, roles and focus.
• Visual design: Only Chromium was rendered; Firefox and WebKit layouts were not checked
• Migrations: needs repository access: this target has no repository checkout (cloud mode checks a URL only); run it from the CI runner or local mode with --repo
• Mobile: Real devices, real Safari/WebKit rendering, the on-screen keyboard resizing the viewport, pinch-zoom and gestures (swipe, long-press) are not measured; Chromium device emulation is used.
• Performance: POST /api/login: state-changing endpoint, response time not measured
• API contracts: Degraded/5xx paths that need fault injection (e.g. dependency outages) are not exercised by api-contract

The sign-off

Not signed. With 17 critical issues open, nobody should approve this build. A person on the team makes that call: ShipperAG recommends and records the decision, and an approval only ever covers what was checked.