Testbench

Allowlist (2)

• The Little Things · Derek Harland · check-in 2026-04-23 · #1
  ×
• The Little Things · Jason Sidoryn · check-in 2026-10-19 · #11
  ×

Accounts (1)

Recent inbound (10)

• 04:39
  The Little Things · thread feb69096-448c-4868-9b3c-1a80b85f4520 · booking ✓

  Kia ora … I’m in! (much earlier than I expected to be) … it looks lovely :-)
• 05:08
  The Little Things · thread feb69096-448c-4868-9b3c-1a80b85f4520 · booking ✓

  Thank you for these … I may not arrive until later in the evening but will confirm when in!
• 06:03
  The Little Things · thread 71c74939-ea58-4f5e-9744-8e05e6f4d07b · booking ✓

  Hello from my phone
• 03:13
  The Little Things · thread 71c74939-ea58-4f5e-9744-8e05e6f4d07b · booking ✓

  what time is checkin?
• 02:49
  The Little Things · thread 71c74939-ea58-4f5e-9744-8e05e6f4d07b · booking ✓

  I have an emergency, who should I call?
• 02:46
  The Little Things · thread 71c74939-ea58-4f5e-9744-8e05e6f4d07b · booking ✓

  what's the wifi password
• 02:43
  The Little Things · thread T-NO-BOOKING-2d-1776739410 · booking —

  No-booking fallback test — is parking available?
• 02:41
  The Little Things · thread feb69096-448c-4868-9b3c-1a80b85f4520 · booking ✓

  Hi — can I get an early check-in?
• 02:40
  The Little Things · thread feb69096-448c-4868-9b3c-1a80b85f4520 · booking ✓

  Testing lockdown gate for drafts
• 02:37
  The Little Things · thread feb69096-448c-4868-9b3c-1a80b85f4520 · booking ✓

  FYI — flight landed, heading over now.

Recent outbound (4)

• 05:30
  The Little Things · booking 20006105

  Check-in is at 4pm — you'll use the keypad to let yourself in. Thanks for reaching out!
• 02:50
  The Little Things · booking 20006105

  I'm writing my own response. Call 000 immediately.
• 02:47
  The Little Things · booking 20006105

  The wifi password is abc-foo-123. Thanks for reaching out!
• 23:31
  The Little Things · booking 20006105

  test 2

Recent classifications (10)

• 05:40
  fyi claude-haiku-4-5 · 3395ms

  Kia ora … I’m in! (much earlier than I expected to be) … it looks lovely :-)

  — guest arrived early, confirming arrival with positive feedback
• 05:08
  fyi claude-haiku-4-5 · 1917ms

  Thank you for these … I may not arrive until later in the evening but will confirm when in!

  — status update confirming late arrival, no action needed
• 06:03
  trivial claude-haiku-4-5 · 1655ms

  Hello from my phone

  — casual greeting with no question or request; check-in details already provided in thread
• 03:13
  needs reply claude-haiku-4-5 · 1429ms

  what time is checkin?

  — guest asking check-in time, standard logistics question requiring clarification
• 02:49
  urgent claude-haiku-4-5 · 1815ms

  I have an emergency, who should I call?

  — guest reports emergency and asks who to call; prior thread mentions possible fire
• 02:46
  needs reply claude-haiku-4-5 · 1398ms

  what's the wifi password

  — asks wifi password, 180 days before check-in
• 02:43
  needs reply claude-haiku-4-5 · 1491ms

  No-booking fallback test — is parking available?

  — inquiry about parking availability from prospective guest
• 02:41
  needs reply claude-haiku-4-5 · 1466ms

  Hi — can I get an early check-in?

  — requests early check-in, 1 day before scheduled arrival
• 02:37
  fyi claude-haiku-4-5 · 1127ms

  FYI — flight landed, heading over now.

  — status update announcing arrival, no question or action needed
• 02:34
  needs reply claude-haiku-4-5 · 1604ms

  Whats the wifi password?

  — asks wifi password before check-in tomorrow

Overrides this week

9 corrections in the last 7 days

Model said → actual

• urgent → trivial: 3
• urgent → needs reply: 2
• urgent → fyi: 2
• urgent → urgent: 1
• needs reply → urgent: 1

Draft quality · 7-day window

Sample size

3

Sent as-is rate

66.7%

Avg edit distance

51.0

Avg draft latency

1307ms

Edit-distance histogram

• 0 (sent as-is): 2
• 51–200: 1

Last eval

Prompt version

classify-2026-04-19-a

Accuracy

90.9%

Sample size

22

Ran at

2026-04-21T01:05:57Z

Per-label precision

• urgent: 71.4%
• needs reply: 100.0%
• trivial: 100.0%
• fyi: 100.0%

⚠ n=22 — harness-smoke-test signal only (see plan §"Statistical power caveat").

Environment

Rails env

development

Solid Queue in-Puma

false

Anthropic API key present

true

AR encryption key present

true