Skip to content
Back to Idea Validation

Listening for Surprises

Actively listen for statements that contradict your beliefs — they are the highest-signal moments in an interview.

Why

Confirmation bias makes you hear what confirms your beliefs and ignore what contradicts them. In an interview, the moments that surprise you ("oh, I did not expect that") are the highest-signal moments — they are where your mental model is wrong, and wrong models are what kill products. If you finish 5 interviews and nothing surprised you, either you have perfect understanding (unlikely) or you were not listening for surprises. The discipline is to enter each interview with a written list of your beliefs, and actively note when a statement contradicts one. These contradictions are gold: they reveal the gap between your model and reality, and closing that gap is the whole point of validation. A interview that confirms everything you believed is a wasted interview — you learned nothing.

When

During every interview. Takes 5 min before (write beliefs) and active attention during. Synthesize after each interview.

How

1. Before the interview, write your beliefs (5 min)
   - "I believe maintainers review on weekends because they have no time during the week"
   - "I believe the pain is the technical difficulty of review"
   - "I believe they would trust AI to pre-vet simple PRs"
   - List 5-10 beliefs; these are your hypotheses to test

2. During the interview, listen for contradictions
   - When the user says something that contradicts a belief, note it immediately
   - Mark it: "SURPRISE: [belief] was wrong because [what they said]"
   - Do not argue or correct — just note and ask "tell me more about that"

3. Listen for intensity and emotion
   - "Reviewing is fine" said flatly → not a pain
   - "I spent my ENTIRE Saturday on PRs" said with frustration → real pain
   - Emotion is signal; flatness is not

4. Listen for workarounds
   - "I wrote a script to..." → the pain is real enough that they built a tool
   - "I just skip it" → the pain is real but they have given up
   - Workarounds are the strongest evidence a pain is real

5. After the interview, review your beliefs
   - Which were confirmed? (good, but low learning)
   - Which were contradicted? (high learning — update your model)
   - Which were not addressed? (ask in the next interview)

6. Synthesize surprises across 5 interviews
   - A surprise heard once is an anecdote; heard 3+ times is a model update
   - The biggest surprises often reframe the problem entirely

Example

Beliefs before interview 1:
  1. Maintainers review on weekends because they have no time during the week
  2. The pain is the technical difficulty of review
  3. They would trust AI to pre-vet simple PRs
  4. They use the GitHub PR UI for review
  5. They want faster review

Interview 1 surprises:
  - Belief 1 confirmed: "I review on Saturday because workdays are for my day job"
  - SURPRISE: belief 2 wrong — "reviewing is not hard, it is just tedious and long"
  - Belief 3 not addressed
  - SURPRISE: belief 4 wrong — "I pull PRs into VS Code because the GitHub UI is too slow for 200-line diffs"
  - SURPRISE: belief 5 wrong — "I do not want faster review, I want review to NOT happen on my weekend. Even if it took the same time, if it happened on Tuesday afternoon I would be happier."

The biggest surprise (belief 5): the pain is not speed, it is timing. The maintainer wants review moved off the weekend, not made faster on the weekend. This reframes the solution entirely: not "AI that reviews in 30 seconds" but "AI that reviews on Tuesday and queues the result for the maintainer to approve on their schedule".

This surprise, heard in interview 1, was confirmed in interviews 2, 4, and 5 (4/5 pattern). It reframed the product from "fast AI review" to "async AI review that gives weekends back" — a fundamentally different product.

Common Problems

  • ⚠Not writing beliefs before the interview — without a written list, you cannot identify contradictions; you will just feel the interview "went well".
  • ⚠Hearing what you want to hear — "she said reviewing is hard, that confirms my belief" when she actually said "reviewing is tedious, not hard"; listen to the exact words.
  • ⚠Ignoring surprises as one-offs — a surprise in interview 1 is an anecdote; ask about it in interviews 2-5 to see if it is a pattern.
  • ⚠Not updating your model after surprises — if you finish 5 interviews with 3 surprises but your beliefs are unchanged, you wasted the interviews; update the model.

Best Practices

  • ✓Write 5-10 beliefs before each interview — this is the list you check against; without it, confirmation bias runs unchecked.
  • ✓Mark surprises explicitly in notes — "SURPRISE: [belief] wrong because [quote]" makes the learning visible and actionable later.
  • ✓Probe surprises with "tell me more about that" — the first statement is the surface; the follow-up reveals the depth and confirms it is real.
  • ✓Treat the biggest surprise as the most valuable output — the interview that confirms everything is low-value; the one that surprises you is high-value.