Keegan Choudhury

Keegan Choudhury / Dreamcatcher

Dreamcatcher Beta

A prototype for giving AI answers a paper trail, with the source behind each claim and a way to challenge the result.

I wanted to be able to argue with an answer instead of just believing it or throwing it away. Dreamcatcher is my test of whether a paper trail helps. It is a prototype, and this page says plainly what it does not yet prove.

Figure 1. A silent, edited preview of the Beta. The sources and reviewer actions in it are fictional sample data. MP4

Walkthrough

  1. Read the captured passage behind a claim and check its wording.
  2. Inspect the evidence score and the reviews recorded for an answer.
  3. Open a challenge, and the score is withheld while the objection is reviewed.
  4. Explore whether the confidence estimates match outcomes on separate examples. Independent human labels are the next step.

At a glance

Source passages
behind each claim

Every claim points to a captured passage.

Recorded reviews
for each answer

Reviews are kept with the answer they cover.

Challenge
the result

Question a claim and follow its evidence.

Not measured
real-world accuracy

The evidence score is not a probability that an answer is correct.

System diagram

An evidence trail

An evidence trailIllustration of the idea: captured source passages support claims, with recorded reviews and a way to challenge the result. Scores describe recorded checks and reviews, not correctness probabilities.Captured source passageClaim and recorded reviewsChallenge the resultAn evidence trailIllustration of the idea: captured source passages support claims, with recorded reviews and a way to challenge the result. Scores describe recorded checks and reviews, not correctness probabilities.Captured source passageClaim and recorded reviewsChallenge the result

An illustration of the idea, not a screenshot of the app.

Figure 2. The evidence trail behind one claim.

How it's built

Each claim is linked to the source passage that was captured for it, and the answer keeps a record of its checks and reviews across 150 automated tests (99% line coverage). The evidence score describes that record.

I also tried to automate the checking. In my model lab I tested whether a model's internal features could tell a checked claim from a guessed one, or spot when an agent drifts from its instructions. The best signal I found only worked because it fired on stock-market words, and the drift detector scored AUROC 0.412 against a bar of 0.60 I had set in advance. Neither was good enough, so neither shipped. I would rather show the failed test than pretend the shortcut worked.

Limits

Dreamcatcher is a prototype for testing evaluation designs, and the experiment so far uses synthetic labels. I haven't measured real-world accuracy yet.

Next I want to collect independently checked examples, test how well the scores predict correctness, and learn which explanations help people validate an answer for themselves.