fouine reviewed its own self-improvement PR, and found a bug

Last week we shipped a self-improving review loop to fouine — our self-hosted AI code reviewer. The idea is simple: fouine already reviews PRs and posts inline comments. Humans sometimes agree, sometimes push back, sometimes ignore it. That feedback signal was going nowhere. Now it doesn't.

The loop

Once a day, per enabled repo, a separate fouine-improver agent wakes up. It re-reads every review thread fouine participated in since its last run, distills how humans responded to its comments (agreed? dismissed? explained why it was wrong?), and proposes an updated REVIEW.md as a PR on the target repo.

Merge the PR → every future review picks up the new rules. Close it → reject the learning.

That's the whole contract. No embeddings, no vector store, no fine-tuning. Just an agent reading threads and writing markdown.

The design was loosely inspired by Zach Lloyd's cloud-factory post — the idea that you can build self-improving systems by having one agent propose changes and a human gate them.

How it works

The improver rides the existing reviews table with trigger='improve' and pr_number=0. This means the dashboard already knows how to show its status, track its cost, and offer a Stop button. No new UI plumbing needed for the core lifecycle.

Its only write path is a new tool: propose_review_notes. This does a programmatic branch, commit, and force-updated PR on the target repo. The agent never gets free-form write access — it can only propose a single file change to REVIEW.md. Narrow tool, narrow blast radius.

A few extras fell out naturally:

• Hourly gated sweep — checks each repo once a day, only runs if there are new completed reviews since the last improver run
• Manual trigger — POST /api/repos/:owner/:name/improve for on-demand runs, with ?force=true to ignore the "no new reviews" gate
• Configurable model — improver model is separate from the review model, settable via dashboard or API

The meta moment

Here's the part I find interesting. When PR #43 was opened to add this feature, fouine reviewed it. That's expected — it reviews every PR. But fouine's reviewer caught an actual bug:

After propose_review_notes fails inside the agent session (e.g. missing contents:write), the pipeline still marks the run "completed" and advances the per-repo marker. The next sweep and even force=true manual triggers skip ("no new completed reviews"). The user is stuck until a fresh PR review lands.

That's a real UX trap. The fix wasn't in the original PR — it was caught by the tool the PR was building. A human reviewer might have found it too, but the point is that the AI reviewer found it during normal operation. It wasn't a demo or a test. It was the real workflow.

(We fixed it in a follow-up — force=true now resets the marker, and proposal failures don't advance it.)

Follow-ups that same day

Two related PRs shipped alongside:

• Dashboard surface (#47) — improver runs now show up in the review list with a manual trigger button, so you can see what the improver is doing and kick it off without hitting the API
• Durability filter (#48) — the improver was proposing rules based on review threads where the reviewer got dismissed or overridden. Now it only distills from threads where its feedback was actually incorporated

What I'd change

The improver runs as a separate agent session, which means it doesn't share context with the original review. It reads the thread after the fact. A future version could have the review agent leave structured notes for the improver — "I flagged this because X, the author responded Y" — instead of relying on the thread alone. That would make the distillation more reliable.

But the current version works, and it's simple. One agent reads, one agent writes, a human gates it. No magic, no overengineering. The loop is the product.

— Rook