We Didn't Build an Autonomous Coding Agent. We Built a Cockpit.

Every AI coding demo tells the same story: an agent picks up a ticket, writes the code, runs the tests and opens a pull request while the developer gets coffee.

We built a platform that does exactly that. It takes a story from Jira, refines it with the right context, writes the code in an isolated sandbox, spins up the services and mocks it needs to run end-to-end tests, and opens a PR with the evidence. It’s deployed in production.

The most important decision we made, though, was not to make it autonomous.

This post covers the why: the problem we were solving, the bet we made, and what I’d tell another engineering leader thinking about the same thing. The architecture deep dive comes in a follow-up post.

The problem: agents in the wild don’t scale

AI adoption in most engineering teams starts the same way. A few developers install Claude Code, Cursor or OpenCode, get great results, and the tools spread. That’s good. It’s also where the problems begin when you have dozens of developers across many services.

• Quality depends on who writes the prompt. Two developers working on the same story get very different results.
• Context lives in people’s heads. The agent doesn’t know your architecture standards, the service that owns that table, or why a past decision was made.
• There is no governance. Which model ran, with what data, at what cost, producing which change? Nobody can answer that.
• Cost is invisible until the invoice arrives.
• Security is a matter of individual discipline. An agent on a developer’s laptop has access to whatever that developer has.

Individual productivity goes up. The organization doesn’t become more predictable. For a team that ships to production every week, predictability matters as much as speed.

The real bottleneck isn’t code. It’s the story.

This was the insight that changed the design.

When we looked at why agent output was inconsistent, the cause was rarely the model. It was the input. User stories are written for humans who share context: “add validation to the payment form” means something specific to the person who wrote it and to the team that has worked on that form for a year. To an agent, it’s ambiguous.

An agent doesn’t fix ambiguity. It amplifies it. You get a confident, well-structured, fully tested PR that solves the wrong problem.

So the question shifted from “how do we make the agent write better code?” to “how do we make sure the agent receives a story worth executing?”

Why a cockpit

A cockpit isn’t autopilot. The pilot isn’t flying every second, but they decide where the plane goes, they see what the systems see, and they can take control at any moment.

The usual approach puts human judgment at the review of the final PR. That’s the most expensive place for it: reviewing 1,500 generated lines to find out the agent misunderstood the requirement is slow, tiring and demoralizing. A correction before any code exists costs almost nothing.

Same people, same agent, same story. The difference is where the humans spend their judgment. In the autopilot model, they spend it untangling a diff. In the cockpit, they spend it deciding what and how, and the final review becomes a confirmation, not an investigation.

In the platform, every story lives on one pipeline. Each row shows exactly where it is, from functional refinement to a PR with evidence, and who owns the next step. The same pipeline is seen through the role you’re working as: Product Owner, Architect or Developer.

Interface recreated for this post. Story names, services and data are illustrative.

That shift only works if two things are true: the input is refined well, and the agent can prove its own output. The next sections cover both.

Two refinements, two kinds of knowledge

A story has two questions to answer before anyone writes code: what should be built, and how it should be built in our system. Those are different questions, owned by different people, and they need different knowledge.

flowchart TD
    A[Jira story] --> B

    subgraph F["1. Functional refinement"]
        B{{"Product Owner<br/>+ agent"}}
        K1[("RAG: documentation<br/>by product")]
        K1 -.-> B
    end

    B --> C[Refined story<br/>clear behavior + acceptance criteria]
    C --> D

    subgraph T["2. Technical refinement"]
        D{{"Architect<br/>+ agent"}}
        K2[("RAG: documentation<br/>by microservice")]
        K2 -.-> D
    end

    D --> E[Execution brief]
    E --> G[Isolated sandbox<br/>code + full environment]
    G --> H[E2E tests against<br/>real services + mocks]
    H --> I[Pull request]
    I --> J{{Human review}}

    style B fill:#f5d76e,stroke:#b8962e,color:#000
    style D fill:#f5d76e,stroke:#b8962e,color:#000
    style J fill:#f5d76e,stroke:#b8962e,color:#000

The first refinement is functional. The Product Owner works with an agent grounded in a RAG over the documentation of each product: business rules, existing flows, edge cases that were decided long ago. The output is a story whose behavior and acceptance criteria are unambiguous. No technical decisions yet.

The second refinement is technical. The architect takes that refined story and works with an agent grounded in a RAG over the documentation of each microservice: contracts, dependencies, data ownership, standards. The output is the execution brief: which services change, how, under which constraints, and how it will be tested.

For the Product Owner, the functional refinement is a conversation, not a form. The assistant reads the story and the product documentation, asks only what isn’t clear, and points to the document behind each question. Every answer lands immediately in a brief written in plain language. When nothing is left open, the PO signs it off and it moves to the architect.

Here’s what the next step looks like for the architect. On the left, the Product Owner’s functional brief, signed off and locked. On the right, the technical plan: the approach, the path across services, tasks, edge cases and what “done” means. Each side shows which documents it was grounded on. The architect can ask the AI to revise, edit by hand, and only then approve the plan for build.

Splitting the knowledge this way matters more than it looks:

Mixing both in one RAG gives you an agent that answers business questions with implementation details and technical questions with product marketing. Keeping them separate means each human gets the context that matches their decisions, and each refinement catches a different kind of mistake.

There are three human checkpoints, and they aren’t equal. The two refinements are where the real decisions happen. The PR review should mostly confirm that the plan was executed.

The execution brief: a contract between intent and execution

The output of the cockpit isn’t code. It’s an execution brief: a structured document that turns a refined story into something an agent can execute without guessing.

mindmap
  root((Execution brief))
    Scope
      Functional goal
      Explicitly out of scope
    Acceptance criteria
      From functional refinement
    Technical plan
      Affected microservices
      Contracts to respect
      Standards and constraints
    Test plan
      Unit and integration
      QA-style scenarios
    Traceability
      Source story
      Who refined it
      Knowledge used

The brief is what makes execution reproducible. The same brief should produce an equivalent result regardless of which developer triggered it or which model ran it. It’s also an audit trail: when a PR looks wrong, you can tell whether the problem was the requirement, the technical plan or the execution.

I’ll go deeper into what goes inside a brief in the architecture post.

None of this works if the agent only writes code

Here’s the part that’s easy to miss, and the one that makes the whole model hold together.

If the agent just writes code and runs unit tests, you’ve moved the problem, not solved it. Someone still has to pull the branch, start the service, wire up its dependencies and click through the flow to know if it actually works. The human review goes back to being an investigation, and the cockpit loses its point.

So the agent, Claude Code or OpenCode running inside the sandbox, doesn’t stop at the code. It brings up the whole environment and runs end-to-end tests against it.

For every task, the sandbox starts the microservice being changed, along with the infrastructure it depends on, and mock servers for everything outside its boundary: other services, third-party providers, external APIs. Then the agent runs the end-to-end scenarios from the brief’s test plan. When something fails, it reads the logs, fixes the code and runs again, until every test passes.

The PR that comes out isn’t “here’s some code, please check it”. It’s code plus evidence: E2E results and logs showing the change behaves as the refined story says it should.

Look at the timeline: the first E2E run failed on a rounding rule, and the agent fixed it and ran the suite again before anyone looked at it. And if the reviewer wants something changed, they ask for an adjustment in plain language and the agent continues in the same session, with the same environment.

This is what turns the final human review into a confirmation. The reviewer isn’t asking “does this work?” anymore. They’re asking “is this the right way to have done it?”, which is a much better use of an engineer’s time.

It’s also the hardest part to build. Spinning up a realistic environment per task, keeping mocks faithful to the real contracts, and doing it fast enough to be useful is real platform work. I’ll cover how we approached it in the architecture post.

From personal rules to platform

In I Didn’t Write the Code. I Wrote the Rules. I described how I build products with agents as a single developer: the value is in the rules and context you give the agent, not in typing the code.

This platform is the same idea at team scale.

flowchart LR
    A["One developer<br/>rules in a file"] --> B["One team<br/>shared prompts"] --> C["Organization<br/>rules as infrastructure"]
    C --- C1[Product knowledge]
    C --- C2[Microservice knowledge]
    C --- C3[Refinement as a process]
    C --- C4[Governed execution]

The rules stop being a file on my laptop and become infrastructure: two knowledge bases, a refinement process everyone goes through, and a controlled environment where execution happens.

What “in production” means right now

It’s deployed and working. It isn’t yet used massively, and I want to be honest about that, because I think it’s the most underrated part of building these platforms.

flowchart LR
    A[Built] --> B[Deployed] --> C[Trusted] --> D[Adopted] --> E[Default way of working]
    style A fill:#7bc67b,stroke:#3d8b3d,color:#000
    style B fill:#7bc67b,stroke:#3d8b3d,color:#000
    style C fill:#f5d76e,stroke:#b8962e,color:#000

Building the platform was the easier half. Adoption is the harder one. A developer who already gets good results from their own agent setup needs a reason to go through refinement. A Product Owner needs to see that the functional agent actually knows the product. An architect needs to trust that the brief reflects the standards. A team lead needs to see that the PRs coming out are worth reviewing.

That’s not a technology problem. It’s a product problem, and an internal platform has to be treated like a product: with users, onboarding, feedback loops and a reason to choose it over the alternative.

What I’d tell another engineering leader

If you’re thinking about building something similar, these are the decisions I’d make again:

1. Start with refinement, not generation. Code generation is getting commoditized fast. Good input is not.
2. Separate the “what” from the “how”. Different owners, different knowledge, different refinement.
3. Put human judgment where it’s cheapest. Reviewing a plan beats reviewing a diff.
4. Make knowledge a shared asset, organized by who uses it. Product docs for product decisions, microservice docs for technical ones.
5. Make cost and usage visible from day one. A model gateway with routing, fallback and cost control (we use LiteLLM, which I wrote about separately) is not optional at scale.
6. Isolate execution. Agents run in sandboxes, not on laptops.
7. Make the agent prove its work. Full environment, mock servers, end-to-end tests. A PR without evidence is just a longer review.
8. Treat it as a product. Shipping to production is the starting line, not the finish.

Autonomous agents make great demos. In a real engineering organization, what you need is a system where humans decide what gets built and how, agents do the building, and there’s a clear contract between the two.

That’s the cockpit.

In the next post: the architecture behind it, with four planes (experience, control, execution and knowledge), the tools we chose, and the trade-offs we made.