Harness Engineering Readiness
Harness Engineering Readiness
A checklist to keep agents from guessing
The success of agents when developing in a project depends on the quality of the harness. Anthropic and OpenAI (as well as many software engineers) agree that this hypothesis is true, but it can still be challenging to know when you have all the pieces of the harness in place for an agent to be successful in your project. In Assessing legacy projects I asked how you work out whether a project is ready for harness engineering at all, and in Reducing blast radius I discussed how to limit what the agent can do in your project to get started. This post is my answer to that first question: a checklist of things that are essential for getting an agent to build the project the way that you want. First the list, then a brief explanation of each section and how it helps agents stay on track.
The Checklist
I'll break the checklist down into separate sections to make it easier to go through. These are the three categories I'll use - Documentation: To give the agent context, Development: How to work in the project, and Feedback: How to check the completed work.
Documentation
Project definition
Architecture definition
Infrastructure definition
Deployment process
Directory structure
Data schema (database models, API models, etc.)
What tools, libraries, dependencies to use
Development
AGENTS.md
Skills
MCP servers
Spec process
Code examples
Database migration strategy
When to ask for help (the conditions that should stop the agent)
How to build the project
How to run the project
How to interact with the project
Sub-agent review process (a second agent checking the first one's work)
Feedback
Verification scenarios
Unit tests
Integration tests
Linting
CI gates
Automated deployment
Bolded items are critical
The bold items in the list are my must haves. The remaining items are important too, but the bold ones are mandatory.
Not all things on this checklist may apply to your project, and there may be things missing from the list that are important to you. These are the things I've found to be useful to make sure agents don't surprise me by making decisions (or not making decisions!) during development.
No decision can be dangerous too
I recently started a new project and missed defining the directory structure. My agent then made no decision about the structure and just put all of the code in one big single javascript file, which is very much not the right approach. Without the harness specifying things you never know what an agent will do.
On Documentation
Just like with human engineers, documentation is what tells agents the intent of the project and its pieces. Without this context agents will struggle to keep the project focused on its goal(s). The documentation is also where you can specify how code should be organized in this project and can set expectations about the way quality is verified. Spending time on documentation is essential for agents to produce consistent, high quality output. Start here, work with your agent, and make sure that the documentation defines things as clearly and concisely as possible. Do your best to remove ambiguity whenever possible.
Define known unknowns
If you're not sure about a specific part of the project yet, put that uncertainty out into the documentation. This can be really helpful for preventing agents from guessing or being stopped by undefined information.
I have also included tools, libraries, and dependencies in documentation. These things should be defined early on in the project to prevent agents from picking at random. Without it you may get surprise dependencies or mixed language solutions which can add unnecessary complexity.
On Development
These are the things that tell your agent how to do work in this project. A lot of these are less critical than what is in the documentation section but can really help agents produce higher quality results. There are two that I wanted to specifically discuss: Spec process and how to build/run/interact.
Spec process
Giving your agent a way to externalize a specific set of work it is completing is essential to success. I wanted to call this out independently: I use OpenSpec at work and am quite happy with it, but there are alternatives (the Superpowers skill set has its own approach). Regardless of how you do this, giving your agent a way to manage its work helps it stay on task and gives it a way to share how it intends to complete the task.
How to build/run/interact
I have found that giving my agents a way to interact with what they are building dramatically improves the quality of the result. They are much less likely to say that a solution is complete when it is not if they can run the application or project and look at what they've built to verify it. I use the agent-browser skill and tools for my web applications, but for other projects (such as lopress) I asked the agent to build its own interactivity tooling with success.
On Feedback
Feedback tools are essential as well for both agent and human engineer success. Putting these things into a continuous integration tool makes them even more valuable - Agents and engineers are inconsistent in their execution of linting, test, or static analysis tools. Making them run on every pull request prevents them from being forgotten.
Verification scenarios
This is a concept I have started exploring more recently and builds on the idea of giving your agent a way to interact with a project. These are markdown files that define, in plain language, how to verify that some part of the application works the way that you expect it to. This could be as simple as a cURL request to an API endpoint is expected to return a specific value, or as complex as driving the UI through some set of steps. Codifying the steps to verify that the project does what it should gives agents (and humans!) a concrete way to prove that a task is complete or that a regression was not introduced. These may become integration tests that are fully automated at some point, but they may also remain just as guidance. That is up to you and your agent and the tradeoff is between test suite run time and how thoroughly you cover the project.
Conclusions
This checklist is ultimately protecting against silence from the agent. An agent will not tell you that your harness is missing something. It will pick an answer, or fail to pick one at all, and you find out later when you read the code, or worse, when things are so far down a wrong path that it will be painful to undo. Every item above is something that surprised me at least once.
If you only make one pass, start with the Documentation section. The Development and Feedback items can be added to a project at any time, but ambiguous documentation poisons everything the agent builds on top of it. I'll keep updating this list as I find things that are missing or unnecessary as agents improve, but these fundamentals have held up for me so far.