Assessing legacy projects for Harness Engineering

Harness Engineering is a hugely popular topic at the moment. It seems to be the latest way to talk about agentic coding - Using an agent to do most of the software development, with humans giving direction. There are lots of articles about "what harness engineering is" or "how to do harness engineering." As a Staff Engineer working on a legacy codebase, with a passion for using LLMs to generate quality software, I've been considering a different question: How can I determine what the "readiness" of a project is for adopting harness engineering?

To answer this question, we need to dig a bit deeper into what makes a project ready to adopt harness engineering, and then we need to take a look at specific parts of the project to determine the viability of autonomous code generation (or agentic coding, or whatever you'd like to call it). I'll start with answering what readiness means, then break down specific targets to focus on based on that definition.

What makes a project ready for Harness Engineering

The answer to this question varies from project to project based on the risk tolerance of the team and the project. I think that looking at the existing development process can provide some good insight. Here are some questions to consider about the existing approach that can give us a sense of the project's readiness for autonomy:

1. How does the team decide that a change should be merged?

2. Does the project have any autonomous quality checks already?

3. Does the project have any manual quality checks?

4. How does a new developer start working on this project?

5. What expectations are defined for developers in this project?

6. How do developers know what to build next?

The answers to these questions give us a better sense of what it means for a project to be ready for automation. I like to think of these buckets: Setup (getting the environment ready), Development (Actually doing the work), and Validation (making sure everything works). The questions above map onto them like this: question 4 is about Setup, questions 5 and 6 are about Development, and questions 1 through 3 are about Validation. A project that is ready for the team to adopt harness engineering has simple setup steps (maybe an "init" script), clear development expectations, and automated validation (Runs tests, lints, or other measures in Pull Requests).

The questions posed above should give you a good sense of the readiness of the project, but it's up to you or the team to make the call on exactly what the requirements are. However, I think that there are two questions that your answer must be "yes" to before harness engineering will work for you:

• Do I trust my CI checks to verify changes before they are merged?

• Can my development environment be set up automatically?

If you cannot answer those two questions with "yes", you will not get good results consistently from automated agentic development. While development is the "meat" of software, it's the Setup and Validation that are essential to doing harness engineering, since the agent does Development for you. Once you are confident in those two sides of the development process (starting and finishing) it's much easier to know that your agent has done the work you want them to.

Getting the "start" working is sometimes simple and sometimes not. In a greenfield project a single "init" script might be enough, but in an older codebase it can mean untangling undocumented dependencies, database seeding, or secrets before the environment comes up cleanly. It's usually worth the effort, but don't assume it's free. Either way, the finish is where trust is hardest to earn, so that's the part I'll focus on. Let's look at some basic questions to use to make the end trustworthy.

Building trust in Validation

It's easy to build validation steps - unit tests, linters, and other deterministic tools have been around for a long time. They have been fundamental to maintaining quality in legacy applications for a long time. When I say the validation steps need to be trustworthy I don't mean they need to be reliable (although that helps). What it means for validation to be trustworthy is that when all the tools pass the change is ready for production. How to get there depends on the project, so think carefully about what shortcomings your existing validation has and how you and the team need those to improve before you can trust them. Once you know what to do, have your agent do it for you!

It's up to you / the team to define what level of trust you want - but it's good to write down what you think your validation does for you, and how those tools help you trust an incoming change. For me, I want my validation steps to check a few things:

• Does this change break existing functionality? (unit tests, e2e tests)

• Is this code written in the same way as the rest of the project? (linters)

• Does this code contain any blatant security issues? (static analysis)

• Does this code contain any obvious bugs? (also static analysis)

• Is the code tested to prevent future changes from breaking it? (unit tests)

There are other questions you may want your validation to address but these are the main things that come to mind for me. Once you have defined your expectations for the validation steps you can verify that the existing mechanisms in the project do these things for you. If they don't coding agents make it easy to add new validation! Agents are excellent at writing unit tests and setting up automated scripts - Github workflows, Jenkins, or any other CI solutions are simple to define and configure with the help of an Agent.

Conclusions

The biggest thing I have learned when determining whether a project is ready to go "all in" on agentic coding and start doing harness engineering is that trusting the automated validation is fundamental to trusting the output of an agent. Without trust in the checks it is very hard (read: impossible) to trust the results of automated coding. When starting on the path to harness engineering on an existing project, get the CI pipeline right and the rest will follow easily.