The 4-week audit: a fixed-scope engagement for AI initiatives that stalled
A walkthrough of my fixed-scope audit: what I look at, what I write, and how to know if the AI initiative is worth continuing — or starting over.
I get a version of the same email every month. Someone at a mid-sized LATAM company — a CTO, a head of engineering, sometimes a founder — has an AI initiative that has stalled. The team built a prototype six months ago, the prototype worked, the demo went well, and then the initiative slowed to a crawl. The prototype never became a product. The team is now spending maintenance on a thing that does not pay for itself. The CTO wants to know whether to keep going or to stop.
The honest answer is “it depends”, and the reason it depends is that the team usually does not have the data to decide. They know the prototype works. They know the production path is unclear. They do not know how big the gap is, or which parts of the gap are technical and which are organisational. They are making a binary decision with continuous information.
I built a fixed-scope engagement to fix that. It is called the audit, it takes four weeks, it costs a fixed price, and it ends with a written recommendation. This post is what I do in those four weeks, what I write at the end, and how to use the output to make the keep-or-stop decision.
Week 1: read the code, not the slides
The first week is the code week. I read every line of the prototype. I read the orchestration code, the prompt templates, the tool definitions, the evaluation harness, the deployment scripts, the error logs. I do not read the slides, the design docs, or the strategy memos. The code is the truth.
I am looking for three things. First, the shape of the system: what are the boundaries between the agent, the tools, and the data? Where does the prompt meet the runtime? Where does the runtime meet the infrastructure? Second, the failure modes: what happens when an LLM call returns garbage? What happens when a tool returns an error? What happens when a user sends something the agent was not trained for? Third, the test surface: what is covered, what is not, and what is the cost of adding coverage.
I write a private technical memo. The memo is a map of the prototype as it actually is, not as the team thinks it is. I send it to the CTO at the end of the week. The memo is the first artifact the team sees, and it is often the first time the team has seen their own prototype described accurately.
Week 2: talk to the people, not the org chart
The second week is the people week. I talk to the engineers who built the prototype, the product manager who is supposed to ship it, the customer success person who is supposed to support it, and the executive who is supposed to fund it. I do not talk to the org chart. I talk to the people who do the work.
I am looking for the gap between the design and the reality. The design said the agent would be a copilot. The engineers think it is a search engine. The product manager thinks it is a chatbot. The customer success person thinks it is a thing they will have to support but no one has told them how. The executive thinks it is a competitive moat. None of these are wrong, but they cannot all be true at once.
I write a second memo, this one focused on the human side. Who is the intended user? What problem are they trying to solve? What is the current workaround? How much would they pay to not have the workaround? The answers are almost never what the team assumed.
Week 3: measure the production path
The third week is the production week. I measure the distance between the prototype and a thing that can run in production at the volume the business needs. I do not build anything. I measure.
The measurements are concrete. How long does a single request take, end to end, including LLM calls, tool calls, and database queries? How does that latency change under 10x load? What is the cost per request, in dollars, including LLM tokens, infrastructure, and engineering time? What is the rate of hallucinations, measured against a labelled set of 100 examples? What is the rate of tool failures, measured against a representative trace?
I write a third memo, this one focused on the production gap. The memo lists the engineering work that has to happen, the order it has to happen in, and a rough estimate of how long it would take a senior team to do it. The memo does not recommend whether to do the work. That is week 4.
Week 4: write the recommendation
The fourth week is the writing week. I produce a single document, twenty to thirty pages, that is the engagement’s deliverable. The document has four sections.
The first section is the technical state: a one-paragraph summary of the prototype, a one-paragraph summary of the failure modes, and a one-paragraph summary of the test surface. The second section is the user state: a one-paragraph summary of who the intended user is, what they are trying to do, and how much they would pay. The third section is the production state: a one-paragraph summary of the gap between the prototype and production, the work to close the gap, and the rough timeline. The fourth section is the recommendation: keep going, stop, or pivot.
The recommendation is not a guess. It is a derivation from the first three sections. If the technical state is solid, the user state is real, and the production gap is small, the recommendation is keep going. If any of the three is broken, the recommendation is stop or pivot, with a clear reason.
How to use the output
The output is a written document, not a presentation. The CTO takes it to the executive team. The executive team reads it, asks questions, and makes the keep-or-stop decision with the data on the table. The decision is still theirs. The audit just makes the decision possible to make.
If the recommendation is keep going, the document also serves as the roadmap. The work to close the production gap is the next quarter’s engineering plan. If the recommendation is pivot, the document is the rationale for the pivot. If the recommendation is stop, the document is the post-mortem.
The audit is not for every stalled AI initiative. It is for the ones where the team is willing to look at the prototype honestly and the executive is willing to make a real decision. If either of those is missing, the audit will produce a document that no one reads. In that case, stop is the right answer and the document is not needed.