The Agent Ran for 13 Hours

I had built a job-posting pipeline with two AI agents. The Normalizer cleaned up messy role names, locations, and skills. The Analyzer read the original posting alongside those facts and explained the company, the work, and the requirements.

A few jobs looked good when I checked them by hand. That did not tell me what would happen across hundreds of real postings.

So I asked Codex, a separate coding agent, to evaluate both agents across 500 jobs, improve them, and continue until the run was complete.

One hour in, I checked its progress. Codex had found bugs, changed tests, and built review machinery. It had evaluated zero jobs.

That was the first sign that it was working hard on the wrong problem.

What Codex was evaluating

The first three steps collected the source data. The final two were the AI agents Codex had to evaluate.

The Analyzer used both the original posting and the normalized facts. Codex reviewed the outputs of both agents. Valid JSON was not enough; the facts and requirements also had to survive the process.

If the Normalizer chose the wrong standard role, the stored job was mislabelled. If the Analyzer turned a nice-to-have qualification into a hard requirement, the job would look more demanding than it really was. A simplified example would be turning "you will work on payment systems" into "payment systems experience is required." The output sounds reasonable, but the source never said it.

How I set up the task

Codex could read the repository, edit code, run the pipeline, and create smaller subagents for focused reviews. It managed the evaluation; the Normalizer and Analyzer were the systems under test.

My brief was simple:

• Run ten batches of 50 unseen jobs through the complete pipeline.
• Review the real output, not only test results.
• Fix repeated problems on the same batch, then test the fix on fresh jobs.
• Prefer general improvements over rules for one awkward example.

I also instructed Codex to maintain three Markdown files without rewriting old entries. progress.md tracked the run, decisions.md kept proposed fixes and decisions, and results.md stored jobs and failures. They gave a long-running agent memory outside the chat.

The four steers that mattered

Steer 1 — Run the experiment, not just the laboratory

Codex created temporary reviewer subagents for source quality, output quality, runtime failures, prompt review, and evidence integrity. They quickly found real bugs.

A green test could accept an empty analysis. A failed job could be silently replaced. A timeout was shorter than a normal model response. One saved result did not even show whether the reviewers had inspected that run's output.

All of this was useful. It was also not the experiment I had asked for.

Codex kept polishing the evaluation machinery because those problems were clean and easy to prove. Is the schema valid? Did the test pass? Does the file exist? The difficult question was still waiting: is the AI output actually good?

I added a rule that a failed job had to stay in its batch. Then I gave the first steer: stop preparing forever and run the actual system.

Steer 2 — Keep the system generative

Once real outputs started arriving, the reviewers found mistakes in meaning. A responsibility could become prior experience the job falsely required. An example technology could become compulsory. A list of alternatives could become a list of requirements.

Codex did what engineers often do. It added exact phrase checks, section parsers, negation rules, links back to source sentences, and more tests.

Then the adversarial reviewers found new edge cases, so Codex added more rules. Soon we were building a small programming language for job descriptions.

This is where I got worried. The Normalizer and Analyzer were generative. There was no single perfect sentence they had to reproduce. I wanted Codex to judge whether the meaning was faithful, not force every output through a growing pile of English-language rules.

I asked it to remove the phrase-level checks, keep the basic format checks, and change the prompt only when the same meaningful failure appeared more than once.

Codex rolled back the clever parser.

I realised it was approving fixes simply because the reviewers sounded confident. If this continued, we would spend the entire run improving the evaluator instead of the two product agents.

Steer 3 — Judge a batch, not one weird output

The run used the model label gpt-5.6-sol with high reasoning in Codex Desktop. Inside the pipeline, the two product agents started on gpt-5-mini and later moved to gpt-5.6-terra with high reasoning. Along the way, Codex also tried different prompts, timeouts, and worker settings.

But the evaluation had another problem. One debatable omission in the first few jobs could stop the whole attempt. The batch received no score, and the loop started again from job one.

Seven hours into the run, Codex had reached attempt 24 without completing a single batch. gpt-5.6-terra had produced 13 outputs before the review stopped. Of those same 13, one reviewer passed five and the other passed six.

I read the outputs myself. Many were useful and faithful. They were simply not identical to the reviewer's imagined perfect answer.

At that point, I was frustrated. It felt like the evaluator was failing more than the product agents.

I asked Codex to review at least 25 random jobs together and judge the batch as a whole. A real factual mistake should still fail. Normal variation in wording should not kill the run.

Nothing had changed in the product agents. I had only stopped the evaluator from treating every difference in wording as a failure.

About 18 minutes later, the first full batch of 50 finished. It was not a clean pass. The reviewers still found 15 to 18 material failures, including missing constraints and misleading role information.

That result was more useful than the previous seven hours. We finally had a picture of the real problems across 50 jobs instead of one argument about the first strange output.

GPT-5.6 Sol worked for 13 hours without giving up. But working for a long time does not mean working on the right thing.

Steer 4 — Teach with examples, then finish the run

Now the failures were useful. Repeated mistakes became short examples in the system prompts. One-off oddities stayed in the notes instead of becoming new production rules.

I gave Codex one final push: finish all ten batches, review both agents, move faster, and make the remaining decisions without waiting for me.

The loop finally became simple:

pick 50 jobs and keep that batch fixed
→ run both agents
→ two reviewers inspect every result
→ Codex resolves disagreements
→ did the same important failure appear again?
  yes: fix it and rerun the fixed batch
  no: record it and finish the batch
→ validate the fix on 50 fresh jobs

The fixed batch showed whether a change helped. The fresh batch showed whether the change worked beyond the examples that inspired it.

Two separate reviewers inspected every output. In one batch, they looked at the same outputs and produced scores of 48 out of 50 and 37 out of 50. The gap was not in the data. It was in how they read the rules.

The stricter reviewer blamed the AI for contradictions already present in the source and treated small omissions as major failures. Codex compared both reports and kept only the mistakes that met our final material-failure rubric.

After that, two subagents reviewed one batch while Codex started the next. The work finally moved faster.

The run covered 500 unique jobs. On the final validation batch of 50, Codex's reconciled review scored both the Normalizer and Analyzer 49 out of 50. That is not 98% accuracy across all 500 jobs. It is one result from one batch, on this data, under a rubric that took us several hours to make clear.

The complete task took 13 hours and 46 minutes. Codex created 46 reviewer and specialist subagent tasks, then sent them 131 follow-up assignments. That much parallel work made the shared rubric and memory files necessary; otherwise each reviewer could pull the run in a different direction.

More subagents gave me more feedback, but not always more clarity. A clear definition of ‘good’ mattered more than the number of agents.

Codex left behind a reusable evaluation harness and better prompts. It also produced 1.4 GB of temporary evidence, along with progress.md, decisions.md, and results.md. I kept the useful machinery and deleted the experiment's debris.

What I would change next time

First, I would demand five real outputs immediately. A small sample would have exposed the important questions before Codex spent hours building evaluation machinery.

Second, I would use different stopping rules for different failures. A broken output, dead worker, or missing result file can stop a run immediately. A disagreement about meaning needs a proper batch. Failing fast on opinion is an easy way to overfit to one unusual example.

Before asking an agent to fix anything, I would check four things:

• Was the source itself wrong or contradictory?
• Did the model misunderstand it?
• Did another part of the pipeline drop or change it?
• Was the evaluator simply being too strict?

The agent knew how to keep going. My job was to decide when it should continue, when it needed steering, and when it was solving the wrong problem.