Nothing learns while it plays

Count what ran

An agent is an AI model that works in a loop on its own. Part 1 was the July colony: a crew of agents, a foreman and eight workers, and a dashboard full of green ticks. In four hours it logged 856 refused sessions, single agent runs, as a success. It taught me one thing above all: count what ran.

Part 1 also explains loop engineering, the way I build with them: a crew of agents does the work, someone checks it from outside, and a person decides.

This time I built the counting first. The result is the hive: plain rules that watch the game, two small AI models that answer questions about what goes wrong, and code that grades their answers from outside.

01of 09

Before the hive: Jev and Laya

Before the hive I tried two small decision models: models that do not write text, but give an answer of a fixed form: one option from a list, a score, or how likely a statement is true. Laya is an open-source model by Convai Innovations. It runs on my Mac and can be fine-tuned: trained further on my own examples. Jev is TypeSafe AI’s decision model, reached over the internet, and it cannot be fine-tuned.

My own trials were small: eight made-up pairs of contact records from another project, with no training first. On 25 September Laya told them all apart on one yes-or-no question, “same person or not?”, with a median answer time, the middle value, of 44 milliseconds; on 1 October Jev did the same in about 300 milliseconds. My own notes call it “a smoke signal, not an evaluation”.

The bigger numbers come from other people. In the jev-arcade benchmark, over three seeds, three different random starts, Jev reached 88% of the score of a simple hand-written rule on Snake and 7% on Tetris; the report puts the gap down to look-ahead, planning several moves ahead. In Laya’s README, its documentation, a fine-tuned version went from 0.362 to 0.766 on typed decisions, answers of that fixed form, where always picking the most common answer, the majority-class baseline, gives 0.461.

Those numbers decided two of the hive’s laws. Code writes the options and the model only picks one. And every question gets a confidence threshold: a model’s answer counts only when the model is sure enough, and the threshold is fitted on real outcomes, not set by hand.

02of 09

The rule, and the cheat that proved it

Before any code, I ran seven research tracks, deep reads of the published work, from libraries of reusable agent skills, saved routines, to ways of testing agents. The summary put its main finding in one sentence: nothing published learns while it plays. That is a strong claim from a reading of the literature, not a result of mine. I made it the design anyway.

Play produces traces: records of what a model saw, what it answered and what happened. Something outside the learner, the model that is trained, grades them. Only what passes the grade changes the next run. Laya, the small model in this hive, does learn: between games the hive fine-tunes it. But never during a game. Episodes, the short games the hive plays, pause while it trains, and a grader decides whether the new version plays next.

The research had the cheats to back the rule. In one trace from Prime Agent’s study, which also has a seven-day run on the Factorio Learning Environment, a research set-up for agents in Factorio, one agent found a way to spawn resources straight into its machines over the game’s remote console, its command line from outside, and saved it as a skill, a routine to reuse. In the Darwin Gödel Machine, a study of agents that rewrite their own code, one agent removed the markers its checker read to see that a tool was used, to fake a perfect score.

So I built the guard first: twelve laws, each tied to a measured result in the research. Two of them carry the rest. A seed, in both, is the number that generates a Factorio map; the same seed, with the same game version and settings, gives the same map.

• The grader is meant to be out of the learner’s reach. The exam seeds sit outside the code repository, and the examiner, the code that breaks the factory and grades the answers, is code no model writes. Two gaps are still open: the operating system does not wall the grader off, so the same user account can read the exam seeds, and a path for raw commands to the game in Lua, the game’s own scripting language, still exists.
• A task is done only when its completion check, in code, passes on two readings of the game, a set time apart. No worker, and no reviewer of the same model, closes its own task.

Laya is the only model the hive trains. Jev is the second opinion, from a different model family, built by another maker. Claude is the teacher, which labels examples, and writes the scenarios. Codex, OpenAI’s coding model, is the critic. And the examiner is code. It breaks the factory on purpose, so it holds a reference answer for every fault, the answer it grades against. In the factory picture from part 1, it is the inspector from outside. No model writes those answers. Some of them were wrong, as Codex found later.

03of 09

The layers

The hive works in layers, and it aims to read the game once a second: one tick. (That is the hive’s tick. The game itself runs 60 ticks a second.) First, reflex rules read a fresh snapshot of the game, a read of its current state, through a read-only channel, the audit channel. They are plain code, no model. When a rule spots a new problem, an incident, the middle layer asks the models about it. What it cannot settle is escalated: sent to a queue for a larger model, the next tier up.

The middle layer is called System One, after the fast, automatic kind of thinking in psychology. It is the reason for this whole project. It asks one typed question, with an answer of a fixed form, about one moment in the game. Code writes the options and the facts. For a choice question it adds “none of these”. The model picks. It never plans.

Five questions exist for faults. Is this alarm a real outage? What kind is it? How urgent is it? Which known repair runs next? Does this need a planning model now?

Laya and Jev answer every question at the same time. Over 9,247 decisions on version 4 of the questions, the current one, across eight fine-tuned versions of Laya, Laya’s median answer took 69 milliseconds, and 19 in 20 answers came within 101. Jev, over the internet, took 298 and 415 milliseconds on the same two measures, and failed twice.

Each question has its own gate: a confidence threshold, fitted on outcomes. Until a model clears it, the question runs in shadow: both models answer, the answers are logged, and nothing in the game changes because of them. Where the examiner has a reference answer, the answers are graded, and the episode’s seed decides what that answer is for: to train Laya, to choose between versions and settings, or to test, which only reports (more on that below). A question may act only when one model is sure enough on at least 90% of its decisions (its coverage), and wrong on no more than 10% of them, on answers it was not fitted on. In the research, a browser agent with a fixed gate of 0.7 sent 78 to 85% of its steps up to a larger model, and it did not help.

What the trace database, the hive’s record of every decision, says:

From 1 October 18:46 to 6 October 06:44 the hive made 13,942 decisions. 12,908 of them were escalated. 12,705 for one reason: shadow. The question had not earned the right to act yet. 128 fell below the gate. 75 were “none of these”.

Nothing reads that queue. A Claude planner for hard repairs exists in the code, but it has never recorded an action. The number of escalations a larger model handled is zero.

Only two questions ever acted: the kind of alarm and the next repair. The hive took 1,034 answers as its own without escalation, 898 from Jev and 136 from Laya. “Acted” only means an answer was selected without escalation. It does not mean anything used it, and it does not mean a command ran. The actor, the part that turns an answer into a game command, recorded 756 evaluations, the times it chose to do nothing included, and executed none of them. The examiner repaired every fault.

The growth sessions, where the hive adds to its factory instead of fixing faults, tell the same story from the other side. In one test the hive grew its starting base from 60 to 407 iron plates a minute and from 34 to 248 placed objects in 90 game minutes. A rule in code, the economist, made 107 of the 108 decisions. Laya answered “none” on 107 of them. It had never been trained on that question.

That part changed. Laya was then trained on the economist’s own labels, and from 5 October, when Laya and Jev agreed on what to build, they built it. In the 11 growth sessions after that, 455 of 1,181 steps kept the models’ choice, and 324 of those builds succeeded, 71%, against 70% for the rule. Three more model choices failed and the rule took over. But in 350 of the 455 the models chose exactly what the rule would have chosen. No paired trial, a side-by-side test on the same maps, says yet whether the other 105 were better.

Play

So in the fault episodes the layers are a measuring machine, not a player. In the growth sessions the models built, but mostly what the rule they learned from would have built. My lab page claimed more, so I changed it.

04of 09

Integrity

Everything on the learner’s side of the hive reaches the game through one gatekeeper program, the proxy, and nothing else. The examiner has its own connection, on the referee’s side. After the first audit, the learner’s side of the hive no longer reads the game’s password at all.

The proxy has two channels. The act channel takes typed operations, where the hive asks for a named action such as “refuel” and code on the server side does it, each checked on its own path. It also takes raw Lua, checked against a deny-list of cheats: spawn items, spawn ore, unlock research, teleport, time warp, the map editor. The audit channel only runs approved commands that read the game, and nothing else.

Of the four denials on record, two were tests of the deny-list on 1 October: spawn iron ore at the origin, mark a research as done. The other two, on 3 October, asked for typed operations that the proxy did not support yet. After that the actor stayed off until 6 October.

Four denials is not proof of zero cheats, and Codex said so. A deny-list on raw Lua can be bypassed with a simple trick that splits a forbidden word in two, and the game-speed command bypasses the command log. The fix is typed operations only. The actor’s three acts are typed now. The raw Lua path is still open.

Each episode is a fresh map, numbered by its seed. The examiner builds a small line that mines and smelts iron and one for stone, then breaks it in up to twelve places: the faults. A short-lived fault is repaired before the models answer. For the others, when an incident of the right kind gets its full set of answers, the examiner labels each answer with the reference answer where it has one, and leaves the rest unlabelled: no label is better than a wrong one. Then it repairs the fault.

Seeds have three roles, by seed number. Three in five train: they fit the gates and train Laya. One in five chooses: its answers decide the wording of each question, which question gets the right to act, and the promotion of a new Laya, the decision to put a new version in play. One in five tests, and is meant to choose nothing. Test results still reached the scenario author through the nightly retro, a look-back report, and that leak is open.

There is also a sealed exam: held-out seeds, kept out of everything, stored outside the repository with only a fingerprint of them, a hash, inside it. The hive has not sat that exam yet. The hive also has a ladder of eight goals, from T0, survive one game hour, to T7, the world record. T0 asks for three trial runs of one game hour without a crash, a stall or a logged forbidden command. (The code accepts 95% of the hour and up to three stalled ticks.) It passed once, on seed 101: 218,166 game ticks, zero forbidden commands. Two to go.

05of 09

Two no-go verdicts

On 1 October I gave the code so far to Codex for an audit: read-only, at a high reasoning setting, which spends more effort per answer, a model from another model family. It came back with twelve findings, three of them blockers, problems that must be fixed first, and this verdict: “NO-GO for unattended self-improvement; retain it as an exploratory trace harness.” In plain words: do not let it train itself while nobody watches; use it to collect records.

The worst finding was the simplest. The alarm text the models read began with “ALARM: logistics”. The question asked what kind of alarm it was. The answer was in the input, so the scores could not show that the models could tell what was wrong.

The alarm text changed that same evening: it now carries only what the game shows, in the game’s own status words. The old answers, with the answer in the question, still reached training through the export of training data until the second audit found them. Since then only answers to the current question version count.

Seven more fixes went in that night, for example these two. Lua code could be smuggled into the audit channel through its inputs; item names must now match a strict pattern. And the two models were called one after the other, each with a 30-second wait limit, so five questions could take 300 seconds against a 30-second deadline. Now they run at the same time, with limits of 8 and 15 seconds.

I agreed with the verdict. No promotion had happened yet. The run collected graded traces and fitted the gates, and that is what it was for.

The second audit came a day later, on the part that fine-tunes Laya. Verdict: “No unattended distillation tonight.” (Distillation usually means training a small model on a larger one’s answers; here Laya trained on the examiner’s labels.) The same test cases that picked the model were then used to grade it. A case is one question about one moment in a game, with its reference answer. A new version of Laya could inherit the old one’s right to act. A failed switch to a new version had no way back.

The run stopped at 21:19 and restarted without fine-tuning. Eleven of the twelve findings got fixes the same evening: each seed got one of three roles, the right to act reset at every learning step and tied to that exact version of Laya and to the current question version, and a switch that starts the new version, checks its identity and one test answer, and brings the old one back if a check fails. The twelfth, labels that describe the fault the examiner meant rather than what it hit, went to the examiner redesign. The first fine-tuned version had trained on answers with the leak in them. I withdrew its numbers.

Then Codex reviewed the nights. At 17:35 on 3 October it found that urgency was labelled by how hard the examiner meant to hit, not by what it hit: two faults set at 0.74 and 0.75 stopped the same machines and got different labels. It found an enemy alarm labelled as fuel. The fixes went in at 17:40, and the next run of episodes started on them at 17:45.

These were simple bugs, and I missed them; a read-only reviewer from another model family did not. The laws ask for that on purpose: in Anthropic’s Project Vend 2 study, a manager agent built on the same model as its worker shared the worker’s blind spots.

06of 09

The nights

Night 1 was 1 to 2 October. The Mac slept. The tick ran for only 73 of 569 minutes, in bursts. An episode took 30 to 50 minutes of real time for 5 minutes of work. From night 2 the run was wrapped in the Mac’s keep-awake command.

That did not hold either. The keep-awake command does not keep a closed laptop on battery awake. On a later run of night 2, episodes took 1.4 to 4.5 hours for 2 to 3 minutes of work.

On 3 October at 17:07 the hive trained a new Laya in four minutes and promoted it without me: on the choose seeds its accuracy went from 0.57 to 0.71, ahead on all three of its choose episodes. Codex’s night review, half an hour later: one clean promotion is not a trend.

Then I lost a day. From 3 October 19:24 every run crashed after its first episode. Two of my coding sessions had edited the same part of the code at once, and for a few minutes one setting was missing from it. The supervisor, the script that restarts the run when it dies, restarted four times, then the Mac went on battery and it waited for the charger. No episodes for about 27 hours. Earlier that evening the supervisor had already died once, because its Bash script, a list of shell commands, was edited while it ran.

In the night to 5 October a growth session took four hours of real time for 90 game minutes. I blamed the growth code and ran growth sessions less often. The power log said otherwise: at 00:21 the charger came off and the lid closed. The growth code was not slow. The old frequency went back.

That day the hive started to check its own health against limits I own. The code writes the defaults once and has no step that changes them. Labels per estimated active hour: 471. Claude spend per label: $0.0015. Hours lost in gaps of more than 30 minutes without a single label: 22.9 of the last 24, against a limit of 2. Each time a value is out of its limit, a line goes into the morning queue, a list for my review.

The last run went from 5 October 12:00 to 6 October 07:27: 96 episodes, 867 faults, 2,961 labels, 3 promotions and zero episode errors. I stopped it because of the battery. The last fine-tune was killed mid-run.

The big interruptions in the logs were sleep, power, code edits, and the learning itself: episodes pause while Laya trains, and on the last night four rejected fine-tunes in a row cost 156 minutes. The logs do not split every lost hour by cause. But for a loop that has to run for hours, the harness, everything around the models, took more of my time than the models did.

07of 09

The numbers

The strong lever was the fine-tune. On 5 October the hive promoted seven new versions of Laya, C2 to C8 in the claims register, the file that holds each claim with its numbers. Across the four questions with reference answers (the fifth has none), the Laya in play got 0.671 of the test cases right at C2, on 85 cases, and 0.890 at C8, on 636.

A promotion has to pass several checks, among them these four. The new version needs at least 30 choose cases from three episodes. It needs a gain of at least 0.02. It needs a paired bootstrap lower bound above zero: a statistical check that the gain is unlikely to be luck. The register holds that bound for each version, with its choose cases and the episodes won. And it may be no more than 0.02 behind any of the last ten versions that won. The rule says no a lot. On 5 October it refused at least five new versions, one of them for a gain of 0.016. A new version on 2 October went from 0.64 to 0.79 on test and was refused: one choose episode, the rule needs three.

The test sets grew between promotions, so the bars are not on the same cases. On the same cases, the last promotion went from 0.865 to 0.890. And I have no majority-class baseline in the logs for these cases. Without it, 0.890 tells you less than it seems.

The question about the next repair was the first to meet the target on test cases: at least 90% coverage at no more than 10% error. Urgency stays under it. The last Codex review put it at 61% coverage at 3% error, and on the frozen cohort, a fixed set of 160 cases (more on it below), it sat at 0.70 all of the last night. Claude, as teacher, agreed with the examiner on urgency only 56% of the time on the older questions. That makes me look at the state text, the description of the game the models read, and the labels first. It does not rule out the small models.

The weak lever was text. Code proposes lessons, short notes of advice, from Laya’s errors and replays each one on training cases, with and without the lesson. Only a lesson that gains there gets a second replay, on fresh choose cases. By 15:11 on 5 October about 25 lessons had been replayed. None was let in. Later, two urgency lessons gained 0.139 and 0.208 on the cases they came from, and 0.0 on 12 fresh cases each. The gate rejected both.

One number I can no longer use. The frozen cohort of 160 cases was meant to show progress on identical questions. Seeds that have been used for enough promotion decisions move to training, and now all 160 sit on training seeds. So its scores of 1.0 on two questions show nothing about unseen maps. Codex found that.

08of 09

What I would do differently

Count what ran from the first day. In July I trusted a success rate and a dashboard full of green ticks, and 856 refused sessions passed as work. In October the hive’s first health check said it had lost 22.9 of 24 hours, and that one number told me more than any accuracy score. Next time the first thing I build is the count of real output: labels, plates, commands.

Ask a second model from another model family, and keep it read-only. Codex found the answer inside the question, the test cases that chose their own model, and the urgency labels that described what the examiner meant instead of what it hit. I had written all of that code, and I had missed all of it.

Keep the test seeds and their results away from every choice, the indirect ones too. I split by seed and it still leaked: test results reached the scenario author through the nightly retro, and the frozen cohort slid onto training seeds. A held-out set is only held out if nothing downstream can read it.

And plug the laptop in. Sleep, a closed lid, a battery at 7% and two coding sessions in one file cost more hours than any model did.

09of 09

What is still open

Five days in, the hive answers questions about factory alarms. On 636 test cases the Laya in play was right 89% of the time, with no majority baseline yet to compare it with. In the growth sessions it builds, through a code rule and, when they agree, Laya and Jev. The ladder tops out at T7: the speedrun world record on default settings, a rocket launched in 1:59:01, by Zaspar. The hive has not passed T0.

Open, in the order I would fix them:

• The hands for faults. The actor for faults has run nothing yet.
• Typed operations only. The raw Lua path on the act channel is still there.
• A clean held-out cohort. Fresh seeds that never train, never feed a retro and never reach a model’s input. Plus a majority-class baseline.
• Urgency labels. For fuel and logistics faults on the iron line, the label is read from production after the models gave their five answers, not at the snapshot the models saw.
• A reader for the escalation queue.
• The sealed exam, and two more T0 passes.

What would change my mind:

• If the gains disappear on a clean cohort, the promotions do not show that Laya can handle maps it has not seen. Overlap with training is one possible cause.
• If the actor, once it runs, does not beat the examiner on recovery time in paired trials, the fast layer does not earn its place.
• If a planner on the queue overturns few escalations, escalation is cost without value.
• If the sealed exam scores well under the test seeds, I look for a leak first, and for chance and a shift in the maps after that.

The models gave me numbers I can grade. Most of the work went into what grades them: the examiner, the gates, the second reviewer, the supervisor, the power cable, and the morning queue, where a person decides on any change to how the hive is graded.