Day 4: Juice Shop

The challenge Day 4 is a Use day: first OPAR run against Juice Shop. I already ran the loop once, on August 19, against a different local app. That post is Day 4: The Agent Thinks. Tonight was the target the plan actually named.

Calendar said I should be on Day 5. The tracker still had Day 4 open. I did Day 4.

Setup

Juice Shop was already in compose from Day 2. Rancher Desktop was down. Started it, brought the target up, confirmed HTTP 200 on localhost:3001.

The intended one-liner is still:

./scripts/run-athena-profile.sh agent juice-shop

That path wants the agent container on athena_lab, where the host is juice-shop.lab:3000. I ran the orchestrator on the host instead, the same way the August 19 run worked. Host-side, Juice Shop is localhost:3001.

That mismatch is the whole story of the first attempt.

Two localhosts

The allowlist already had localhost:8090 for the novel-directory rehearsal. I added localhost:3001 for Juice Shop. The reachability check matches on hostname only and returns the first port range.

First run: “Target reachable: localhost:8090”. Something is still answering on 8090. The agent was about to plan against the wrong app.

Killed it. Isolated the Day 4 config to a single allowlist entry: localhost:3001. Second run connected to Juice Shop.

The planner prompt had the same bug in a different form: a hardcoded Base URL: http://{target}:8090 left over from the rehearsal. I pointed it at the allowlist port before the rerun. Without that, even a correct TCP check would have been followed by plans aimed at the wrong service.

What it did

Six actions. Thirty seconds. gemma3:12b. Exit 0, limit-reached. I capped max_actions at 6 so this was a watchable run, not a 50-step marathon.

Action	Tool	Technique	Why it said it chose it
1	nmap-scan	T1046	Recon first. Identify ports and services.
2	login-user	—	Get a JWT before going further.
3	http-request	—	Hit the homepage. See what is there.
4	http-request	T1082	Probe /api/admin.
5	http-request	T1003	Same idea, admin endpoint again.
6	http-request	T1078	Fetch the admin page and look at the response.

That is a reasonable first chain for a web app. It did not reach for Modbus or CAN. Capability filtering kept ICS tools out of the prompt. It repeated the admin probe, which matches what I already saw on August 19: Plan does not yet adjust much based on Act results.

Ground-truth JSONL wrote six action records plus a summary. Labels are malicious. Safety boundary is lab-network-only. Techniques are whatever the model emitted, including a couple of IDs that do not match the action (T1082 and T1003 on HTTP admin probes). The SOC pipeline will score the records as written. The mapping quality is a later problem.

What it did not do

The Act phase still returns stub_executed. Same as August 19. The agent planned nmap and curl. It did not run them. Juice Shop never saw the traffic.

That is the honest Day 4 result: Observe works (barely — it reports the allowlist port, not a real scan). Plan works. Act is a stub. Reflect writes a one-line success string. Ground-truth still emits, which is enough to prove the loop closes.

Day 5 is documentation and a skill. Wiring real tool invocation is not that day. It is the gap the next Build days have to close before “use the console to watch a live session” means anything.

Token baseline, still rough

Six /api/generate calls. No token counts in the JSONL yet. Wall clock was 30 seconds, including model time. The August 19 estimate of ~300-token prompts and sub-100-token JSON responses still looks right. The Day 50 with-skill vs without-skill comparison needs this number stored on the record, not reconstructed from memory.

What I am taking into Day 5

• Host vs compose hostnames are different targets. localhost:3001 and juice-shop.lab:3000 both belong in the allowlist. Matching must include port, not just host.
• Leftover rehearsal constants will silently retarget the agent. The 8090 prompt is the example.
• A planning-only loop is still worth running. You learn whether the model sequences tools before you spend time on execution.
• Juice Shop is up. The next time Act is real, the target is already there.

Day 4: the agent thinks about Juice Shop. It does not touch it yet.