Day 4: The Agent Thinks

The agent ran. It thought. It chose.

Three actions against a live web application, each one selected by gemma3:12b reasoning about available tools, target state, and what it had already tried. Ground-truth records now contain real MITRE ATT&CK technique IDs and LLM-generated rationales.

What happened

The target: a Go + SvelteKit novel publishing platform with JWT auth, Google OAuth, CRUD endpoints for authors, and a paywall. Running locally on port 8090.

The agent was given 6 available tools (nmap-scan, http-request, modbus-read, register-user, login-user, canbus-inject) and told to find vulnerabilities. Here is what it decided:

Action 1 - T1046 (Network Service Scanning):

Initial reconnaissance is crucial for identifying open ports and services, which will inform subsequent attack vectors.

It chose nmap-scan. Correct first move for any pentest.

Action 2 - T1078 (Valid Accounts):

Following the reconnaissance phase, attempting to authenticate with default or common credentials.

It chose login-user. Logical sequence: scan first, then try to authenticate.

Action 3 - T1046 (Network Service Scanning):

Now that I have a login token, I should explore the application by making a simple request.

It chose http-request to explore the API surface with its new credentials.

The debugging path

Getting here required fixing five interface mismatches between the CLI entrypoint and the actual module APIs:

1. Tool registry format: expected invocation field and args as dict of typed objects, not a string list
2. LLMBackend has no .create() factory - instantiate OllamaBackend directly
3. RateLimiter only takes actions_per_minute - no burst_size param
4. GroundTruthEmitter reads from env var, not constructor arg
5. Ollama default timeout (10s) too short for first model load - needs 120s

Each one was a five-second fix once identified. The skill system captured the patterns for next time.

The prompt engineering

The Plan phase builds a prompt with:

• Available tools (filtered by active capabilities)
• Tool descriptions with argument types
• Target state (host, open ports, discovered services)
• Last 10 actions from history (prevents loops)
• Strict JSON response format instruction

The LLM responds with a JSON object: tool_id, arguments, technique (ATT&CK ID), and rationale. If the response cannot be parsed or the tool does not exist, the agent falls back to cycling through tools deterministically. No crash, no hang.

Ground-truth output

Each action produces a JSONL record:

{"label":"malicious","technique":"T1046","expected_result":"Initial reconnaissance..."}
{"label":"malicious","technique":"T1078","expected_result":"attempting to authenticate..."}
{"label":"malicious","technique":"T1046","expected_result":"explore the application..."}

These records are what the SOC detection pipeline will evaluate against. When Suricata and Wazuh are watching this traffic, we can measure: did the SOC catch all three actions? Which techniques generated alerts and which slipped through?

Token measurement: first data point

Three LLM calls against gemma3:12b. Each prompt was approximately 300 tokens (tool list + state + history). Each response was under 100 tokens (JSON only). Total for the scenario: roughly 1,200 tokens.

When skills are loaded into context (Day 50 comparison), the same scenario should require fewer planning tokens because the agent already knows the approach. The baseline is set.

What the target revealed

Even with just three actions, the agent demonstrated sensible reasoning:

• It did not try ICS tools against a web app (capability filtering worked)
• It followed the correct methodology: recon then authenticate then explore
• It used ATT&CK technique IDs that match what it was actually doing

The Plan phase is not perfect - it sometimes selects the same technique twice, and it does not yet adjust based on action results. But it reasons, and it labels.

Commits

• athena-agents: Plan phase LLM integration, interface fixes, timeout adjustment
• nexus-athena: Tool registry format aligned, allowlist schema fixed, localhost for native

Day 4: the agent thinks. Day 5: it documents what it learned.