Workflow Run AI Summaries — Design
GitHub issue: #672
Goal
Generate an optional, AI-written summary of a workflow run after it finishes, so users can scan the runs list and understand what a run did at a glance without opening every step.
Requirements
- Trigger: automatic for workflows that opt in (
summarizeRuns), plus an on-demand "Summarize" button for any terminal run. - Automatic scope: only runs that end
completedorfailed. Cancelled runs are summarized only on demand. - Engine/model: global settings
summaryEngine/summaryModel. When unset, fall back to the system default engine and that engine's configured default model. - Format: one-line headline (≤120 chars) for the runs list, plus a 3–8 bullet markdown body for the run detail page, produced by a single agent session that returns JSON.
- Investigation: the agent receives a lean digest and can query a read-only set of Axiom MCP tools for deeper detail (task outputs, logs, trace).
- Non-goals: per-workflow engine/model overrides, deleting summaries, summarizing non-terminal runs, summaries for cancelled runs without a manual request, strict per-run tool scoping (see below).
Architecture Overview
A DB-backed queue (workflow_run_summary table) is populated by a hook at run termination and drained by a
@Scheduled poller. The poller leases an agent from AgentPool, gives it a lean digest plus a read-only,
stateless MCP server (axiom-run-summary), parses the JSON result, stores it, records AI usage, and fires an
SSE update after the transaction commits.
Data Model
Workflow definition opt-in
A new column summarize_runs boolean not null default false on workflow_definition, mapped as
WorkflowDefinitionEntity.summarizeRuns. Summary generation is off by default for every workflow:
- The migration's
default falsecovers existing rows, so no existing workflow is opted in. - New definitions are created with
falseunless the create request setssummarizeRuns: true. - The setting is edited in the definition's existing "Edit Workflow Definition" modal (name/description).
- The global settings never enable summaries for all workflows.
axiom.run-summary.enabledcan only turn the feature off.
Why a column instead of a key in the workflow content: the content is parsed into the external flow-engine
Workflow record, which has no slot for extra properties, and OpenAPI treats content as opaque. A hidden
top-level key could be silently dropped by the editor. The trade-off is that the setting is not versioned
with the content: it is read from the definition as it stands when a run finishes.
Table workflow_run_summary (Flyway V12__workflow_run_summary.sql)
| column | type | notes |
|---|---|---|
run_id |
bigint PK, FK → workflow_run |
ON DELETE CASCADE |
status |
varchar(20) not null | pending, generating, ready, failed |
headline |
varchar(200) | nullable |
body |
TEXT | markdown, nullable |
error |
TEXT | nullable; last failure message |
engine |
varchar(255) | engine that produced the summary |
model |
varchar(255) | model that produced the summary |
trigger_type |
varchar(10) not null | auto or manual |
attempts |
integer not null | failed attempts so far; max 2 |
created_on |
timestamp(6) with time zone not null | |
completed_on |
timestamp(6) with time zone | nullable |
Deleting a run must delete its summary automatically, at the database level. Runs are removed with bulk JPQL
deletes (HistoryCleanup, SubjectDeletionService, and workflow definition deletion), and these bypass JPA
cascades. The summary row is therefore protected in two places:
- Flyway: the migration declares
foreign key (run_id) references workflow_run on delete cascade, following theworkflow_run_labelprecedent inV6. - Entity:
WorkflowRunSummaryEntitymapsrun_idas@OneToOne @MapsIdtoWorkflowRunEntity, annotated@OnDelete(action = OnDeleteAction.CASCADE). The dev and test profiles' Hibernatedrop-and-createschema then gets the same cascade.
No application code deletes summaries explicitly. (deleteWorkflowDefinition has a comment preferring explicit
deletes over DB cascades; this feature deliberately follows the V6 precedent instead, per the requirement.)
System config
SystemConfigEntity gains nullable summaryEngine and summaryModel columns, added in the same migration.
REST API (contract-first, openapi.json)
The name WorkflowRunSummary is already used by the run list item, so the new schema is named
WorkflowRunAiSummary and the bean property is aiSummary.
- New schema
WorkflowRunAiSummary:status,headline,body,error,engine,model,trigger,completedOn. - The full run bean (
WorkflowInstance, returned bygetWorkflowRun) gets an optionalaiSummary. - The run list item (
WorkflowRunSummary) gets an optionalaiSummary, populated with onlystatusandheadline. POST /workflow/runs/{runId}/ai-summary(generateWorkflowRunAiSummary) upserts the row topendingwithtrigger=manual, resetsattemptsto 0, and returns202 Accepted. The response is declared as"202": {"content": {"application/json": {}}}so the generated method returnsResponse.404if the run does not exist.409if the run is not terminal (running/waiting), or if its summary is currentlygenerating.503ifaxiom.run-summary.enabled=false.- Allowed for any terminal status, including
cancelled, whether or not the workflow opted in. WorkflowDefinition,CreateWorkflowDefinitionRequest, andUpdateWorkflowDefinitionRequestget an optional booleansummarizeRuns. On update, an absent value leaves the setting unchanged.SystemConfigandUpdateSystemConfigRequestget optionalsummaryEngineandsummaryModel.UpdateSystemConfigRequestkeepsdefaultEnginerequired. The summary fields are updated only when present in the request, and an empty string clears them back to "use the default".
Generation Pipeline
Enqueue
A new onRunTerminal(WorkflowRunEntity) helper in WorkflowExecutionService is called from every terminal
transition: applyAdvancedInstance, the synchronous-start completion path, and cancelWorkflowRun. It delegates
to RunSummaryService.enqueueIfEnabled(run). That method inserts a pending row with trigger=auto only when:
axiom.run-summary.enabledistrue;- the status is
completedorfailed; - the run's definition has
summarizeRuns = true; - no summary row exists for the run.
The insert happens in the run's transaction, so a rollback enqueues nothing.
Poller (RunSummaryPoller)
@Scheduled(every = "${axiom.run-summary.poll-interval:5s}", concurrentExecution = SKIP); turnedoffin the test profile, like the task queue poller.- Claims up to
axiom.run-summary.batch-size(default 2)pendingrows with a conditional update (status = 'generating' where runId = ? and status = 'pending'), so concurrent pollers cannot double-claim. - At startup, rows stuck in
generatingare reset topending. - If no agent lease is available, the claim is reverted to
pending; work is never dropped. - Generation runs outside the claim transaction because the agent call is slow.
Prompt (RunSummaryPromptBuilder, pure and unit-testable)
A lean digest, budgeted to about 3,000 characters:
- run ID, workflow name, run title, status, duration, failure reason (first lines);
- the ordered steps, each with task ID, node ID, task status, and a one-line output preview (about 150 chars);
- the total AI cost of the run.
Instructions given to the agent:
- The run ID to pass to the tools.
- Use the available tools only when the digest is insufficient, especially to diagnose failures.
- Respond ONLY with JSON
{"headline": "...", "body": "..."}. headlineis at most 120 characters;bodyis 3–8 markdown bullets covering what the run did, key outputs, and the cause of any failure.- Never reproduce secrets or credentials.
Read-only MCP server (axiom-run-summary)
A new Node stdio MCP server template under app/src/main/resources/templates/axiom-run-summary-mcp/, installed
the same way as the other built-in servers. Its environment holds only AXIOM_API_URL. It is stateless:
every tool takes a runId argument, so a single registration can be shared across concurrent summaries and
across all engines (OpenCode registers MCP servers once per shared process; Claude Code and Copilot use a
per-request config file).
| tool | purpose |
|---|---|
get_run(runId) |
inputs, status, timings, labels, failure reason, task list (task ID, node ID, status) |
get_task(runId, taskId) |
full task input and output (the task must belong to the run) |
get_task_log(runId, taskId, offset, limit) |
paginated execution log for that task |
get_trace(runId) |
the run's trace nodes, failed nodes first |
Scoping trade-off (accepted): the tools can read any workflow run on this Axiom instance, not only the run being summarized. The prompt tells the agent which run to examine. The guarantees that do hold:
- The server contains only read-only tools and calls only GET endpoints.
- It can reach only workflow runs and their tasks, logs, and traces; no other Axiom data.
AgentRequest.allowedToolslists exactly these four tools (mcp__axiom-run-summary__*names), and the agent gets no other tools (no shell, file, or web access).
Wiring per engine:
- Claude Code / Copilot:
RunSummaryServicewrites a per-summary MCP config file containing onlyaxiom-run-summaryand passes it asAgentRequest.mcpConfigFile. - OpenCode:
McpConfigGenerator.getMcpServersForOpenCodeincludesaxiom-run-summarywhen the allowed tools reference it, andOpenCodeMcpManagertreats it as a built-in server (registered once, never disconnected).
Execution
- Engine:
summaryEngine, falling back toSystemConfigEntity.defaultEngine, then the registry default. Model:summaryModel, ornullso the agent uses its configured default model. maxSteps=axiom.run-summary.max-steps(default 10); timeout =axiom.run-summary.timeout-seconds(default 300).- The response is parsed leniently: the first JSON object is extracted, tolerating code fences.
headlineis required and truncated to 200 characters;bodyis optional. - Success: store
ready,engine,model(fromAgentResult, falling back to the requested values),completed_on, and record anAiUsageEntityrow withinvocationType = "run-summary", linked to the run and its workflow definition. - Failure (agent error, timeout, or invalid JSON): increment
attempts. Ifattempts < 2, go back topending; otherwise storefailedwitherror. - After commit, fire
SseEvent.workflowUpdated(subjectId, runId, runStatus)through anAFTER_SUCCESSobserver (theSecretExpirySseNotifierpattern).
Configuration (application.properties)
axiom.run-summary.enabled(defaulttrue): a global kill switch for the automatic and manual paths.axiom.run-summary.poll-interval(default5s;%test=off).axiom.run-summary.batch-size(2),axiom.run-summary.max-steps(10),axiom.run-summary.timeout-seconds(300).
UI
Runs list (WorkflowRunsPage.tsx, workbench RunsList.tsx)
ready: the headline appears as a muted line under the run title.pending/generating: a small spinner with "Summarizing…".failedor no summary: nothing is shown.- The list refreshes through the existing
workflow-updatedSSE handling.
Run detail (WorkflowRunDetailPage.tsx)
A Summary card between the header and the tabs:
ready: bold headline, the markdown body, and a muted footer "Generated by {engine}/{model} · {time}". A "Regenerate" action.pending/generating: a skeleton with "Generating summary…".failed: an inline warning with the error and a "Retry" button.- No summary on a terminal run: a "Summarize this run" button.
- Non-terminal run: the card is hidden.
Settings (EngineSettingsPage.tsx)
A "Run summaries" card with engine and model selects. The engine select has a "System default" option, and the model select has an "Engine default" option. A hint explains the fallback.
Workflow definition
A "Generate AI summaries for completed and failed runs" checkbox in the "Edit Workflow Definition" modal,
bound to summarizeRuns and unchecked by default.
Error Handling
- Agent unavailable: the row stays
pendingand is retried on the next poll. - Agent failure, timeout, or bad JSON: up to 2 attempts, then
failedwith the error shown in the UI and a manual Retry. - Process restart:
generatingrows are recovered topendingat startup. - Run deleted by any path: the summary is removed with it by the DB cascade.
- A summary failure never affects the run's own status.
Testing
- Unit (JUnit 5):
RunSummaryPromptBuilder: digest contents, budget, and truncation.RunSummaryResponseParser: plain JSON, fenced JSON, missing headline, overlong headline.@QuarkusTest:enqueueIfEnabledrules: opt-in off by default, terminal status filter, no duplicates, kill switch.- Cascade: deleting runs through each bulk path (
HistoryCleanup,SubjectDeletionService, and workflow definition deletion) leaves noworkflow_run_summaryrows. - POST endpoint: 202, 404, 409, and 503 cases.
- Generation with a mocked
AgentPool/Agent: success storesreadyand records AI usage; failures retry then endfailed; no lease reverts topending; startup recovery. - Bean mapping:
aiSummaryon run detail and list items;summarizeRunson definitions; summary settings on system config. MigrationSchemaValidationTestpasses with V12.- MCP server (
node:test): only GET calls, task-belongs-to-run check, log pagination, tool list is exactly the four tools. - UI (vitest): summary card states, headline rendering in the runs list, settings card, opt-in checkbox.
- Manual live check: an isolated instance
(
AXIOM_HOME=/tmp/axiom-test/home ./dev.sh --portOffset=500 --dataDir=/tmp/axiom-test/data).