Tracing
Axiom's tracing subsystem records a hierarchical tree of every step that occurs during a workflow run. Each trace is a tree of lightweight trace nodes that reference more detailed records elsewhere in the system, keeping nodes small and fast to query while providing full drill-down capability.
This guide covers the data model, service API, REST endpoints, and how to instrument new pipeline stages with tracing.
Design Principles
- Lightweight breadcrumbs — trace nodes carry a summary, status, and timing, but delegate detail to referenced entities (activity logs, tasks, tool executions, etc.)
- Non-fatal — all trace operations are wrapped in exception handlers. Tracing never interrupts the main pipeline.
- Independent transactions — trace writes use
QuarkusTransaction.requiringNew()so data persists even if the caller's transaction rolls back - Real-time — every trace mutation fires an SSE event for live UI updates
- Correlation by UUID —
TraceEntityuses a UUID primary key that doubles as the correlation identifier threaded through environment variables and API callbacks
Data Model
Trace entities live in core/src/main/java/io/apitomy/axiom/core/entities/. Read
TraceEntity.java, TraceNodeEntity.java, and ToolExecutionEntity.java directly for
the exact field list — the concepts below are what you need to work with the API and
build new instrumentation.
TraceEntity
The root of a complete execution trace (table trace). It uses a UUID primary key
(traceId) instead of the standard auto-increment Long used elsewhere, because that
ID doubles as the correlation identifier threaded through subprocess environment
variables and API callbacks. Each trace has a traceType (see below), a status
(in-progress, completed, or failed), a human-readable summary, optional
eventId/subjectId associations, and start/completion timestamps. eventId is the
UUID of the originating stream_event row. It is a soft reference (no foreign key): stream events are
purged by the event retention policy while traces are retained, so the UUID may outlive its event.
Every trace is currently of type workflow, created when a workflow run starts (manually on a subject,
manually from a stream event, by a create-workflow routing rule, by a schedule trigger, or as a re-run of a
failed run). Tasks created by the run's action and human-task
nodes, and tool calls made by their agents, are recorded as nodes of the run's trace.
TraceNodeEntity
A single step/span within a trace tree (table trace_node). Nodes form a parent-child
tree via parentNodeId (null for the root node). Each node has a nodeType, a status,
a summary, start/completion timestamps with a computed durationMs, and an optional
entityType + entityId pair that points to a more detailed record elsewhere in the
system. entityId is stored as a string (VARCHAR) so it can hold either a numeric ID
(e.g. task, activity-log) or a UUID (e.g. event, which references a stream event).
Node Types
Node type strings are chosen by whichever service creates the node — there is no enum, so treat this as the currently observed set rather than an exhaustive contract:
| Node Type | Meaning | Typical Entity Reference |
|---|---|---|
workflow |
Workflow run started (root node of a workflow trace) |
workflow-run |
task |
A workflow step: a task created by an action or human-task node, or a run parked at a wait or receive-event node |
task, workflow-wait, or workflow-event-subscription |
tool-execution |
An MCP tool was invoked | tool-execution |
Entity Types
The entityType field determines which entity table the entityId references:
| Entity Type | Referenced Entity | Detail Content |
|---|---|---|
event |
StreamEventEntity |
Raw normalized event payload |
activity-log |
ActivityLogEntity |
Log entry with type, summary, execution log |
task |
TaskEntity |
Task details — action type, agent, status, output |
tool-execution |
ToolExecutionEntity |
Full JSON input and output |
ai-usage |
AiUsageEntity |
Token counts, cost, model |
workflow-run |
WorkflowRunEntity |
Run status, current node, times, definition and subject names |
workflow-wait |
WorkflowWaitEntity |
Wait type, resume time, status (waiting/resolved/cancelled), run |
workflow-event-subscription |
WorkflowEventSubscriptionEntity |
Awaited event type, status, run, subject |
GET /traces/{traceId}/nodes/{nodeId} resolves these into the node's detail object (see
TraceResourceImpl.resolveDetail). An unknown entity type, a malformed ID or a deleted record yields the
node with a null detail. The one exception is workflow waits and event subscriptions: their rows are
deleted when the parked node resumes, so the detail is rebuilt from the trace node and its workflow run,
with status resolved (node completed) or cancelled. The event that satisfied a subscription is not
recorded, so matchedEventId is always null. The workflow-run detail omits the engine's
instanceState.
ToolExecutionEntity
Detailed record of an MCP tool invocation (table tool_execution), storing the full
JSON input and output for debugging. Referenced by trace nodes via
entityType="tool-execution". Each record carries the associated traceId, the tool
name, its JSON input/output, a status, and a duration.
TraceService API
TraceService is the central service for creating and managing traces. It lives in
the core module so any module can use it without circular dependencies.
Location: core/src/main/java/io/apitomy/axiom/core/tracing/TraceService.java
createTrace()
Creates a new trace and its root node in a single transaction, then fires an SSE event.
public TraceContext createTrace(
String traceType, // e.g. "workflow"
String summary, // human-readable trace description
UUID eventId, // associated stream event ID (nullable)
Long subjectId, // associated subject ID (nullable)
String rootNodeType, // node type for the root node
String rootNodeSummary, // summary for the root node
String rootEntityType, // entity type for the root node (nullable)
Long rootEntityId // entity ID for the root node (nullable)
)
Returns a TraceContext with the root node already pushed onto the stack.
addNode()
Adds a child node under the current parent (top of the context stack).
public Long addNode(
TraceContext ctx, // current trace context
String nodeType, // node type identifier
String status, // initial status (e.g. "in-progress", "completed")
String summary, // human-readable description
String entityType, // referenced entity type (nullable)
Long entityId // referenced entity ID (nullable)
)
Returns the new node's ID. The parent is determined by ctx.currentParentNodeId().
To reference a UUID-keyed entity (such as a stream event, entityType="event"), use
addUuidEntityNode(), which has the same parameters except that entityId is a UUID:
Long nodeId = traceService.addUuidEntityNode(traceCtx, "event-received", "completed",
"Event received: " + event.type, "event", event.id);
completeNode()
Marks a node as completed with a final status and calculated duration. Has two overloads:
// Basic completion
public void completeNode(Long nodeId, String status)
// Completion with entity reference (when entity isn't known at creation time)
public void completeNode(Long nodeId, String status,
String entityType, Long entityId)
completeTrace()
Marks the trace itself as completed or failed.
Trace lifecycle rules
Every unit of work ends with exactly one trace in a final state (completed or failed). No trace or
task node may stay in-progress once its work item is final, and a trace is never completed twice.
| Work item | Who completes the trace |
|---|---|
| Workflow run | WorkflowExecutionService when the run is terminal (completed, failed, or cancelled), including runs that finish synchronously on start |
TaskTraceFinalizer is used by both the agent and script task paths (including failTask). It always
completes the task's task node and never completes a workflow-owned trace — the workflow run owns its
trace lifecycle.
Workflow traces and events
- A run started from a stream event gets a
workflowtrace witheventIdset to the stream event, so the trace appears inGET /traces?filterEventId=.... This covers runs started by acreate-workflowrouting rule, manual runs started from the event (POST /stream/events/{eventId}/runs), and re-runs of a failed run that was started from an event. All of them go throughEventStreamOrchestrator.startRunFromEvent. For a routing rule, the routing outcome'straceIdis the run's trace. Manual runs from an event have no ledger entry or routing outcome. - Runs started manually on a subject or by a schedule trigger have no
eventId. A re-run of such a run has none either. - A re-run (
POST /workflow/runs/{runId}/rerun) is a new run with its own trace. It is linked to the original run only by itsrerunOf:<runId>label, not through the trace. - A
create-workflowrule that is skipped because the subject already has an active run of the definition creates no run and no trace. - When a stream event resumes a run parked at a receive-event step, the receive-event
tasknode (entity typeworkflow-event-subscription) is completed and its summary gets "— resumed by event". The workflow-resumedrouting outcome item'strace_node_idpoints to that node, and aworkflow_run_resumerow records the event (see the event sourcing design).
TraceContext
TraceContext is a mutable context object threaded through workflow and task
processing. It tracks the current position in the trace tree using an internal stack.
Location: core/src/main/java/io/apitomy/axiom/core/tracing/TraceContext.java
| Method | Description |
|---|---|
UUID traceId() |
Returns the trace UUID |
Long currentParentNodeId() |
Returns the top of the stack (current parent) |
void push(Long nodeId) |
Pushes a node onto the stack, making it the current parent |
Long pop() |
Pops the current parent, returning to the previous level |
Push/Pop Pattern
When descending into a child scope, push the new node onto the stack so that any nodes created within that scope become its children. Pop when returning to the parent level.
// Create a parent node
Long parentId = traceService.addNode(traceCtx, "my-step",
"in-progress", "Doing something", null, null);
traceCtx.push(parentId);
try {
// Any nodes created here become children of parentId
traceService.addNode(traceCtx, "my-child-step",
"completed", "Child step", "activity-log", logId);
} finally {
traceCtx.pop();
traceService.completeNode(parentId, "completed");
}
REST API Reference
All trace endpoints are defined in the OpenAPI specification at
common/api/src/main/resources/openapi.json and implemented in
app/src/main/java/io/apitomy/axiom/app/rest/TraceResourceImpl.java.
List Traces
Returns a paginated list of traces with optional filtering.
Query Parameters:
| Parameter | Type | Default | Description |
|---|---|---|---|
page |
integer | 1 | Page number (1-indexed) |
limit |
integer | 20 | Page size |
filterTraceType |
string | — | Filter by trace type (currently always "workflow") |
filterStatus |
string | — | Filter by status (comma-separated, e.g. "in-progress,completed") |
filterEventId |
string (UUID) | — | Filter by stream event ID (400 if not a valid UUID) |
filterSubjectId |
integer | — | Filter by subject ID |
Response: TraceSearchResults
{
"items": [
{
"traceId": "a1b2c3d4-...",
"traceType": "workflow",
"status": "completed",
"summary": "Workflow: Auto-Label New Issues",
"eventId": null,
"subjectId": 7,
"startedOn": "2026-06-29T10:00:00Z",
"completedOn": "2026-06-29T10:00:05Z"
}
],
"totalCount": 1,
"page": 1,
"limit": 20
}
Convenience: List Traces by Entity
The API also provides a convenience endpoint that returns the traces for a specific subject as a JSON
array of Trace objects (not paginated):
This is equivalent to calling GET /traces?filterSubjectId=...,
but are more convenient when you already have the entity ID.
Get Trace Detail
Returns a trace and all of its nodes.
Path Parameters:
| Parameter | Type | Description |
|---|---|---|
traceId |
string (UUID) | The trace UUID |
Response: TraceDetail
{
"trace": {
"traceId": "a1b2c3d4-...",
"traceType": "workflow",
"status": "completed",
"summary": "Workflow: Auto-Label New Issues",
"subjectId": 7,
"startedOn": "2026-06-29T10:00:00Z",
"completedOn": "2026-06-29T10:00:05Z"
},
"nodes": [
{
"id": 1,
"traceId": "a1b2c3d4-...",
"parentNodeId": null,
"nodeType": "workflow",
"status": "completed",
"summary": "Workflow: Auto-Label New Issues",
"startedOn": "2026-06-29T10:00:00Z",
"completedOn": "2026-06-29T10:00:00Z",
"durationMs": 0,
"entityType": null,
"entityId": null
},
{
"id": 2,
"traceId": "a1b2c3d4-...",
"parentNodeId": 1,
"nodeType": "task",
"status": "completed",
"summary": "Task: Auto-Label Issue",
"startedOn": "2026-06-29T10:00:03Z",
"completedOn": "2026-06-29T10:00:05Z",
"durationMs": 2000,
"entityType": "task",
"entityId": "15"
}
]
}
Get Trace Node Detail
Returns a trace node with its resolved entity detail. The detail object's structure
depends on the node's entityType.
Path Parameters:
| Parameter | Type | Description |
|---|---|---|
traceId |
string (UUID) | The trace UUID |
nodeId |
integer | The node ID |
Response: TraceNodeDetail
{
"node": {
"id": 5,
"traceId": "a1b2c3d4-...",
"parentNodeId": 3,
"nodeType": "tool-execution",
"status": "completed",
"summary": "Tool: github_list_issues",
"durationMs": 1200,
"entityType": "tool-execution",
"entityId": "7"
},
"detail": {
"toolName": "github_list_issues",
"toolInput": "{\"repo\": \"owner/repo\", \"state\": \"open\"}",
"toolOutput": "[{\"number\": 1, \"title\": \"Bug report\"}]",
"status": "completed",
"durationMs": 1200
}
}
Create Tool Call
Creates a new tool-call trace node and its associated ToolExecutionEntity. Called by
MCP tool wrappers during task execution.
Request Body: ToolCallRequest
{
"traceId": "a1b2c3d4-...",
"parentNodeId": 3,
"toolName": "github_list_issues",
"toolInput": "{\"repo\": \"owner/repo\", \"state\": \"open\"}"
}
| Field | Type | Required | Description |
|---|---|---|---|
traceId |
string (UUID) | Yes | The parent trace UUID |
parentNodeId |
integer | No | Parent node ID for tree nesting |
toolName |
string | Yes | Name of the MCP tool being invoked |
toolInput |
string | No | Tool input as JSON |
Response: ToolCallCreated
Complete Tool Call
Completes a tool-call trace node with the execution result.
Path Parameters:
| Parameter | Type | Description |
|---|---|---|
nodeId |
integer | The trace node ID returned by the create call |
Request Body: ToolCallCompletion
{
"toolOutput": "[{\"number\": 1, \"title\": \"Bug report\"}]",
"status": "completed",
"durationMs": 1200
}
| Field | Type | Required | Description |
|---|---|---|---|
toolOutput |
string | No | Tool output as JSON |
status |
string | No | Final status ("completed" or "failed") |
durationMs |
integer | No | Execution duration in milliseconds |
Instrumenting New Pipeline Stages
To add tracing to a new pipeline stage or service:
Step 1: Inject TraceService
Step 2: Create a Trace or Add a Node
If your code is the entry point for a new kind of traced work (like WorkflowExecutionService), create a
new trace:
TraceContext traceCtx = traceService.createTrace(
"my-pipeline",
"Processing something: " + description,
eventId, subjectId, // eventId is the stream event UUID (nullable)
"my-root-node", "Root: " + description,
null, null);
If your code runs within an existing trace (like task execution within a workflow run), receive the
TraceContext as a parameter and add nodes to it:
Long nodeId = traceService.addNode(traceCtx, "my-step", "in-progress",
"Doing something important", null, null);
Step 3: Push/Pop for Child Scopes
If your node will have child nodes, push it onto the stack:
traceCtx.push(nodeId);
try {
// child nodes created here become children of nodeId
doWork(traceCtx);
} finally {
traceCtx.pop();
}
Step 4: Complete Nodes and the Trace
// Complete a node
traceService.completeNode(nodeId, "completed");
// Complete a node and set its entity reference
traceService.completeNode(nodeId, "completed", "activity-log", logEntry.id);
// Complete the trace
traceService.completeTrace(traceCtx.traceId(), "completed");
Step 5: Pass Correlation Data to Subprocesses
If your stage launches a subprocess that should report tool calls back to the trace, inject the trace correlation data as environment variables:
Map<String, String> env = new HashMap<>();
env.put("AXIOM_TRACE_ID", traceCtx.traceId().toString());
env.put("AXIOM_PARENT_NODE_ID", String.valueOf(nodeId));
The subprocess (or its MCP tool wrapper) reads these variables and calls back to
POST /api/v1/traces/tool-calls and PUT /api/v1/traces/tool-calls/{nodeId}.
Non-Fatal Pattern
Always wrap trace operations in try/catch when called from the main pipeline:
Long nodeId = null;
if (traceCtx != null) {
try {
nodeId = traceService.addNode(traceCtx, "my-step", "in-progress",
"Description", null, null);
traceCtx.push(nodeId);
} catch (Exception e) {
LOG.warnf(e, "Failed to add trace node");
}
}
// ... do the actual work ...
if (traceCtx != null && nodeId != null) {
try {
traceCtx.pop();
traceService.completeNode(nodeId, "completed");
} catch (Exception e) {
LOG.warnf(e, "Failed to complete trace node");
}
}
MCP Tool Call Integration
When Axiom generates MCP server configurations for AI agent tasks, it includes the trace
correlation environment variables (AXIOM_TRACE_ID and AXIOM_PARENT_NODE_ID). The
generated MCP tool wrappers use these to register tool calls as trace nodes:
AI Agent starts task
│
├─ Axiom sets AXIOM_TRACE_ID and AXIOM_PARENT_NODE_ID in subprocess env
│
▼
MCP Tool Wrapper executes
│
├─ Reads AXIOM_TRACE_ID and AXIOM_PARENT_NODE_ID from environment
├─ POST /api/v1/traces/tool-calls → receives nodeId
├─ Executes the actual tool
└─ PUT /api/v1/traces/tool-calls/{nodeId} → sends output + status
This flow is implemented in TaskExecutionService, which looks up the trace node ID for the current task
and injects it as AXIOM_PARENT_NODE_ID so tool calls become children of the correct node in the tree.
If the trace callback fails (network error, missing trace, etc.), the tool execution continues normally — tracing is best-effort and never blocks work.
Correlation
The trace ID is also Axiom's correlation ID. Activity log rows, AI usage rows, tasks and workflow runs store
the trace ID of the work that produced them, and agents can join their caller's trace through the
X-Axiom-Trace-Id and X-Axiom-Parent-Node-Id headers. See Correlation by Trace ID.
Server log lines also carry the trace ID (MDC key traceId), so a trace can be followed in the logs with
grep traceId=<uuid>. See Correlation by Trace ID.