Flow Execution Engine — Saga Compensation and Retry

The engine that makes SysMARA flows actually run — not just validate

What is the Flow Execution Engine?

The Flow Execution Engine is the runtime that takes a FlowSpec and actually executes it — step by step, capability by capability. It handles the full lifecycle: context threading between steps, retry with exponential backoff, condition evaluation, and saga compensation when things go wrong. Every state transition is recorded as structured JSON for AI consumption.

This is what makes flows real. Without the executor, flows are just declarations. With it, they become executable workflows with built-in resilience.

Core Design Principles

• Steps are capabilities — each step.action maps to a declared CapabilitySpec. The executor validates this at construction time.
• Context threading — the output of step N is available as input to step N+1 via context.stepOutputs.
• Saga compensation — if a step fails with onFailure: "compensate", the executor runs compensation steps in reverse order for all previously completed steps.
• Retry with exponential backoff — steps with onFailure: "retry" are retried up to maxRetries times with exponentially increasing delays.
• Safe condition evaluation — step conditions are evaluated using a recursive descent parser. No eval() is ever used.
• Structured execution log — every execution produces a FlowExecutionRecord with AI-readable summaries.

FlowExecutor API

import { FlowExecutor } from '@sysmara/core';

const executor = new FlowExecutor(specs, {
  maxRetries: 3,           // default: 3
  retryBaseDelayMs: 100,   // default: 100ms
  capabilityHandler: async (capability, input, context) => {
    // Your logic to execute a capability
    return myService.run(capability, input);
  },
});

// Execute a flow
const result = await executor.execute('user_signup_flow', {
  email: 'alice@example.com',
  role: 'admin',
});

// Validate a flow (check all capabilities exist)
const validation = executor.validate('user_signup_flow');

// List all registered flows
const flows = executor.listFlows();

// Access execution history
const log = executor.getExecutionLog();
const summary = log.summarize();

Capability Handler

The capabilityHandler is the pluggable function that actually runs capabilities. This design allows:

• Testing — provide an in-memory handler for unit tests
• Production — wire up to real databases and services
• Logging — the default handler logs invocations without side effects

type CapabilityHandler = (
  capability: string,
  input: Record<string, unknown>,
  context: FlowContext,
) => Promise<unknown>;

Execution Record

Every flow execution produces a FlowExecutionRecord — a complete, machine-readable record of what happened:

interface FlowExecutionRecord {
  id: string;                    // unique execution ID
  flowName: string;
  status: FlowStatus;           // "completed" | "failed" | "compensated"
  triggeredBy: string;
  input: Record<string, unknown>;
  output?: Record<string, unknown>;
  steps: StepExecutionRecord[];  // detailed record per step
  error?: FlowError;
  startedAt: string;             // ISO timestamp
  completedAt?: string;
  totalDurationMs: number;
  summary: FlowSummary;         // AI-readable summary
}

AI-Readable Summary

Each execution record includes a FlowSummary designed for AI agent consumption:

interface FlowSummary {
  stepsTotal: number;
  stepsCompleted: number;
  stepsSkipped: number;
  stepsFailed: number;
  compensationsRun: number;
  capabilitiesInvoked: string[];
  entitiesAffected: string[];
}

Saga Compensation

When a step with onFailure: "compensate" fails, the executor enters compensation mode:

1. Identifies all previously completed steps that declare a compensation capability
2. Runs compensation capabilities in reverse order
3. Records each compensation result in the execution record
4. Sets final status to "compensated" if all compensations succeeded, or "failed" if any compensation also failed

flows:
  - name: user_signup_flow
    trigger: user_signup
    module: auth
    steps:
      - name: create_user
        action: create_user
        onFailure: compensate
        compensation: delete_user      # runs if later step fails

      - name: create_profile
        action: create_profile
        onFailure: compensate
        compensation: delete_profile   # runs if later step fails

      - name: send_email
        action: send_welcome_email
        onFailure: compensate          # if this fails:
        # 1. delete_profile runs (reverse of step 2)
        # 2. delete_user runs (reverse of step 1)

Condition Evaluation

Steps with a condition field are only executed if the condition evaluates to true. The evaluator supports:

• Property access: context.input.role, context.stepOutputs.create_user.id
• Comparison operators: ===, !==, >, <, >=, <=
• Logical operators: &&, ||
• Literals: strings, numbers, booleans, null, undefined

steps:
  - name: admin_setup
    action: setup_admin_dashboard
    onFailure: abort
    condition: 'context.input.role === "admin"'

  - name: premium_onboarding
    action: setup_premium
    onFailure: skip
    condition: 'context.stepOutputs.create_user.tier === "premium"'

Execution Log

The FlowExecutionLog stores all execution records and provides aggregate summaries:

const log = executor.getExecutionLog();

// Query individual executions
const record = log.get('execution-id');

// Get all executions
const all = log.list();

// AI-readable summary of all executions
const summary = log.summarize();
// {
//   totalExecutions: 42,
//   successRate: 0.95,
//   averageDurationMs: 150,
//   mostFrequentFlows: [{ name: 'user_signup_flow', count: 30 }],
//   recentFailures: [...]
// }

CLI Commands

Flow Validation

Before executing a flow, you can validate it to check that all step actions map to existing capabilities:

const result = executor.validate('user_signup_flow');
// {
//   valid: true,
//   errors: [],
//   warnings: [],
//   stepCount: 3,
//   capabilitiesRequired: ['create_user', 'create_profile', 'send_welcome_email']
// }