Run/Flow

Capabilities & The Magic #capabilities

What RunFlow Does

• Hardened Sandbox: Run any Docker image (--cap-drop ALL, read-only rootfs, no-new-privileges, pids-limit, CPU/memory caps, never --privileged) — submit over REST, get a track_id. Firewalled egress: your code can reach the public internet, but not the host, private networks, or other runs.
• Scheduling: Run now, at a future wall-clock time, or strictly within your business-hours calendar (e.g. Mon–Fri 09:00–18:00 → a Friday-evening submit defers to Monday 09:00).
• Full Lifecycle Control: Drive execution remotely with no shell/SSH required: pause / resume / stop / kill.
• Live Observability: Watch it live with streaming logs (Redis hot-tail + on-disk), a CPU/memory/network utilization time-series, and the full state-transition history.
• Reactive Capacity: Submit past your concurrency limit and jobs queue with a position + ETA instead of being rejected; queue-timeout if they wait too long.
• Governance: Includes per-tenant quotas, env-var encryption at rest, secret redaction, per-operation RBAC, auto-kill on timeout, and audit logs.
• Result Collection: Collect exit code, duration, stdout/stderr, and a last-1KB result summary upon completion.

The Most Differentiating Thing

Serverless can't pause a running invocation; CI/cron can't freeze-and-resume a live job; plain docker needs shell access and gives you no governed observability API. RunFlow makes a running container behave like a process you can attach a debugger to — remotely, safely, and with RBAC + quotas around it.

Live Demo: Pause & Resume

Here is a trivial Python "training loop" that keeps its state only in process memory (so continuation proves the state survived the freeze). You can run this live:

# Set up your environment (Use the demo API key to try this live!)
export KEY="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
export BASE="https://runflow.rodmena.app"

# 1) Submit a simulated training job that streams progress
TID=$(curl -s -X POST $BASE/api/v1/runs -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{
  "image":"python:3.12-slim",
  "command":["python","-u","-c","import time\nloss=1.0\nfor epoch in range(1,21):\n    time.sleep(1)\n    loss*=0.95\n    print(f\"epoch {epoch}/20 loss={loss:.4f}\",flush=True)"],
  "timeout_seconds":60,"cpu_limit":0.5,"memory_limit_mb":128,"name":"pause-resume-demo"}' | jq -r .track_id)

# 2) Watch logs stream + profile CPU live
curl -s "$BASE/api/v1/runs/$TID/logs?stream=stdout" -H "Authorization: Bearer $KEY" | jq -r '.lines[-3:][]'
curl -s "$BASE/api/v1/runs/$TID/utilization"        -H "Authorization: Bearer $KEY" | jq '.samples[-1]'

# 3) FREEZE it mid-training → logs stop advancing, state=paused, CPU→0
curl -s -X POST "$BASE/api/v1/runs/$TID/pause"  -H "Authorization: Bearer $KEY"

# 4) RESUME → it continues from the exact epoch it froze at (memory intact)
curl -s -X POST "$BASE/api/v1/runs/$TID/resume" -H "Authorization: Bearer $KEY"

# 5) Collect the result when it finishes
curl -s "$BASE/api/v1/runs/$TID/result" -H "Authorization: Bearer $KEY" | jq

What you'll see (actual run output):

Authentication #auth

As an AI agent, you will use a Bearer Token to authenticate. The backend uses a two-tier model:

• Tier 1 — Master client key: RunFlow's own service key (AUTH_CLIENT_KEY) authenticates RunFlow to auth.rodmena.co.uk. This is configured server-side, never exposed to callers.
• Tier 2 — API keys: Each caller presents a Bearer token (UUIDv4) that must be registered and whitelisted in auth.rodmena.co.uk. Unknown keys are rejected with 401. Known keys without the required permission are rejected with 403.

Your API key should be provided by the human user or accessible in your environment. Do not generate a random UUID. If you do not have one, a free-tier key can be obtained by emailing runflow-freepass@mail.rodmena.co.uk from the address it should be bound to (and runflow-freepass+rotate@mail.rodmena.co.uk to replace a compromised one). Ask your human to send it — the key is replied to that mailbox, so an agent cannot self-serve. Free-tier keys are restricted; call GET /api/v1/usage to see exactly what yours allows. The API key serves as the tenant isolation boundary: all runs, logs, and metadata are scoped to your identity.

Authorization: Bearer <your-api-key>

• 401 — missing, malformed, or unknown API key (not registered in auth.rodmena.co.uk).
• 403 — valid API key but missing the RBAC permission for the endpoint.
• 503 — auth.rodmena.co.uk is unreachable (fail-closed; no requests pass).

RBAC permissions are granted per API key in auth.rodmena.co.uk:

runflow_run_submit   # POST /api/v1/runs
runflow_run_read     # GET /api/v1/runs, /{track_id}, /result, /logs, /utilization, /tenant/quotas
runflow_run_control # POST /pause, /resume, /stop, /kill
runflow_run_delete   # DELETE /api/v1/runs/{track_id}
runflow_tenant_admin # PUT /api/v1/tenant/quotas

curl https://runflow.rodmena.app/api/v1/tenant/quotas \
  -H "Authorization: Bearer $API_KEY"

Quick Start #quickstart

Follow these exact steps to execute a job programmatically:

curl https://runflow.rodmena.app/health
# {"status":"healthy","postgres":"up","redis":"up","docker":"up","workers":{"total":4,"busy":0}}

API_KEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx   # your API key (email sales@rodmena.co.uk to obtain one)
curl -X POST https://runflow.rodmena.app/api/v1/runs \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "image": "python:3.12-slim",
    "command": ["python", "-c", "print(\"hello from runflow\")"],
    "timeout_seconds": 60
  }'
# {"track_id":"01KY0XX71BMX0B88C3A189S2K7","status":"scheduled","eligible_start":"2026-07-20T23:31:17Z","clamped":false}

TRACK=01KY0XX71BMX0B88C3A189S2K7

# status — poll until state is terminal
curl -s https://runflow.rodmena.app/api/v1/runs/$TRACK \
  -H "Authorization: Bearer $API_KEY" | python3 -m json.tool

# result — available when state is "exited" / "failed" / "timeout" / "killed"
curl -s https://runflow.rodmena.app/api/v1/runs/$TRACK/result \
  -H "Authorization: Bearer $API_KEY" | python3 -m json.tool

# logs — stdout only here; use stream=both for both
curl -s "https://runflow.rodmena.app/api/v1/runs/$TRACK/logs?stream=stdout" \
  -H "Authorization: Bearer $API_KEY" | python3 -m json.tool

Comparison & Alternatives #comparison

How does RunFlow compare to other compute platforms? Here is a breakdown of pros, cons, and when to use what.

Serverless Functions (AWS Lambda, Google Cloud Run)

• Pros: Fully managed, infinite horizontal scaling, and deep integration with cloud ecosystems.
• Cons: Strictly stateless and often time-bound (e.g., 15-minute limits). You cannot pause a Lambda mid-flight or easily attach a live debugger over REST.
• Verdict: Use Serverless for high-volume stateless microservices. Use RunFlow for stateful, agent-driven scripts that need mid-flight control and live observability without time limits.

CI/CD Pipelines (GitHub Actions, GitLab CI)

• Pros: Excellent for repository-bound static pipelines and automation.
• Cons: Not designed for dynamic, API-driven single-job execution by AI agents. CI/CD lacks mid-flight freeze/resume capabilities and doesn't offer live CPU/Memory profiling time-series data.
• Verdict: RunFlow provides far superior dynamic API control and observability for autonomous agents.

Raw Kubernetes / Docker API

• Pros: Ultimate flexibility over container infrastructure.
• Cons: Exposing the Docker API is a massive security risk. Kubernetes requires complex RBAC and overhead. Neither provides out-of-the-box tenant quotas or business-hour scheduling natively.
• Verdict: RunFlow abstracts the danger and complexity, providing a hardened, governed wrapper perfect for third-party or agentic consumption.

Native DAG Workflows

RunFlow runs directed-acyclic-graph workflows natively. Submit a graph of container nodes and RunFlow schedules each node as its own governed run the moment its dependencies are satisfied — no sidecar orchestrator required.

• Conditional edges — gate a node on a parent’s outcome (on_success, on_failure, always): “if Job A succeeds, run Job B.”
• Per-node retries — each node carries its own retries (0–10) and retry_backoff_seconds; a transient node failure is retried without restarting the graph.
• JSON message-passing — a node’s last stdout line (JSON) is handed to downstream nodes as RUNFLOW_INPUT_<REF> environment variables. No shared volumes; node outputs are encrypted at rest.
• Approval gates (human-in-the-loop) — a node with "type":"approval" pauses the branch at waiting_approval until you approve (runs the success path) or reject (runs the failure path).
• Runtime control — pause/resume the whole graph, retry a failed stage and everything downstream, and stream every transition from an event log (replay or subscribe by cursor).
• Same governance — every node inherits tenant quotas, business-hour scheduling, and container hardening. One node = one run = one track_id.

POST   /api/v1/workflows                   # submit a DAG (nodes + edges)
GET    /api/v1/workflows                   # list workflows
GET    /api/v1/workflows/{id}              # status + per-node state
GET    /api/v1/workflows/{id}/nodes/{ref}  # one node's detail
GET    /api/v1/workflows/{id}/result       # per-node outcomes when terminal
GET    /api/v1/workflows/{id}/events       # event log (replay/subscribe by ?since=)
POST   /api/v1/workflows/{id}/cancel       # cancel the whole graph
POST   /api/v1/workflows/{id}/pause|resume # halt / continue the graph
POST   .../nodes/{ref}/retry|approve|reject # retry a stage; decide an approval gate

Full reference — states, engine semantics, request models and worked examples — in Workflows. Copy-paste examples: 4g–4j.

Available over both the REST API and MCP. Connect any MCP client to the hosted endpoint at https://runflow.rodmena.app/mcp (streamable-HTTP, Authorization: Bearer <api-key>) — 25 tools at full REST parity: submit/monitor/control/delete runs; submit, monitor, cancel, pause/resume, retry, approve/reject, and event-stream workflows; read/update quotas.

Examples #examples

4a. Hello-world Docker run

Minimal run: pull python:3.12-slim, run a one-liner, exit.

curl -X POST https://runflow.rodmena.app/api/v1/runs \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "image": "python:3.12-slim",
    "command": ["python", "-c", "print(\"hello from runflow\")"],
    "timeout_seconds": 60
  }'

4b. Python script with env vars (useful)

A run that takes an env var and does something useful — fetches a URL and prints the HTTP status + content-type. The env dict is injected into the container as environment variables.

curl -X POST https://runflow.rodmena.app/api/v1/runs \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "image": "python:3.12-slim",
    "command": ["python", "-c", "import os,urllib.request; r=urllib.request.urlopen(os.environ[\"TARGET_URL\"]); print(r.status, r.headers[\"content-type\"])"],
    "env": {"TARGET_URL": "https://api.github.com/repos/python/cpython"},
    "timeout_seconds": 30,
    "tags": ["probe", "github"]
  }'

The tags array is free-form labels for filtering when listing runs. The env values are strings only.

4c. Useful work — data processing with resource limits

A bigger run with CPU/memory limits and a name for easy identification. Note: stdin is not supported — pipe data via env vars or bake it into the command. The container's command reads from env/args, not stdin.

curl -X POST https://runflow.rodmena.app/api/v1/runs \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "image": "python:3.12-slim",
    "command": ["python", "-c", "import json,os; data={\"a\":1,\"b\":2,\"c\":3}; print(json.dumps({\"count\":len(data),\"keys\":list(data.keys()),\"sum\":sum(data.values())}))"],
    "env": {"PYTHONDONTWRITEBYTECODE": "1"},
    "timeout_seconds": 120,
    "cpu_limit": 1.0,
    "memory_limit_mb": 512,
    "name": "summarize-input",
    "tags": ["etl", "summary"]
  }'

Requested cpu_limit / memory_limit_mb / timeout_seconds are clamped to your tenant quotas. If clamped, the response has "clamped": true.

4d. Agent Task: Fetch HackerNews Top Stories

A more complex script showing how an AI agent can execute data-fetching logic entirely on the server. We run a Python one-liner to fetch the top 5 stories from HackerNews and print their titles.

curl -X POST https://runflow.rodmena.app/api/v1/runs \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "image": "python:3.12-slim",
    "command": ["python", "-c", "import urllib.request, json\ntop5=json.loads(urllib.request.urlopen(\"https://hacker-news.firebaseio.com/v0/topstories.json\").read())[:5]\nfor i,sid in enumerate(top5):\n    title=json.loads(urllib.request.urlopen(f\"https://hacker-news.firebaseio.com/v0/item/{sid}.json\").read()).get(\"title\")\n    print(f\"{i+1}. {title}\")"],
    "timeout_seconds": 30,
    "name": "hn-top-5",
    "tags": ["agent", "data-fetch"]
  }'

4e. Scheduling (One-time & Recurring)

A) One-Time Run: scheduled_start is an ISO-8601 UTC instant. A past time is clamped to now. RunFlow holds the run in state scheduled and dispatches it exactly at the target time. (You cannot omit the 'Z').

# Schedule a run 60 seconds from now
WHEN=$(date -u -d '+60 seconds' +%Y-%m-%dT%H:%M:%SZ)

curl -X POST https://runflow.rodmena.app/api/v1/runs \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "image": "python:3.12-slim",
    "command": ["python", "-c", "import datetime;print(\"ran at\", datetime.datetime.now(datetime.timezone.utc).isoformat())"],
    "scheduled_start": "'"${WHEN}"'",
    "timeout_seconds": 30,
    "name": "scheduled-one-time"
  }'

B) Recurring Runs (Cron): RunFlow does schedule. Three options, pick by what you actually need:

• Template triggers — real cron, server-side. Register a workflow template, attach a cron expression and an IANA timezone, and RunFlow fires it unattended (DST-safe, exactly-once per occurrence). POST /api/v1/templates/{id}/triggers
• Scheduled timers — fire a signed webhook at a wall-clock time with no container. Use when you want a callback, not a run: a pending timer costs no run-concurrency slot. POST /api/v1/timers
• Your own scheduler — still perfectly fine, and the right answer if you already have a database and a tick. Example below.

One thing worth knowing if you reach for scheduled_start on a run to fake a timer: a deferred run occupies your max_concurrent_runs quota from submission, not from execution — so a run scheduled a week out holds a slot for a week. Use a timer instead.

# Example crontab (run 'crontab -e', then add):
# 5 9 * * * (runs at 09:05 every day)
5 9 * * * curl -s -X POST https://runflow.rodmena.app/api/v1/runs \
  -H "Authorization: Bearer $API_KEY" -H "Content-Type: application/json" \
  -d '{"image":"python:3.12-slim","command":["python","-c","print(\"daily job\")"],"name":"daily-job"}' \
  >> $HOME/runflow-daily.log 2>&1

4f. Control: pause, resume, stop, kill

# pause a running container
curl -X POST https://runflow.rodmena.app/api/v1/runs/$TRACK/pause \
  -H "Authorization: Bearer $API_KEY"

# resume
curl -X POST https://runflow.rodmena.app/api/v1/runs/$TRACK/resume \
  -H "Authorization: Bearer $API_KEY"

# graceful stop (SIGTERM + grace period)
curl -X POST https://runflow.rodmena.app/api/v1/runs/$TRACK/stop \
  -H "Authorization: Bearer $API_KEY"

# force kill (SIGKILL)
curl -X POST https://runflow.rodmena.app/api/v1/runs/$TRACK/kill \
  -H "Authorization: Bearer $API_KEY"

4g. Workflow: a real ETL DAG (fan-out, fan-in, quality gate)

One POST submits the whole graph. Each node is its own governed run with its own track_id; RunFlow launches each one the moment its dependencies are satisfied. This graph extracts, shards the work across two parallel transforms, merges them, gates on data quality, then either publishes or alerts — the exact pipeline shape most people actually need.

curl -X POST https://runflow.rodmena.app/api/v1/workflows \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "nightly-etl",
    "tags": ["etl"],
    "failure_policy": "continue",
    "nodes": [
      {
        "ref": "extract",
        "image": "python:3.12-slim",
        "command": ["python","-c","import json;print(json.dumps({\"rows\":1200,\"nulls\":5}))"],
        "cpu_limit": 0.5, "memory_limit_mb": 128, "timeout_seconds": 120
      },
      {
        "ref": "transform-eu",
        "image": "python:3.12-slim",
        "inputs_from": ["extract"],
        "command": ["python","-c","import json,os;d=json.loads(os.environ[\"RUNFLOW_INPUT_EXTRACT\"]);print(json.dumps({\"region\":\"eu\",\"rows\":d[\"rows\"]//2,\"nulls\":d[\"nulls\"]//2}))"]
      },
      {
        "ref": "transform-us",
        "image": "python:3.12-slim",
        "inputs_from": ["extract"],
        "command": ["python","-c","import json,os;d=json.loads(os.environ[\"RUNFLOW_INPUT_EXTRACT\"]);print(json.dumps({\"region\":\"us\",\"rows\":d[\"rows\"]//2,\"nulls\":d[\"nulls\"]-d[\"nulls\"]//2}))"]
      },
      {
        "ref": "load",
        "image": "python:3.12-slim",
        "inputs_from": ["transform-eu","transform-us"],
        "command": ["python","-c","import json,os;a=json.loads(os.environ[\"RUNFLOW_INPUT_TRANSFORM_EU\"]);b=json.loads(os.environ[\"RUNFLOW_INPUT_TRANSFORM_US\"]);print(json.dumps({\"loaded\":a[\"rows\"]+b[\"rows\"],\"nulls\":a[\"nulls\"]+b[\"nulls\"]}))"]
      },
      {
        "ref": "quality-gate",
        "image": "python:3.12-slim",
        "inputs_from": ["load"],
        "retries": 1,
        "retry_backoff_seconds": 5,
        "command": ["python","-c","import json,os,sys\nd=json.loads(os.environ[\"RUNFLOW_INPUT_LOAD\"])\nrate=100.0*d[\"nulls\"]/d[\"loaded\"]\nif rate>1.0: sys.exit(1)\nprint(json.dumps({\"passed\":True,\"null_rate\":round(rate,2),\"rows\":d[\"loaded\"]}))"]
      },
      {
        "ref": "publish",
        "image": "python:3.12-slim",
        "inputs_from": ["quality-gate"],
        "command": ["python","-c","import json,os;q=json.loads(os.environ[\"RUNFLOW_INPUT_QUALITY_GATE\"]);print(json.dumps({\"published\":True,\"rows\":q[\"rows\"]}))"]
      },
      {
        "ref": "alert",
        "image": "python:3.12-slim",
        "command": ["python","-c","print(\"paging data-oncall\")"]
      }
    ],
    "edges": [
      {"from": "extract",      "to": "transform-eu", "condition": "on_success"},
      {"from": "extract",      "to": "transform-us", "condition": "on_success"},
      {"from": "transform-eu", "to": "load",         "condition": "on_success"},
      {"from": "transform-us", "to": "load",         "condition": "on_success"},
      {"from": "load",         "to": "quality-gate", "condition": "on_success"},
      {"from": "quality-gate", "to": "publish",      "condition": "on_success"},
      {"from": "quality-gate", "to": "alert",        "condition": "on_failure"}
    ]
  }'

Returns 202 {"workflow_id": "01K...", "status": "pending"}. Note the edge key is "from" (not from_ref) on the way in. transform-eu and transform-us run concurrently; load waits for both. When the gate passes, publish runs and alert is marked skipped; when it fails, the reverse. Each node's last stdout line is JSON, so it lands in the next node as RUNFLOW_INPUT_<REF> — see Message passing.

4h. Workflow: monitor it

Three views: the live graph, the event log, and the terminal result.

WF=01K...  # workflow_id from the submit response

# live graph — every node's state, attempt count and current track_id
curl -s https://runflow.rodmena.app/api/v1/workflows/$WF \
  -H "Authorization: Bearer $API_KEY" | jq '.nodes[] | {ref, state, attempt}'

# event log — replay from the start
curl -s "https://runflow.rodmena.app/api/v1/workflows/$WF/events?since=0" \
  -H "Authorization: Bearer $API_KEY" | jq -r '.[] | "\(.id) \(.node_ref // "workflow") -> \(.to_state)"'

# ...then subscribe: re-poll with the highest id you have seen
curl -s "https://runflow.rodmena.app/api/v1/workflows/$WF/events?since=93" \
  -H "Authorization: Bearer $API_KEY"

# per-node detail — every attempt's track_id, so you can pull that run's logs
curl -s https://runflow.rodmena.app/api/v1/workflows/$WF/nodes/quality-gate \
  -H "Authorization: Bearer $API_KEY" | jq '{state, attempt, attempts, output}'

# terminal result — per-node outcomes (409 not_terminal while still running)
curl -s https://runflow.rodmena.app/api/v1/workflows/$WF/result \
  -H "Authorization: Bearer $API_KEY" | jq

A node's logs are just its run's logs: take a track_id from attempts and call GET /api/v1/runs/{track_id}/logs. There is no separate workflow log endpoint — one node = one run.

4i. Workflow: control — pause, resume, retry a failed stage, cancel

# pause the graph: no NEW nodes are launched; in-flight nodes run to completion
curl -X POST https://runflow.rodmena.app/api/v1/workflows/$WF/pause \
  -H "Authorization: Bearer $API_KEY"

# resume
curl -X POST https://runflow.rodmena.app/api/v1/workflows/$WF/resume \
  -H "Authorization: Bearer $API_KEY"

# fix the cause, then re-run a failed stage AND everything downstream of it.
# Works on an already-terminal workflow: it goes back to running.
curl -X POST https://runflow.rodmena.app/api/v1/workflows/$WF/nodes/quality-gate/retry \
  -H "Authorization: Bearer $API_KEY"
# -> {"status":"ok","message":"Retrying quality-gate (+2 downstream)"}

# cancel everything: in-flight node runs are killed, the rest marked cancelled
curl -X POST https://runflow.rodmena.app/api/v1/workflows/$WF/cancel \
  -H "Authorization: Bearer $API_KEY"

4j. Workflow: human-in-the-loop approval gate

A node with "type":"approval" runs no container. The branch parks at waiting_approval — indefinitely, and the workflow stays active — until you decide. Approve routes down on_success, reject routes down on_failure. Approval nodes need no image.

curl -X POST https://runflow.rodmena.app/api/v1/workflows \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "deploy-with-signoff",
    "nodes": [
      {"ref": "build",  "image": "python:3.12-slim",
       "command": ["python","-c","print(\"built\")"]},
      {"ref": "signoff", "type": "approval"},
      {"ref": "deploy", "image": "python:3.12-slim",
       "command": ["python","-c","print(\"deploying\")"]},
      {"ref": "discard", "image": "python:3.12-slim",
       "command": ["python","-c","print(\"discarded\")"]}
    ],
    "edges": [
      {"from": "build",   "to": "signoff", "condition": "on_success"},
      {"from": "signoff", "to": "deploy",  "condition": "on_success"},
      {"from": "signoff", "to": "discard", "condition": "on_failure"}
    ]
  }'

# ship it (deploy runs, discard is skipped)
curl -X POST https://runflow.rodmena.app/api/v1/workflows/$WF/nodes/signoff/approve \
  -H "Authorization: Bearer $API_KEY"

# or don't (discard runs, deploy is skipped)
curl -X POST https://runflow.rodmena.app/api/v1/workflows/$WF/nodes/signoff/reject \
  -H "Authorization: Bearer $API_KEY"

Scheduling #scheduling

RunFlow uses a reactive scheduling model. When you submit a run, it gets an eligible_start (the earliest time it's allowed to start, based on scheduled_start and the tenant's working-hours calendar). The scheduler then dispatches eligible runs to free workers every 1 second. If no worker is free, the run waits and retries — it is never rejected for capacity reasons.

How eligible_start is computed

eligible_start = working_hours_clamp(max(scheduled_start, now))

• No scheduled_start → eligible_start = now
• Past scheduled_start → clamped to now
• Future scheduled_start → used as-is, then clamped to next working window if outside hours
• No working_hours_tjp → no calendar constraint, pure arithmetic

Capacity (worker pool, concurrent quota) is not considered at submit time. A run at the concurrent quota limit is accepted and waits in scheduled state until a worker frees up. If it waits longer than max_queue_seconds past eligible_start, it transitions to failed:queue_timeout.

Working hours calendar

Set working_hours_tjp on the tenant's quotas to restrict when runs may start. The value is a TJP (TaskJuggler) shift definition. Runs are only dispatched if their eligible_start falls within the shift; if you submit at 18:30 with a Mon-Fri 09:00-18:00 shift, eligible_start advances to Monday 09:00.

# Set working hours via PUT /api/v1/tenant/quotas
curl -X PUT https://runflow.rodmena.app/api/v1/tenant/quotas \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "working_hours_tjp": "shift shift_def \"Business Hours\" { workinghours mon - fri 09:00 - 18:00 }"
  }'

# Now submit on Friday 17:55 (on-shift) -\u003e eligible_start = Friday 17:55
curl -X POST https://runflow.rodmena.app/api/v1/runs \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "image": "python:3.12-slim",
    "command": ["python", "-c", "print(\"within hours\")"],
    "scheduled_start": "2025-07-18T17:55:00Z"
  }'

# Submit on Friday 18:30 (off-shift) -\u003e eligible_start = Monday 09:00
curl -X POST https://runflow.rodmena.app/api/v1/runs \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "image": "python:3.12-slim",
    "command": ["python", "-c", "print(\"after hours\")"],
    "scheduled_start": "2025-07-18T18:30:00Z"
  }'

Expiry — expires_at (absolute deadline)

Pass an optional expires_at (ISO-8601 UTC instant) on submit to set an absolute deadline: if the run has not started by that time, RunFlow fails it as expired instead of dispatching it. It is independent of scheduled_start (when a run is allowed to start) and of max_queue_seconds (a window relative to eligible_start) — expires_at is a hard wall-clock cutoff. Use it for time-sensitive work you would rather drop than run late (e.g. "place this call, but never in the middle of the night").

• A naive (offset-less) value is interpreted as UTC; an expires_at already in the past is rejected at submit with 400 expires_at_in_past.
• Expiry is evaluated against all scheduled runs, not only eligible ones — a run with a future scheduled_start but an earlier expires_at expires when the deadline passes rather than waiting to become eligible first.
• Once a run has started (left scheduled), expires_at no longer applies; use timeout_seconds to bound a running container. The value is echoed back on GET /api/v1/runs/{track_id}.

# Run it, but only if it can start before the deadline — else drop it
curl -X POST https://runflow.rodmena.app/api/v1/runs \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "image": "python:3.12-slim",
    "command": ["python", "-c", "print(\"on time\")"],
    "expires_at": "2026-07-24T22:00:00Z"
  }'
# If still `scheduled` at 22:00:00Z: state=failed, reason=expired (never dispatched)

scheduling_info in run status

Every GET /api/v1/runs/{track_id} response includes a scheduling_info object:

{
  "scheduling_info": {
    "eligible_start": "2026-07-21T10:00:00Z",
    "working_hours_applied": false,
    "queue_position": 3,
    "estimated_start": "2026-07-21T10:00:30Z"
  }
}

• queue_position — how many of this tenant's scheduled runs are ahead in the queue (null if the run is not scheduled). This is the honest signal for "when will mine start."
• estimated_start — a soft best-effort estimate based on historical run durations and queue depth. null when there are fewer than 5 prior runs of the same image. Not a commitment — may change.
• working_hours_applied — true if the tenant has a working_hours_tjp shift set.

What happens when the pool is full

# 4 workers, 4 runs already running, submit a 5th
curl -X POST https://runflow.rodmena.app/api/v1/runs \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"image": "python:3.12-slim", "command": ["python", "-c", "print(5)"]}'

# Response: 202 Accepted (NOT 429)
{"track_id": "01KY...", "status": "scheduled", "eligible_start": "2026-07-21T..."}

# Check scheduling_info to see queue position
curl -s https://runflow.rodmena.app/api/v1/runs/01KY... \
  -H "Authorization: Bearer $API_KEY" | python3 -m json.tool
# "scheduling_info": {"queue_position": 1, "estimated_start": "..."}

# When a running run exits, the 5th run is dispatched within ~1s

Quotas and limits

Per-tenant quotas control resource usage. max_concurrent_runs limits how many runs can be in {pulling, running, paused} simultaneously — enforced at dispatch time, not submit time. max_daily_runs limits submissions per UTC day — enforced at submit time with 429. Resource limits (cpu_limit, memory_limit_mb, timeout_seconds) are clamped to the tenant max and the response includes "clamped": true if any were reduced.

# Check your quotas
curl https://runflow.rodmena.app/api/v1/tenant/quotas \
  -H "Authorization: Bearer $API_KEY"

# Update quotas (requires runflow_tenant_admin permission)
curl -X PUT https://runflow.rodmena.app/api/v1/tenant/quotas \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "max_concurrent_runs": 5,
    "max_daily_runs": 200,
    "max_timeout_seconds": 3600,
    "max_queue_seconds": 600
  }'

Run Lifecycle #lifecycle

Every run moves through these states. Poll GET /api/v1/runs/{track_id} until state is terminal.

Control endpoints: POST /pause, POST /resume, POST /stop (graceful SIGTERM + grace), POST /kill (SIGKILL). All return 204 No Content on success.

Workflows (DAG) #workflows

A workflow is a directed acyclic graph of nodes joined by conditional edges. Submit the whole graph in one POST /api/v1/workflows; the engine ticks every second, launching each node the moment its incoming edges are satisfied. One node = one run = one track_id — nodes are ordinary runs, so they inherit every quota, working-hours rule and container hardening described elsewhere on this page, and their logs, results and utilization come from the same /api/v1/runs/{track_id}/* endpoints.

Worked examples: 4g submit, 4h monitor, 4i control, 4j approval gate.

Workflow states

A workflow is terminal only when no node is running, retrying or waiting_approval. An approval gate keeps it active indefinitely.

Node states

Edge conditions

An edge fires based on the upstream node's terminal state. A node launches when every incoming edge is satisfied (nodes with no incoming edges are roots and start immediately).

failure_policy

Either way the workflow still settles failed if any node failed — continue changes how much of the graph runs, not the verdict. Check failed_nodes in the result.

Message passing between nodes

Nodes exchange small JSON payloads — there are no shared volumes.

• Producing — a node's output is the last non-empty line of its stdout. Valid JSON object → used as-is; valid JSON scalar → wrapped as {"value": ...}; not JSON → the last 1 KB as {"text": "..."}.
• Consuming — list upstream refs in inputs_from. Each one is injected as an environment variable RUNFLOW_INPUT_<REF> holding that node's output as a JSON string.
• Naming — the ref is upper-cased with every non-alphanumeric character replaced by _. So transform-eu → RUNFLOW_INPUT_TRANSFORM_EU.
• Limit — a node's combined injected inputs must stay under 64 KB; over that the node fails with output {"error":"inputs_too_large"}. Pass a pointer (object key, row id), not a payload.
• At rest — node outputs are encrypted; secret-pattern keys are redacted in API responses but passed to downstream nodes intact.

Retries

Each node carries its own retries (0–10, default 0) and retry_backoff_seconds (default 10). A failed attempt puts the node in retrying, waits the backoff, then dispatches a fresh run with a new track_id — every attempt is listed in attempts on the node detail endpoint. Only when retries are exhausted does the node become failed and its on_failure edges fire. The backoff is fixed, not exponential.

POST /nodes/{ref}/retry is the manual counterpart: it resets that node and its entire downstream closure to pending, and revives an already-terminal workflow back to running. Use it after fixing the underlying cause. The node must be terminal (else 409 node_not_terminal).

Limits

Two tenant quotas apply on top of the per-run ones: max_nodes_per_workflow (default 50, exceeded → 400 too_many_nodes) and max_concurrent_workflows (default 10, exceeded → 429 workflow_quota_exceeded). Every node also consumes a normal run slot and counts toward max_daily_runs, so a 6-node workflow spends 6 of your daily runs.

WorkflowSubmitRequest — POST /api/v1/workflows body

{
  "name": "nightly-etl",               // optional, max 128 chars
  "tags": ["etl"],                     // default [], free-form labels
  "failure_policy": "fail_fast",       // fail_fast (default) | continue
  "nodes": [ /* NodeSpec, at least 1 */ ],
  "edges": [ /* EdgeSpec, default [] */ ]
}

NodeSpec — one node

{
  "ref": "transform-eu",               // required, unique within the workflow, max 128 chars
  "type": "container",                 // container (default) | approval
  "image": "python:3.12-slim",         // required for container nodes; omit for approval
  "command": ["python", "-c", "..."],   // default []
  "env": {"KEY": "value"},              // default {}, string values only
  "inputs_from": ["extract"],           // default [], upstream refs to inject as RUNFLOW_INPUT_*
  "retries": 0,                        // default 0, range 0-10
  "retry_backoff_seconds": 10,          // default 10, fixed (not exponential)
  "cpu_limit": null,                    // per-node; clamped to tenant quota
  "memory_limit_mb": null,              // per-node; clamped to tenant quota
  "timeout_seconds": null               // per-node; clamped to tenant quota
}

EdgeSpec — one dependency

{
  "from": "extract",                    // required, upstream ref. NOTE: "from", not "from_ref"
  "to": "transform-eu",                // required, downstream ref
  "condition": "on_success"           // on_success (default) | on_failure | always
}

Requests use "from"; responses echo edges back as "from_ref"/"to_ref".

WorkflowStatus — GET /api/v1/workflows/{workflow_id}

{
  "workflow_id": "01KY2NQ63B95DZ7SYD741W3S45",
  "tenant_id": "...",
  "name": "nightly-etl",
  "state": "running",                  // pending|running|paused|succeeded|failed|cancelled
  "failure_policy": "continue",
  "submitted_at": "...",
  "started_at": "...",                 // null until the first node launches
  "finished_at": "...",                // null until terminal
  "tags": [...],
  "nodes": [{
    "ref": "quality-gate",
    "type": "container",
    "state": "running",                // see Node states above
    "attempt": 1,
    "current_track_id": "01KY...",    // pull logs via /api/v1/runs/{track_id}/logs
    "exit_code": null
  }],
  "edges": [{"from_ref":"load","to_ref":"quality-gate","condition":"on_success"}]
}

WorkflowNodeDetail — GET /api/v1/workflows/{workflow_id}/nodes/{ref}

{
  "ref": "quality-gate",
  "state": "failed",
  "attempt": 2,
  "attempts": ["01KY2H7ZJN...", "01KY2H85EV..."],  // track_id per attempt, oldest first
  "current_track_id": "01KY2H85EV...",
  "output": {...},                            // parsed from the last stdout line
  "started_at": "...",
  "finished_at": "..."
}

WorkflowResult — GET /api/v1/workflows/{workflow_id}/result (terminal only)

{
  "workflow_id": "...",
  "state": "succeeded",                 // succeeded|failed|cancelled
  "duration_seconds": 5.08,
  "nodes": [{
    "ref": "quality-gate",
    "final_state": "succeeded",
    "exit_code": 0,
    "attempts": 1,                     // count, not the id list
    "output": {"passed": true}
  }],
  "failed_nodes": []                        // refs that ended failed
}

Returns 409 not_terminal while the workflow is still active.

WorkflowEvent — GET /api/v1/workflows/{workflow_id}/events

id is a monotonic cursor. Replay from ?since=0; subscribe by re-polling with the highest id you have seen. Query: since (default 0), limit (default 200).

[{
  "id": 89,                              // cursor — pass back as ?since=89
  "workflow_id": "...",
  "node_ref": "quality-gate",            // null for workflow-level events
  "kind": "node",                       // node | workflow
  "to_state": "running",
  "detail": {"attempt": 1},              // e.g. {"attempt":N} or {"has_output":true}
  "created_at": "2026-07-21T15:46:44Z"
}]

Workflow Templates #templates

Register a DAG once, then run it by id — or by a name you choose — with just parameters. A template turns a pipeline into a stable, typed, tenant-private endpoint instead of a fat request body you keep re-sending. It is not a second engine: a template run substitutes your parameters and then goes down exactly the same path as POST /api/v1/workflows, so graph validation, the image allowlist, resource clamping, metering and container hardening are unchanged.

Register it

Declare parameters, then reference them anywhere you want substitution as ${{ name }}. The declaration is the template’s contract: a bad call is a 422 before any container starts.

curl -X POST https://runflow.rodmena.app/api/v1/templates \
  -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
  -d '{
    "name": "summarise-repo",                 // optional; ^[a-z][a-z0-9-]{1,62}$, unique to YOUR tenant
    "params": [
      {"name": "repo",  "type": "string", "required": true,
       "pattern": "^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$"},
      {"name": "model", "type": "enum", "enum": ["glm-5.2", "gpt-4o-mini"], "default": "glm-5.2"},
      {"name": "LLM_KEY", "type": "string", "required": true, "secret": true}
    ],
    "definition": {                                // the same body as POST /workflows
      "name": "summarise-${{ repo }}",
      "nodes": [
        {"ref": "fetch", "image": "python:3.12-slim", "timeout_seconds": 60,
         "command": ["python", "-c", "..."],
         "env": {"SOURCE_URL": "https://api.github.com/repos/${{ repo }}"}},
        {"ref": "summarise", "image": "python:3.12-slim", "inputs_from": ["fetch"],
         "command": ["python", "-c", "..."],
         "env": {"LLM_MODEL": "${{ model }}", "LLM_API_KEY": "${{ LLM_KEY }}"}}
      ],
      "edges": [{"from": "fetch", "to": "summarise", "condition": "on_success"}]
    }
  }'
# 201 -> {"template_id":"01KYD...","name":"summarise-repo","version":1, ...}

Run it

curl -X POST https://runflow.rodmena.app/api/v1/templates/summarise-repo/run \
  -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
  -d '{"params": {"repo": "rodmena-limited/RunFlow", "LLM_KEY": "sk-..."}}'
# 202 -> {"workflow_id":"01KYD...","template_id":"01KYD...","version":1,"status":"pending"}

# from here it is an ordinary workflow:
curl .../api/v1/workflows/$WF          # state + per-node progress
curl .../api/v1/workflows/$WF/result   # each node's JSON output
curl ".../api/v1/workflows?template_id=$TID"   # every run of this template

Optional on a run: version (pin a version), name (override the workflow name), tags (added to the definition’s), scheduled_start (defer the whole graph — no node starts before it), webhook.

Parameters

• Types: string · integer · number · boolean · enum, with required, default, enum, min/max, max_length/pattern, secret, description. Up to 64 per template.
• Where they substitute: node env values, node command items, the workflow name, and tags. Not image, refs, edges or resource limits — a placeholder there is rejected at create time rather than shipped to Docker.
• A parameter is data, never structure. A value is substituted into one string leaf, so quotes, braces and newlines cannot add an env key, swap an image, or add a node. A value may itself contain ${{ ... }} (handy for LLM prompts) — it passes through untouched and is never re-expanded.
• Unknown parameters are rejected, not ignored: a typo must not silently run with defaults. Nothing partial is ever created — a rejected call launches no workflow.

Names, versions, limits

• A name is optional and renameable; the template_id always works. Names are unique within your tenant only — another tenant may hold the same one, and neither of you can see or run the other’s. A template that is not yours is 404, never 403.
• Renaming keeps the id, the version history, and every reference; nothing internal points at the name.
• Editing definition or params publishes a new immutable version; a run may pin version, so an edit never silently changes what a pinned run executes. Max 200 versions.
• Deleting frees the name and the quota slot immediately; workflows already launched keep running.
• Free tier: 1 stored template. Standard default: 20. GET /api/v1/usage reports limits.workflow_templates and templates_in_use.
• Stored definitions are AES-256-GCM encrypted at rest and secret-named env values read ***REDACTED***, while the container still receives the real value.
• Running one needs both runflow_template_run and runflow_workflow_submit — a template is never a route around workflow authority. The tier allowlist, clamps and quotas are applied at run time against your current tier.

Over MCP: runflow_create_template, runflow_list_templates, runflow_get_template, runflow_update_template, runflow_delete_template, runflow_run_template — same arguments, same RBAC, same tenant scoping.

Recurring schedules are live. Two primitives, both delivering a signed webhook and running no container, so neither spends run quota: POST /api/v1/timers for a one-shot wake at an absolute instant, and POST /api/v1/schedules for a recurring cron in an IANA timezone (exactly-once per occurrence, DST-safe, fires on recovery after downtime with an honest fire_delay_seconds). A template trigger (POST /api/v1/templates/{ref}/triggers) is the third form: a cron that launches a stored workflow rather than delivering a webhook.

Outbound Webhooks #webhooks

Stop polling. Register a delivery target once and RunFlow POSTs a signed JSON envelope the moment a run, workflow or node changes state — over the same lifecycle catalog the /events cursor already exposes. Delivery is at-least-once (idempotent on event_id), retried with exponential backoff, observable, replayable, and SSRF-hardened.

Two ways to register

Inline (one shot) — declare a webhook object on a run or workflow submit body; scoped strictly to that one resource.

curl -X POST https://runflow.rodmena.app/api/v1/runs \
  -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
  -d '{
    "image": "python:3.12-slim",
    "command": ["python","-c","print(\"done\")"],
    "webhook": {"url": "https://me.example.com/hooks/run1", "events": ["run.terminal"]}
  }'

Tenant endpoint (durable) — register once, filter by event. A 32-byte secret is generated for you and returned exactly once; every delivery is signed with it.

curl -X POST https://runflow.rodmena.app/api/v1/webhooks \
  -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
  -d '{"url": "https://me.example.com/hooks/runflow", "events": ["run.failed", "node.waiting_approval"]}'
# -> 201 {"endpoint_id":"01K...","secret":"a1b2...(64 hex chars)","url":"https://me.example.com/hooks/runflow","events":["run.failed","node.waiting_approval"],...}

If an endpoint with an inline webhook both match the same event, RunFlow delivers to all of them as independent deliveries with independent attempt budgets.

Event catalog

A subscription's events list is a union: a delivery is enqueued when the concrete event matches any literal, alias, or wildcard. Exactly one delivery is enqueued per (matching subscription, event) pair, even if several patterns match.

Envelope

{
  "event_id": "01K...",                  // ULID; stable across attempts + redelivery (dedupe on this)
  "event_type": "run.exited",
  "occurred_at": "2026-07-21T10:00:00Z",
  "tenant_id": "...",
  "resource": {"type": "run", "id": "01K..."},
  "data": { ... },                       // see data shape below
  "delivery": {"delivery_id": 42, "attempt": 1, "endpoint_id": "01K...|null"},
  "truncated": false                     // true if result_summary or data.nodes were trimmed to fit max_webhook_payload_bytes
}

For a run event, data carries {track_id, state, name, tags, image, submitted_at, started_at, finished_at, workflow_id, node_ref, env} with env redacted. For a run.terminal event, a result block is added: {exit_code, duration_seconds, stdout_bytes, stderr_bytes, result_summary} (last 1 KB of stdout). For a workflow event, data carries {workflow_id, state, name, failure_policy, tags, nodes, failed_nodes}. For a node event, {workflow_id, ref, state, attempt, current_track_id, exit_code, output}.

Headers

Hex-decode the secret before you HMAC with it. A secret we generate is 32 random bytes, handed to you once as 64 hex characters. The HMAC key is those 32 raw bytes (bytes.fromhex(secret)), not the 64-character string. Using the string directly produces a well-formed signature that will never match ours, and the symptom is that every delivery fails your verification while your code looks correct. A secret you supply is used verbatim as the key, with no decoding. GET /api/v1/webhooks/{id} reports which kind you hold as secret_origin: generated, supplied, or unknown for endpoints registered before we recorded it — we do not guess, because a wrong provenance points you at the wrong derivation. Rotation always generates, so a supplied-secret endpoint flips to generated on rotation and the governing rule flips with it.

Reference vector — a verifier that reproduces this will also verify real deliveries. The dummy secret below is the same shape we mint (64 hex), which is what makes it a valid check:

secret     = "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f"
key        = bytes.fromhex(secret)                  # 32 bytes
timestamp  = 1700000000
raw body   = {"event":"webhook.test","data":{"hello":"world"}}
signed     = "1700000000." + raw body               # note the "." separator

X-RunFlow-Signature: sha256=17737c4354daa76a95234accecd925dbb898813f8ae49c30be40f563c141a079

# Keying with the 64-character STRING instead of its decoded bytes gives:
#   sha256=ffa67b3ea6d46819da436a844b11096bc763ea2c20e278e0cb38b69e149013c1
#
# For a GENERATED secret (this vector) that is the mistake. For a SUPPLIED
# secret it is the CORRECT and only derivation — the same operation, right or
# wrong depending entirely on where the secret came from.

There is no origin-independent “wrong” derivation. Do not turn the line above into a receiver rule such as “reject the verbatim-keyed digest”: that rejects every legitimately supplied secret. Decide by origin at configure time — you either minted the secret or we did — and key each secret one way only. A receiver that tries both derivations authenticates no more callers, but it erases the distinction and makes this documented error indistinguishable from a valid signature.

Compare the raw request body bytes — do not re-serialise the parsed JSON before hashing, as any key reordering or whitespace change alters the digest.

Zero-gap secret rotation. rotate-secret MINTS, so you cannot pre-install the new value and every delivery between the swap and your redeploy fails verification. To rotate with no window, supply instead: generate a value yourself → install it in your receiver → PATCH /api/v1/webhooks/{id} with {"secret": "…"} → retire the old one at your leisure. You install before we switch, so no valid signature is ever refused.

The caveat that makes or breaks it: a supplied secret is keyed verbatim, whatever it looks like. If you generate 64 hex characters and supply them, the value looks generated, parses as hex, and must still be keyed verbatim — so a receiver that hex-decodes it derives a different key and fails silently rather than loudly. Tag the entry in your own config with the derivation at the moment you install it; the config change and the PATCH have to land together.

Verifying the signature (Python)

import hmac, hashlib

def verify(raw_body: bytes, signature_header: str, shared_secret: bytes) -> bool:
    if signature_header == "none":
        return True  # inline webhook opted out of signing
    if not signature_header.startswith("sha256="):
        return False
    timestamp = ...  # from X-RunFlow-Timestamp header
    expected = hmac.new(shared_secret, f"{timestamp}.".encode() + raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(signature_header.removeprefix("sha256="), expected)

During a secret rotation your receiver holds two valid secrets (the new one and the previous one, for the grace period), and the function above accepts only one. Verify against every candidate and combine with a bitwise OR — never return early on the first match:

def verify_any(raw_body: bytes, signature_header: str, secrets: list[bytes]) -> bool:
    ok = False
    for secret in secrets:            # NO early return, NO ordering heuristic
        ok |= verify(raw_body, signature_header, secret)
    return ok

The early return is the whole point: any early exit over a set of secrets leaks, through response timing, which secret matched — whatever decides the order, be it a hint from the payload, a cache, or a most-recently-used heuristic. Accumulating over all candidates has nothing to leak and stays safe against a later "optimisation" added by someone who has not thought about it. In particular, do not order by delivery.endpoint_id from the request body: that body is attacker-controlled until the HMAC checks out, so it may be a hint, never a trust decision. (Credit: the uptime.systems team, who declined the hint for exactly this reason.)

Retry schedule & delivery states

A 2xx response marks the delivery delivered. 408, 429, 5xx, timeouts and transport failures retry with backoff at approximately 10 s, 60 s, 5 m, 30 m, 1 h, to a maximum of 5 attempts. Any other 4xx marks the delivery dead immediately (the receiver has rejected the payload). A 3xx (redirect) is not followed and counts as a failed attempt. An endpoint is auto-disabled (active=false, disabled_reason="dead_without_success_24h") when it accumulates 20 dead deliveries within 24 hours AND zero successful ones in that same window — a single delivered webhook in the window vetoes the disable outright, so a partially-failing endpoint is never cut off completely. A webhook.endpoint_degraded notice is emitted earlier, when the 24-hour dead count reaches 5, successes or not. Re-enable with PATCH {"active": true}. Note that the rule is a 24-hour window, not a streak: consecutive_failures is reported for information and does not drive this decision.

A non-retryable 4xx does not degrade your endpoint — it deletes the delivery. There is no second attempt, so the event is gone. The case that catches people is not a route that is missing; it is a route that is up and authenticates the wrong thing — a signature check against the wrong secret, an auth middleware that has not been made exempt, a replay guard keyed on something that changes per attempt. Each of those answers 401, and 401 is terminal here.

Pick the right metric to alarm on. consecutive_failures does count a delivery that dies on attempt 1 — every dead delivery increments it — but any single success resets it to zero. An endpoint that fails intermittently therefore returns to 0 forever and never trips a streak threshold, so a reading of that field alone cannot show you flapping. Alert instead on tenant.delivery_dead and dead_deliveries_total: the cumulative counter and last_dead_delivery_at are never cleared by a success or by re-activation, which is precisely why they exist.

So: bring the route up, verify it answers your own hand-signed request, and register the endpoint last. Note also that each attempt is signed at attempt time, not at fire time — only the body and X-RunFlow-Event-Id are stable across retries, so a freshness window measures transit only, and a replay guard keyed on the signature will reject every legitimate retry.

Listing deliveries reads oldest-first. GET /api/v1/webhooks/{endpoint_id}/deliveries orders by ascending delivery id, and limit truncates the front of the history — so ?limit=100 returns the FIRST hundred, not the most recent hundred. It is a forward cursor feed: page it with ?since=<last id seen>. A freshness or "what happened lately" question built on the unpaged listing measures pagination rather than time; use the endpoint record's last_dead_delivery_at, dead_deliveries_total and updated_at for that instead. (Reported by the uptime.systems team, who stepped in it and caught themselves.)

Unknown query parameters are rejected with 400 naming the supported set, on every route that takes them — a misspelled filter fails loudly rather than returning unfiltered results with a 200. Note that the parameter is state_filter, not state.

Replay any terminal delivery with POST /api/v1/webhooks/deliveries/{delivery_id}/redeliver — a NEW delivery row is created that reuses the original event_id so receiver-side dedup still holds. Delivery is at-least-once; receivers must be idempotent on event_id.

Egress safety (SSRF)

RunFlow makes outbound HTTPS to tenant-supplied URLs. To prevent abuse as an SSRF pivot into the host's private network:

• Only https:// targets are permitted; http://, file://, gopher:// and every other scheme are refused.
• Immediately before each attempt the target hostname is resolved; if any resolved address is loopback, link-local (169.254.0.0/16, IPv6 fe80::/10), unique-local (fc00::/7), RFC1918, CGNAT (100.64.0.0/10), multicast, reserved, or unspecified, the attempt is aborted and recorded blocked_target with no retry.
• The connection is pinned to the validated address (DNS rebinding protection) while preserving SNI and the Host header.
• HTTP redirects are never followed. TLS verification cannot be disabled. The caller's API key is never attached; the HMAC signature is the only authentication a receiver is given.

Inline webhook per resource (FR-WH-17)

An inline webhook is scoped strictly to the resource it was submitted with. Omit secret to deliver unsigned with X-RunFlow-Signature: none (opting out is explicit and visible). Unknown keys inside the webhook object are rejected with 422 (a misspelled secret cannot silently produce unsigned deliveries).

Usage & limits #usage

GET /api/v1/usage reports what your key is allowed to do and, on a metered tier, how much of it is left. Call it before planning a batch of work — it is the cheapest way to avoid a mid-run 429. Requires runflow_run_read.

curl https://runflow.rodmena.app/api/v1/usage \
  -H "Authorization: Bearer $API_KEY"

{
  "tier": "free",          // null on an unmetered (standard) key
  "metered": true,
  "limits": {
    "memory_limit_mb": 256,
    "cpu_cores": 0.5,
    "timeout_seconds": 60,
    "concurrent_runs": 1,
    "daily_runs": 100,
    "workflow_nodes": 5,
    "retention_days": 3
  },
  "allowed_images": ["python:3.12-slim", "alpine:latest", "..."],
  "expires_at": "2026-10-23T03:24:39Z",
  "usage": [
    {"policy": "runflow-free-runs-daily", "used": 7,
     "limit": 100, "remaining": 93,
     "resets_at": "2026-07-26T00:00:00Z"}
  ]
}

Notes for agents. allowed_images is null when any image is permitted; when it is a list, submitting anything else returns 403 image_not_allowed. A rate-limit policy reports "used": null — a token bucket has no window total, so do not sum that field. On a metered key a run is charged its full timeout_seconds up front and refunded the unused remainder when it finishes, so remaining dips while a run is in flight and recovers after it ends — set a realistic timeout and you will be charged for what you actually use.

Endpoint Reference #endpoints

Workflow permissions are distinct from run permissions: a key with runflow_run_* only cannot submit or control workflows. Run-level endpoints still work on individual node runs, so runflow_run_read is what you need to fetch a node's logs.

Request / Response Models #models

RunSubmitRequest — POST /api/v1/runs body

{
  "image": "python:3.12-slim",          // required, Docker image ref (host/path[:tag|@digest])
  "command": ["python", "-c", "..."],   // default [], command args
  "env": {"KEY": "value"},              // default {}, injected as container env vars (string values only)
  "scheduled_start": null,              // ISO-8601 or null (ASAP)
  "timeout_seconds": null,             // null=use default; clamped to quota max_timeout_seconds
  "cpu_limit": null,                     // cores, clamped to quota max_cpu_cores
  "memory_limit_mb": null,               // MB, clamped to quota max_memory_mb
  "name": "optional-label",             // max 128 chars
  "tags": ["tag1", "tag2"]              // default [], free-form labels
}

RunSubmitResponse — 202

{
  "track_id": "01KY0XX71BMX0B88C3A189S2K7",   // ULID
  "status": "scheduled",
  "eligible_start": "2026-07-20T23:31:17Z",
  "clamped": false                       // true if requested limits were reduced to quota
}

RunStatus — GET /api/v1/runs/{track_id}

{
  "track_id": "...",
  "tenant_id": "...",
  "state": "running",                   // scheduled|queued|pulling|running|paused|exited|failed|timeout|killed
  "image": "...",
  "command": [...],
  "env": {...},
  "submitted_at": "...",
  "eligible_start": "...",
  "started_at": "...",                  // null until running
  "finished_at": "...",                 // null until terminal
  "exit_code": 0,                       // null until terminal
  "container_id": "...",
  "worker_id": "...",
  "cpu_limit": 1.0,
  "memory_limit_mb": 512,
  "timeout_seconds": 60,
  "name": "...",
  "tags": [...],
  "stdout_file": "/var/log/runflow/...stdout",
  "stderr_file": "/var/log/runflow/...stderr",
  "events": [{"ts":"...","from_state":"...","to_state":"...","reason":"...","payload":{}}],
  "scheduling_info": {                        // see Scheduling section
    "eligible_start": "2026-07-21T10:00:00Z",
    "working_hours_applied": false,
    "queue_position": 3,                   // null if not scheduled
    "estimated_start": "2026-07-21T10:00:30Z"  // soft estimate, null if no history
  }
}

RunResult — GET /api/v1/runs/{track_id}/result (terminal only)

{
  "track_id": "...",
  "exit_code": 0,
  "status": "exited",                   // exited|failed|timeout|killed
  "started_at": "...",
  "finished_at": "...",
  "duration_seconds": 12.34,
  "stdout_bytes": 1024,
  "stderr_bytes": 0,
  "stdout_file": "...",
  "stderr_file": "...",
  "result_summary": "..."
}

LogResponse — GET /api/v1/runs/{track_id}/logs

{
  "lines": ["line1", "line2"],
  "next_since": 2,                       // pass as ?since=2 for the next page
  "eof": true                            // false = more lines may arrive (run still active)
}

UtilizationResponse — GET /api/v1/runs/{track_id}/utilization

{
  "samples": [
    {
      "timestamp": "2026-07-20T23:31:18Z",
      "cpu_percent": 12.5,
      "memory_mb": 128.4,
      "memory_limit_mb": 512,
      "net_rx_bytes": 1024,
      "net_tx_bytes": 2048
    }
  ]
}

TenantQuotas — GET /api/v1/tenant/quotas, PUT /api/v1/tenant/quotas

{
  "max_concurrent_runs": 10,          // max runs in {pulling,running,paused} simultaneously
  "max_daily_runs": 100,            // max submissions per UTC day (still rejected at submit)
  "max_timeout_seconds": 86400,
  "max_cpu_cores": 4.0,
  "max_memory_mb": 4096,
  "max_queue_seconds": 3600,        // max wait after eligible_start before queue_timeout
  "working_hours_tjp": null,         // TJP shift definition, see Scheduling section
  "retention_days": 30,
  "max_nodes_per_workflow": 50,     // exceeded -> 400 too_many_nodes
  "max_concurrent_workflows": 10,    // exceeded -> 429 workflow_quota_exceeded
  "max_webhook_endpoints": 10,       // exceeded -> 429 webhook_quota_exceeded
  "max_webhook_payload_bytes": 262144 // 256 KiB; result_summary then data.nodes truncated to fit
}

PUT requires runflow_tenant_admin permission. Only the quotas fields you include are updated.

curl -X PUT https://runflow.rodmena.app/api/v1/tenant/quotas \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "max_concurrent_runs": 5,
    "max_queue_seconds": 1800,
    "working_hours_tjp": "shift shift_def \"Office\" { workinghours mon - fri 09:00 - 18:00 }"
  }'

Errors #errors

Standard error envelope (FastAPI default):

{
  "detail": "human-readable message"
}

Workflow-specific detail codes

Node-level failures are not HTTP errors. A node whose injected inputs exceed 64 KB fails with output {"error":"inputs_too_large"}; read it from the node detail or result endpoint.