---
icon: 🏃
---

# Flow Runs

A Flow Run records one execution of a specific flow version, from trigger to terminal state. It stores compressed step-by-step logs, supports pause/resume for delay and webhook waits, offers retry strategies, and emits WebSocket + application events for real-time UI.

### Entities & services
- **FlowRun** — id, projectId, flowId, flowVersionId, environment (PRODUCTION/TESTING), status, logsFileId, parentRunId (subflows), `parentWaitpointId` + `dispatchIndex` (barrier attribution — which barrier counts this run, and its ordinal within that barrier), failedStep (JSONB `{name, displayName, message?}`), timeline (JSONB), archivedAt (soft delete).
- **11 statuses**: 3 non-terminal (QUEUED, RUNNING, PAUSED) + 8 terminal (SUCCEEDED, FAILED, TIMEOUT, CANCELED, QUOTA_EXCEEDED, MEMORY_LIMIT_EXCEEDED, INTERNAL_ERROR, LOG_SIZE_EXCEEDED).
- **Waitpoint** — row per paused step: `type` (DELAY|WEBHOOK|BARRIER), `version` (V0|V1), `status` (PENDING|COMPLETED), unique on `(flow_run_id, step_name)`. A BARRIER adds `sealed` (bool) and a nullable `policy` jsonb, and any row may carry `deadLetteredAt`. A partial index on `resumeDateTime` (`idx_waitpoint_live_deadline`, `WHERE status = 'PENDING' AND "resumeDateTime" IS NOT NULL AND "deadLetteredAt" IS NULL`) serves the deadline sweep; every read path leads with `flowRunId` or the primary key, but `idx_waitpoint_project_id` stays because `fk_waitpoint_project_id` is `ON DELETE CASCADE` and a project delete seq-scans the table without it.
- **WaitpointSignal** — one row per thing a BARRIER awaits, created up front: `status` (PENDING|SUCCEEDED|FAILED|REJECTED|CANCELED|NOT_DISPATCHED), `refId` (the child run or link it stands for), nullable `sequence` (the producer's ordinal, partial-unique per barrier), nullable `label`, small `result` jsonb. Release is the pure `shouldReleaseBarrier({ policy, sealed, counts })` in `core-execution` — see decision [000015](../decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md).
- **LogsFile** — zstd-compressed File (type FLOW_RUN_LOG) holding the full executor context.

### How it works
- **Endpoints**: `GET /` (cursor paginated by composite `(created DESC, id DESC)`, filters incl. `failedStepMessage` ILIKE), `GET /count-by-status` (the status chart above the runs table), `GET /:id`, `POST /:id/retry`, `POST /retry|cancel|archive` (bulk), waitpoint resume routes.
- **Retry strategies**: `FROM_FAILED_STEP` (rebuild context from logs, re-run from failure, prior outputs kept) or `ON_LATEST_VERSION` (a new run on the latest locked version, with the same trigger payload). Both resolve the trigger payload via `resolveStepOutput`. If the trigger itself failed, they switch to `executeTrigger: true` to reprocess the raw event.
- **Pause/resume (V1 waitpoints)**: pieces call `createWaitpoint` + `waitForWaitpoint`. Any waitpoint carrying a `resumeDateTime` upserts a `RESUME_DELAY_WAITPOINT` job keyed **per waitpoint** (`systemJobIds.resumeDelay`), removed when it resumes or is deleted; WEBHOOK resumes on an HTTP call to `/:id/waitpoints/:waitpointId[/sync]`; a BARRIER resumes on its own predicate, or on its deadline via the one-a-minute `WAITPOINT_DEADLINE_SWEEP`.
- Logs backed up every 15s during execution for crash recovery; uploaded via 7-day JWT-signed URLs.
- **RUN_TELEMETRY job**: `flow-run-module.ts` registers a BullMQ system job (cron `50 23 * * *`, once daily at 23:50 UTC) that aggregates the day's run counts by `(projectId, flowId, environment)` in one transaction (5-minute statement timeout) and emits a `FLOW_RUN_CREATED` telemetry event per group. No-op when telemetry is disabled. The cron was `0/50 23 * * *` until GIT-1632, which also fired at 23:00 with partial counts.

### Gotchas
- **A Delay inside a Loop pauses and requeues the whole run once per iteration, so no fixed sync-webhook timeout can cover it.** The delay is not a sleep inside the step: each iteration arms a waitpoint, the run goes `PAUSED`, and the resume comes back through the queue, so every item costs its delay plus queue latency and the total scales with the item count. Worked case on dev: `Catch Webhook → Code → Loop { Delay For 8s } → Return Response` over 6 items reported `stepsCount` 9 (Code + Loop + 6 delays + Return Response) and ran 53.6s, of which 48s was delay and 3s was the initial queue leg. With `AP_WEBHOOK_TIMEOUT_SECONDS` at 30 the `/sync` caller was answered 408 mid-loop; Return Response then ran ~23s later and published to a listener that had already resolved and been deleted, so it was a no-op. Putting the response step *after* slow work is the bug: respond before it (respond-and-continue rather than `stop`) or go async with a callback, because raising the timeout only works until someone sends more items. See the sync-response gotchas in [webhooks](../eventing-webhooks/webhooks.md).
- **A worker OOM-kill leaves the run stuck in RUNNING forever, and Cancel is greyed out.** The flow timeout is enforced *inside* the worker, so if the pod dies (OOM) nothing ever transitions the run to a terminal state; Cancel only applies to paused/queued runs, so the UI offers no way out and the run can't be retried either. Bug: activepieces#14372, fix PR #14374. Manual unblock on the customer's Postgres: `UPDATE flow_run SET status = 'CANCELED', "finishTime" = NOW(), updated = NOW() WHERE id = '<run id>' AND status = 'RUNNING';` (run id = last path segment of the run URL), then "Retry on latest version" replays the original payload.
- **A per-signal confirm link's only guard is that every segment of it is unguessable.** `/:id/signals/:signalId/confirm` is `securityAccess.unscoped(ALL_PRINCIPAL_TYPES)`, so the URL carries the signal's random `apId` and never a caller-supplied key — a semantic segment (`approver-a`, an email, an index) is guessable by string substitution, and **one legitimate approver holding their own link could cast a whole K-of-N quorum**. `label` is the audit name in the summary, never the credential.
- **A barrier's deadline is anchored to the run's start, and reaching it releases the barrier rather than failing the run.** `defaultBarrierDeadline` takes `flowRun.created + PAUSED_FLOW_TIMEOUT_DAYS`, not `now + PAUSED_FLOW_TIMEOUT_DAYS`: a barrier opens partway through a run, so a creation-anchored deadline always lands *after* the run's own pause-timeout boundary. That ordering made `handleResumeDelayWaitpoint`'s run-age guard unconditionally true for every naturally scheduled barrier deadline, so the run was marked `FAILED` and the `release({ timedOut: true })` branch below it was dead code — the timeout summary, and every signal already collected, were thrown away. The handler now routes `PauseType.BARRIER` to `release` *before* the run-age guard; the guard still governs `DELAY` and `WEBHOOK`. Tests that backdate only `waitpoint.resumeDateTime` and leave `flowRun.created` fresh cannot see this — backdate the run too.
- **A waitpoint's deadline is set when it is created, never at seal.** A barrier that took its deadline at seal was invisible to the sweep until sealed — safe only while the parent went PAUSED *after* the seal returned. The moment a producer pauses the parent before its awaited things exist, that accident ends: a PAUSED run, an unsealed barrier, no deadline, and no sweep coverage until retention deletes it.
- **The `RESUME_DELAY_WAITPOINT` job id is per waitpoint, not per run.** It was `resume-delay-${flowRunId}`, which collided the moment one run held two timed waitpoints — a delay inside a loop upserted iteration 2's timer over iteration 1's, and BullMQ keeps the *existing* one-time job, so the second iteration's deadline silently never fired. `systemJobIds.resumeDelay({ waitpointId })` is the fix; `systemJobIds.legacyResumeDelay` exists only so `remove` can still find a job scheduled under the old id.
- **A timeout job is usually removed by itself, while it is still running.** `handleResumeSignal` removes the waitpoint's timeout job after every dispatch, and when the timer is what fired, that job is locked by the worker running it. `Job.remove()` throws on a locked job, which logged a false warning on every timer-driven resume; `systemJobsSchedule.removeJob` uses `Queue.remove()`, which answers 0 instead, and leaves the job to `removeOnComplete`.
- **The deadline sweep pages on a keyset cursor, because its rows survive the pass.** An overdue waitpoint stays `PENDING` with the same `resumeDateTime`, so the repo's delete-and-repeat drain loop would re-read the first page forever. `sweepOverdueDeadlines` pages oldest-first on `("resumeDateTime", "id")`, and `idx_waitpoint_live_deadline` carries both columns so a page inside a large tie group stays cheap. A tick that spends its arm quota (`MAX_ARMED_PER_TICK`) or page budget (`MAX_SCAN_PAGES_PER_TICK`) stores the last row it handled in `distributedStore` (`waitpoint:deadline-sweep:cursor`, 10-minute TTL) and the next tick resumes there; a tick that reaches the end deletes it. The scan has no lower bound on purpose: a row overdue past `PAUSED_FLOW_TIMEOUT_DAYS` is exactly the stuck run it exists to find, and nothing deletes waitpoint rows on a schedule.
- **A dead-lettered deadline is stamped in Postgres so the scan can skip it.** Whether a job exhausted its attempts lives in Redis, which a `WHERE` cannot read, so the probe stamps `waitpoint.deadLetteredAt` and the partial `idx_waitpoint_live_deadline` (`deadLetteredAt IS NULL`) drops the row; without it, a burst of failed resumes fills the head of every scan. `createForPause` clears the stamp when its re-arm retries the failed job or finds the row stamped, and the stamp is guarded on `updated <= <scan start>` read from the database clock (`SELECT now()`), so a probe that ran before that retry cannot re-hide the row. It is not grounds to fail the run: two attempts cannot tell a dead flow from a database outage, and for a barrier the deadline is the timeout branch.
- **`armed` counts only what `upsertJob` reports as `'added'`.** `upsertJob` keeps an existing live one-time job and silently drops the requested schedule, so pushing to the result list unconditionally reported re-arms that never happened — and made the sweep's own test vacuous, since `barrierService.create` already schedules a job at the 30-day deadline and the test only backdated the DB column. `upsertJob` now returns a `UpsertJobResult` status (`added` | `kept` | `retried` | `scheduler-upserted`). Note the sweep still probes with its own `getJob` first rather than reading `'retried'`: `upsertJob` calls `retry()` *before* it returns, which is the very thing the probe exists to prevent.
- **A confirm link is open while the run is `PAUSED`, `RUNNING` *or* `QUEUED` — not `PAUSED` alone.** The piece emails the links during the step, so the barrier and its signals exist and are `PENDING` for as long as it takes the engine to finish and the metadata worker to write `PAUSED`. Gating the link on `PAUSED` meant a fast approver got a 200 "Already responded" page and their decision was never recorded or logged. Those three are exactly the states `waitpointService.handleResumeSignal` treats as resumable (it completes the waitpoint in place while `RUNNING`/`QUEUED` and lets the `PAUSED` upload trigger the resume), so the link's gate and the resume path now agree.
- **Resume Confirmation Page (scanner guard)**: the `/confirm` route serves an HTML Approve/Disapprove page on `GET`/`HEAD` (never consumes) and only resumes on `POST` — stops email security scanners (Safe Links, Mimecast, Proofpoint) prefetching approval links. The deprecated bare `GET /:id/waitpoints/:waitpointId` still resumes for old emails. Slack is unchanged (server-side POST from webhook).
- **`POST /confirm` with an empty JSON body is rejected before it reaches the resume code.** Sending `Content-Type: application/json` with no body gets Fastify's `FST_ERR_CTP_EMPTY_JSON_BODY` 400, so the waitpoint is never consumed and the run just stays PAUSED — which reads exactly like a resume regression. `curl -X POST .../confirm?action=approve` is fine because curl sends no content-type at all; an HTTP client that defaults to JSON is not. Either omit the content-type or send a real body (`{}`). Hit while scripting the `/test-waitpoints` flow 4 race ordering, 2026-08-30.
- **Cross-project isolation (subflow parent-fail)**: `markParentRunAsFailed` scopes its parent lookup to `{ id: parentRunId, projectId }` using the child run's authenticated `projectId`. `parentRunId`/`failParentOnFailure` arrive from spoofable webhook headers (`ap-parent-run-id`/`ap-fail-parent-on-failure`) on the public webhook endpoint, so without the scope a failed child in project A could complete a paused parent's waitpoint and resume it in project B. A cross-project parent id now matches nothing and the fail is a no-op; legitimate subflows are always same-project (Call Flow only targets flows in the caller's project).
- **A resume link resumes the run, not the step it was minted for.** Waitpoints are unique on `(flowRunId, stepName)`, so a flow that runs `create_approval_links` and then `wait_for_approval` holds **two PENDING rows at once** — and the link handed out by `create_approval_links` points at the *first* row while the run blocks on the *second*. `handleResumeSignal`'s PAUSED branch looks the row up by `{ id, flowRunId }` with no check that it belongs to the step being waited on, so that link resumes the run anyway and its payload lands on whichever step is paused. Verified 2026-08-30 on a local instance: POSTing the step_1 link while PAUSED on step_3 resumed the run and gave step_3 `approved: true`. Upshot: an approval link is a **run-level** capability, not a step-level one.
- **A resume that arrives while the run is still RUNNING is parked, and delivered later by the `runsMetadataQueue` worker.** `handleResumeSignal`'s RUNNING/QUEUED branch takes `complete()`, which marks the row COMPLETED rather than deleting it; the worker then picks it up on writing PAUSED via `findUndeliveredCompletedWaitpoint` (`flow-runs-queue.ts`) and enqueues the resume. Because delivering a resume never leaves the row at COMPLETED (both `handleResumeSignal`'s PAUSED branch and the race branch in `resumeTrustedWithoutLock` go through `waitpointService.consume`, which deletes a non-barrier and tombstones a barrier as CONSUMED), any row still sitting at COMPLETED is by construction one whose signal was never delivered — which is why that lookup does not need to care how many other waitpoints the run holds. It declines only when the run holds a PENDING **barrier**, since a barrier may be released solely by its predicate or deadline. **Gotcha for anyone narrowing this check:** an earlier form returned null whenever *any* PENDING row existed, which hung every run holding a second waitpoint — `create_approval_links` + `wait_for_approval` sat in PAUSED forever with the approval already consumed (`step_3` output `{approved: true}`, status `PAUSED`). Reproduced and fixed 2026-08-30 on `feat/barrier-service` with flow 4 of `/test-waitpoints`. Single-waitpoint runs (the common subflow-callback case) never showed it, so unit tests and every other flow in the suite stayed green.
- **`create_approval_links` hands out an approval and a disapproval URL that are byte-identical** — both are the bare `/confirm` route with no `?action=`. The verdict comes from the button the human clicks on the confirmation page (which POSTs back with the action), not from which link they were given, and `wait_for_approval` reads `queryParams.action` with anything other than `approve` counting as a disapproval.
- **A barrier's waitpoint row survives its release, and the legacy by-run resume guard keys off that row's existence.** Until 2026-08-30 `legacyResume` / `legacySyncResume` (the deprecated V0 `/:id/requests/:requestId` routes, reached whenever `findPendingByVersion('V0')` returns null) checked `hasPendingBarrier`, which filters `status: PENDING` — so the guard stopped guarding the instant the barrier was released. `closeBarrier` commits COMPLETED in its own transaction and only *then* does `release()` call the trusted resume under the `runs_metadata_${flowRunId}` lock, and the legacy paths take no lock at all. Through that gap **and** the much wider one after delivery (the run stays PAUSED until the engine's next progress upload — `flow_run.status` is written only by the `runsMetadataWorker`), the run was PAUSED with no PENDING barrier, so a legacy resume was accepted and enqueued a **second** job with a request-controlled payload: `enqueueResume` namespaces job ids as `${runId}-resume-legacy` vs `${runId}-resume-${barrierId}` and the worker queue has no BullMQ `deduplication`, so nothing collapsed them. Reproduced against the dev DB on `feat/barrier-service`, and identical on all four stacked branches. **Fixed** by making delivery explicit rather than encoded as row-absence: a delivered barrier is tombstoned `CONSUMED` (see [decision 000033](../decisions/000033-a-released-barrier-leaves-a-tombstone-until-the-run-ends.md)) and the legacy guard became `hasBarrier` — existence, no status filter. **Gotcha for anyone touching these two methods:** `hasPendingBarrier` still exists and `findUndeliveredCompletedWaitpoint` must keep calling *it*, not `hasBarrier`; switching that call site to existence would return null whenever a CONSUMED barrier is present and hang a later non-barrier delivery on the same run. Two different questions, two methods. The V1 addressed path never had the hole — `addressedWaitpointIsBarrier` rejects on `waitpoint.type === BARRIER` whatever the status.
- **`handleResumeSignal`'s PAUSED branch must reject `CONSUMED` only — never require `PENDING`.** The three statuses are a delivery state machine, not a lifecycle: PENDING is *not yet owed*, COMPLETED is *owed*, CONSUMED is *delivered*. Requiring PENDING therefore breaks the pre-completed path, because `findUndeliveredCompletedWaitpoint` hands this branch a COMPLETED row on purpose whenever a callback beat the pause. Tried on `feat/barrier-service` 2026-09-22 while answering a review that proposed exactly that: it failed **nine** tests, four of which predate the barrier work, including `consumes a COMPLETED waitpoint when race recovery fires` and `does not leave a stale COMPLETED row to poison the next pause cycle`. Before that fix the branch had no status filter at all, so a CONSUMED barrier tombstone could be dispatched a second time.
- **That `CONSUMED` check does not need a conditional `UPDATE ... WHERE status = ... RETURNING`, and two reviewers have now asked for one.** The branch already runs `SELECT ... FOR UPDATE` inside `transaction()`, and TypeORM/Postgres default to READ COMMITTED, where a `FOR UPDATE` that blocks on a concurrent writer does **not** return its original snapshot on release — EvalPlanQual follows the update chain and re-checks the `WHERE` against the newest committed version. The predicate is `id` + `flowRunId` + `projectId`, all immutable, so the row comes back with the committed status. Read-then-branch-then-write in that transaction has exactly the atomicity of a conditional UPDATE. A conditional UPDATE is also actively worse here: `consume` is polymorphic (DELETE for a non-barrier, UPDATE for a barrier), and `onReady` has to run *between* the status check and the status write, which a single `RETURNING` statement cannot host.
- **`onReady` runs inside that transaction deliberately — do not move it after the commit.** The order is dispatch → consume → commit, so a crash in the window leaves a resume job plus an unconsumed row: recoverable, and the row is the evidence that recovery is owed. Committing first would leave a `CONSUMED` tombstone (or, for a non-barrier, nothing at all) with no job, which is an unrecoverable hang and a tombstone that lies. The real cost of the current order is that `enqueueResume` does a platform lookup and an unconditional payload offload — taking a *second* pool connection — while holding `FOR UPDATE`. Fix that by hoisting the offload above the transaction, never by reordering the commit.
- **Widening `releaseIfReady`'s lookup to COMPLETED recovers nothing on its own.** It still routes into `release` → `closeBarrier`, which filters `status: PENDING`, returns null, and makes `release` return *before* `resumeTrusted`. A crash between `closeBarrier`'s commit and `resumeTrusted` otherwise strands the run PAUSED with a COMPLETED barrier forever: `releaseIfReady` used to filter PENDING, the deadline handler returns unless the waitpoint is PENDING, and `closeBarrier` deleted the signals so nothing re-enqueues an evaluation. The recovery has to branch on status and re-dispatch the summary already persisted on the row, bypassing `closeBarrier` — routed through the same `resumeTrusted` call the normal path uses, so it inherits the run lock and the CONSUMED guard instead of enqueuing beside them.
- **That recovery is bounded, not complete.** The only automatic re-invoker is BullMQ's stalled check on `BARRIER_EVALUATION`, and the worker takes the default `maxStalledCount: 1` — exactly one free redelivery after a SIGKILL. A second stall retires the job to `failed` permanently, and a Redis flush removes the trigger altogether while Postgres still holds a COMPLETED barrier on a PAUSED run. `attempts: 5` and the 8-minute backoff do not help: they govern *failed* jobs, and a retry just re-runs `releaseIfReady`. The durable backstop is a sweep, and a deadline sweep only closes this if it selects `type = BARRIER AND status = COMPLETED` — the natural `status = PENDING` reading never sees the state. The same hole exists for non-barrier waitpoints (`complete()` commits COMPLETED, the caller dies before dispatch) and is covered only opportunistically by the metadata worker.
- **The deadline sweep carries a second, separate scan for undelivered barriers, and it cannot be folded into the first.** `findOverdueWaitpoints` selects `status = PENDING AND resumeDateTime < now`; a barrier closed a second ago is COMPLETED with its deadline still 30 days out, so widening that query would not see it until the deadline passed. `redeliverUndeliveredBarriers` therefore scans `type = BARRIER AND status = COMPLETED` on a PAUSED run, keyed on `updated` rather than `resumeDateTime`, and calls `releaseIfReady` — which re-dispatches the stored summary. A two-minute grace window keeps it off barriers the normal release is still delivering (that path takes milliseconds), and the scan is covered by the partial index `idx_waitpoint_undelivered_barrier`. A barrier that cannot be delivered at all — its run row gone, say — stays COMPLETED and is retried every tick within the per-tick cap rather than dead-lettered; it is loud in the logs and rare, and retrying is what makes a transient failure self-heal.
- **`releaseIfReady` rethrows a failed re-dispatch, so every loop that calls it in a batch must guard each row.** `redeliverClosedBarrier` logs and rethrows: that is what spends the `BARRIER_EVALUATION` job's `attempts: 5`, and a swallowed error there marks the job successful and throws the retry budget away. The cost is that the sweep calls the same method for up to a hundred rows, so a bare rethrow aborts the tick on the first bad row — and since the scan orders by `updated ASC`, every later tick meets that same row and aborts again, starving every other undelivered barrier. `redeliverUndeliveredBarriers` therefore wraps each row in `tryCatch` and logs. The two belong together: the rethrow is the queue's signal, the per-row guard is what keeps it from becoming a head-of-line block.
- **A waitpoint inside a loop is keyed by its full execution path**, e.g. `step_1:0/step_2`, not the bare `step_2` — so each iteration gets its own row under the `(flowRunId, stepName)` unique constraint instead of colliding with the previous iteration's. Set by `step-execution-path.ts`.
- **ResumeReason** (`WAITPOINT`|`RETRY`) discriminates whether FAILED steps are restored on resume: waitpoint resumes preserve them, retry resumes drop them so the failed step re-executes.
- **Pause timeout is cumulative from `flowRun.created`, not per-pause.** `waitpointService.createForPause` (`clampWaitpointResumeDeadline`) rejects any resume date past `flowRun.created + AP_PAUSED_FLOW_TIMEOUT_DAYS`, throws `ErrorCode.PAUSED_FLOW_TIMEOUT_EXCEEDED`, and `waitpointClient` in the engine translates that to `PausedFlowTimeoutError` (USER → clean FAILED). This catches stacked delays (e.g. `1 + 10 + 10 + 25 + 25 = 71 days` on a 30-day timeout) that individually pass the engine-side per-pause `assertDelayWithinTimeout` but cumulatively outlive retention. Webhook waitpoints without an explicit `resumeDateTime` are defaulted to the same run deadline so they can't wait indefinitely past retention; when their timer fires the `RESUME_DELAY_WAITPOINT` handler branches on `PauseType.WEBHOOK` and marks the run FAILED (expiry, not resume). The same handler also marks FAILED if the timer fires past pause-timeout for a `DELAY` waitpoint — catches BullMQ dispatch lag on ancient queued jobs. `AP_EXECUTION_DATA_RETENTION_DAYS` must be strictly greater than `AP_PAUSED_FLOW_TIMEOUT_DAYS` (a 1+ day margin) or a valid resume firing on the retention boundary can still race the file-cleanup job.
- **`executionJournal.upsertStep` must stay immutable, and step retry is why.** `runWithExponentialBackoff` re-uses the *same* `executionState` for every attempt, so a failed attempt writing its `FAILED` output must not reach back into that state. While the journal still mutated the shared `steps` map (fixed in #14453), the write cleared the step's `PAUSED` status, the next attempt read `isPaused === false` and ran `BEGIN` instead of `RESUME`, armed a **new** waitpoint and paused again — a fresh engine run per resume means `attemptCount` restarts at 1, so `maxAttempts` is unreachable. For Call Flow with wait-for-response that re-invoked the subflow every 4-5s forever with the parent stuck `PAUSED` (GIT-1712, ≤0.86.3, same defect at the old `packages/shared/...` path in 0.85.5). Pinned by the retry-on-failure subflow case in `execute-flow-e2e.test.ts`.
- **Retrying a step that failed on a waitpoint resume is intentional, not an oversight.** Attempts 2-4 re-run the RESUME branch against the same stored resume payload, which is pointless for a piece that just rethrows (Call Flow burns ~28s of backoff before failing) but is exactly right for one that does real work on resume — AssemblyAI's `transcribe` fetches the transcript in its RESUME branch, so a transient API failure there is worth retrying. Suppressing retry for resumed steps would trade that away.
- **Failed-trigger payload** survives past BullMQ job completion only because `buildFailedTriggerContext` writes it into the trigger step's `output` slot.
- **The trigger step's status IS the raw-vs-extracted discriminator for retry** — there is no separate field (a `payload` field was tried and removed as redundant). `FAILED` means "`output` holds a raw event, re-run `run()` on it" (`executeTrigger: true`); `SUCCEEDED` means "`output` is already the trigger's result, replay as-is" (`executeTrigger: false`). So any code that *fabricates* a trigger step without the engine having run — the `QUOTA_EXCEEDED` admission gate is the first — must pick the status from where its payload came: raw for sync webhooks, extracted for anything sourced from the worker RPC `submitPayloads` (which passes post-`TriggerHookType.RUN` output). Get it wrong on a polling trigger and retry re-polls against an already-advanced `lastPoll` cursor, so the run gets `undefined` or an unrelated newer item *and* silently consumes those fresh items' own runs.
- **Big step outputs**: over 32 KB inline → stored as a `LogSliceRef` pointer to a `FLOW_RUN_LOG_SLICE` file (`outputType === SLICE`); missing backing file throws `ENTITY_NOT_FOUND` (loud retry failure). Step *inputs* over 2 KB (`AP_FLOW_RUN_LOG_INPUT_TRUNCATE_THRESHOLD_KB`) become a display-only truncation placeholder.
- **An INTERNAL_ERROR's cause lives in three places — check them in this order.** (1) **The run log.** `addToQueue` always assigns a `logsFileId`, `reportFlowStatus` forwards it with the `internalError`, and `engineRunCallbackService.uploadRunLog` writes the error into the log file, so a platform admin sees it through the runs table's **View error** button (`getPopulated`). On Cloud only `ENGINE`-source errors are kept (`internalErrorEnabled`); `WORKER`-source ones are dropped there. (2) **The worker's `job.execute` event** (since 2026-09): `reportFlowStatus` sets `flowRun.status`, `flowRun.willRetry`, `flowRun.environment`, `flowRun.internalErrorSource`, the failed step's `step.name` when known, and attaches the error, so `service:activepieces-worker @flowRun.id:<id>` shows the cause next to `flow.id`/`project.id`. `outcome` still means the worker's handling verdict and stays `success` for an engine-reported failure — filter on `flowRun.status`, not `outcome`. (3) **The failed BullMQ job's `failedReason`** (engine message plus stderr, set by `jobBroker.completeJob`). On a dedicated worker group that job lives in `platform-<platformId>-jobs`, **not** `workerJobs` — pass `--queue` to `debug-failed-job.js` or it reports "job not found". An earlier version of this note said `logsFileId` is nil on every BEGIN run and the run log shows nothing; that was wrong.
- Retries only allowed on terminal states within `EXECUTION_DATA_RETENTION_DAYS`.
- **The builder's run state is single-run scoped: anything reading a step's run data must go through `useStepOutputInRun`, not `state.run.steps`.** `loopsIndexes` is one flat `Record<stepName, index>` and a batch child is a separate `FlowRun`, so resolving an in-batch step against the *parent* run yields `undefined` — which for a loop inside a batch meant `totalIterations === 0`: the canvas control rendered inert (`max=0`, tooltip "1/0", every input clamped back to iteration 0), so loop-inside-batch could not be viewed past its first iteration even though the panel honours `loopsIndexes` when reading child steps. `useStepOutputInRun` (`run-details/use-step-output-in-run.ts`) is the seam that picks the right steps map; `LoopIterationInput` and the run detail panel use it. The canvas **status and duration badges** go through the seam too since #14898, so a step inside a batch body shows the status of the batch selected in the run detail panel; before, they read the parent run and showed nothing for an in-batch step. `selectFailedStep`/`useFocusOnStep` still read the parent run and can only ever land on the batch step, because the parent's `failedStep` structurally cannot name a step inside a child. Cross-child bleed is handled in the store, not the seam: `setBatchIndex` drops the batch's descendant loop indexes (it has `flowVersion`, so it needs no run data — which is why `loopsIndexes` is *not* keyed by child run id). The store's other loop writer goes through the seam by *parameter*: `setLoopIndex({ stepName, index, steps })` takes the steps map as a **required** argument and re-clamps the loop's descendant loop indexes against it, so descendants clamp against whichever map resolved the parent loop — `LoopIterationInput` passes what `useStepOutputInRun` returned, and the parameter is mandatory precisely because reading the wrong map is the defect class here. It used to read `state.run.steps` unconditionally, which snapped a loop nested in a batch back to iteration 1 every time its enclosing in-batch loop moved. A descendant the passed map cannot resolve is still forced to 0, and that is the *correct* answer across a batch boundary (`loop_outside → batches → loop_in_batch`): a different outer iteration is a different batch output, barrier and children, so there is no index to preserve. Still unfixed and benign for the same reason: `batchesIndexes` is keyed by step name only, so changing an enclosing loop's iteration keeps "batch 4 selected" against a different barrier — a coherent selection, just different items. Pinned by `packages/web/test/app/builder/run-details/batch-loop-scope.test.ts`.
- **`PROCESS_IN_BATCHES` cannot be nested inside another, and the run detail panel quietly depends on that.** The guard is `hasNestedBatch` in `flowOperations.apply` (`core/execution/flows/operations`), checked *after* the operation switch so it covers add, paste, move, duplicate, update-in-place and import alike — the web picker's `excludeProcessInBatches` is only the friendly half. This is what makes `enclosingBatchStep` (`run-details/use-step-output-in-run.ts`) correct: it takes the *first* ancestor batch from a pre-order `getAllSteps`, i.e. the outermost, which is the only one for every structure that can exist. Were nesting ever allowed, that lookup returns the wrong batch *and* `useBatchStepRun` breaks harder — it resolves the summary from `state.run.steps`, but an inner batch's summary lives in the outer child's run, so the inner batch's Batches tab would sit on a skeleton forever while its Output tab rendered fine. Pinned by `core/execution/test/flow/process-in-batches.test.ts` ("refuses a batch step anywhere inside another batch step", plus retyping a body step into a batch). One residual: the guard only blocks *introducing* nesting (`hasNestedBatch(cloned) && !hasNestedBatch(original)`), so a version that somehow already has one is grandfathered.
- **`placeholderData: keepPreviousData` on a per-item query is a stale-data bug when a gate reads `isLoading`.** On a key change to an uncached key TanStack v5 hands back the previous key's data and *rewrites the status*: `queryObserver.js` sets `status = 'success'` when placeholder data is used, so `isPending`/`isLoading` are `false` and only `isPlaceholderData` is true — an `isLoading ? skeleton : data` branch then renders the *old* item's payload under the *new* item's label. It bit the batch child-run query in the run detail panel (selecting an unvisited batch showed the previous batch's steps under the new batch's breadcrumb) and was fixed by deleting the option: it only ever applies when a key has no data, and the only thing that changes that key is the user switching batch, so the option's entire effect was the bug. Never gate on `isLoading` alone — gate on `isPlaceholderData`, or don't keep previous data. Pinned by `packages/web/test/features/flow-runs/use-flow-run.test.tsx`.
- **Credit metering (Autumn)**: on terminal runs (paid editions), `onFinish` does two tryCatch-wrapped billing steps that never break run completion. (1) A PRODUCTION run not in `QUOTA_EXCEEDED` charges +1 apCredit via `billingProvider.trackCredits` with idempotency key `{runId}:run`. (2) `flowRunAiUsageTracker` pre-scans the flow version for `@activepieces/piece-ai` steps, extracts per-provider/model usage from step outputs (`flow-run-ai-usage-extractor` — recurses into loops, fetches `FLOW_RUN_LOG_SLICE` files, falls back to flow-version settings on `**REDACTED**` models), meters `Σ(messages × model credit weight) + toolCalls` to Autumn (`{runId}:ai`, plus `{runId}:appSumoAi` for the managed-ACTIVEPIECES AppSumo cap), then emits the `AI_USAGE_PER_RUN` PostHog event — the license key is only the PostHog distinctId, no longer a gate on metering.
- **Credit gate is fail-open at admission**: the worker RPC `submitPayloads` checks `shouldBlockOnCredits` (blocks only when the platform is `billingEnforced` AND the cached balance is exhausted; CE default and Autumn-outage behavior is false). A blocked run is still admitted — as a `QUOTA_EXCEEDED` run with the trigger payload persisted in its log — so it stays retryable once credits return instead of being dropped. **`AP_EDITION=ee` skips the gate entirely** (`shouldBlockRunOnCredits` returns `false` before any provider call) so self-hosters pay no Redis/Autumn latency on admission — a temporary measure, see decision 000020.
- **A new PRODUCTION run has no row yet when its id comes back.** `queueOrCreateInstantly` saves a TESTING run, but hands a PRODUCTION run to `runsMetadataQueue` and returns the object. So the id from a webhook, `start` or a retry (`POST /retry`) can answer 404 on `GET /flow-runs/:id` for a moment, and the run is missing from the list. A reader that polls must treat 404 as "not yet" (`fixtures/runs-list.ts` in the E2E suite does).
- **A cancelled run's terminal status is only *queued*, so `flow_run.status` still reads `PAUSED` for a moment — read the Redis intent, not the row.** `cancelSingleRun` holds `runs_metadata_${id}`, removes the jobs and deletes every waitpoint, then hands `CANCELED` to `runsMetadataQueue` rather than writing it. For that window a resume request sees a `PAUSED` run with no waitpoint and cannot tell it from a legacy pre-waitpoint pause, and nothing downstream stops it — `execute-flow.ts` has no terminal-status guard, so the resume actually runs and overwrites the cancellation. The intent *is* observable: `runsMetadataQueue.add` writes the pending status into the `runs_metadata:<runId>` hash synchronously, inside that same lock, before enqueuing, and the metadata worker clears the key with `deleteKeyIfFieldValueMatches` once it lands. So `resumeService` refuses when `hgetJson(redisMetadataKey(id))` already holds a terminal status, and every resume path — the legacy by-run ones included — takes `runs_metadata_${id}` so the two orderings are both safe: cancel-first makes the resume wait and then see it, resume-first lets cancel's `removeAllFlowRunJobs` reap the job it just enqueued. The lock alone is not enough, and neither is re-reading `flow_run` inside it.
- **The `RESUME_DELAY_WAITPOINT` job id is keyed per waitpoint, and it has to stay that way.** `waitpointService.createForPause` schedules it as `resumeDelayJobId(waitpoint.id)` — never per run — because `systemJobsSchedule.upsertJob` only adds when that id is free (`if (isNil(existingJob))`), so a run-keyed id means the first waitpoint to arm owns the run's **only** timer and every later one silently gets none. Since decision 000015 one run legitimately holds several waitpoints, so that is a hang, not an edge case: `create_approval_links` (WEBHOOK, +30d) followed by `delayFor 45s` (DELAY) left the delay with no timer and the run sat `PAUSED` until the 30-day webhook expiry marked it FAILED. Two `WEBHOOK` rows hide it — identical `AP_PAUSED_FLOW_TIMEOUT_DAYS` deadlines, and each is resumed by its own callback anyway — which is why flow 4 of `/test-waitpoints` never caught it and flow 9 was added. Found and fixed 2026-09-10; the bug had been live on `main`, not introduced by the barrier work. Per-waitpoint keying also fixes the `removeOnFail: { age: ONE_MONTH }` case, where `upsertJob` retries a month-old failed job and would otherwise replay it against a **different** waitpoint's id. Pinned by "should give each waitpoint of one run its own resume job id" in `waitpoint.test.ts`.
- **Post-run metering window**: AI usage is metered only at `onFinish`, so a long run can spend past the credit limit before anything lands; interim by design — see decision 000016.
- **A step that arms a waitpoint and then asks the API to fill it can leave the row behind if that second call fails, and the leftover is safe only while the run stays out of PAUSED.** Both the AI pieces (`createWaitpoint` then `POST /v1/ai/execute`) and the AI Router (`waitpointClient.create` then `POST /v1/engine/ai-router`) create the row first, because the request has to carry the waitpoint id. A failed second call fails the step and the run, and `onFinish` then deletes the PENDING row and its `RESUME_DELAY_WAITPOINT` timer, because the row belongs to the step that failed (see the next bullet for what a failed run keeps). `onFinish` never runs for `INTERNAL_ERROR` (it calls `isFlowRunStateTerminal` with `ignoreInternalError: true`); `cancel` deletes every row (`cancelSingleRun` calls `deleteByFlowRunId`). `handleResumeDelayWaitpoint` skips a timer whose run is not PAUSED or whose row is not PENDING, so a failed run is never touched. The one corner is a run that ends `INTERNAL_ERROR` and so keeps its rows: a retry of that run within the deadline (10 min for the router) that pauses somewhere else is PAUSED when the stale WEBHOOK timer fires, and `isWebhookExpiry` marks it FAILED. Reviewed on #15707 and left as is. A retry that clears every waitpoint was tried on #14888 and dropped, because it also kills an approval link an earlier step handed out; the fix, if this ever bites, is a delete of the row on the failure path.

- **A failed run keeps the waitpoints of the steps that finished, so a retry can still honour a link it handed out.** `FROM_FAILED_STEP` keeps every earlier step's output, including a `create_approval_links` link that a later step may already have emailed. So `onFinish` (`waitpointService.deleteForFinishedRun`) deletes all rows only for a run that succeeded, was cancelled or names no failed step; a run in a failed state (`isFailedState`) loses its barriers and the rows of its failed step — matched on the last segment of the row's `stepName`, so every loop iteration counts — and keeps the rest with their timers. The retry leaves the rows alone, and `createForPause` re-arms a step through `.orIgnore()` on `(flowRunId, stepName)`, so a step that runs again gets its old row and its old link back. Kept rows are inert while the run is terminal: the deadline sweep and `handleResumeDelayWaitpoint` act only on PAUSED runs, and a click answers "expired". Accepted cost: a failed run that is never retried keeps those rows for good — `waitpoint` has no foreign key to `flow_run` and nothing sweeps them — which is one row per handed-out link, not per failure, because a failed step's own rows (an expired approval, an AI leftover) are still deleted. A barrier parent needs a fresh barrier on retry; that is why barriers always go, and any re-dispatch reset belongs next to the code that re-dispatches its children. `ON_LATEST_VERSION` copies `parentRunId` but must never copy `parentWaitpointId`: the barrier already counted the original child, so attributing the retry to it would inflate `terminal` and risk a premature release.
- **`parentWaitpointId IS NOT NULL` is the one test for "this run is a barrier child". `dispatchIndex` is ordering data, never an identity test.** The two columns are written together by the single dispatcher and are set together or not at all; treat that as the invariant and keep any new producer to it. The only place that reads both is `flowRunService.retry`, which refuses a run carrying *either* column — belt and braces, because a retry that slipped through would re-signal a barrier that already counted the original child. Retrying a child is meaningless in both strategies anyway: `FROM_FAILED_STEP` resumes from the trigger, so it would re-run the *parent's* whole prefix inside the child, and `ON_LATEST_VERSION` starts a fresh top-level run that no barrier is counting. `retry` throws `VALIDATION` before the retention check.
- **The bulk helper hides children from a filter, never from an explicit id.** `filterFlowRunsAndApplyFilters` backs `cancel`, `bulkArchive` *and* `bulkRetry`, so an unconditional `parentWaitpointId IS NULL` there made `cancel({ flowRunIds: [childId] })` return 200 and do nothing while the child ran on and the barrier waited. Cancelling a stuck child directly is a legitimate operator action, and it is already safe end to end: `cancelSingleRun` hands `CANCELED` to `runsMetadataQueue`, which is terminal, so the barrier receives the signal. The child filter therefore applies only when the caller passes no `flowRunIds`. It is not load-bearing for retry safety — `bulkRetry` calls `retry()` per run, and that guard is the real lock; a filter-driven bulk retry would otherwise return one error row per child after a wide fan-out.
- **`parentWaitpointId` (and any new run field) must be added to `RUNS_METADATA_UPSERT_KEYS`** in `runs-metadata-queue-factory.ts`. It is an allowlist — an unlisted key is silently dropped between the worker and the row with no error, the same failure class as forgetting `getEntities()`.
- **A new index on a mostly-NULL `flow_run` column must be partial.** Postgres indexes NULLs (unlike Oracle), and a non-HOT UPDATE inserts a tuple into *every* index on the table including ones on columns it did not touch — `flow_run` updates are essentially all non-HOT already (`status` sits in five composite indexes, `logsFileId` is indexed and the 15s log backup touches it). So a full index on a column that is NULL for ~every row adds permanent write amplification to the hottest write path in the product for reads that almost never happen. `WHERE <col> IS NOT NULL` makes that maintenance **zero**, not smaller, and the planner still uses the index for `col = $1` because `x = c` implies `x IS NOT NULL`. `idx_run_parent_waitpoint_id` is the worked example: partial, and widened to `(parentWaitpointId, projectId, status)` so the barrier's existence probe and release aggregate are index-only. `idx_run_parent_run_id`, `idx_run_logs_file_id` and `idx_run_triggered_by` are still full indexes on mostly-NULL columns.
- **Barrier children are unlisted, never invisible — and the discriminator is `parentWaitpointId IS NOT NULL`, nothing cheaper.** Children share the parent's flow id, flow version id *and* display name. Of the eight run list/count surfaces, five filter them out and three deliberately keep them: **user-facing and billing surfaces exclude children, ops and load surfaces keep them.** The five are `flowRunService.list` (one edit covering the runs endpoint, the builder's list, the agent tool and the MCP tool), `countByStatus`, `filterFlowRunsAndApplyFilters`, the billing usage report's daily-execution count and `platformAnalyticsReportService`'s `listRuns`. The three that keep counting are `healthMetricsService`, the telemetry daily rollup in `flow-run-module.ts` and the admin platform retry. Two of the five are correctness, not consistency: the shared filter-and-apply helper is the *only* thing stopping a filter-driven bulk retry from being a second door onto the refused mid-graph retry path, and the analytics filter multiplies a user-set per-trigger constant, so counting children makes a flow batching ten items claim eleven times the time saved. Reaching a child by id is the zero-work default (`getOnePopulatedOrThrow` already scopes by project) and `ListFlowRunsRequestQuery.parentWaitpointId` is the opt-back-in, ordered by `dispatchIndex` — a filter on the existing endpoint, not a new one. That listing also requires `dispatchIndex IS NOT NULL`, and the reason is the cursor, not taste: `Paginator` slices its composite cursor at the **first field missing from the decoded cursor object**, so a page ending on a NULL-index child encodes no `dispatchIndex`, emits *no* cursor predicate at all, and returns page 1 again forever. Any ordering added on a nullable column has this failure mode — either exclude the NULLs or do not sort on it. A child with no dispatch index is not a batch (only the webhook header path can make one) and stays reachable by id. `archivedAt` is a separate user-driven axis and is **not** reused for hiding.
- **The status-count query plans on `idx_run_project_id_environment_status_created_archived_at`, not the `created`-leading one — but only once the visibility map is healthy.** `EXPLAIN` it on a table whose VM is stale (right after a bulk `UPDATE`) and the planner refuses the index-only scan, falls back to a bitmap heap scan on `idx_run_project_id_environment_created_archived_at`, and you extend the wrong index. `VACUUM ANALYZE` first, then `EXPLAIN`. Appending `parentWaitpointId` as the trailing column keeps the scan index-only and pushes the child exclusion into the Index Cond instead of a post-heap Filter — measured on 1M rows / 50k per project: 42ms and 17.8k buffers → 5.5ms and 4k buffers, `Heap Fetches: 0`. A btree cannot be extended in place, so the migration creates the widened index under a new name **concurrently first** and only then drops the old one; dropping first would leave the hottest table in the product without that index for the length of a concurrent build. The children-listing query needs no new index — the partial `idx_run_parent_waitpoint_id` already covers it. **But the widened index only covers the status-count query, and the runs *list* was left behind.** `flowRunService.list` with no status filter orders by `created DESC` and plans on `idx_run_project_id_environment_created_archived_at`, which has no `parentWaitpointId` — so there the child exclusion is exactly the post-heap Filter this bullet says was eliminated, and every child row created since the newest visible parent is fetched and discarded before the page fills. Cost scales with children ever created, not with page size: right after a wide fan-out the first page walks the whole batch to return ten rows, and the table re-polls it every 15s while anything is non-terminal. **Resolved by making the three `created`-leading list indexes partial on `WHERE "parentWaitpointId" IS NULL`** — the only shape the list asks for, and a child costs nothing to maintain there (see the mostly-NULL bullet above): children are absent from the index entirely, so cost is proportional to page size instead of fan-out size, and a fan-out writes zero entries into them. A normal run is in every one of them, so until the four old list indexes are dropped, each insert and non-HOT update of a normal run maintains eight list indexes instead of five. Measured on 1.02M rows / 200 projects / a 20k-child fan-out as the newest activity: the no-filter list 4.5ms → 0.11ms, flow+status 6.2ms → 0.79ms, `countByStatus` 20.5ms → 1.2ms and the billing daily count 10.7ms → 2.3ms (both now `Index Only Scan`, `Heap Fetches: 0`). Trailing the column instead of a partial predicate was measured too and is the weaker fix — it moves the exclusion into the Index Cond but still walks every child's leaf pages, so it only bought 2-3x where partial buys 40x.
- **The status-leading run index must stay NON-partial, and that asymmetry is deliberate.** It is the only index left that can serve the ops aggregates in `healthMetricsService`, which by design count children (see the unlisted-children bullet) and therefore carry no `parentWaitpointId` predicate. Make all four partial and those seven aggregates fall to a `Parallel Seq Scan` on the whole table. The alternative — adding the predicate to `healthMetricsService` — was rejected because it silently reclassifies an ops surface as a user-facing one. So: `created`-leading and `flowId`-leading indexes are partial (list-only), the `status`-leading one is full and keeps `parentWaitpointId` as a trailing column so `countByStatus` stays index-only.
- **The `flow_run` list-index reshape runs in the background track, not at boot.** `AddBarrierChildAttribution` (blocking) only adds the two nullable columns; `ReshapeFlowRunListIndexesForBarrierChildren` (background) builds the five new indexes after the app listens, so a large `flow_run` no longer holds every replica's boot for five concurrent builds. It then drops only `idx_run_project_id_environment_created_archived_at`: with that narrow index present, the planner picks it over the partial `created` index and reads every child again (PGlite, 1M rows and a 20k fan-out: 0.14ms → 3.2ms). Its queries stay covered under the old image, because `idx_run_project_id_environment_created_status_archived_at` has the same leading columns. The other four old indexes stay declared on the entity until a later release drops them, because a Cloud rollback runs the old image on the reshaped table (see the rollback gotcha on the Database page). The new code is fast on the old indexes meanwhile: the list query keeps its index and adds only a `parentWaitpointId IS NULL` filter that matches every row until children exist. **The blocking `down()` restores the old indexes, drops the new ones and deletes the background row before it drops the columns**, because the background track is forward-only and `DROP COLUMN` silently drops all five new indexes with it (each one references `parentWaitpointId`), which would leave `flow_run` with only its primary key. Deleting the row makes a re-upgrade run the reshape again.
- **A run read must never widen its `flow_version` join, and the paginator is why it costs double.** `queryBuilderForFlowRun` is the single builder behind `list`, `getOne`, `getOnePopulatedOrThrow` *and* `findFlowRunOrThrow`, and the only thing any caller ever reads off the relation is `displayName` — the runs table column and the failed-step dialog. It used to be `leftJoinAndSelect`, which selects all thirteen `flow_version` columns including three `jsonb` ones: `trigger` (the whole flow graph, routinely TOASTed), `backupFiles`, `notes`. The `.addSelect(['"flowVersion"."displayName"'])` sitting beside it was raw SQL, so TypeORM mapped it to nothing — a duplicate column in the SELECT list that read as intent and did no work. Three things compounded it: `Paginator.paginate` uses `getRawAndEntities()` and `mergeAdditionalColumns` copies **every raw key not prefixed `<rootAlias>_` or `ap_cursor_`** onto the entity, so each row carried the graph twice (nested `flowVersion.trigger` *and* flattened `flowVersion_trigger`) — that flattening applies to any paginated list with a join, not just runs; `convertDatesToStrings` in `server.ts` then deep-walks both copies before zod strips them, so the wire payload stayed clean while the DB read, the heap and the per-request CPU did not; and `releaseBarrierWithoutLock` calls `findFlowRunOrThrow` up to 3× per barrier release just to read `status`. The fix is `leftJoin` + `addSelect(['flowVersion.id', 'flowVersion.displayName'])`, and **the PK is load-bearing, not decoration** — `RawSqlResultsToEntityTransformer.group()` keys joined rows by the alias's primary columns, so a partial relation select without the id silently fails to hydrate. Use entity-property notation (`'flowVersion.displayName'`); the quoted raw form is not mapped back onto the relation. Pinned by `list-flow-runs.test.ts`, which asserts on the **service** result, not the HTTP response — zod strips the heavy keys at serialization, so an HTTP-only assertion passes against the wide query too and proves nothing. Measured on 1M runs / 200 projects / 10k flow versions with a ~6.4 KB-on-disk TOASTed trigger, fully warm: a 50-row page went **1.59 MB → 15 KB** returned and 4.63ms → 0.56ms median, and the single-row read behind `findFlowRunOrThrow` 0.63ms → 0.30ms. **`EXPLAIN (ANALYZE, BUFFERS)` cannot measure this and will tell you there is nothing to fix** — it discards output rows, so it never detoasts the wide columns; the before/after execution-buffer difference was 9 pages and the only four-figure number in the plan was *planning* buffers. Time the query with its rows actually materialized (`\timing` + `\o /dev/null`). Two further traps in seeding it: give each project its own flow versions, or a page's lookups collapse into a handful of `Memoize` hits and the join looks free, and fill `trigger` with random text rather than repeated characters, or pglz compresses the payload away.
- **Every path that creates a *terminal* run must carry `parentWaitpointId`, not just the paths that execute one.** The barrier releases from a `GROUP BY` over `flow_run.parentWaitpointId`, so a child row that lands terminal without it is invisible: `terminal >= expected` never holds and the parent hangs to the barrier's `timeoutAt`. The `QUOTA_EXCEEDED` admission gate is the trap — `createQuotaExceededRun` writes a finished row straight to `runsMetadataQueue`, bypassing `flowRunService.start`, so it needs **both** `parentWaitpointId` and `dispatchIndex` threaded explicitly at all three call sites (worker RPC `submitPayloads`, sync webhook, manual trigger) — `dispatchIndex` is not in `RUNS_METADATA_UPSERT_KEYS`, so add it there first or the allowlist drops it and the row lands half-attributed. Found while merging the gate into the fan-in branch: neither side was wrong alone. Any future admit-without-executing path inherits the same obligation.
- **A run-scoped poll needs a stop condition, and "the run is over" is not automatic.** The batch panels poll their children every 7s (`useBatchChildren`, `useBatchChildByIndex` in `flow-run-hooks.ts`); shipped without a terminal check they kept polling a run that finished months ago for as long as the panel stayed open, and because every step node inside a batch body subscribes to those same queries, each tick re-rendered the whole batch region too. Worse on the list: it is an infinite query with no `maxPages`, so one tick refetches **every page the user has scrolled through**, not just the newest. The list stops on `batchUtils.areAllBatchesTerminal` — the step wrote its summary **and** that summary's `stillRunning` is 0 — never on its own rows: a barrier can release on timeout with children still running, and the "Failed only" list holds no running row, so a row check stopped it before a late failure could appear. A release with `stillRunning > 0` therefore polls for as long as the panel is open. It also refetches once *after* that point before it stops, because the summary usually arrives seconds after the list's last tick, and that tick predates the last child's failure. Each observer records the query's `dataUpdateCount` when `allBatchesTerminal` turns true and stops once the shared count moves past it. Keep that state out of the `queryFn`: a ref written there only reaches the observer whose options supplied the function, so a second observer of the same key never sees the flag flip and polls forever. A single child (`useBatchChildByIndex`) can trust its own row, but only a row fetched *after* the summary arrived: it stops when that fetch returns a terminal row or no row. A stop on the pre-summary data left a batch blank forever when its child was created between the last poll and the summary (Greptile P1 on #14898). `useFlowRun` (formerly `useBatchChildRun`) had the shape right from the start; copy its `refetchInterval: (query) => … | false` rather than a bare interval.
- **Cancelling a parent DOES cancel its batch children — but only the ones sitting in `PAUSED` or `QUEUED`.** `flowRunService.cancel` walks a `WITH RECURSIVE` descendant query over `parentRunId`, and batch children carry both `parentRunId` and `parentWaitpointId`, so they are in scope; `cancelSingleRun` then removes their queue jobs *and* calls `waitpointService.deleteByFlowRunId`, so a child parked on an approval or a delay is dropped and that approval is never reachable again. The recursion filters on `CANCELLABLE_STATUSES` (`PAUSED`/`QUEUED`) only, so a child already `RUNNING` survives and finishes — and a child that flips `QUEUED`→`RUNNING` between the parent's cancel and the descendant query survives too. The shipped confirm-dialog copy claimed the opposite ("batches keep running to completion") and was rewritten; if you are writing user-facing copy for cancel, the true shape is *queued batches never start, paused batches are dropped, executing batches finish*.
- **Anything the run list says *about a run* must be read from that run's own `flowVersionId`, never from a version fetched some other way.** Both list surfaces span versions while their version source does not: the builder sidebar lists every run of the flow but reads the single `state.flowVersion` (the viewed run's version on `/runs/:id`, the draft on `/flows/:id`), and the runs table classifies by `flowHooks.useFlows`, which defaults to `versionState = DRAFT` server-side and is capped at one un-paginated page of 1000 flows. That is wrong in the direction that matters for retry warnings, because `FROM_FAILED_STEP` re-executes `oldFlowRun.flowVersionId` — remove a step from the draft and an old run retried from the table gets the new version's answer about the old version's graph. `flowsApi.get(flowId, { versionId: run.flowVersionId })` is the correct read; for bulk selections spanning versions (and for select-all, which has no row list at all) the only honest options are a per-version fetch at click time or a flag on the run DTO, so over-warn deliberately rather than classify from the draft. A boolean on the run DTO looks like the tidy fix and is not: computing it needs `flowVersion.trigger` in `queryBuilderForFlowRun`, which is the join-widening this page measures at 1.59 MB → 15 KB per 50-row page. A denormalized `flow_version` column is the only version of that idea worth having.
- **`streamStepProgress: WEBSOCKET` means "stream to the builder", not "this is a test run".** A manually triggered PRODUCTION run streams too, on its first execution and on a waitpoint resume (`streamStepProgressUtils`). The first execution of a retry does not stream, since no builder follows it; but a `FROM_FAILED_STEP` retry keeps the row's `triggeredBy`, so its resume after a waitpoint streams again. Anything test-only keys on `runEnvironment === TESTING`, never on `EngineConstants.isStreamingToBuilder`: the engine's `maxTestWaitMs` cap (`submitWaitpoint` in `piece-executor.ts`) once read the streaming flag and cut a production agent step's 3 h wait to 15 min. The cap also applies to TESTING runs that stream to nobody, such as a draft-webhook run or a batch child of a test run.

### Editions
CE has full run tracking. Cloud may enforce retention windows; bulk-retry admin endpoint is Cloud-only.

### Key files
Entry point: `flowRunService`, defined in `flow-run-service.ts` and wired through `flow-run-module.ts`.

- `packages/server/api/src/app/flows/flow-run/` — controller, service, entity, hooks, side effects, runs queue, AI usage extractor/tracker
- `packages/server/api/src/app/waitpoints/` — the waitpoint module: entity, service, resume routes, the `/confirm` page, its theme hooks, and the `RESUME_DELAY_WAITPOINT` handler
- `packages/core/execution/src/lib/flow-run/` — `FlowRun` type, request dtos, execution types (`StepOutput`, `FlowExecution`), zstd log serializer
- `packages/server/engine/src/lib/helper/logging-utils.ts` — produces the truncated-input placeholder the web run-details tab detects
- `packages/server/api/src/app/ee/license-key-usage-report/` — daily EE job emitting per-platform run counts to PostHog (`TOTAL_RUNS_PER_DAY`, captured and flushed in platform batches)
- `packages/web/src/features/flow-runs/` — `flowRunsApi`, run query/mutation hooks, runs table and its dialogs
- `packages/web/src/app/routes/runs/` — runs list and run detail pages
- `packages/web/src/app/builder/run-details/` — step input/output inspector inside the builder
- `packages/web/src/app/builder/run-list/` — recent runs sidebar in the builder
- `packages/web/src/app/builder/state/` — run state and canvas state, including live-follow control

Paths verified 2026-07-26. An earlier version pointed at `packages/core/shared/src/lib/automation/flow-run/` (moved to `packages/core/execution/src/lib/flow-run/`) and `packages/server/api/src/app/ee/flow-run-tracking/` (renamed to `packages/server/api/src/app/ee/billing-usage-report/`, then to `packages/server/api/src/app/ee/license-key-usage-report/`).
