Run Lifecycle & Execution

Run Lifecycle & Execution

You have declared a type and written a handler. This page is what happens to it after that: who calls execute_chunk(), how often, with what budget, what is already done for you, and what a run looks like when it is over.

The eight steps, end to end

One minute of wall clock produces at most one tick per tenant, and that tick does two things: it turns due automations into run rows (Phase A) and it pushes open run rows forward (Phase B). The two phases are deliberately separate. Claiming is cheap, bounded and must always finish; executing is expensive and is allowed to stop halfway. Splitting them means a slow automation can never prevent a fast one from becoming due.

Read the stepper below — each step names its owner, the table it writes, and, more usefully, how it fails. Most of the design lives in that last column: a runtime is defined by what it does when things go wrong, not by the happy path.

Who owns what

This split is the whole reason a new automation type is a few hundred lines instead of a few thousand. Read the left column as "already solved, do not re-solve", and the right column as "the only part that is actually about your domain".

Idempotency — at three levels

A minute-resolution scheduler that runs on every tenant every minute will be invoked twice for the same moment eventually: a slow tick, a restart, a manual trigger, a clock adjustment. The design answer is not to prevent that; it is to make the second attempt a no-op at every level.

One occurrence, one run row

automation_runs.idempotency_key carries a UNIQUE index (uniq_automation_runs_idem) and create_run() writes through INSERT IGNORE, then re-selects. Concurrency is handled by the database, not by a SELECT-then-INSERT that has a race in the middle. The key itself is deliberately narrow:

// modules/Automations/helpers/automations_helper.php

/**
 * The key that makes one due occurrence exactly one Run, however many times the
 * dispatcher is invoked for that minute.
 *
 * The configuration revision is DELIBERATELY EXCLUDED: one scheduled occurrence
 * must stay one occurrence even when the automation is edited after it became due.
 * Including it would let an edit mid-tick manufacture a second run of the same
 * moment (plan §9).
 */
function automations_run_idempotency_key($automation_uuid, $scheduled_for_utc, $trigger_kind = 'schedule') {
    return hash('sha256', $automation_uuid . '|' . $scheduled_for_utc . '|' . $trigger_kind);
}

That exclusion is the interesting decision, and it is worth stating plainly because it looks like an oversight: the revision is left out on purpose. If the key included it, an operator saving an edit while a tick was in flight would mint a second key for the same minute, and the same occurrence would run twice against two configurations. One scheduled moment is one occurrence, whatever anyone does to the rule afterwards.

The run row is written before the pointer moves

Phase A inserts the run and then advances next_run_at_utc. That order is the only crash-safe one. Crash between the two and the same occurrence is re-claimed next tick, where the unique key makes the retry harmless. Advance first and a crash silently loses the occurrence — the worse failure, because nothing anywhere records that it should have happened.

The pointer is advanced whether or not this process created the run. If the row already existed, the occurrence is accounted for and the pointer must still move, or the automation would be re-claimed on every tick forever.

One record, one decision per run

automation_run_items carries UNIQUE(run_id, subject_type, subject_id) and record_item() also uses INSERT IGNORE. A resumed chunk that re-observes a record it already decided writes nothing the second time. Your handler therefore does not need to remember what it has seen within a run — but it does need a cursor that does not re-offer records unboundedly, or you will simply spend the budget re-deciding.

Budgets, caps and the real numbers

Every one of these is a real constant in the source, not a guideline. They exist so that a single automation cannot monopolise a tick, a bad clock cannot flood the queue, and a mistake that survived Observe costs a reviewable handful of records rather than a thousand.

Unfinished is not failed

This is the single most important thing to internalise about the loop, because the instinct to "clean up" a run that is still running after a tick is exactly wrong. When the deadline passes or the chunk cap is reached, the executor returns status: 'running' and simply stops. The run keeps its cursor, its counts and its lease, and the next tick continues it.

// modules/Automations/libraries/Automation_executor.php — run_chunks()

while ($chunks < self::MAX_CHUNKS_PER_TICK) {
    if (time() >= $deadline) {
        break;   // out of budget: stay running, resume next tick
    }

    $remaining = $max_items - ($already + $processed_now);
    if ($remaining <= 0) { $has_more = false; break; }

    $limit = min($batch_size, $remaining);
    $chunk = $handler->execute_chunk($context, $config, $cursor, $limit);

    // ... record items, merge reason counts and notices ...

    $cursor = get_array_value($chunk, 'next_cursor');
    $this->CI->Automation_runs_model->save_progress($run_id, $cursor, $counts);

    $has_more = (bool) get_array_value($chunk, 'has_more');
    if (!$has_more) { break; }
}

if ($has_more) {
    // Unfinished is not failed. Leave it running with its cursor.
    return array('status' => 'running', 'processed' => $processed_now, 'chunks' => $chunks);
}

Note the ordering inside the loop: the cursor is saved after the items are recorded. A crash between the two re-offers the same records, and the unique item index absorbs them. Saving the cursor first would skip records that were never decided.

Known gap, named: Automation_runs_model::get_stale_running() exists — it finds runs left running by a worker whose lease has expired — but nothing calls it yet. Today it does not matter: the Phase B query selects status IN ('queued','running') without consulting the lease, so an abandoned run is simply advanced again on the next tick, and the cron runtime's per-(job, tenant) lock already prevents two executors from touching one tenant concurrently. The lease is therefore defence-in-depth for a future out-of-band run path, not load-bearing today. Do not delete it, and do not assume it is enforced.

Cursors — make them absolute and JSON-serializable

Your next_cursor round-trips through the database as automation_runs.cursor_json between ticks, and is handed back to you verbatim. Two constraints follow, and both are hard.

The shipped attendance handler is the reference shape — it reads $cursor['after_id'], orders by id ascending, and returns array('after_id' => $last_id) with has_more = count($candidates) >= $limit. That last expression is worth copying too: it costs one extra empty chunk at the end of a run, and in exchange it never claims "done" while records remain.

Preview and live must be one evaluator

The interface says it outright: "A preview that can disagree with the live run it predicts is worse than no preview at all." A wrong preview does not merely fail to help — it manufactures confidence. So preview() and execute_chunk() are required to select candidates and apply rules through the same code path, differing only in whether they mutate.

The shipped handler implements this as one private method with an $apply flag: preview() passes false, and execute_chunk() passes $context->run_mode === 'live'. That single boolean is the entire difference, which is also what makes Observe mode trustworthy — Observe is not a simulation of the live path, it is the live path with the write suppressed.

The promotion gate, and why it is checked twice

Going Live is not a mode toggle; it is a decision with four preconditions, all enforced in Automations::_live_refusal() before the automation's mode can change.

Those four are a publish-time gate. Authority, however, is also revalidated at run time, and only for live runs:

Run-mode reconciliation — the current mode governs

A run is stamped with the automation's mode when it is claimed, alongside the config snapshot. But at execution time the automation's current mode is what governs, and the run row is corrected to match.

That direction looks inconsistent with the frozen config snapshot, so it is worth being explicit about why it is the opposite way round: it only ever moves toward safety. An operator who demotes a live automation to Observe expects records to stop changing now, including for runs already queued. A demotion that took effect one tick later would be useless in precisely the situation it exists for. Promotion mid-queue is honoured equally, but that path is gated by _live_refusal() before it can happen at all.

The correction is written back to the run row rather than merely applied in memory, so that history reports what the run actually did — not what it was queued to do. A run listed as dry_run genuinely wrote nothing.

How a run ends

A run closes when a chunk reports has_more = false, or when the accumulated item count reaches max_items_per_run. Then:

Closing a run does four things in order: finish_run() writes the status, finished_at, duration_ms and the rolled-up reason counts while clearing the lease; set_last_run() updates the automation's summary columns; the audit row is written; and the outcome is returned to the dispatcher, which logs one line per run into the cron output.

The counts split by mode, and the distinction is deliberate: in Observe a positive outcome is a prediction, so it is counted as would_count — never as approved_count, which must only ever mean a record was actually changed. If you are reading a dashboard and see approved_count, something really happened.

Runtime checklist for a new handler