Incremental & Scheduled Modules

Let a command run unattended and pick up only what is new since the last successful run. You declare the semantics; the station owns the position.

Requires station v0.62 or later. On an older station the contract is ignored — the command still runs, but every tick re-fetches everything.

The rule

The module declares that it is incremental. The station stores the watermark. You write no state-handling code, and you must not keep your own cursor.

A module already receives a workspace and a save helper, so it could persist its own position. If it did, nothing would receipt the movement. When the mark drifts you find out because a customer was never contacted, with no record of when the gap opened — and a module update silently resets it. The station keeps the position inside the signed run receipt, so a gap is provable rather than merely suspected.

Declare it

Two blocks on the command in module.json. incremental requires schedulable: a command that is never scheduled has no watermark to advance, and the manifest is rejected at install rather than quietly ignored.

json

{
  "id": "acme.list_new_orders",
  "input_schema": {
    "since":        { "type": "string",  "required": false },
    "exclude_ids":  { "type": "array",   "required": false },
    "limit":        { "type": "integer", "required": false }
  },
  "output_schema": { "count": "number", "orders": "array" },

  "schedulable": {
    "min_interval_minutes": 5,
    "max_runtime_minutes": 10,
    "concurrency": "skip"
  },

  "incremental": {
    "since_param":     "since",         // station injects the cutoff here
    "seen_param":      "exclude_ids",   // station injects what it already delivered
    "items_field":     "orders",        // where the station reads results
    "cursor_field":    "id",            // stable per-item identity
    "watermark_from":  "created_at",    // ISO-8601 UTC on each item
    "watermark_type":  "timestamp",
    "initial":         "today",         // now | today | epoch
    "lookback_seconds": 300,
    "seen_window_seconds": 604800
  }
}

Normalise your own timestamps

watermark_from must name a field carrying ISO-8601 UTC. If your provider renders something else — a 12-hour clock, a local time with no offset — convert it in the module and emit both. Teaching the station to parse a provider’s display format would put provider trivia in the governance layer and break the moment that provider changes it.

A timestamp with no timezone is a correctness hazard, not a formatting one. Read as UTC when the account renders Eastern, every record looks hours older than it is — and records older than the watermark are dropped. Take an explicit timezone input, and when it cannot be resolved, widen your comparison and say so in the output rather than silently guessing.

What the station does

Around every scheduled run, in this order:

text

before   inject since_param  = watermark minus lookback_seconds
         inject seen_param   = cursors already delivered

run      your command executes normally

after    read items_field from your output
         drop anything whose cursor_field is already known
         advance the watermark to the newest watermark_from seen
         write all of it into the signed run receipt

The watermark advances only after a run completes. A run that fails, crashes, or rolls back holds its position, so the next run re-fetches — duplicates the seen-set absorbs, rather than records nobody ever sees.

Why seen_param matters

since_param is deliberately set earlier than the stored watermark by lookback_seconds, so records created while the previous run was in flight are not missed. That overlap is intentional — but without seen_param every record in it is delivered again on every run. Filtering after the fact is too late: for a spreadsheet that is a duplicate row; for an email or a charge it is worse.

seen_param is optional, and a command without it will re-deliver its overlap window on every tick. If you skip it, set lookback_seconds to 0 and accept that a record created mid-run can be missed.

Treat limit as a cap, not a selector

Once a watermark is supplied, a limit that still selects the newest N will silently truncate a busy morning — and because the watermark advances past what you returned, the remainder is never seen again. Return everything after the cutoff, use limit only as a safety ceiling, and when you hit it, say so:

python

out = {"count": len(items), "orders": items}
if hit_the_cap and more_remain:
    out["truncated"] = True    # the station will REFUSE to advance the watermark

truncated: true makes the station hold the position rather than step over rows you never returned, and surfaces the reason in the receipt.

First run

initial decides where a never-run schedule starts. now yields one record, epoch floods the workflow with years of history at 3am, and todaysits between them — computed from the operator’s local midnight, which is what “today’s records” actually means to them. For a backlog or queue, today is usually right.

Manual runs

A hand-clicked Run reads the current position but never writes one, so testing a workflow at noon cannot move the cursor a 3am tick depends on. Expect a manual run to re-deliver records a scheduled run will deliver again; the receipt says so explicitly rather than leaving it to be inferred.

Failure modes worth designing for

The station refuses to advance — and records why — when your output is unreadable, when items_field is absent, when an item lacks cursor_field, or when you report truncated. Refusing to move is the safe failure. An empty result is not a failure: return {"count": 0, "orders": []} cleanly rather than raising, or a quiet day will look like an outage.

What the operator sees

Every scheduled run is badged Scheduled in the Runs tab and filterable there. Opening one shows the position it consumed and where it left the cursor — fetched-since, returned, new, suppressed, and the watermark before and after — all inside the receipt seal, so it cannot be edited afterwards to claim coverage that never happened.