Build Queue

api does not run a deployment operation at once. It enqueues it, and the available builders process it. Thus the request and the execution are separate.

Queue semantics

• Replace by default: a submit in replace mode, while a build of the same deployment runs, is refused as full, and the running build continues (FR-009 of 001). A program can run a TF graph that is not atomic, so to stop it part-way can leave the resources in a partial state. A queued build does not block the submit and is not dropped: the new build queues after it.
• Explicit enqueue: in enqueue mode, api does not do this check and always queues a new build, also while a build of the same deployment runs (for example, for staged rollouts).
• Builder assignment: the queue gives builds to the available builders. When the backend supports it, the choice of a builder can follow cache locality (a builder that already has the relevant store paths).

Backend selection criteria

The backend should satisfy all of these:

• Cancel of a queued job.
• Per-line progress events that api can bridge to pub/sub without polling.
• Historical logs with offset, to catch up after a reconnect, for running and completed jobs.
• Queue position and builder identity, given by the backend.
• Replace and enqueue modes, with a way to skip replace for staged rollouts.
• Backpressure: a signal when the capacity is full.
• OpenTelemetry: trace context that can travel through progress events.
• Reconnect catch-up by job identifier after any delay, with no state for each client on the server.
• Notifications while the operator is offline: terminal events after a closed tab or a gap of days.
• License: fully open source, with no open-core gates and no source-available licenses.
• Footprint: moderate or less, and no Kubernetes control plane for the job backend. This criterion is about the queue only: the k8s deployment method can deploy nodes to a Kubernetes cluster of the hosting provider.
• A language-neutral REST client API.
• Layer separation: on the deployment plane, separate from development CI.

Queue backends

The queue backend is a deployment choice. The API hides it: clients use only the procedures of the published contract.

Comparison

The first four criteria are hard gates: the license, the layer, a job backend that does not need Kubernetes, and a REST client API. A backend that fails one is disqualified. SSO is secondary, because the deployment gates the queue UI at the edge with forward authentication, so a backend that cannot log a user in itself is not disqualified. Only Windmill is verified against its source code (see below); the other entries come from upstream documentation.

Windmill

Windmill is selected because it covers cancel, log streaming, log offset, queue position and OpenTelemetry natively, with a light, fully open (AGPL-3.0) footprint, on the deployment plane and with a REST client API.

SSO is not available in any configuration: in the open-source build the code path is absent, not behind a license tier. Thus the deployment gates the whole UI name at the edge with forward authentication, and the gate sends the API token of Windmill upstream, so that the portal is the only login of an operator. The deploy submitter of api uses a separate name that serves only the API, on the private CA, behind bearer authentication. Identity for each user inside Windmill is accepted as not delivered: it is an internal deploy runner, and humans open its UI only to read job logs.

api maps a submit to a job of the deploy script, a get to the job and job list (states, queue position, worker), a cancel of a job that no worker started to the native cancel, and the progress of a job to Centrifugo, line by line.

• Replace semantics: the concurrency key of Windmill is a property of the script, so a submitter cannot set one for each job. Thus api drops no queued build. To supersede one would need a key template on the deploy script, or a cancel-then-submit.
• Reconnect and offline notifications: Windmill keeps job state and logs by job identifier after completion, and can run webhook flows on terminal states. Its live stream does not replay, so on reconnect api reads the earlier log, then subscribes.
• Log offset: only the endpoint that gives the whole log reads both places where Windmill keeps a log (the recent output and the flushed output). Thus api reads the whole log, takes the state from the job, and counts the offset as a position in the text. A resumed read can start in the middle of a line, so a client keeps the last partial line.
• Job state: Windmill has only a queued and a completed type, so api reads running and canceled from flags.
• OpenTelemetry: Windmill emits traces, and a progress event can carry the trace context.