Deployments

Deployment identifiers

A deployment identifier is opaque to api: api stores and returns the token that a front-end gives, and does not parse a structure out of it. The identifier names the resources of the deployment, thus it must be a DNS label of at most 40 characters (FR-002 of 001). The panel uses a UUID, but that is a convention of the panel, not a contract of api.

A list takes the identifiers to list, so that a front-end can scope a list to the deployments of one user and api does not know its tenancy model.

This opacity is load-bearing for the reconcile design below. The one place where api could want to open the identifier (to find the row of a front-end after a destroy) uses a callback that the front-end gives instead. Thus api does not need the structure of the identifier or the tenancy model of a front-end.

Two-store model and deletion

api and a front-end keep complementary facts about a deployment. Each is the authority for its own:

api owns the runtime fact: the build history, the derived state (building, active, destroying, idle, failed) and the live infrastructure that the builds made.
The front-end owns the intent fact: the draft or desired configuration of the operator, its human name, and its owner. This fact exists before a build, stays after a failed build, and is what the editor of the front-end changes. It is a concern of the web application, not of api (see Unstaged configurations).

The configuration that api stores is a snapshot of the intent at the last submit, not a second authority over intent. Most operations touch only one fact. Deletion is the exception: it removes the runtime fact and the intent fact. The two stores have no shared transaction, thus a delete is a sequence that the browser orders, and each step waits for the confirmation of the step before it.

Browser-orchestrated delete

The browser is the only participant with a live, authenticated relation to the two authorities. Thus it orders the two writes:

Phase 1, runtime teardown (api). The browser sends the delete to api with its bearer token. api enqueues a destroy build and returns at once; it does not tear down in the request. When api does not know the deployment, the browser counts phase 1 as done. Thus the operation is idempotent under retries and reconciliation.
Wait. The browser follows the channel of the deployment until a terminal state.
Phase 2, intent removal (front-end). On success (idle), the browser calls the delete of the front-end, with its session, to remove the intent record. On failure, the intent record stays, so that the operator can try again: the desired configuration is still valid.

This order is deliberate. It prefers the recoverable failure (a destroyed deployment whose intent record stays) to the failure that cannot be recovered (live infrastructure that no front-end knows). A phantom intent record is visible and can be deleted again. Orphaned infrastructure is an invisible leak.

The normative rules are FR-005 of 001 and FR-006 of 004.

Durability of the intent record

A browser tab is a temporary orchestrator. When it closes between the phases, a "phantom" intent record stays after its runtime is gone. Two mechanisms converge the stores without a durable browser:

Reconcile push. When a destroy reaches idle, api calls a reconcile endpoint of the front-end once, without a retry. api must not derive where that call goes: the identifier is opaque, and one configured URL would contain the tenancy of one front-end. Thus the browser gives a callback URL and a callback token on the delete, and different front-ends can share one api. A callback URL from a caller is a server-side request forgery primitive, so two limits apply:
- Origin allowlist. The callback origin must be in the CORS allowlist: the front-ends that can call in are the ones that api calls back. api checks it when it accepts the delete, and again before the call, so that an old row cannot reach an origin that is no longer trusted. A callback that is not allowed gives no push.
- Scoped token, no shared secret. The front-end makes the token for one deployment and checks that its scope is the row that it deletes. api only stores and sends it back. Thus identity stays in one authority, and a leaked token can delete only one row.
The push is best effort. The session-authenticated confirm of phase 2 and the token-authenticated reconcile are sibling endpoints, so each has one credential: the session path keeps its CSRF protection, and a header token is not ambient.
Idempotent re-delete (escape hatch). Phase 1 counts an unknown deployment as done, so a later delete of a phantom goes directly to phase 2.

A periodic sweep from the front-end was a possible third backstop, and is deliberately omitted: at this scale it adds a scan of all deployments for little more coverage.

What a configuration serves

A stored configuration is opaque to a front-end, and where a value sits in it follows the module source (below the branch of the deployment method, or at the root when the build pins the method). Thus api derives the facts that a front-end shows, and serves them as typed fields that do not change with that shape: the domain, and the instances of the last submit and of the last successful build, with their domains, URLs and sections (FR-004 of 001). The catalog components with a web UI of their own are instances too, and are always on, because a deployment runs its whole group.

For a configuration that api does not store, such as a draft, two public procedures read only the request body: the application paths (with the changed applications, each side read at its own path, so that a change of the method branch does not mark all applications as changed), and the sections that a form edits one at a time. A front-end does not parse a section identifier; at present it is the name of an application.

Names eval at submit

The schema does not hold all the rules of a deployment: a module of the deployment can refuse a configuration that the schema accepts. Thus a submit also evaluates the configuration as the deploy does, before it stores it (FR-011 of 001). The eval gives the nodes that the deploy stands up, the DNS names that each node publishes, and for a submit the store path of the configuration.nix of each node (SBOM of the deployed nodes). It does not evaluate the systems of the nodes. Thus it is fast.

The eval shares the timeout and the concurrency cap of the schema builds (Cost and latency). When all slots are in use, the submit does not wait and does not fail: it continues without the eval, and the deploy does the eval itself.
A preview does the same checks and the same eval, stores nothing and queues no build. It gives the nodes with their names, and the applications that the configuration disables. When all slots are in use, the preview reports that the service is unavailable, because a preview without the eval has no result.

The eval does not check the plans of the nodes, and the deploy does not use its result: the deploy evaluates the configuration again.

A migration import also evaluates the system of each node before it stores anything (Dry eval of a migration import).

SBOM of the deployed nodes

api gives the CycloneDX SBOM (system.build.sbom, core/setups/nixos/sbom.nix) of each node of a deployment (FR-017 of 001):

GET /sbom-list/{deployment_id} gives the state of the SBOM of each node: ready, or unavailable with a reason, with the build whose deploy gave it and the time.
GET /sbom/{deployment_id}?node= gives the SBOM of one node as application/vnd.cyclonedx+json. It gives a 404 with the reason when the SBOM is unavailable.

Both have the same authorization as the other reads of a deployment.

The SBOM describes the software that the deploy put on the node, not the software of the configuration that is stored now. Thus the deploy makes it (core/effects/common/deploy.sh): one evaluation gives the system of the node and its SBOM, and the deploy builds the SBOM in the background while it deploys the system. The eval cache keeps both derivations, thus a cached evaluation also gives the SBOM. Before the deploy exits, it waits for the SBOM build, for at most DEPLOY_SBOM_TIMEOUT seconds (600). The first SBOM of a set of pins takes about 15 seconds to build, a later one about 5 seconds. An SBOM that does not evaluate or build becomes the reason of its node, and the deploy continues.

The deploy writes each SBOM to SBOM_OUT_DIR, and the list of the nodes of the deployment to nodes.json. _runner.read_sbom_dir reads that directory: the Windmill job gives the value as its result, and the subprocess path reads it directly. When the build succeeds, api stores the SBOMs before it marks the build as succeeded, thus a client that reacts to the terminal state finds them.

api keeps the SBOM in its state store, not as a path in the Nix store: api cannot write a Nix store of its own, and on a shared store the hypervisor removes indirect GC roots. The store keeps each component one time, by the SHA-256 of its JSON text, and the remainder of each document with its node. Many nodes and deployments run the same packages, thus most components are shared. GET /sbom/{deployment_id}?node= puts the document together again, with its components in their order.

A node that the deploy does not change keeps its SBOM, because its software did not change. A node that the deployment no longer has loses its SBOM, and a destroy removes all of them. A component that no SBOM uses then goes. A node that no deploy changed since api keeps SBOMs is unavailable until a deploy changes it. A failed build changes no SBOM.

Secrets of the operator

Some settings of an application are secrets that the operator gives, for example the API key of a translation service. The schema marks each of them writeOnly: true and format: password, and gives the application a readOnly option secretsSet and a secretsClear option beside its settings (core/lib/operator-secrets.nix). The vars instance is <application>-operator-secrets, where <application> is the key of the application in applications.

A submit removes each secret value from the configuration before it stores it or gives it to the deploy (fediversity_api.operator_secrets):

A value that is not empty goes to the OpenBao KV entry of the deployment, <mount>/data/<deployments root>/<id>/secret/<instance>/<field>. The configuration that api/ stores and gives to the deploy holds only the name of the field, in secretsSet.
An empty or null value keeps the entry that OpenBao has. api/ reads which entries exist, so secretsSet also holds the fields that an earlier submit set.
When the rename table of the source gives a field a new name, and OpenBao has no entry of the new name, api/ lists the entries of the instance and copies the value from the entry of the old name. The entry of the old name stays: the node reads it until its next deploy. A value in the submit replaces the copy.
A field in secretsClear that the submit gives no value is cleared: api/ leaves it out of secretsSet and does not read its entry or the entry of an old name. api/ deletes nothing. The stored configuration keeps the list, and the deploy deletes the entry of each field in it, with all versions, after the nodes run the configuration that does not name the secret (core/effects/tf/operator-provision/kv-clear.tf).
A value that is not empty wins over a clear: api/ takes the field out of secretsClear. api/ also removes from the list each name that is not a secret field. A client that keeps its own copy of the configuration must take the field out of its list too, or its next submit clears the new value.
api/ writes to OpenBao only after all checks pass. A submit that a check refuses writes nothing.
When OpenBao is not configured (OPENBAO_ADDR is not set), a submit with a secret value gets 400 invalid_configuration. When OpenBao refuses a read or a write, the submit gets 503 service_unavailable and stores nothing.

A preview does not read or write OpenBao: its secretsSet holds only the fields that the body sets.

A configuration that api/ stored before an application had a secret can still hold the value in its settings. The next submit of that deployment moves it to OpenBao. api/ does not change the stored builds of earlier submits, so their snapshots keep the value. The Nix store of a node keeps the value of an earlier deploy until a garbage collection removes that generation. A clear does not delete the value that a rename left in the entry of an old name.