Schema and migration sources

The form of a front-end comes from the options of a NixOS module: core makes a JSON Schema of that module, and a table of option renames to migrate stored configurations to the current shape. A per-deployment module source selects the module. This lets an operator use their own module (a fork) and not only the default, which matters for moving between hosting providers (#100): the configuration of an operator must not be hostage to the module of one provider.

A fork is subject to the trusted-source allowlist of the hosting provider (see Why the */get_for variants are authenticated and allowlisted): the deploy runs the Nix of the fork, so a provider can limit which sources it fetches and runs.

This section covers the schema half of the pinned-source boundary: what a submit validates and migrates against the pin. The deploy half (which inputs come from the pin and which from the current revision of the provider) is in The deploying-vs-deployed version boundary (#972). The normative rules are in 002-schema-and-sources.

Endpoints

The schema and the renames each have two procedures:

The default procedures are public. They return the files that the build of api made for the default module. They are safe without authentication because they run no eval: their only parameters select a part of a document that is already built.
The get_for procedures are authenticated and allowlisted. They take a module source and, on a cache miss, fetch the source tree and build the schema or rename output of the core in it, with one nix-build that imports the fetched core/default.nix. Only the coordinates of the serving provider that shape the schema go into that build, because a collapse of the form is a property of the serving deployment and not of the pin; the deploy credentials do not. This evaluates Nix from the fetched source, so the allowlist gates it, on the same basis as the deploy. The result stays in an in-memory cache for each source. A source without the output, or whose build fails, is an error.

Serving one application's schema

The schema procedures can return the schema of one application, with only the definitions that it references, so that a front-end validates one value without the full schema. core gives all application schemas as one output and finds the applications in the form, so api does not know the structure of the form, and a source costs one build, not one for each application. A parity check makes sure that the baked table equals the output of core. A source whose core is older than this output gets a clear error, not a build log.

At present a section is an application. The rename procedures take a section and give only the renames inside its value, with relative paths, so the table applies to that value alone; a rename outside the section, such as the move of the method branch, is dropped (FR-007 of 002).

Migration semantics

A rename maps a from path to a to path; a segment is a literal key or a wildcard for each key at a map or list level. api moves the stored value before validation. Migration is stable in order (a chain resolves when it is listed in order) and idempotent (FR-008 to FR-010 of 002).

The entries have two sources, in one list:

Option renames of the module system, recovered from the alias markers that the rename helpers of NixOS leave. Nothing is declared: a move of an option makes the entry.
Renames of values inside an option that carries a JSON Schema as it is, declared as data beside the component. There, a whole subtree is one option, so a key that upstream moves leaves no alias marker.

The harvested kind is a snapshot: it holds the markers that the pinned nixpkgs still has, and upstream drops an alias some releases after the move. The declared kind is a log that we only add to. Thus an old configuration migrates across the schema side in one pass, but across the NixOS side only as far as the current pin remembers.

The build checks the declared kind against the narrowed schema that the operator gets, so that an entry that cannot migrate anything is a build failure, not a silent no-op. The check looks at the end of a chain, not at each step: the last to must be a key that the form offers. A from must be gone from the schema only where the chain does not bring the value back to it, because upstream can move a key away and later restore the name; a stricter rule would make us falsify the record of upstream. The check refuses wildcards, because the narrowed schema seals each object, and the seal is what makes the selection binding. The entries resolve before the lift of the deployment method branch.

A key that upstream removes is not a rename: no entry can carry its value. The selection check fails the build, and a stored value with that key then fails validation. The refusal is the whole response: to drop the key on read would discard operator data, so only the operator decides, with a re-save of the form. The message lists each stored path that the source has no place for, and the remedy, because the sealed schema alone names only one unexpected property.

The panel finds these keys before the operator edits: after migration on read, it prunes them from the edited value and lists them, and the stored row stays until a save. It keeps the old path of a rename that could not apply, does not prune when it could not load the renames, and in a union lists only a path that no branch offers.

An entry that cannot apply is skipped, not guessed, and the validation error names it with its reason: the wildcards of the two paths do not align, or the to already holds a value. When the renames cannot load, migration passes the value through, and the schema validation stays the authority, so a temporary build failure does not turn a submit into a server error.

Default-source baseline freeze

This is the self-pin of SELF_MODULE_SOURCE.

A deployment without a module source once stored no source, and a later read resolved to the current built-in schema. After a rebuild of api with a changed schema, the "old" side of the schema-change view of the panel (#213) moved too, and the diff could not see the drift that it must show.

Thus api resolves a submit without a source to a pin of its own revision, and stores that concrete source (FR-011 of 002). Each submit resolves its own absent source, so a deployment without a source follows the revision of the provider, as the form does.

A revision is a property of the deploy, not of the tree: to record it in a tracked file is self-referential, because the write changes the revision. Thus the pin comes in at run time, in three equal ways: from the locked revision of the flake, as an argument to the plain Nix entry point, or from the environment outside Nix. Without a pin (for example, a dirty tree, which is not reproducible as a fetch anyway), the source floats; a malformed pin counts as no pin.

The schema fetch of the self-pin is best effort: on a failure, a submit without a source validates against the built-in schema, while the row stores the pin. An explicit source has no fallback, because validation against another schema would silently accept a configuration that the named source refuses.

Why the `get_for` variants are authenticated and allowlisted

A module source from a caller, fed into a fetch and a build, is two risks. Both the schema path and the deploy path evaluate Nix from the fetched source.

Server-side request forgery and code execution. The fetch gets any source that the caller names, from the network position of api, and then evaluates its Nix. The trusted-source allowlist of the hosting provider closes this: api applies it on the deploy and on the get_for procedures, before any fetch (FR-004 of 002). The allowlist is a lever of the hosting provider, not a gate of the Fediversity project on forks. It holds only the project repository by default, and an explicit empty list allows all, which gives back the freedom to use a fork to move between providers. An absent source is always allowed. The allowlist adds to authentication: authentication still limits who can start a build.
Compute and disk exhaustion. Each new source is a fetch plus a Nix build. Authentication limits build capacity to sessions, and Cost and latency bounds the rest.

The public procedures are safe for the opposite reason: no caller input, no fetch, no build.

Form hints are in the schema of each source

The schema document also tells the form when to show an option: the if, then and else keywords of an application. core makes these keywords in the evaluation that makes the schema, from the same source. A front-end migrates a stored configuration to the module source of the deployment, and shows the form with the schema of that source. Thus the hints always agree with the schema version that the configuration migrates to, and they need no migration and no separate store. core makes the conditions after it removes the old paths of the renames, so a condition names only the current paths.

core does not make a separate JSON Forms UI schema, because no form needs one at present. A UI schema names options by path, so a separate document would need its own migration across renames. Put a form hint that a future form needs into the schema document. The hint then uses the per-source build and cache of the schema, and an older source that does not have the hint shows the default form.

Transport: schema and rename are computed on demand

api builds the schema and renames of a fork itself, from the fetched source, with the same outputs that the built-in default uses. A fork commits no asset and runs no CI step, so there is no drift to police: the schema always comes from the module that the operator named. The fetch is the same as on the deploy path, so it does not depend on the forge.

An earlier design committed the schema to the source tree, to avoid running Nix that a caller names. That reason stopped once the allowlist gated get_for on the same basis as the deploy, and the committed asset then gave only a drift risk.

A fetch pins the revision that the source names; see Tracking rolling revisions of a fork.

Cost and latency

A cache hit is a file read. A cache miss runs a fetch and a Nix build, which can take minutes cold. Mitigations, in order of priority:

Timeout and concurrency cap (implemented; FR-003 of 002). When all slots are in use, a miss is refused at once as unavailable, so a flood cannot pile up threads. One deadline covers the fetch and the build, and a build that overruns it is killed and reported as a timeout. This closes the unbounded hang that existed with or without authentication.
Async miss (implemented, with HTTP polling). A miss on a get_for procedure answers that the result is computing, and the form polls. A submit must have the schema, so it blocks; it resolves the schema and the renames and then evaluates the nodes (Names eval at submit), so it can hold three deadlines. The ingress timeout must allow that, or its own answer replaces the answer of api (see CORS); the deployment sets both from one value. Centrifugo was rejected here: the build gives one result with no progress, so polling avoids the channel and its authorization for no loss.
A dedicated priority lane for schema builds, separate from the deployment queue. Deferred until measurement shows that schema builds compete with deploy builds.

"No build ever" is not possible without a committed asset, so a bounded, timed build that does not block the form is kept by design.