Migrations

Export of a migration

An export is not safe to repeat, because it changes the source:

Before it stops a service, it does the checks of a suspend: while a build of the deployment runs, it is refused as queue_full, and when the deployed revision cannot suspend, as invalid_configuration.
It stops the app services on each node of the deployment, and then makes the snapshot of each app. Thus no write after the snapshot is lost. When a snapshot fails, it starts the services again.
api sets the export record of the deployment, and then queues a suspend. When that submit fails, api sets the record back, starts the services again and returns the error.

The source stays suspended until a target restores the snapshots, and the services are down from the export until the restore. To roll back the move, resume the source: the boot of the instances starts the services again. A second export of a suspended deployment is refused as conflict, thus the string shows once. To export again, resume the source first.

The export action asks the operator to confirm that the export stops the apps and suspends the deployment. While a build of the deployment runs, the action is not enabled.

A configuration with no enabled app has no snapshot, thus it has no write to lose. Its export sets the export record, but does not do the checks of a suspend and does not suspend the deployment.

The deployment row keeps the export record. A resume keeps it, and a destroy that succeeds clears it. Each build of the deployment gets it next to the suspend value. When the DNS service of the hosting provider holds the zones of the operators, the graph of tf-incus-operator marks the zone of a suspended deployment for the follow job of the primary. The follow job then copies the records of the target into the zone, until the public DNS delegates the zone to the target.

Follow of a deSEC zone after an export

When the curation holds the zone of the registered domain of the source at deSEC, the export that suspends the source also marks that zone. The export of a configuration with no enabled app does not suspend the source, thus it does not mark the zone. When the submit of the suspend fails, api removes the mark. The target holds the zone on its own nameservers after the move. A resolver that has the old delegation in its cache asks deSEC until the NS records of the parent expire, thus the zone at deSEC must give the records of the target for that time.

api follows each marked zone: a background task does a step each 300 seconds. A step:

Asks the authoritative servers of the parent zone, not the system resolver, for the NS records of the zone. When no server answers, the step does nothing.
While the parent delegates to the nameservers of the zone at deSEC, or to the nameservers of the registration in the curation, does nothing.
When the parent delegates to other nameservers, records the time of the move and the cache time: the longest TTL of the NS records of the parent and of the zone at deSEC, at most 604800 seconds.
Until the cache time and 300 seconds pass after the move, reads the A, AAAA, CNAME, MX, TXT, CAA and SRV records of the zone at the new nameservers. It writes the differences to the zone at deSEC in one bulk change, with the token of the deployment. A TTL below the minimum TTL of the domain becomes that minimum. The step does not change the NS, SOA and DNSSEC records. The step reads the zone in one of three ways:
- It asks each new nameserver for a zone transfer (AXFR), first signed with the transfer key of the export, then without a signature. The target of a migration lets the holder of that key, and the addresses of api, transfer the zone, see the transfer sources of a migration below.
- When no new nameserver gives the transfer, it walks the NSEC chain of the zone (RFC 4034 section 4) with authoritative queries: each NSEC record gives the next name of the zone and the types at its owner. The walk follows at most 10000 names.
- When no new nameserver gives the walk, for example for a zone with NSEC3 or a zone that is not signed, the step asks the new nameservers for the apex, for each name that the zone at deSEC holds, and for each label of services.fediversity-api.zoneFollow.probeLabels. These labels are by default the labels under which Fediversity writes records. When a server gives no authoritative answer, the step keeps the records. It then does not see another name that only the new nameservers hold.
The transfer and the walk also give a name that only the new nameservers hold, and the step deletes the records of a name or a type that they do not give. A delegation inside the zone makes the transfer and the walk fail, thus the step then keeps the records at deSEC of a name at or below the delegation.
Then ends the follow.

A destroy of the source that succeeds during a follow keeps the deSEC tokens of the deployment. A step of a destroyed source makes the token again when the destroy removed its KV entry. When the follow of a destroyed source ends, api deletes the tokens of the deployment at deSEC and their KV entry. This follow ends at most 604800 seconds after the destroy, also when the parent does not delegate to other nameservers. A deploy that succeeds after the destroy ends the follow and keeps the tokens. The follow of a source that is not destroyed ends but keeps the tokens, because the destroy of the source uses them. The follow also ends when the deSEC account does not hold the zone.

A resume of a marked source is refused as conflict while the parent delegates the zone to other nameservers. When the parent delegates to the nameservers of the zone at deSEC, or when no server answers, the resume removes the mark.

api keeps one follow record for each marked deployment: the zone, the deSEC API, the nameservers of the registration, the time of the mark, the time of the destroy, the time of the move, the cache time, and whether the follow ended. A destroy does not delete the record.

Domain claim of a migration import

A deployment that registers its domain claims it in the ledger of the hosting provider. The target of a migration import from another hosting provider that uses the same registrar account finds the domain held and no claim. Thus:

A migration import submits with a flag to claim the held domain. The submit keeps the flag in the deployment row before the build starts. A plain submit cannot set the flag.
Each build of the deployment gets the flag of the row. With it, a build claims a domain that the account holds and that no deployment claims. A claim that names another deployment still fails the build.
The next build of the deployment that succeeds, a deploy or a destroy, clears the flag. The claim is then in the state of the deployment, or the destroy removed it. A build that failed keeps the flag.

Domain transfer of a migration

The target of a migration deploys the domain of the operator under the registrar account of its hosting provider. When the registrar account of the source holds the domain, the target transfers the domain to its own account, with the auth code of the domain at the registry. Thus:

An export of a deployment that has the domain that its latest build registered (status held), when its configuration names that domain, gives the auth code in domain_transfer of the string. api reads the code at the registrar with the registrar credentials of the operator deployer, unlocks the domain, and removes its DNSSEC keys, thus the DS records of the source do not stay after the transfer. When the registrar has no code, api makes a new one. api also writes a new marker in the comments of the domain at the registrar, and keeps the other comments. Each time that api locks the domain again and resets its code, it also removes the marker. It does not set the DNSSEC keys again. This occurs before the export stops a service. When the snapshot or the submit of the suspend fails, api locks the domain again and resets its code.
When the account does not hold the domain, the string has null. When api has no registrar command, or the registrar fails, the export is runtime_error, and does not change the source. Each registrar that the operator lane registers at has a registrar command.
api records each domain whose auth code an export gave, with the deployment and the marker. A destroy of the deployment keeps the record. When a deployment that took the domain back exports it again, the records of the domain for other deployments stay, and are superseded: api refuses those deployments as the target took over the domain, without a read of the registrar.
An import refuses a domain_transfer that is not an object with a non-empty domain and auth_code as invalid_configuration. When the configuration registers that domain, the import submit writes the domain and the code to the KV of the deployment, at <deployment prefix>/secret/migration/domain-transfer. The builds of the deployment then transfer the domain when a different account holds it, and fail until the transfer is complete, or when the registry refused the transfer. The build that finds the transfer complete deletes the entry. A submit whose configuration registers a different domain, or no domain, deletes the entry after it stores the configuration; a failure of that delete is a line in the log, not an error of the submit. Else, the destroy of the deployment deletes it.
When the record names the domain, or the registrar account of this hosting provider holds the domain, the domain stays in the account. The import then writes no auth code. After the submit, it locks the domain again and resets its code. The target adopts the domain with the flag to claim a held domain. The record stays, unless the import is into the deployment of the record.
After the export, the target of the migration took over the domain when the record names the domain, and the account does not hold the domain with the marker of the record: a transfer moved the domain to a different account, or a target with the same account took the domain back, also when it then exported the domain from a different api. A lock that a person set at the registrar does not change this result. A change of the marker that a person made at the registrar has the same result as a takeover.
A record without a marker, from a registrar command that writes no marker, uses the transfer lock: the target took over the domain when the account does not hold the domain without a transfer lock. For such a record, a lock that a person set at the registrar has the same result as a takeover.
While the target took over the domain, api refuses each deploy and each action of the deployment of the record as conflict. After a destroy of that deployment, api refuses only a deploy whose configuration registers the domain. When the registrar fails, api refuses with runtime_error.
A destroy of the deployment of the record, while the target took over the domain, first removes the domain from the TF state of the deployment. Thus the destroy does not change the domain at the registrar. When the registrar fails, the destroy also removes the domain from the TF state. A new export of the domain clears this mark.
After the export and before the takeover, api does not refuse the source. A deploy of the source in this period changes the domain at the registrar when its configuration changed, for example the name servers, the DNSSEC keys or the contacts. The deploy does not lock the domain again, because the TF state does not manage the transfer lock. A destroy of the source in this period does not change the renewal of the domain. The deploy of the target sets the renewal to the value of its configuration.
A resume of the source locks each domain of its records again, resets its code and removes the record. A change of the curation after the build or the export does not change which domain the export or the resume takes. When a transfer moved the domain to a different registrar account, the resume is conflict. When api has no registrar credentials or the registrar fails, the resume continues, the record stays, and the code stays valid.
While the record names the domain, domain_list reads the registrar account. When the target took over the domain, and the claim names no other deployment of api than the deployment of the record, the status of the domain is transferred, and domain_claim refuses it as conflict. domain_claim for the deployment of the record also refuses it when the claim names another deployment. Else, when domain_claim writes the claim, it locks the domain again and resets its code. When the claim is for the deployment of the record, it also removes the record. When the registrar fails, domain_list gives the status of the claim, and domain_claim gives runtime_error.

Transfer sources of a migration

The follow job of the source copies the zone of the target with a zone transfer when the target lets it. Else it walks the NSEC chain of the zone, as step 4 above does, and when the zone has no NSEC chain, it asks for each name that the zone holds at the source and for each probe label (operatorZones.probeLabels of the BIND provider, zoneFollow.probeLabels of api). Only the zone transfer and the walk give each name that the target adds after the move. Thus:

An export puts the transfer sources in the string. When the export marks the zone at deSEC for a follow, these are the global IP addresses from which api connects to other networks: for each IP version, the local address of the route to a root server. When that address is not global, for example behind NAT, api asks the address services of services.fediversity-api.zoneFollow.whoamiUrls in turn over that IP version, and takes the first global address of that version that one gives. Else these are the global IP addresses of the primary nameserver of the source, when the DNS service of the hosting provider is BIND. Another DNS service, a name that does not resolve, or no hosting configuration, gives no addresses. The export also puts the transfer time in the string: the longest time that the follow job of the source needs the transfer, 604800 + 604800 + 300 seconds. This is the longest wait for a move of the delegation, the longest cache time, and the margin.
An export that suspends the source also puts a new transfer key in the string: an hmac-sha256 TSIG secret, the base64 of 32 random bytes. The name of the key is fediversity-transfer.<zone>. Each export makes a new key. The key gives only the transfer of the zone, which the public can query too, and only until the follow job of the target removes it. It does not let its holder change a record. The source keeps the key with the export: in the follow record of the zone at deSEC, or, on a BIND lane, in the keys zone of the primary. The keys zone is not in the catalog, and only the apex key and the operator key may read it or transfer it.
An import refuses transfer sources that are not a list of IP addresses as invalid_configuration, a transfer time that is not an integer from 0 to 2419200 seconds, and a transfer key that is not the base64 of 32 bytes. A string without them gives no addresses, the time 0 and no key. The import submit keeps the addresses, the time and the key in the deployment row before the build starts. A plain submit cannot set them.
Each build of the deployment gets the addresses, the time and the key of the row. On a BIND lane, the graph writes the addresses and the time into the catalog, and the key into the keys zone. The primary of the target then lets only the holder of the key and these addresses transfer the zone. The primary uses only the global unicast addresses, and the addresses in the prefixes that its configuration allows, for example a test network. A key that is not the base64 of 32 bytes stops the build, and the primary does not use it.
The next build of the deployment that succeeds clears the addresses, the time and the key, with the flag to claim a held domain. A build that failed keeps them.
The follow job of the target removes the addresses, the time and the key after its first run that sees them: after the transfer time, or after its own time when that is longer. Its own time is 604800 + 604800 + 300 seconds, as for the source. The import comes after the export, thus the follow job of the source then no longer needs the transfer. A release of the zone also removes the key.
The follow job of the source, and the follow in api, first ask for a transfer signed with the key, then for a transfer without a signature. The follow job of the source removes its key when it releases the zone. A resume of the source, and a release of its zone, remove the key from the keys zone.

The signed transfer does not depend on the address that the follow connects from. Without the key, for example from a string of an export before the key, the target lets only the transfer sources transfer the zone. The follow connects from the address of its route to the target, thus a source behind NAT that no address service sees, or a primary of a BIND source that connects from an address that its name does not give, then gets no transfer. Some targets give no transfer at all:

A target that holds the zone at deSEC does not use the key or the addresses: deSEC gives a zone transfer only to its own secondaries.
A target that is not a Fediversity hosting provider does not import the string, thus it does not use the key or the addresses.

Then the follow walks the NSEC chain or asks for each name. When the zone has no NSEC chain, for example a zone with NSEC3 or a zone that is not signed, the follow asks for each name, and a name that only the target holds is not in the copy, unless it is a probe label. No query gives the names of such a zone.

Dry eval of a migration import

The string of a migration import holds a configuration made against the schema of the source. The names eval at submit does not evaluate the systems of the nodes, thus a configuration can pass it and then fail at the eval of the first build, for example on an option that this provider does not have. That build has the claim of the domain and the pointers in the KV. Thus a migration import submits with a dry eval: after the names eval and before it stores anything, the submit evaluates the deploy as its first build does.

The dry eval runs the package of the deployment in dry-run mode, in the tree and environment of the deploy, with the attribute and version of the submit. In dry-run mode, the package evaluates the system of each node, connects to no host and runs no OpenTofu. The dry eval builds the package as the deploy does, thus a package that does not build also refuses the import.
When the dry eval fails, the import is refused as invalid_configuration. The message names the nodes that do not evaluate, the options that do not exist and the options that have no value. When it names no option, it holds the last Nix error. It does not hold the definition values of the Nix error, because they can hold secrets of the configuration. A refused import makes no zone token, writes no pointers to the KV, makes no deployment row (thus no claim of the domain) and starts no build.
The dry eval uses the concurrency cap of the schema builds, and waits for a slot: api does not refuse or skip an import when all slots are in use. When the dry eval takes more than 1800 seconds, the import is refused as build_timeout.
The dry eval takes minutes, thus api runs the import in the background. The first import request starts it and gets 202 with the status checking. A request with the same deployment and string gets that answer while the import runs, and then the result or the error of the import once. A request after that starts a new import. The panel sends the same request again until the answer is not 202.

The dry eval has no deployment row, thus it does not have the values of the row: the flag to claim a held domain and the snapshot references. The OpenTofu graph reads them, and the systems of the nodes do not. The dry eval does not plan or apply the OpenTofu graph, thus a failure there, or a failure at run time on the nodes, still fails the first build.

Preview of a migration import

The import refuses a configuration that this provider does not cover, and the operator configures each gap. POST /migration-import-preview/{deployment_id} shows the gaps before the import. It takes the body of the import, does the checks of the import with the dry eval, and stops before it changes anything: it makes no zone token, writes no pointers to the KV, makes no deployment row and starts no build. It gives:

deployment_method: the method that the import deploys with (the pinned method of this provider, or else the method of the string), and offered_methods: the methods of this provider.
applications: the applications that the string enables, and restores: the applications that have state to restore.
unknown_options, options_without_value and nodes: the options that do not exist on this provider, the options that have no value, and the nodes that do not evaluate. The import gives the same lists as the details of its invalid_configuration refusal.
refusal: the error, the message and the details of the refusal that the import gets, or null when the import passes its checks.

A refusal of the checks is in the 200 body. A string that does not parse, or a module_source that is not on the trusted-source allowlist, is a 400, as at the import. The preview runs in the background as the import does, with 202 and the status checking until it has a result. It does not keep the result of the eval: the import after it evaluates again. It changes nothing, thus it records no audit event. The panel shows the preview in the import dialog before the operator imports.

Restore of a migration import

The string of a migration import has a state pointer for each backup unit of each app, keyed by app and then by unit, for example the database and the storage of PeerTube. An app can have more than one unit. A state pointer holds the S3 location and the password of the snapshot repository of the source, the snapshot reference to restore, and the keys that decrypt the data of the snapshot. A pointer from a source with an older revision holds no keys, and the import accepts it: the restore continues with the keys of the target when the snapshot has no key tags, and writes a warning to the deploy log. The restore fails when the snapshot has key tags and the pointer holds no value for them. The import refuses a string with one pointer for each app as invalid_configuration: export it again from the source. The target is on a different network from the source, thus the S3 location is the public HTTPS endpoint of the object store of the source, with the bucket and the path of the repository in it. The password, the S3 keys and the keys of the snapshot are secrets. Thus they do not go into the deployment row, the audit events, the logs of the api, the OpenTofu state, the plan or the arguments of a Windmill job:

Only a deployment method that can restore gets the pointers. The schema of the module source of the import must declare the restore action, with the method of the deployment in its methods. Else the submit refuses the pointers as invalid_configuration, and nothing changes. An import without pointers does not need the action.
The import submits with the pointers. After validation and before the deployment row changes, the submit writes the pointers to the OpenBao KV of the deployment, with the operator deployer token of the builds. A failed write is runtime_error, and the deployment does not change. A plain submit cannot set the pointers.
The deployment row keeps only the snapshot reference of each unit of each app. Each build of the deployment gets this map.
When the map is not empty and the deployment is not suspended, the operator graph restores the apps after the deploy. The restore reads the pointers from the KV, and fails when their apps, units or snapshot references are not those of the map, or when the target does not have a unit. It restores all units of an app, then restarts the instance of the app once. A failed restore fails the build. The next deploy restores again.
After all apps restore, the graph deletes the KV entry. The next build that succeeds and is not suspended, or a destroy that succeeds, clears the map. A destroy deletes the KV of the deployment, and thus also an entry that no restore deleted.

api and the graph each build the KV path of the pointers on their own side, with the same mount and prefix. A unit test holds that they agree.