Migration workflow — translate, review, prove parity, cut over

Migrating a legacy warehouse is not a translation exercise — it’s an evidence exercise. Kirimana’s migration workflow is built around one question the rest of the migration cannot proceed without answering:

Does the new platform produce the same business-meaningful rows as the legacy platform, against the same input window?

Everything below — translation, review, parity, cutover — exists to answer that question with an exit code and an audit trail, not a paragraph of caveats.

Translate the legacy estate

The funnel starts read-only against the live legacy warehouse:

# Scaffold sources/*.yml from the legacy warehouse's system views
kiri migrate analyze --output sources/legacy.yml

Encrypted procedures are forced out of scope; low-confidence classifications carry a # REVIEW: flag so nothing is silently trusted. From there, legacy SQL translation runs in two tiers:

• Deterministic first. Small, trivially translatable bodies are converted to Spark SQL with no AI calls at all; residual constructs the translator can’t prove are surfaced as warnings rather than guessed at.
• AI for the residue. Legacy business-vault objects whose logic the deterministic path can’t cover are routed to an AI translator. Every call is audited — trace id, prompt hash, model, token counts, cost estimate — and capped per run, so an out-of-bounds burst fails before it bills.

Source-system-specific translation verbs (per legacy dialect and per legacy metadata tool) are documented in the product repo. Apply the medallion layer-policy validator across the generated models with kiri migrate lint-models.

Crucially, no AI output ships itself. Every AI translation lands in a review queue; the contract emitter is the only path that writes a silver-business contract, and it refuses candidates that were never reviewed. “Prove this Spark SQL was reviewed by a human” is a single join from audit row to review decision.

The review queue: KEEP / DROP / MERGE / REDESIGN

A migration that ships the legacy model’s complexity wholesale fails its own goal. The review queue therefore gives the SME four first-class decisions per legacy object, not just accept/reject:

Decision	Meaning	Effect
KEEP	”Migrate this 1:1.”	A silver-business contract is emitted; a parity test guards it
DROP	”Don’t migrate this at all.”	No contract, no parity test; an audit row links the decision to the legacy object
MERGE_INTO	”Fold this into another target.”	No own contract; a consolidation-map row records source → target; parity tests the target only. A merge target that doesn’t resolve to a real contract fails at apply time, never silently
REDESIGN	”The concept stays; the logic is rewritten.”	The SME supplies a hand-written Spark SQL body; the machine translation is discarded

Each decision is a terminal state in the review state machine (ACCEPTED, REJECTED, DROPPED, MERGED, REDESIGNED), and each one writes an audit row — no silent drops, no silent keeps.

Flipping the burden of proof. A project-level flag in kiri.yml — simplification_mode: true — changes the default outcome of a never-reviewed entry from KEEP to DROP. The question stops being “why remove this legacy view?” and becomes “why migrate it?”. On a 66-object migration, an overlooked entry at default-KEEP migrates one extra legacy object; at default-DROP it migrates one fewer — the safe direction when the explicit goal is less code, not more.

The parity harness

Parity is what makes “the translation looks right” load-bearing. The harness enforces five invariants, each pinned by a regression test:

1. Input snapshots are pinned. Before any comparison, the harness records the input window, the legacy input version, the shadow side’s Delta version (via time-travel), the translator version, and a hash of the tolerance config. Re-running with the same snapshot produces a byte-identical report; a snapshot whose pins can’t be resolved raises instead of silently scanning the latest state.
2. One closed-menu row hash. Every row on both sides is hashed by the same pure function: columns sorted by name, NULL as a sentinel that can’t collide with the string "NULL", decimals at declared scale, timestamps normalised to UTC ISO-8601. Column order and row order can never cause a phantom diff.
3. Exactly four diff buckets. match, new (PK only in the shadow), missing (PK only in legacy), mutated (same PK, different hash). There is no fifth bucket.
4. Tolerance is declarative and exact by default. Per-table kiri.transformation.parity_tolerance accepts three modes — exact (default), epsilon, ordering_independent. An unknown mode fails at load time, not at diff time.
5. Mismatches block promotion. A non-zero mutated count (or any new/missing on a strict table) exits 1 and refuses to mark the snapshot accepted. There is no --ignore-failures flag; the only override is editing the tolerance contract in git, which leaves an audit trail. Sample rows appear in reports only when the operator asks and the contract’s classification permits — a confidential table reports counts and PKs, never values.

Dual-flow bronze backfill

Migrations usually need two append flows into one bronze table: a one-time historical seed exported from the legacy estate, and the continuous operational feed. Both share one contract via kiri.ingest.flows:

- property: kiri.ingest.flows
  value:
    backfill:
      landing_zone: /Volumes/main/landing/orders_backfill/
      format: parquet
      once: true
    continuous:
      landing_zone: /Volumes/main/landing/orders_continuous/
      auto_loader: true
      schema_evolution: add_new_columns

The two landing zones must differ, backfill.once must be true, and a lint rule (no-full-refresh-with-backfill) refuses any full-refresh setting on a backfilled table — the historical seed must never be reloadable by accident.

Cutover: shadow → diff → promote → rollback

Cutover is four verbs, not a midnight RENAME TABLE:

kiri migrate shadow   --run-id <id> [--window <iso_pair>]
kiri migrate diff     --run-id <id>          # delegates to the parity harness
kiri migrate promote  --run-id <id> --evidence <approval-token>
kiri migrate rollback --run-id <id> [--within <hours>]

• shadow materialises every accepted contract into an isolated namespace, <schema>_shadow_<run_id>, without touching production reads. The shadow never claims the contract’s canonical identity; re-running the same run-id fails closed unless you pass --reset.
• diff assembles the pinned inputs and calls the parity harness — it does not reimplement comparison. The report path is recorded in the cutover state so promotion can bind to it.
• promote is atomic and rename-based: production renames to a pre-cutover archive, the shadow renames to production, and the namespace claim transfers — one transaction on Databricks Delta. It is also evidence-bound: the --evidence approval token must reference the exact parity report from diff, carry signed approvals, and be unexpired. Any mismatch refuses before a single rename runs.
• rollback reverses the rename pair within a window (default 168 hours, configurable via kiri.yml::cutover.rollback_window_hours). Before renaming back, it re-hashes the pre-cutover archive and compares against the hash recorded at promote time — if someone wrote to the archive, rollback refuses rather than restore corrupted state. Late rollback requires an explicit override approval.

Quarantine reprocess: a typed four-state machine

Rows rejected by silver data-quality gates are a flow state, not a terminal one. Each rejects table gets a sibling reprocess table, and every row walks a closed state machine:

State	Meaning	Terminal?
submitted	Operator/SME inserted a correction; awaiting reprocess	No
reingested	Corrected row re-inserted into bronze	Success on next DQ pass
rejected_again	Silver DQ re-rejected the corrected row	Loops until the attempt limit
abandoned	Hit the 3-attempt limit; no further automated retry	Yes

The operator surface is four verbs — kiri migrate quarantine-init / quarantine submit / quarantine reprocess / quarantine status — and corrected rows go back through the same DQ pipeline that rejected them. There is no bypass lane, and every transition writes an audit row.

The reconciliation sign-off gate

The final gate before business sign-off runs in a locked environment: the legacy process and the migrated pipeline both run from the same frozen input batch, and kiri reconcile compares them — row counts, PK uniqueness, null rates, date bounds — in --report-mode locked, which omits raw values so the report can leave the environment. Raw values, customer names, and production samples never cross into tickets, prompts, or review comments.

Sign-off is cryptographically bound to the evidence:

kiri approve reconcile-hash --report locked-report.json \
  --environment prod --snapshot-id <frozen-batch-id>
# → sha256:<hex> used through kiri approve open / grant / issue, then:
kiri approve consume-reconcile --token <issued-token> \
  --report locked-report.json --environment prod --snapshot-id <frozen-batch-id>

Change the report, the environment, or the snapshot id and the token is rejected. A green, signed reconcile report is the artefact you show the business to prove the new warehouse is faithful to the old one.

Names survive verbatim

Migrated objects keep their source names exactly — tables and columns land under their legacy identifiers, with no generated <source>__ prefix. A dedicated gate enforces it:

kiri migrate verify-names --contract contracts/customer.yml --schema sales

Table-name findings are NAME-001, column-name findings NAME-002. For bronze contracts, kiri.naming.bronze_ref_verbatim: true makes downstream refs use the bare table name too. Ten years of saved queries, reports, and muscle memory keep working on day one.