Migration workflow — translate, review, prove parity, cut over

Migrating a legacy warehouse is not a translation exercise — it’s an evidence exercise. Kirimana’s migration workflow is built around one question the rest of the migration cannot proceed without answering:

Does the new platform produce the same business-meaningful rows as the legacy platform, against the same input window?

Everything below — translation, review, parity, cutover — exists to answer that question with an exit code and an audit trail, not a paragraph of caveats.

What is in the free CLI, and what is the Migration Pack. The free kiri-cli covers source analysis (kiri migrate analyze), deterministic T-SQL → Spark SQL translation (kiri migrate translate-tsql), the layer-policy and verbatim-name checks, the reject lane, and the kiri reconcile sign-off gate. AI translation with its review queue, the parity harness, and the shadow → diff → promote → rollback cutover are the Migration Pack — an add-on for design-partner migrations, marked below. Ask about it.

Translate the legacy estate

The funnel starts read-only against the live legacy warehouse:

# Scaffold sources/*.yml from the legacy warehouse's system views
kiri migrate analyze --output sources/legacy.yml

Encrypted procedures are forced out of scope; low-confidence classifications carry a # REVIEW: flag so nothing is silently trusted. From there, legacy SQL translation runs in two tiers:

• Deterministic first. Small, trivially translatable bodies are converted to Spark SQL with no AI calls at all; residual constructs the translator can’t prove are surfaced as warnings rather than guessed at.
• AI for the residue (Migration Pack). Legacy business-vault objects whose logic the deterministic path can’t cover are routed to an AI translator. Every call is audited — trace id, prompt hash, model, token counts, cost estimate — and capped per run, so an out-of-bounds burst fails before it bills.

Source-system-specific translation verbs (per legacy dialect and per legacy metadata tool) are documented in the product repo. Apply the medallion layer-policy validator across the generated models with kiri migrate lint-models.

With the Migration Pack, no AI output ships itself. Every AI translation lands in a review queue; the contract emitter is the only path that writes a silver-business contract, and it refuses candidates that were never reviewed. “Prove this Spark SQL was reviewed by a human” is a single join from audit row to review decision.

The review queue: KEEP / DROP / MERGE / REDESIGN (Migration Pack)

A migration that ships the legacy model’s complexity wholesale fails its own goal. The review queue therefore gives the SME four first-class decisions per legacy object, not just accept/reject:

Decision	Meaning	Effect
KEEP	”Migrate this 1:1.”	A silver-business contract is emitted; a parity test guards it
DROP	”Don’t migrate this at all.”	No contract, no parity test; an audit row links the decision to the legacy object
MERGE_INTO	”Fold this into another target.”	No own contract; a consolidation-map row records source → target; parity tests the target only. A merge target that doesn’t resolve to a real contract fails at apply time, never silently
REDESIGN	”The concept stays; the logic is rewritten.”	The SME supplies a hand-written Spark SQL body; the machine translation is discarded

Each decision is a terminal state in the review state machine (ACCEPTED, REJECTED, DROPPED, MERGED, REDESIGNED), and each one writes an audit row — no silent drops, no silent keeps.

Flipping the burden of proof. A project-level flag in kiri.yml — simplification_mode: true — changes the default outcome of a never-reviewed entry from KEEP to DROP. The question stops being “why remove this legacy view?” and becomes “why migrate it?”. On a 66-object migration, an overlooked entry at default-KEEP migrates one extra legacy object; at default-DROP it migrates one fewer — the safe direction when the explicit goal is less code, not more.

The parity harness (Migration Pack)

Parity is what makes “the translation looks right” load-bearing. The harness enforces five invariants, each pinned by a regression test:

1. Input snapshots are pinned. Before any comparison, the harness records the input window, the legacy input version, the shadow side’s Delta version (via time-travel), the translator version, and a hash of the tolerance config. Re-running with the same snapshot produces a byte-identical report; a snapshot whose pins can’t be resolved raises instead of silently scanning the latest state.
2. One closed-menu row hash. Every row on both sides is hashed by the same pure function: columns sorted by name, NULL as a sentinel that can’t collide with the string "NULL", decimals at declared scale, timestamps normalised to UTC ISO-8601. Column order and row order can never cause a phantom diff.
3. Exactly four diff buckets. match, new (PK only in the shadow), missing (PK only in legacy), mutated (same PK, different hash). There is no fifth bucket.
4. Tolerance is declarative and exact by default. Per-table kiri.transformation.parity_tolerance accepts three modes — exact (default), epsilon, ordering_independent. An unknown mode fails at load time, not at diff time.
5. Mismatches block promotion. A non-zero mutated count (or any new/missing on a strict table) exits 1 and refuses to mark the snapshot accepted. There is no --ignore-failures flag; the only override is editing the tolerance contract in git, which leaves an audit trail. Sample rows appear in reports only when the operator asks and the contract’s classification permits — a confidential table reports counts and PKs, never values.

Dual-flow bronze backfill

Migrations usually need two append flows into one bronze table: a one-time historical seed exported from the legacy estate, and the continuous operational feed. Both share one contract via kiri.ingest.flows:

- property: kiri.ingest.flows
  value:
    backfill:
      landing_zone: /Volumes/main/landing/orders_backfill/
      format: parquet
      once: true
    continuous:
      landing_zone: /Volumes/main/landing/orders_continuous/
      auto_loader: true
      schema_evolution: add_new_columns

The two landing zones must differ, backfill.once must be true, and a lint rule (no-full-refresh-with-backfill) refuses any full-refresh setting on a backfilled table — the historical seed must never be reloadable by accident.

Cutover: shadow → diff → promote → rollback (Migration Pack)

Cutover is four verbs, not a midnight RENAME TABLE:

kiri migrate shadow   --run-id <id> [--window <iso_pair>]
kiri migrate diff     --run-id <id>          # delegates to the parity harness
kiri migrate promote  --run-id <id> --evidence <approval-token>
kiri migrate rollback --run-id <id> [--within <hours>]

• shadow materialises every accepted contract into an isolated namespace, <schema>_shadow_<run_id>, without touching production reads. The shadow never claims the contract’s canonical identity; re-running the same run-id fails closed unless you pass --reset.
• diff assembles the pinned inputs and calls the parity harness — it does not reimplement comparison. The report path is recorded in the cutover state so promotion can bind to it.
• promote is atomic and rename-based: production renames to a pre-cutover archive, the shadow renames to production, and the namespace claim transfers — one transaction on Databricks Delta. It is also evidence-bound: the --evidence approval token must reference the exact parity report from diff, carry signed approvals, and be unexpired. Any mismatch refuses before a single rename runs.
• rollback reverses the rename pair within a window (default 168 hours, configurable via kiri.yml::cutover.rollback_window_hours). Before renaming back, it re-hashes the pre-cutover archive and compares against the hash recorded at promote time — if someone wrote to the archive, rollback refuses rather than restore corrupted state. Late rollback requires an explicit override approval.

Rejected rows: a reject lane with an audited reprocess loop

Rows that break a contract’s column rules are a flow state, not a terminal one. A rule marked on_violation: reject makes kiri apply capture every violating row into a rejects table — the full row, the rule it broke, and the run that caught it — before it reaches silver.

Fixing one is a correction, never an edit of history:

Verb	What it does
kiri quality rejects status / list	See what was rejected, by check and run, leading with the value that broke the rule
kiri quality rejects submit	Stage a corrected row, with a reason
kiri quality rejects reprocess	Append the correction to bronze as a new arrival that records which rejected row it fixes
kiri quality rejects abandon	Stop trying to fix a row, with a reason

The correction goes back through the same rules that rejected it on the next apply — there is no bypass lane. Each row ends resolved, rejected again, or abandoned; only principals the contract allow-lists can move it, and every transition is audited before it takes effect.

The reconciliation sign-off gate

The final gate before business sign-off runs in a locked environment: the legacy process and the migrated pipeline both run from the same frozen input batch, and kiri reconcile compares them — row counts, PK uniqueness, null rates, date bounds — in --report-mode locked, which omits raw values so the report can leave the environment. Raw values, customer names, and production samples never cross into tickets, prompts, or review comments.

Sign-off is cryptographically bound to the evidence:

kiri approve reconcile-hash --report locked-report.json \
  --environment prod --snapshot-id <frozen-batch-id>
# → sha256:<hex> used through kiri approve open / grant / issue, then:
kiri approve consume-reconcile --token <issued-token> \
  --report locked-report.json --environment prod --snapshot-id <frozen-batch-id>

Change the report, the environment, or the snapshot id and the token is rejected. A green, signed reconcile report is the artefact you show the business to prove the new warehouse is faithful to the old one.

Names survive verbatim

Migrated objects keep their source names exactly — tables and columns land under their legacy identifiers, with no generated <source>__ prefix. A dedicated gate enforces it:

kiri migrate verify-names --contract contracts/customer.yml --schema sales

Table-name findings are NAME-001, column-name findings NAME-002. For bronze contracts, kiri.naming.bronze_ref_verbatim: true makes downstream refs use the bare table name too. Ten years of saved queries, reports, and muscle memory keep working on day one.