Troubleshooting

When a migration fails, stop the deployment and identify where it stopped: acquisition, ledger preparation, a user step, bookkeeping, commit, or cleanup. The safe next action depends on that boundary. Keep the original error and deployment identity, but do not publish credentials or sensitive native error text.

This page assumes a configured migrator with the same definitions, table, and lock name as the failed run. Deployment explains the transaction and session boundaries that determine whether work can already be final.

Acquisition Failures

An error matching migrate.ErrLockTimeout means the configured budget expired while acquiring a session or waiting for the lock, before protected work began. It also matches context.DeadlineExceeded. A caller cancellation or deadline that ended first is returned without the lock-timeout marker.

Check whether another approved runner is active and whether unfinished cursors, transactions, sessions, or root named locks occupy the pool. Let the legitimate holder finish or resolve the resource leak before increasing timeouts. Do not acquire the migration lock manually before calling the migrator: it is not reentrant. After correcting acquisition, verify that the next run reaches protected preparation; a larger timeout does not fix a scope bypass or deadlock.

If PostgreSQL lock behavior is inconsistent behind PgBouncer, check pooling mode. Use direct or session pooling for session advisory locks. For SQLite, verify that the database and host profile support cross-process file locks; memory databases cannot coordinate independent migration processes.

Definition Failures

ErrInvalidMigration can indicate missing Up steps, invalid IDs, a dialect restriction, invalid configuration, or MySQL ledger incompatibility. Read the wrapped context rather than treating every instance as a malformed definition. ErrDuplicateID means registration reused an ID. Fix construction before execution; do not rename a previously applied migration to make registration pass.

For ErrOutOfOrder, compare pending and recorded definitions within the affected group. Restore the intended append-only order and investigate why the earlier definition is pending. Deliberately allowing a late change requires reviewing its dependencies and effects on already-applied changes, not merely switching off the error. The default policy warns and runs late work.

History Mismatches

Status marks recorded IDs absent from registration as Missing. Restore the original definitions and their relative order. Rollback rejects unknown targets with ErrUnknownMigration before executing Down steps; do not delete ledger rows or guess the ordering from timestamps to force it through.

ErrIrreversible means a rollback target has no Down steps. Restore from an appropriate backup or deploy a reviewed forward correction. Inventing a Down step after data has been discarded does not make that data recoverable.

With arkoren.dev/core/service/database/migrate imported, inspect history after resolving connectivity and acquisition problems:

statuses, err := migrator.Status(ctx)

Handle the error before using statuses. Status is protected by the migration lock and can create the ledger on a fresh database; it is not a read-only probe. Preparation, metadata, session-loss, or cleanup failure returns nil. On success, compare IDs, batches, and Ran values with the deployed definitions and actual schema/data, not only the last log line.

Partial Changes

On MySQL or with WithoutTransaction, a failed step or ledger insert can leave changes applied but unrecorded. Up sees that ID as pending and will run it again. A later failure also leaves earlier successfully committed migrations in place. Do not treat the whole batch as rolled back.

Before another execution:

1. Quiesce competing migration runners and relevant application writers.
2. Preserve a backup and capture the error, registered order, history, and actual objects/data touched by the failed step.
3. Determine which statements completed and whether the final commit or session outcome is known. On connection loss, consult server evidence and reconcile state rather than assuming the client error proves rollback.
4. Choose an owner-approved recovery: safely complete the pending idempotent change, restore an appropriate backup, or prepare a reviewed corrective path. Do not casually edit ledger rows to claim work ran.
5. Verify schema, data invariants, and ledger together before allowing deployment or replay. Ensure old and new application versions remain compatible.

An error after confirmed commit can leave both schema and ledger final. database.ErrAfterCommit reports that boundary when produced by transaction cleanup or hooks, but a step can itself return an error carrying the same marker. Classify the origin and inspect state; a marker in a step’s error is not proof that its enclosing migration committed. The migrator does not reconnect or replay after losing its retained session.

MySQL Ledger

MySQL operations verify actual ID-column and index metadata under the retained session and migration lock before user steps or history-row mutations. New ledgers use id VARBINARY(191) NOT NULL with a full, single-column primary key on id. ID matching is byte-exact, consistent with Go, rather than collation-based.

An existing character column is unsupported even with a _bin or NO PAD collation. Fixed BINARY, nullable or generated ID columns, composite/prefix/ expression primary keys, and additional unique indexes are also rejected. Nonunique secondary indexes are permitted. The library never automatically ALTERs an existing ledger. Preparation can already issue CREATE TABLE IF NOT EXISTS before rejecting a layout; rejection does not mean no SQL ran.

Column and index inspection failures preserve their cause. Missing or hidden metadata is an inability to establish compatibility, not proof that storage needs conversion. First verify the selected schema, metadata visibility, and permissions. SHOW FULL COLUMNS and SHOW INDEX inspect the addressed target; an unqualified configured table is resolved to the retained session’s current database before steps. Later USE statements cannot redirect bookkeeping. ID insert/delete bindings use the original UTF-8 bytes.

Authorized Repair

Changing a legacy ledger can corrupt migration identity or conceal applied work. Do not run a generic ALTER or delete history to bypass validation. Require explicit owner authorization and a reviewed, database-specific plan that preserves ID bytes and every history tuple.

Before an approved conversion, quiesce all migrators, ledger writers, and conflicting administrator DDL. The migration lock coordinates cooperating runners; it does not exclude arbitrary schema administration. Back up the full schema and every (HEX(id), batch, ran_at) tuple.

Prove that stored ID bytes match the intended UTF-8 application identities, fit within 191 bytes without truncation, and remain uniquely represented. Use OCTET_LENGTH, not a character count, to check that byte limit. Non-UTF-8 legacy storage can require an owner decision: physical bytes and application identity cannot both be preserved by blindly changing a charset. Never normalize, trim, rename, or discard IDs as an incidental conversion step.

Inspect constraints, triggers, generated values, indexes, and column attributes before selecting the repair. Use strict SQL mode during the approved change. There is no universally safe ALTER recipe; attributes and dependencies can change what must be preserved or restated. The library performs no conversion, and the ordinary migration role does not need new ALTER privileges for this check.

After repair, compare the row count and every saved history tuple byte-for-byte, including full timestamp precision. Inspect canonical column and primary-key metadata, and confirm there are no extra unique indexes. Run Status with the original definitions and target to verify the protected inspection succeeds. Reconcile any earlier failed nontransactional user steps before authorizing Up: repairing the ledger does not undo or complete them. Reconcile deployment records too: conversion cannot recover history already lost or conflated under the old equality rules. Stop on conversion warnings or changed tuples rather than accepting coercion or reconstructing timestamps.

For error identities and ledger behavior, see the migrate API reference.