COCKROACH-DEBUG-RECOVER(1)

NAME

cockroach-debug-recover - commands to recover unavailable ranges in case of quorum loss

SYNOPSIS

cockroach debug recover [command] [flags]

DESCRIPTION

Set of commands to recover unavailable ranges.

If cluster permanently loses several nodes containing multiple replicas of the range it is possible for that range to lose quorum. Those ranges can not converge on their final state and can't proceed further. This is a potential data and consistency loss situation. In most cases cluster should be restored from the latest backup.

As a workaround, when consistency could be sacrificed temporarily to restore availability quickly it is possible to instruct remaining replicas to act as if they have the most up to date view of data and continue operation. There's no way to guarantee which replica contains most up-to-date information if at all so heuristics based on raft state are used to pick one.

As a result of such recovery, some of the data could be lost, indexes could become corrupted and database constraints could be violated. So manual recovery steps to verify the data and ensure database consistency should be taken ASAP. Those actions should be done at application level.

'debug recover' set of commands is used as a last resort to perform range recovery operation.

Loss of quorum recovery could be performed in two modes half-online and offline. Half-online approach is a preferred one but offline approach is preserved for compatibility with any tooling that may pre-exist. The main advantage of half-online approach is that replica info could be collected and recovery plan could be staged without stopping the cluster. Only nodes that contain replicas of affected ranges needs to be subsequently restarted to complete the recovery.

To perform recovery using half-online approach one should perform this sequence of actions:

    1.  Run 'cockroach debug recover make-plan' in a half-online mode to collect replica information from surviving nodes of a cluster and decide which replicas should survive and up-replicate.

    2.  Run 'cockroach debug recover apply-plan' in half online mode to distribute plan to surviving cluster nodes for application. At this point plan is staged and can't be reverted.

    3.  Follow instructions from apply plan to perform a rolling restart of nodes that need to update their storage. Restart should be done using appropriate automation that used to run the cluster.

    4.  Optionally use 'cockroach debug recover verify' to check recovery progress and resulting range health.

If it was possible to produce distribute and apply the plan, then cluster should become operational again. It is not guaranteed that there's no data loss and that all database consistency was not compromised.

If for whatever reasons half-online approach is not feasible or fails when collecting info or distributing recovery plans, one could perform this sequence of actions:

    1.  Decommission failed nodes preemptively to eliminate the possibility of them coming back online and conflicting with the recovered state. Note that if system ranges suffer loss of quorum, it may be impossible to decommission nodes. In that case, recovery can proceed, but those nodes must be prevented from communicating with the cluster and must be decommissioned once the cluster is back online after recovery.

    2.  Stop the cluster

    3.  Run 'cockroach debug recover collect-info' on every node to collect replication state from all surviving nodes. Outputs of these invocations should be collected and made locally available for the next step.

    4.  Run 'cockroach debug recover make-plan' providing all files generated on step 1. Planner will decide which replicas should survive and up-replicate.

    5.  Run 'cockroach debug recover apply-plan' on every node using plan generated on the previous step. Each node will pick relevant portion of the plan and update local replicas accordingly to restore quorum.

    6.  Start the cluster.

If it was possible to produce and apply the plan, then cluster should become operational again. It is not guaranteed that there's no data loss and that all database consistency was not compromised.

Example run #1 (half-online mode):

If we have a cluster of 5 nodes 1-5 where we lost nodes 3 and 4. Each node has two stores and they are numbered as 1,2 on node 1; 3,4 on node 2 etc. Recovery commands to recover unavailable ranges would be (most command output is omitted for brevity):

[cockroach@admin ~]$ cockroach debug recover make-plan --host cockroach-1.cockroachlabs.com --certs-dir=root_certs -o recovery-plan.json

[cockroach@admin ~]$ cockroach debug recover apply-plan --host cockroach-1.cockroachlabs.com --certs-dir=root_certs recovery-plan.json

Proceed with staging plan [y/N] y

Plan staged. To complete recovery restart nodes n2, n3.

[cockroach@admin ~]$ # restart-nodes 2 3 as instructed by apply-plan.

[cockroach@admin ~]$ cockroach debug recover verify --host cockroach-1.cockroachlabs.com --certs-dir=root_certs recovery-plan.json

Loss of quorum recovery is complete.

Example run #2 (offline mode):

If we have a cluster of 5 nodes 1-5 where we lost nodes 3 and 4. Each node has two stores and they are numbered as 1,2 on node 1; 3,4 on node 2 etc. Recovery commands to recover unavailable ranges would be:

Decommission dead nodes and stop the cluster.

[cockroach@node1 ~]$ cockroach debug recover collect-info --store=/mnt/cockroach-data-1 --store=/mnt/cockroach-data-2 >info-node1.json [cockroach@node2 ~]$ cockroach debug recover collect-info --store=/mnt/cockroach-data-1 --store=/mnt/cockroach-data-2 >info-node2.json [cockroach@node5 ~]$ cockroach debug recover collect-info --store=/mnt/cockroach-data-1 --store=/mnt/cockroach-data-2 >info-node5.json

[cockroach@base ~]$ scp cockroach@node1:info-node1.json . [cockroach@base ~]$ scp cockroach@node2:info-node1.json . [cockroach@base ~]$ scp cockroach@node5:info-node1.json .

[cockroach@base ~]$ cockroach debug recover make-plan --dead-store-ids=5,6,7,8 info-node1.json info-node2.json info-node5.json >recover-plan.json

[cockroach@base ~]$ scp recover-plan.json cockroach@node1: [cockroach@base ~]$ scp recover-plan.json cockroach@node2: [cockroach@base ~]$ scp recover-plan.json cockroach@node5:

[cockroach@node1 ~]$ cockroach debug recover apply-plan --store=/mnt/cockroach-data-1 --store=/mnt/cockroach-data-2 recover-plan.json Info:
  updating replica for r1 Proceed with above changes [y/N]y Info:
  updated store s2 [cockroach@node2 ~]$ cockroach debug recover apply-plan --store=/mnt/cockroach-data-1 --store=/mnt/cockroach-data-2 recover-plan.json [cockroach@node5 ~]$ cockroach debug recover apply-plan --store=/mnt/cockroach-data-1 --store=/mnt/cockroach-data-2 recover-plan.json

Now the cluster could be started again.

OPTIONS

-h, --help[=false]
    help for recover

OPTIONS INHERITED FROM PARENT COMMANDS

--enterprise-require-fips-ready[=false]
    abort if FIPS readiness checks fail

--log=

Logging configuration, expressed using YAML syntax. For example, you can
 change the default logging directory with: --log='file-defaults: {dir: ...}'.
 See the documentation for more options and details. To preview how the log
 configuration is applied, or preview the default configuration, you can use
 the 'cockroach debug check-log-config' sub-command.

--log-config-file=

File name to read the logging configuration from. This has the same effect as
 passing the content of the file via the --log flag.

--log-config-vars=[]

Environment variables that will be expanded if present in the body of the
 logging configuration.

--log-dir=

--log-dir=XXX is an alias for --log='file-defaults: {dir: XXX}'.

--logtostderr[=UNKNOWN]

--logtostderr=XXX is an alias for --log='sinks: {stderr: {filter: XXX}}'. If
 no value is specified, the default value for the command is inferred: INFO for
 server commands, WARNING for client commands.

--redactable-logs=false

--redactable-logs=XXX is an alias for --log='file-defaults: {redactable:
 XXX}}'.

--vmodule=
    comma-separated list of pattern=N settings for file-filtered logging (significantly hurts performance)

SEE ALSO

cockroach-debug(1), cockroach-debug-recover-apply-plan(1), cockroach-debug-recover-collect-info(1), cockroach-debug-recover-make-plan(1), cockroach-debug-recover-verify(1)

HISTORY

13-May-2026 Auto generated by spf13/cobra