Suspended ZFS Pool: Recovered Without Disk Replacement

When a ZFS pool enters the SUSPENDED state, it can look like a catastrophic hardware failure. Faulted disks, a RAIDZ vdev full of failed members, and one drive marked as removed often trigger an immediate and expensive reflex: replace disks. However, a suspended pool is as much a connectivity and path problem as it is a disk problem. In many real-world incidents, the storage media is healthy while the data path is not.

This article describes a fault-first approach to getting a suspended TrueNAS (SCALE) pool back online without replacing a single disk. It focuses on what to verify, how to interpret ZFS status output, and the safest sequence for recovery. The scenario described involves a single RAIDZ1 vdev of 11 disks where multiple devices were faulted and one member was listed as removed after a rack outage. The goal is not to replace disks automatically, but to confirm whether the fault pattern points to shared infrastructure.

How ZFS Suspends a Pool After I/O Failures

A suspended pool is a protective mode. It occurs when ZFS detects that one or more devices repeatedly fail in response to I/O operations. The console messaging typically indicates two key actions: verify that devices are connected and then clear the suspended condition.

In the incident, the pool state was reported as:

state: SUSPENDED
status: one or more devices are faulted in response to IO failures
action: ensure affected devices are connected, then run zpool clear

After reconnecting and confirming system services were again responsive, the next step was inspecting the vdev and device status. The output showed an 11-disk RAIDZ1 where many members appeared faulted and one drive listed as removed. At first glance, that output resembles “multiple disk failures,” which would normally lead to replacements.

The Key Diagnostic Insight: Look for Shared Failure Patterns

The most valuable lesson is a diagnostic reframe: multiple disks faulting at once is often a shared-path failure, not independent media failures. Independent disk failures do not usually cluster in time. Shared failures do.

When multiple members of the same RAIDZ vdev fail simultaneously, the likely shared components include:

HBA ports or HBA driver issues
SAS cables, backplane connections, or zoning mistakes
Controller problems or queue/addressing resets
Transient power or backplane power stabilization after an outage

This matters because ZFS fault reporting can reflect loss of reachability rather than actual disk death. A suspended pool can be triggered by paths failing during an outage, followed by partial recovery where disks reappear but ZFS has already marked them faulted.

Rule of thumb: Before replacing any disk, evaluate what the faulted devices have in common. If they share an HBA path, backplane segment, or controller, the odds strongly favor infrastructure recovery over disk replacement.

Resist the First Reflex: Avoid Actions That Make Recovery Harder

The recovery sequence should avoid changes that could complicate ZFS’s understanding of the hardware. The safest early behavior is to treat the suspended state as a connectivity issue until proven otherwise.

The most important “do not” items are:

Do not immediately add disks or attempt to “complete” the vdev based on what the GUI labels as available. Devices shown in TrueNAS may still belong to the pool or may not be safe to reuse.
Do not start replacement workflows while the system still shows widespread “faulted” members. Replacements should be driven by verified disk-level evidence, not by the presence of faulted status alone.

The incident emphasized that the pool was in a suspended state after a rack outage. Once shared infrastructure is restored, ZFS often needs to be told to reassess via zpool clear, rather than forcing data-path changes.

What to Check Before Clearing the Pool

A successful clear depends on devices being back online and reachable. The recovery checks should include both software and hardware validation.

Confirm device reachability

Verify that the TrueNAS system sees the disks again after the outage.
Check for consistent device identifiers rather than random disappear/reappear cycles.

Validate that the fault pattern matches a shared component

Confirm whether all faulted members belong to the same enclosure/backplane segment.
Check whether the same HBA or controller channel connects those disks.
Look for signs of controller resets, SAS link renegotiation, or cabling instability around the outage window.

Minimize changes until the pool is stable

Instead of replacing hardware immediately, the focus should remain on restoring a stable view: disks recognized, links stable, and ZFS ready to reconsider its faulted device list.

Clearing the Suspended State

Once the devices are confirmed connected, the pool can be cleared. In the incident, the recommended action matched the console guidance: run zpool clear so ZFS exits the suspended state and reassesses IO failures.

Practically, the method is:

Ensure all previously faulted members are present and reachable.
Run zpool clear for the affected pool.
Re-check zpool status to confirm the pool is no longer suspended and that devices no longer show as faulted unnecessarily.

If the faulted devices were the result of a transient connectivity outage, this step can return the pool to service without any physical drive replacement.

Why This Approach Works (and When It Should Not)

This method works because ZFS suspension is triggered by IO failures, and IO failures frequently come from shared infrastructure. Clearing allows ZFS to transition from “persistent IO failure observed” back to “re-evaluate current health,” provided the underlying path is restored.

However, this approach should not be used as a substitute for disk-level diagnostics. If physical drives continue to fail SMART tests, repeatedly drop offline, or show confirmed reallocation or CRC errors that persist after stabilization, the situation likely includes real media failure. In those cases, replacement is appropriate.

Takeaway Checklist for a Suspended RAIDZ Pool

Inspect zpool status and note which devices are faulted and how many.
If many devices fault at once within the same vdev, treat it as a shared-path hypothesis.
Validate that cabling, enclosure/backplane, HBA ports, and controller paths are stable.
Avoid immediate disk replacement or pool manipulation until reachability is restored.
After confirming connectivity, use zpool clear and re-check pool health.

A suspended pool can be intimidating, but it is often a recoverable state when the underlying issue is connectivity. Using a fault-first playbook helps reduce unnecessary disk replacements and aligns recovery actions with how ZFS reports failures.