VPLEX: Disk marked hardware dead due to SCSI check condition 3/11/0 from storage-array (2024)

This event is triggered when VPLEX performs a read request to the underlying storage array LUN, and the array is unable to serviceI/O on that block of the LUN, which triggers the 3/11/0check condition (bad block on the array)

This is commonly seen in situations for times of heavy read-I/O such as:

• VPLEX Extent/Device Migration
• Backup operations
• Database Integrity Checks

VPLEX storage-volume is marked "hardware-dead", but shows healthy on the storage-array interface.

Sample output from cli command ll /clusters/cluster-2/storage-elements/storage-volumes/storage-volume name>

VPlexcli:/> ll /clusters/cluster-2/storage-elements/storage-volumes
/VNX_LUN_25

/clusters/cluster-2/storage-elements/storage-volumes/VNX_LUN_25:
Name Value
----------------------------- ------------------------------------------------
application-consistent false
block-count 1073741824
block-size 4K
capacity 4T
description -
free-chunks []
health-indications [hardware dead] <<
health-state critical-failure <<
io-status dead <<
itls 0x50001442a03c0810/0x5006016b08603879/9,
0x50001442a03c0811/0x5006016308603879/9,
largest-free-chunk 0B
locality -
operational-status error <<
provision-type legacy
storage-array-name EMC-CLARiiON-123456789
storage-volumetype normal
system-id VPD83T3:xxxxxxxxxxxxxxxxxxxxx
thin-capable false
thin-rebuild true
total-free-space 0B
underlying-storage-block-size 512
use unusable <<
used-by [extent_VNX_LUN_25]
vendor-specific-name DGC

VPLEX device/extent migration (mobility job) gets stuck at a certain percent.

Sample output from cli command ll data-migrations/device-migrations/<device_migration_name>

VPlexcli:/> ll data-migrations/device-migrations/D__Migrate_LUN_1

/data-migrations/device-migrations/D__Migrate_LUN_1:
Name Value
--------------- ----------------------------
from-cluster cluster-1
percentage-done 7
source device_VNX_LUN25_1
source-exported -
start-time -
status error <<
target device_SYMM_DEV1234_1
target-exported -
to-cluster cluster-2
transfer-size 2M
type full

Host sees VPLEX storage go offline or marked dead, and VPLEX storage-volume is also marked critical-failure or hardware-dead.

Sample data as noted in firmware log,
amf/45 disk VPD83T3:xxxxxxxxxxxxxxx: read failure: marking this in-use disk dead

VPLEX firmware logs will show streaming or intermittent scsi/27 (Check Condition) with SCSI Sense Code entries for 3/11/0 which translates to "Medium Error - unrecovered read error"

Sample output as noted in firmware log during incident,

2016/06/09 02:46:23.67: scsi/27 tgt VPD83T3:6006016011663200b058c25a984de511 cmd 0x28 status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x3 info 0x0 alen 10 csi 0x0 asc 0x11 ascq 0x0 fru 0x0 sks 0x0
2016/06/09 02:46:23.68: scsi/27 tgt VPD83T3:6006016011663200b058c25a984de511 cmd 0x28 status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x3 info 0x0 alen 10 csi 0x0 asc 0x11 ascq 0x0 fru 0x0 sks 0x0
2016/06/09 02:46:23.69: scsi/27 tgt VPD83T3:6006016011663200b058c25a984de511 cmd 0x28 status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x3 info 0x0 alen 10 csi 0x0 asc 0x11 ascq 0x0 fru 0x0 sks 0x0

To confirm this issue, the following will always be true:
key = 0x3
asc = 0x11
ascq = 0x0

When VPLEX sends an I/O read request (0x28) to the storage-array, the array is not able to successfully service the I/O request and responds with check-condition 3/11/0 for "unrecovered read error".

VPLEX attempts to read from a bad block on the storage-array, and since the storage-array is unable to service this I/O VPLEX marks the storage as dead.

This is not array or array-code specific.

The cause for this is external to VPLEX and is a problem on the storage-array with LUN.

The storage-array that is sending the scsi check condition, 3/11/0, to VPLEX needs to be investigated by the respective array vendor. This issue is triggered by the array not being able to service the read I/O request due to an "unrecovered read" issue on the storage-array.

If this is a VNX2 please check if ETA 477140, ETA 477140: VNX2: Uncorrectable data errors may be promoted into FAST Cache which may result in potential data loss or data unavailability., applies to this issue. If this ETA does not apply then VNX Support

be engaged

The following cli command can be run on the VPLEX Management-Server to get a list of top 50 logical-units impacted by the 3/11/0 check conditions:

grep "key 0x3 " /var/log/VPlex/cli/firmware.log_* | awk '{print $3,$5,$18,$19,$26,$27,$28,$29}' | sort | uniq -c | sort -nr | head -50

Example:

This is a block-layer issue on the storage-array and can only be resolved by taking action on the storage-array itself.

This is not a VPLEX issue, but the VPLEX reporting a symptom seen from the back-end array.

The use of "storage-volume resurrect --force" is not valid here.
This command will force the dead storage-volume to appear as "alive" in VPLEX regardless of its current IO-status or issues on the underlying storage-array
This command forces the storage-volume to come back online until the next IO fails to the underlying storage-array.
When the host requests the same block of data that has the 3/11/0 issue on the underlying storage-array, the storage-volume will be marked dead again.
This is expected behavior and not an indication of a VPLEX issue

Presenting the problematic storage-volume directly from the storage-array to the host (bypassing VPLEX) may let the host use some of the data. However this action is directly presenting possible data corruption to the host. The host will continue to have issues reading from the specific block(s) with the 3/11/0 check condition issue.