Код: Выделить всё
Apr 1 03:42:43 blvirt1 kernel: __ratelimit: 8 callbacks suppressed
Apr 1 03:42:43 blvirt1 kernel: end_request: I/O error, dev dm-11, sector 6683623296
Apr 1 03:42:43 blvirt1 kernel: end_request: I/O error, dev dm-11, sector 6683623408
Apr 1 03:42:43 blvirt1 kernel: end_request: I/O error, dev dm-11, sector 0
Apr 1 03:42:43 blvirt1 kernel: end_request: I/O error, dev dm-14, sector 209715072
Apr 1 03:42:43 blvirt1 kernel: end_request: I/O error, dev dm-14, sector 209715184
Apr 1 03:42:43 blvirt1 kernel: end_request: I/O error, dev dm-14, sector 0
Apr 1 03:42:43 blvirt1 kernel: end_request: I/O error, dev dm-92, sector 2147483520
Apr 1 03:42:43 blvirt1 kernel: end_request: I/O error, dev dm-92, sector 2147483632
Apr 1 03:42:43 blvirt1 kernel: end_request: I/O error, dev dm-92, sector 0
Apr 1 03:42:43 blvirt1 kernel: end_request: I/O error, dev dm-11, sector 6683623296
Apr 1 03:42:44 blvirt1 multipathd: 3600000e00d11000000113629001b0000: sdq - tur checker reports path is down
Apr 1 03:42:44 blvirt1 multipathd: 3600000e00d1100000011362900210000: sdr - tur checker reports path is down
Apr 1 03:42:47 blvirt1 multipathd: 3600000e00d11000000113629001b0000: sdg - tur checker reports path is down
Apr 1 03:42:47 blvirt1 multipathd: 3600000e00d1100000011362900210000: sdh - tur checker reports path is down
Apr 1 03:42:47 blvirt1 multipathd: 3600000e00d1100000011362900260000: sdv - tur checker reports path is down
Apr 1 03:42:47 blvirt1 multipathd: 3600000e00d1100000011362900260000: sdu - tur checker reports path is down
Apr 1 03:42:48 blvirt1 kernel: __ratelimit: 26 callbacks suppressed
Apr 1 03:42:48 blvirt1 kernel: end_request: I/O error, dev dm-11, sector 6683623296
Apr 1 03:42:48 blvirt1 kernel: end_request: I/O error, dev dm-11, sector 6683623408
Apr 1 03:42:48 blvirt1 kernel: end_request: I/O error, dev dm-11, sector 0
Apr 1 03:42:48 blvirt1 kernel: end_request: I/O error, dev dm-14, sector 209715072
Apr 1 03:42:48 blvirt1 kernel: end_request: I/O error, dev dm-14, sector 209715184
Apr 1 03:42:48 blvirt1 kernel: end_request: I/O error, dev dm-14, sector 0
Apr 1 03:42:48 blvirt1 kernel: end_request: I/O error, dev dm-92, sector 2147483520
Apr 1 03:42:48 blvirt1 kernel: end_request: I/O error, dev dm-92, sector 2147483632
Apr 1 03:42:48 blvirt1 kernel: end_request: I/O error, dev dm-92, sector 0
Apr 1 03:42:49 blvirt1 multipathd: 3600000e00d11000000113629001b0000: sdq - tur checker reports path is down
Apr 1 03:42:49 blvirt1 multipathd: 3600000e00d1100000011362900210000: sdr - tur checker reports path is down
Apr 1 03:42:49 blvirt1 kernel: end_request: I/O error, dev dm-11, sector 6683623296
Apr 1 03:42:52 blvirt1 multipathd: 3600000e00d11000000113629001b0000: sdg - tur checker reports path is down
Apr 1 03:42:52 blvirt1 multipathd: 3600000e00d1100000011362900210000: sdh - tur checker reports path is down
Apr 1 03:42:52 blvirt1 multipathd: 3600000e00d1100000011362900260000: sdv - tur checker reports path is down
Apr 1 03:42:52 blvirt1 multipathd: 3600000e00d1100000011362900260000: sdu - tur checker reports path is down
Apr 1 03:42:54 blvirt1 multipathd: 3600000e00d11000000113629001b0000: sdq - tur checker reports path is down
Apr 1 03:42:54 blvirt1 multipathd: 3600000e00d1100000011362900210000: sdr - tur checker reports path is down
Apr 1 03:42:57 blvirt1 multipathd: 3600000e00d11000000113629001b0000: sdg - tur checker reports path is down
Apr 1 03:42:57 blvirt1 multipathd: 3600000e00d1100000011362900210000: sdh - tur checker reports path is down
Apr 1 03:42:57 blvirt1 multipathd: 3600000e00d1100000011362900260000: sdv - tur checker reports path is down
Apr 1 03:42:57 blvirt1 multipathd: 3600000e00d1100000011362900260000: sdu - tur checker reports path is down
Apr 1 03:42:59 blvirt1 multipathd: 3600000e00d11000000113629001b0000: sdq - tur checker reports path is down
Apr 1 03:42:59 blvirt1 multipathd: 3600000e00d1100000011362900210000: sdr - tur checker reports path is down
Apr 1 03:43:02 blvirt1 multipathd: 3600000e00d11000000113629001b0000: sdg - tur checker reports path is down
Apr 1 03:43:02 blvirt1 multipathd: 3600000e00d1100000011362900210000: sdh - tur checker reports path is down
Apr 1 03:43:02 blvirt1 multipathd: 3600000e00d1100000011362900260000: sdv - tur checker reports path is down
Apr 1 03:43:02 blvirt1 multipathd: 3600000e00d1100000011362900260000: sdu - tur checker reports path is down
Apr 1 03:43:04 blvirt1 multipathd: 3600000e00d11000000113629001b0000: sdq - tur checker reports path is down
Apr 1 03:43:04 blvirt1 multipathd: 3600000e00d1100000011362900210000: sdr - tur checker reports path is down
Apr 1 03:43:07 blvirt1 multipathd: 3600000e00d11000000113629001b0000: sdg - tur checker reports path is down
Apr 1 03:43:07 blvirt1 multipathd: 3600000e00d1100000011362900210000: sdh - tur checker reports path is down
dm-11 dm-14 dm-92 - это отображение одного LVM в Овирте. Хранилище на эти диски не ругается, значит смарт дисков в порядке.
Вот такие ошибки пошли, когда началась глобальная проблема и всё поотваливалось, только там номер проблемного LUN dm-9 dm-13 dm-15:
Вырезки:
Код: Выделить всё
Mar 27 12:55:15 blvirt2 multipathd: 3600000e00d1100000011362900210000: sdh - tur checker reports path is down
Mar 27 12:55:16 blvirt2 multipathd: 3600000e00d1100000011362900260000: Entering recovery mode: max_retries=10
Mar 27 12:55:16 blvirt2 multipathd: 3600000e00d1100000011362900210000: Entering recovery mode: max_retries=10
Mar 27 12:55:16 blvirt2 multipathd: 3600000e00d11000000113629001b0000: Entering recovery mode: max_retries=10
Mar 27 12:55:16 blvirt2 kernel: sd 0:0:0:10: [sdk] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Mar 27 12:55:16 blvirt2 kernel: sd 0:0:0:10: [sdk] Sense Key : Illegal Request [current]
Mar 27 12:55:16 blvirt2 kernel: sd 0:0:0:10: [sdk] Add. Sense: Logical unit not supported
Mar 27 12:55:16 blvirt2 kernel: sd 0:0:0:10: [sdk] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Mar 27 12:55:16 blvirt2 kernel: __ratelimit: 35 callbacks suppressed
Mar 27 12:55:16 blvirt2 kernel: end_request: I/O error, dev sdk, sector 0
Mar 27 12:55:16 blvirt2 kernel: device-mapper: multipath: Failing path 8:160.
Mar 27 12:55:16 blvirt2 kernel: sd 0:0:0:7: [sdh] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Mar 27 12:55:16 blvirt2 kernel: sd 0:0:0:7: [sdh] Sense Key : Illegal Request [current]
Mar 27 12:55:16 blvirt2 kernel: sd 0:0:0:7: [sdh] Add. Sense: Logical unit not supported
Mar 27 12:55:16 blvirt2 kernel: sd 0:0:0:7: [sdh] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Mar 27 12:55:16 blvirt2 kernel: end_request: I/O error, dev sdh, sector 0
Mar 27 12:55:16 blvirt2 kernel: device-mapper: multipath: Failing path 8:112.
Mar 27 12:55:16 blvirt2 kernel: sd 1:0:0:10: [sdx] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Mar 27 12:55:16 blvirt2 kernel: sd 1:0:0:10: [sdx] Sense Key : Illegal Request [current]
Mar 27 12:55:16 blvirt2 kernel: sd 1:0:0:10: [sdx] Add. Sense: Logical unit not supported
Mar 27 12:55:16 blvirt2 kernel: sd 1:0:0:10: [sdx] CDB: Read(10): 28 00 00 00 00 00 00 00 08Mar 27 12:58:53 blvirt2 vdsm root ERROR Process failed with rc=1 out='\nudevadm settle - timeout of 5 seconds reached, the event queue contains:\n /sys/devices/virtual/block/dm-13 (970102)\n /sys/devices/virtual/block/dm-13 (970103)\n /sys/devices/virtual/block/dm-9 (970104)\n /sys/devices/virtual/block/dm-9 (970105)\n /sys/devices/virtual/block/dm-13 (970108)\n /sys/devices/virtual/block/dm-9 (970109)\n /sys/devices/virtual/block/dm-13 (970110)\n /sys/devices/virtual/block/dm-9 (970111)\n' err=''
00
Mar 27 12:55:16 blvirt2 kernel: end_request: I/O error, dev sdx, sector 0
Mar 27 12:55:16 blvirt2 kernel: device-mapper: multipath: Failing path 65:112.
Mar 27 12:55:16 blvirt2 kernel: sd 1:0:0:7: [sdu] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Mar 27 12:55:16 blvirt2 kernel: sd 1:0:0:7: [sdu] Sense Key : Illegal Request [current]
Mar 27 12:55:16 blvirt2 kernel: sd 1:0:0:7: [sdu] Add. Sense: Logical unit not supported
Mar 27 12:55:16 blvirt2 kernel: sd 1:0:0:7: [sdu] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Mar 27 12:55:16 blvirt2 kernel: end_request: I/O error, dev sdu, sector 0
Mar 27 12:55:16 blvirt2 kernel: device-mapper: multipath: Failing path 65:64.
Mar 27 12:55:16 blvirt2 kernel: set_rtc_mmss: can't update from 0 to 55
Mar 27 12:55:16 blvirt2 multipathd: 3600000e00d1100000011362900260000: sdx - tur checker reports path is down
Mar 27 12:55:17 blvirt2 multipathd: 3600000e00d1100000011362900260000: Entering recovery mode: max_retries=10
Mar 27 12:55:17 blvirt2 multipathd: 3600000e00d1100000011362900260000: Entering recovery mode: max_retries=10
Mar 27 12:55:17 blvirt2 multipathd: 3600000e00d1100000011362900210000: Entering recovery mode: max_retries=10
Mar 27 12:55:17 blvirt2 multipathd: 3600000e00d1100000011362900210000: Entering recovery mode: max_retries=10
...
Mar 27 12:57:55 blvirt2 multipathd: 3600000e00d1100000011362900210000: Disable queueing
Mar 27 12:57:55 blvirt2 multipathd: 3600000e00d1100000011362900260000: Disable queueing
Mar 27 12:57:55 blvirt2 kernel: end_request: I/O error, dev dm-9, sector 209715072
Mar 27 12:57:55 blvirt2 kernel: end_request: I/O error, dev dm-9, sector 0
Mar 27 12:57:55 blvirt2 kernel: __ratelimit: 30 callbacks suppressed
Mar 27 12:57:55 blvirt2 kernel: Buffer I/O error on device dm-9, logical block 0
Mar 27 12:57:55 blvirt2 kernel: end_request: I/O error, dev dm-13, sector 0
Mar 27 12:57:55 blvirt2 kernel: Buffer I/O error on device dm-13, logical block 0
Mar 27 12:57:55 blvirt2 kernel: end_request: I/O error, dev dm-9, sector 209715192
Mar 27 12:57:55 blvirt2 kernel: Buffer I/O error on device dm-9, logical block 26214399
Mar 27 12:57:55 blvirt2 kernel: end_request: I/O error, dev dm-13, sector 2147483640
Mar 27 12:57:55 blvirt2 kernel: Buffer I/O error on device dm-13, logical block 268435455
Mar 27 12:57:55 blvirt2 kernel: end_request: I/O error, dev dm-9, sector 0
Mar 27 12:57:55 blvirt2 kernel: Buffer I/O error on device dm-9, logical block 0
Mar 27 12:57:55 blvirt2 kernel: end_request: I/O error, dev dm-13, sector 0
Mar 27 12:57:55 blvirt2 kernel: Buffer I/O error on device dm-13, logical block 0
Mar 27 12:57:55 blvirt2 kernel: end_request: I/O error, dev dm-13, sector 0
Mar 27 12:57:55 blvirt2 kernel: end_request: I/O error, dev dm-9, sector 0
Mar 27 12:57:55 blvirt2 kernel: Buffer I/O error on device dm-13, logical block 0
Mar 27 12:57:55 blvirt2 kernel: Buffer I/O error on device dm-9, logical block 0
Mar 27 12:57:55 blvirt2 kernel: end_request: I/O error, dev dm-13, sector 24
Mar 27 12:57:55 blvirt2 kernel: Buffer I/O error on device dm-13, logical block 3
Mar 27 12:57:55 blvirt2 kernel: Buffer I/O error on device dm-9, logical block 3
Mar 27 12:57:55 blvirt2 multipathd: 3600000e00d11000000113629001b0000: Entering recovery mode: max_retries=10
Mar 27 12:57:55 blvirt2 kernel: sd 0:0:0:6: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Mar 27 12:57:55 blvirt2 kernel: sd 0:0:0:6: [sdg] Sense Key : Illegal Request [current]
Mar 27 12:57:55 blvirt2 kernel: sd 0:0:0:6: [sdg] Add. Sense: Logical unit not supported
Mar 27 12:57:55 blvirt2 kernel: sd 0:0:0:6: [sdg] CDB: Read(16): 88 00 00 00 00 01 8e 5f ff 80 00 00 00 08 00 00
Mar 27 12:57:55 blvirt2 kernel: device-mapper: multipath: Failing path 8:96.
Mar 27 12:57:55 blvirt2 kernel: sd 1:0:0:6: [sdt] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Mar 27 12:57:55 blvirt2 kernel: sd 1:0:0:6: [sdt] Sense Key : Illegal Request [current]
Mar 27 12:57:55 blvirt2 kernel: sd 1:0:0:6: [sdt] Add. Sense: Logical unit not supported
Mar 27 12:57:55 blvirt2 kernel: sd 1:0:0:6: [sdt] CDB: Read(16): 88 00 00 00 00 01 8e 5f ff 80 00 00 00 08 00 00
Mar 27 12:57:55 blvirt2 kernel: sd 1:0:0:6: [sdt] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Mar 27 12:57:55 blvirt2 kernel: sd 1:0:0:6: [sdt] Sense Key : Illegal Request [current]
Mar 27 12:57:55 blvirt2 kernel: sd 1:0:0:6: [sdt] Add. Sense: Logical unit not supported
Mar 27 12:57:55 blvirt2 kernel: sd 1:0:0:6: [sdt] CDB: Read(16): 88 00 00 00 00 01 8e 5f ff 80 00 00 00 08 00 00
Mar 27 12:57:55 blvirt2 kernel: device-mapper: multipath: Failing path 65:48.
Mar 27 12:57:56 blvirt2 multipathd: 3600000e00d1100000011362900210000: sdh - tur checker reports path is down
Mar 27 12:57:56 blvirt2 multipathd: 3600000e00d11000000113629001b0000: Entering recovery mode: max_retries=10
...
Mar 27 12:58:53 blvirt2 vdsm root ERROR Process failed with rc=1 out='\nudevadm settle - timeout of 5 seconds reached, the event queue contains:\n /sys/devices/virtual/block/dm-13 (970102)\n /sys/devices/virtual/block/dm-13 (970103)\n /sys/devices/virtual/block/dm-9 (970104)\n /sys/devices/virtual/block/dm-9 (970105)\n /sys/devices/virtual/block/dm-13 (970108)\n /sys/devices/virtual/block/dm-9 (970109)\n /sys/devices/virtual/block/dm-13 (970110)\n /sys/devices/virtual/block/dm-9 (970111)\n' err=''
...
Mar 27 13:09:15 blvirt2 multipathd: dm-117: remove map (uevent)
Mar 27 13:09:15 blvirt2 multipathd: dm-117: devmap not registered, can't remove
Mar 27 13:09:15 blvirt2 multipathd: dm-117: remove map (uevent)
Mar 27 13:09:15 blvirt2 multipathd: dm-117: devmap not registered, can't remove
Mar 27 13:09:15 blvirt2 multipathd: dm-118: remove map (uevent)
Mar 27 13:09:15 blvirt2 multipathd: dm-118: devmap not registered, can't remove
Mar 27 13:09:15 blvirt2 multipathd: dm-118: remove map (uevent)
Mar 27 13:09:15 blvirt2 multipathd: dm-118: devmap not registered, can't remove
Mar 27 13:09:15 blvirt2 multipathd: 3600000e00d1100000011362900210000: sdu - tur checker reports path is down
Mar 27 13:09:16 blvirt2 vdsm vm.Vm ERROR vmId=`6b9efb28-3998-4c2e-88f6-28559f9e531f`::Stats function failed: <AdvancedStatsFunction _sampleVcpuPinning at 0x23fecb0>#012Traceback (most recent call last):#012 File "/usr/share/vdsm/virt/sampling.py", line 484, in collect#012 statsFunction()#012 File "/usr/share/vdsm/virt/sampling.py", line 359, in __call__#012 retValue = self._function(*args, **kwargs)#012 File "/usr/share/vdsm/virt/vm.py", line 351, in _sampleVcpuPinning#012 return vCpuInfos[0]#012TypeError: 'NoneType' object is unsubscriptable
Mar 27 13:09:16 blvirt2 vdsm vm.Vm ERROR vmId=`cd5d3e58-fd6b-483e-bc2c-ce69f6216bfb`::Stats function failed: <AdvancedStatsFunction _sampleVcpuPinning at 0x23fecb0>#012Traceback (most recent call last):#012 File "/usr/share/vdsm/virt/sampling.py", line 484, in collect#012 statsFunction()#012 File "/usr/share/vdsm/virt/sampling.py", line 359, in __call__#012 retValue = self._function(*args, **kwargs)#012 File "/usr/share/vdsm/virt/vm.py", line 351, in _sampleVcpuPinning#012 return vCpuInfos[0]#012TypeError: 'NoneType' object is unsubscriptable
Mar 27 13:09:16 blvirt2 multipathd: dm-119: remove map (uevent)
Mar 27 13:09:16 blvirt2 multipathd: dm-119: devmap not registered, can't remove
Mar 27 13:09:16 blvirt2 multipathd: dm-119: remove map (uevent)
Mar 27 13:09:16 blvirt2 multipathd: dm-119: devmap not registered, can't remove
Mar 27 13:09:16 blvirt2 multipathd: 3600000e00d1100000011362900210000: sdh - tur checker reports path is down
Mar 27 13:09:16 blvirt2 multipathd: dm-120: remove map (uevent)
Mar 27 13:09:16 blvirt2 multipathd: dm-120: devmap not registered, can't remove
Mar 27 13:09:16 blvirt2 multipathd: dm-120: remove map (uevent)
Mar 27 13:09:16 blvirt2 multipathd: dm-120: devmap not registered, can't remove
Mar 27 13:09:16 blvirt2 vdsm vm.Vm WARNING vmId=`b5b94ea6-0a39-43ef-a506-002c25c68c42`::trying to set state to Powering down when already Down
Mar 27 13:09:17 blvirt2 multipathd: dm-35: remove map (uevent)
Mar 27 13:09:17 blvirt2 multipathd: dm-35: devmap not registered, can't remove
Mar 27 13:09:17 blvirt2 multipathd: dm-35: remove map (uevent)
Mar 27 13:09:17 blvirt2 multipathd: dm-35: devmap not registered, can't remove
если на хранилке отключить эти LUN, как Овирт поведёт себя?