Jump to: navigation, search
MithiWiki Home > ConnectXf Home > ConnectXf Administration > Configuration > Identifying which drive has large IOWaits


Troubleshooting Icon.png
Troubleshooting
Product ConnectXf
Version All
Applies to Administrators
Level Advanced



Identifying which drive has large IOWaits

Use following steps to identify the drive with large IOWaits :

1. Use iostat -dfx 60 - it will generate stats about each disk repeatedly in format

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await   svctm  %util
sda               0.08    16.17    0.87    9.03    11.67   105.13    23.60     5.17   522.47  62.78  62.15
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00   0.00    0.00   0.00
dm-1              0.00     0.00    0.00    4.80     0.00    19.20     8.00     5.13   1068.37 58.16  27.92
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00   0.00    0.00   0.00
dm-3              0.00     0.00    0.97   20.40    11.73    85.93     9.14    13.64   638.15  27.47  58.70
dm-4              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00   0.00    0.00   0.00


2. Calculate the average time spent in queue using following formula

Note

Take 4 - 5 readings using above command to average out the values

%util :

When this figure is consistently approaching above 80% you will need to take any of the following actions - increasing RAM so dependence on disk reduces increasing RAID controller cache so disk dependence decreases increasing number of disks so disk throughput increases (more spindles working parallely) horizontal partitioning

(await-svctim)/await*100:

The percentage of time that IO operations spent waiting in queue in comparison to actually being serviced. If this figure goes above 50% then each IO request is spending more time waiting in queue than being processed. If this ratio skews heavily upwards (in the >75% range) you know that your disk subsystem is not being able to keep up with the IO requests and most IO requests are spending a lot of time waiting in queue.

For Example:

sda = ((522.47-62.78)/522.47)*100= 87%
dm-1 =((1068.37-58.16)/1068.37)*100= 94%
dm-3 = ((638.15 -27.47)/638.15)*100= 95%

3. If the setup is of LVM type, iostat does not give the name of logical volume directly. It gives dm-0, dm-1 etc.

4. So run following command to find logical volume name :

lvdisplay|awk  '/LV Name/{n=$3} /Block device/{d=$3; sub(".*:","dm-",d); print d,n;}'

It will give output like :

dm-0 /dev/ROOTVG/LogVol04
dm-2 /dev/ROOTVG/LogVol01
dm-1 /dev/ROOTVG/LogVol00
dm-3 /dev/ROOTVG/LogVol03
dm-4 /dev/ROOTVG/LogVol02

Find the volume name using following command

df -h

which gives output as follows

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ROOTVG-LogVol00
79G 22G 53G 30% /
tmpfs 3.9G 72K 3.9G 1% /dev/shm
/dev/mapper/ROOTVG-LogVol01
30G 277M 28G 1% /backup
/dev/sda1 97M 85M 7.1M 93% /boot
/dev/mapper/ROOTVG-LogVol02
5.0G 138M 4.6G 3% /drbdmetadata
/dev/mapper/ROOTVG-LogVol03
60G 16G 42G 27% /mcsdata

References: