Emulate a slow block device with dm-delay

When debugging a problem caused by high I/O latency on Linux, it may be interesting to emulate a slow or congested block device. The device mapper driver which manages logical volumes on Linux has a solution for that: the dm-delay target.

In this article, we will use the dm-delay target to delay reads and writes to a block device. We will first create a ramdisk which is an extremely fast block device. Then, we will stack the dm-delay target on top of it and measure the I/O latency it introduces.

Creating a ramdisk

A ramdisk is a RAM backed disk. Since data written to RAM do not persist without power, do not store real data on a ramdisk. Compared to a hard disk drive, a ramdisk is much smaller in size: its size cannot exceed the computer RAM size. But a ramdisk is much faster than a hard disk drive.

On Linux, loading the brd kernel module creates a set of ramdisks. Passing arguments to the modprobe command configures the number of ramdisks and their size:

• rd_nr sets the maximum number of ramdisks.
• rd_size sets the size of each ramdisk in KiB.

The command below creates a 1 GB ramdisk:

$ sudo modprobe brd rd_nr=1 rd_size=1048576
$ ls -l /dev/ram0
brw-rw---- 1 root disk 1, 0 Aug 24 20:00 /dev/ram0
$ sudo blockdev --getsize /dev/ram0 # Display the size in 512-byte sectors
2097152

Creating a delayed target with dm-delay

The kernel documentation explains how to configure a delayed target with dmsetup. For instance, you can use a script like this one to stack a delayed block device on top of a given block device:

#!/bin/sh
# Create a block device that delays reads for 500 ms
size=$(blockdev --getsize $1) # Size in 512-bytes sectors
echo "0 $size delay $1 0 500" | dmsetup create delayed

Checking the latency of dm-delay

Let's check the latency introduced by dm-delay. We use fio to compare the latency of the ramdisk (/dev/ram0) with the latency of the delayed device (/dev/dm-0). The job file for fio that describes the I/O workload is as follows:

[random]
# Perform 4K random reads for 10 seconds using direct I/Os
filename=/dev/dm-0
readwrite=randread
blocksize=4k
ioengine=sync
direct=1
time_based=1
runtime=10

At the end of the run, fio displays a bunch of statistics. One of them is the completion latency (denoted as clat):

• ramdisk: 1.33 µs
• delayed block device: 499735.14 µs

The latency of the ramdisk is a few microseconds whereas the latency of the delayed block device backed by the ramdisk is close to the 500 ms delay we requested.

A similar experiment for writes shows that dm-delay also delays writes to the device. To delay writes with dm-delay, give a second set of parameters to dmsetup:

#!/bin/sh
# Create a block device that delays reads for 500 ms and writes for 300 ms
size=$(blockdev --getsize $1) # Size in 512-bytes sectors
echo "0 $size delay $1 0 500 $1 0 300" | dmsetup create delayed

Suspending I/Os

The device mapper can also be requested to suspend and resume I/Os.

$ sudo dmsetup suspend /dev/dm-0
$ sudo dmsetup resume  /dev/dm-0