OpenZFS 6531 - Provide mechanism to artificially limit disk performance

Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Ported by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>

OpenZFS-issue: https://www.illumos.org/issues/6531
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/97e8130

Porting notes:
- Added new IO delay tracepoints, and moved common ZIO tracepoint macros
  to a new trace_common.h file.
- Used zio_delay_taskq() in place of OpenZFS's timeout_generic() function.
- Updated zinject man page
- Updated zpool_scrub test files

man/man8/zinject.8

@@ -42,6 +42,39 @@ Cancel injection records.
 .TP
 .B "zinject \-d \fIvdev\fB \-A <degrade|fault> \fIpool\fB
 Force a vdev into the DEGRADED or FAULTED state.
+.TP
+.B "zinject -d \fIvdev\fB -D latency:lanes \fIpool\fB
+
+Add an artificial delay to IO requests on a particular
+device, such that the requests take a minimum of 'latency'
+milliseconds to complete. Each delay has an associated
+number of 'lanes' which defines the number of concurrent
+IO requests that can be processed.
+
+For example, with a single lane delay of 10 ms (-D 10:1),
+the device will only be able to service a single IO request
+at a time with each request taking 10 ms to complete. So,
+if only a single request is submitted every 10 ms, the
+average latency will be 10 ms; but if more than one request
+is submitted every 10 ms, the average latency will be more
+than 10 ms.
+
+Similarly, if a delay of 10 ms is specified to have two
+lanes (-D 10:2), then the device will be able to service
+two requests at a time, each with a minimum latency of
+10 ms. So, if two requests are submitted every 10 ms, then
+the average latency will be 10 ms; but if more than two
+requests are submitted every 10 ms, the average latency
+will be more than 10 ms.
+
+Also note, these delays are additive. So two invocations
+of '-D 10:1', is roughly equivalent to a single invocation
+of '-D 10:2'. This also means, one can specify multiple
+lanes with differing target latencies. For example, an
+invocation of '-D 10:1' followed by '-D 25:2' will
+create 3 lanes on the device; one lane with a latency
+of 10 ms and two lanes with a 25 ms latency.
+
 .TP
 .B "zinject \-d \fIvdev\fB [\-e \fIdevice_error\fB] [\-L \fIlabel_error\fB] [\-T \fIfailure\fB] [\-F] \fIpool\fB"
 Force a vdev error.