Persistent L2ARC

This commit makes the L2ARC persistent across reboots. We implement
a light-weight persistent L2ARC metadata structure that allows L2ARC
contents to be recovered after a reboot. This significantly eases the
impact a reboot has on read performance on systems with large caches.

Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: George Wilson <gwilson@delphix.com>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Co-authored-by: Saso Kiselkov <skiselkov@gmail.com>
Co-authored-by: Jorgen Lundman <lundman@lundman.net>
Co-authored-by: George Amanakis <gamanakis@gmail.com>
Ported-by: Yuxuan Shui <yshuiv7@gmail.com>
Signed-off-by: George Amanakis <gamanakis@gmail.com>
Closes #925 
Closes #1823 
Closes #2672 
Closes #3744 
Closes #9582

man/man8/zdb.8

@@ -212,18 +212,24 @@ If specified multiple times, display counts of each intent log transaction type.
 Examine the checkpointed state of the pool.
 Note, the on disk format of the pool is not reverted to the checkpointed state.
 .It Fl l Ar device
-Read the vdev labels from the specified device.
+Read the vdev labels and L2ARC header from the specified device.
 .Nm Fl l
 will return 0 if valid label was found, 1 if error occurred, and 2 if no valid
-labels were found. Each unique configuration is displayed only once.
+labels were found. The presence of L2ARC header is indicated by a specific
+sequence (L2ARC_DEV_HDR_MAGIC). Each unique configuration is displayed only
+once.
 .It Fl ll Ar device
-In addition display label space usage stats.
+In addition display label space usage stats. If a valid L2ARC header was found
+also display the properties of log blocks used for restoring L2ARC contents
+(persistent L2ARC).
 .It Fl lll Ar device
-Display every configuration, unique or not.
+Display every configuration, unique or not. If a valid L2ARC header was found
+also display the properties of log entries in log blocks used for restoring
+L2ARC contents (persistent L2ARC).
 .Pp
 If the
 .Fl q
-option is also specified, don't print the labels.
+option is also specified, don't print the labels or the L2ARC header.
 .Pp
 If the
 .Fl u

man/man8/zpool-labelclear.8

@@ -48,7 +48,10 @@
 .Xc
 Removes ZFS label information from the specified
 .Ar device .
-The
+If the
+.Ar device
+is a cache device, it also removes the L2ARC header
+(persistent L2ARC). The
 .Ar device
 must not be part of an active pool configuration.
 .Bl -tag -width Ds

man/man8/zpoolconcepts.8

@@ -323,8 +323,28 @@ If a read error is encountered on a cache device, that read I/O is reissued to
 the original storage pool device, which might be part of a mirrored or raidz
 configuration.
 .Pp
-The content of the cache devices is considered volatile, as is the case with
-other system caches.
+The content of the cache devices is persistent across reboots and restored
+asynchronously when importing the pool in L2ARC (persistent L2ARC).
+This can be disabled by setting
+.Sy l2arc_rebuild_enabled = 0 .
+For cache devices smaller than 1GB we do not write the metadata structures
+required for rebuilding the L2ARC in order not to waste space. This can be
+changed with
+.Sy l2arc_rebuild_blocks_min_l2size .
+The cache device header (512 bytes) is updated even if no metadata structures
+are written. Setting
+.Sy l2arc_headroom = 0
+will result in scanning the full-length ARC lists for cacheable content to be
+written in L2ARC (persistent ARC). If a cache device is added with
+.Nm zpool Cm add
+its label and header will be overwritten and its contents are not going to be
+restored in L2ARC, even if the device was previously part of the pool. If a
+cache device is onlined with
+.Nm zpool Cm online
+its contents will be restored in L2ARC. This is useful in case of memory pressure
+where the contents of the cache device are not fully restored in L2ARC.
+The user can off/online the cache device when there is less memory pressure
+in order to fully restore its contents to L2ARC.
 .Ss Pool checkpoint
 Before starting critical procedures that include destructive actions (e.g
 .Nm zfs Cm destroy