Implement a new type of zfs receive: corrective receive (-c)

This type of recv is used to heal corrupted data when a replica
of the data already exists (in the form of a send file for example).
With the provided send stream, corrective receive will read from
disk blocks described by the WRITE records. When any of the reads
come back with ECKSUM we use the data from the corresponding WRITE
record to rewrite the corrupted block.

Reviewed-by: Paul Dagnelie <pcd@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Paul Zuchowski <pzuchowski@datto.com>
Signed-off-by: Alek Pinchuk <apinchuk@axcient.com>
Closes #9372

man/man4/zfs.4

@@ -1870,6 +1870,17 @@ This setting will not reduce the write size below a single block.
 Capped at a maximum of
 .Sy 32 MiB .
 .
+.It Sy zfs_recv_best_effort_corrective Ns = Ns Sy 0 Pq int
+When this variable is set to non-zero a corrective receive:
+.Bl -enum -compact -offset 4n -width "1."
+.It
+Does not enforce the restriction of source & destination snapshot GUIDs
+matching.
+.It
+If there is an error during healing, the healing receive is not
+terminated instead it moves on to the next record.
+.El
+.
 .It Sy zfs_override_estimate_recordsize Ns = Ns Sy 0 Ns | Ns 1 Pq ulong
 Setting this variable overrides the default logic for estimating block
 sizes when doing a

man/man8/zfs-receive.8

@@ -29,7 +29,7 @@
 .\" Copyright 2018 Nexenta Systems, Inc.
 .\" Copyright 2019 Joyent, Inc.
 .\"
-.Dd March 16, 2022
+.Dd April 26, 2022
 .Dt ZFS-RECEIVE 8
 .Os
 .
@@ -57,6 +57,12 @@
 .Fl A
 .Ar filesystem Ns | Ns Ar volume
 .
+.Nm
+.Cm receive
+.Fl c
+.Op Fl vn
+.Ar filesystem Ns | Ns Ar snapshot
+.
 .Sh DESCRIPTION
 .Bl -tag -width ""
 .It Xo
@@ -393,6 +399,24 @@ restrictions (e.g. set-once) apply equally to
 Abort an interrupted
 .Nm zfs Cm receive Fl s ,
 deleting its saved partially received state.
+.It Xo
+.Nm zfs
+.Cm receive
+.Fl c
+.Op Fl vn
+.Ar filesystem Ns | Ns Ar snapshot
+.Xc
+Attempt to correct data corruption in the specified dataset,
+by using the provided stream as the source of healthy data.
+This method of healing can only heal data blocks present in the stream.
+Metadata can not be healed by corrective receive.
+Running a scrub is recommended post-healing to ensure all corruption was
+healed.
+.Pp
+It's important to consider why corruption has happened in the first place
+since if you have slowly failing hardware periodically healing the data
+is not going to save you from data loss later on when the hardware fails
+completely.
 .El
 .
 .Sh EXAMPLES