Make zc_nvlist_src_size limit tunable

We limit the size of nvlists passed to the kernel so a user cannot make
the kernel do an unreasonably large allocation.  On FreeBSD this limit
was 128 kiB, which turns out to be a bit too small when doing some
operations involving a large number of datasets or snapshots, for
example replication.

Make this limit tunable, with a platform-specific auto default.
Linux keeps its limit at KMALLOC_MAX_SIZE. FreeBSD uses 1/4 of the
system limit on user wired memory, which allows it to scale depending
on system configuration.

Reviewed-by: Matt Macy <mmacy@FreeBSD.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Ryan Moeller <freqlabs@FreeBSD.org>
Issue #6572 
Closes #10706

man/man5/zfs-module-parameters.5

@@ -1074,6 +1074,23 @@ pool import (only in read-only mode).
 Default value: \fB0\fR
 .RE
 
+.sp
+.ne 2
+.na
+\fBzfs_max_nvlist_src_size\fR (ulong)
+.ad
+.RS 12n
+Maximum size in bytes allowed to be passed as zc_nvlist_src_size for ioctls on
+/dev/zfs. This prevents a user from causing the kernel to allocate an excessive
+amount of memory. When the limit is exceeded, the ioctl fails with EINVAL and a
+description of the error is sent to the zfs-dbgmsg log. This parameter should
+not need to be touched under normal circumstances. On FreeBSD, the default is
+based on the system limit on user wired memory. On Linux, the default is
+\fBKMALLOC_MAX_SIZE\fR .
+.sp
+Default value: \fB0\fR (kernel decides)
+.RE
+
 .sp
 .ne 2
 .na