c/mirror_zfs
1
0
Fork 0
mirror of https://git.proxmox.com/git/mirror_zfs.git synced 2024-12-26 03:09:34 +03:00
Code Issues Packages Projects Releases Wiki Activity
30af21b025
mirror_zfs/module/zfs
History
Paul Dagnelie 30af21b025 Implement Redacted Send/Receive 
Redacted send/receive allows users to send subsets of their data to 
a target system. One possible use case for this feature is to not 
transmit sensitive information to a data warehousing, test/dev, or 
analytics environment. Another is to save space by not replicating 
unimportant data within a given dataset, for example in backup tools 
like zrepl.

Redacted send/receive is a three-stage process. First, a clone (or 
clones) is made of the snapshot to be sent to the target. In this 
clone (or clones), all unnecessary or unwanted data is removed or
modified. This clone is then snapshotted to create the "redaction 
snapshot" (or snapshots). Second, the new zfs redact command is used 
to create a redaction bookmark. The redaction bookmark stores the 
list of blocks in a snapshot that were modified by the redaction 
snapshot(s). Finally, the redaction bookmark is passed as a parameter 
to zfs send. When sending to the snapshot that was redacted, the
redaction bookmark is used to filter out blocks that contain sensitive 
or unwanted information, and those blocks are not included in the send 
stream.  When sending from the redaction bookmark, the blocks it 
contains are considered as candidate blocks in addition to those 
blocks in the destination snapshot that were modified since the 
creation_txg of the redaction bookmark.  This step is necessary to 
allow the target to rehydrate data in the case where some blocks are 
accidentally or unnecessarily modified in the redaction snapshot.

The changes to bookmarks to enable fast space estimation involve 
adding deadlists to bookmarks. There is also logic to manage the 
life cycles of these deadlists.

The new size estimation process operates in cases where previously 
an accurate estimate could not be provided. In those cases, a send 
is performed where no data blocks are read, reducing the runtime 
significantly and providing a byte-accurate size estimate.

Reviewed-by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Prashanth Sreenivasa <pks@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Chris Williamson <chris.williamson@delphix.com>
Reviewed-by: Pavel Zhakarov <pavel.zakharov@delphix.com>
Reviewed-by: Sebastien Roy <sebastien.roy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #7958
2019-06-19 09:48:12 -07:00
..
abd.c single-chunk scatter ABDs can be treated as linear 2019-06-11 09:02:31 -07:00
aggsum.c OpenZFS 9688 - aggsum_fini leaks memory 2018-10-19 12:08:03 -07:00
arc.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
blkptr.c Undo c89 workarounds to match with upstream 2017-11-04 13:25:13 -07:00
bplist.c Change KM_PUSHPAGE -> KM_SLEEP 2015-01-16 14:41:26 -08:00
bpobj.c Stack overflow in recursive bpobj_iterate_impl 2019-03-06 09:50:55 -08:00
bptree.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
bqueue.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
cityhash.c OpenZFS 8484 - Implement aggregate sum and use for arc counters 2018-06-06 09:35:59 -07:00
dataset_kstats.c port async unlinked drain from illumos-nexenta 2019-02-12 10:41:15 -08:00
dbuf_stats.c Prefix all refcount functions with zfs_ 2018-10-01 10:42:05 -07:00
dbuf.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
ddt_zap.c fat zap should prefetch when iterating 2019-06-12 13:13:09 -07:00
ddt.c ztest: scrub ddt repair 2019-01-17 15:25:00 -08:00
dmu_diff.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
dmu_object.c Fix send/recv lost spill block 2019-05-07 15:18:44 -07:00
dmu_objset.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
dmu_recv.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
dmu_redact.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
dmu_send.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
dmu_traverse.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
dmu_tx.c Prefix all refcount functions with zfs_ 2018-10-01 10:42:05 -07:00
dmu_zfetch.c Linux 5.2 compat: rw_tryupgrade() 2019-05-23 13:46:33 -07:00
dmu.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
dnode_sync.c Reinstate raw receive check when truncating 2019-06-06 13:47:33 -07:00
dnode.c Fix incorrect assertion in dnode_dirty_l1range 2019-05-19 17:30:33 -07:00
dsl_bookmark.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
dsl_crypt.c Better user experience for errata 4 2019-03-14 16:48:30 -07:00
dsl_dataset.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
dsl_deadlist.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
dsl_deleg.c Update build system and packaging 2018-05-29 16:00:33 -07:00
dsl_destroy.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
dsl_dir.c Fix TXG_MASK cstyle 2019-04-12 11:30:59 -07:00
dsl_pool.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
dsl_prop.c Update build system and packaging 2018-05-29 16:00:33 -07:00
dsl_scan.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
dsl_synctask.c OpenZFS 9166 - zfs storage pool checkpoint 2018-06-26 10:07:42 -07:00
dsl_userhold.c zfs should optionally send holds 2019-02-15 12:41:38 -08:00
edonr_zfs.c DLPX-44812 integrate EP-220 large memory scalability 2016-11-29 14:34:27 -08:00
fm.c OpenZFS 9580 - Add a hash-table on top of nvlist to speed-up operations 2018-07-30 11:30:03 -07:00
gzip.c Update build system and packaging 2018-05-29 16:00:33 -07:00
hkdf.c Encryption patch follow-up 2017-10-11 16:54:48 -04:00
lz4.c Reword comment in lz4_compress_zfs 2019-05-02 16:46:04 -07:00
lzjb.c Change KM_PUSHPAGE -> KM_SLEEP 2015-01-16 14:41:26 -08:00
Makefile.in Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
metaslab.c looping in metaslab_block_picker impacts performance on fragmented pools 2019-06-13 13:06:15 -07:00
mmp.c MMP interval and fail_intervals in uberblock 2019-03-21 12:47:57 -07:00
multilist.c Update build system and packaging 2018-05-29 16:00:33 -07:00
objlist.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
pathname.c Update build system and packaging 2018-05-29 16:00:33 -07:00
policy.c Take user namespaces into account in policy checks 2018-03-07 15:40:42 -08:00
qat_compress.c Code improvement and bug fixes for QAT support 2019-04-16 12:38:36 -07:00
qat_crypt.c Code improvement and bug fixes for QAT support 2019-04-16 12:38:36 -07:00
qat.c Code improvement and bug fixes for QAT support 2019-04-16 12:38:36 -07:00
qat.h Code improvement and bug fixes for QAT support 2019-04-16 12:38:36 -07:00
range_tree.c Restrict kstats and print real pointers 2019-04-04 18:57:06 -07:00
refcount.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
rrwlock.c Prefix all refcount functions with zfs_ 2018-10-01 10:42:05 -07:00
sa.c Prefix all refcount functions with zfs_ 2018-10-01 10:42:05 -07:00
sha256.c SHA256 QAT acceleration 2018-03-15 10:53:58 -07:00
skein_zfs.c DLPX-44812 integrate EP-220 large memory scalability 2016-11-29 14:34:27 -08:00
spa_boot.c Add linux kernel module support 2010-08-31 13:41:58 -07:00
spa_checkpoint.c Get rid of space_map_update() for ms_synced_length 2019-02-12 10:38:11 -08:00
spa_config.c Remove vn_set_fs_pwd()/vn_set_pwd() (no need to be at / during insmod) 2019-05-29 16:18:14 -07:00
spa_errlog.c Update build system and packaging 2018-05-29 16:00:33 -07:00
spa_history.c Create /proc/sys/kernel/spl/gitrev with git hash 2018-10-08 21:57:02 -07:00
spa_misc.c Fix coverity defects: CID 186143 2019-05-23 19:17:00 -07:00
spa_stats.c Restrict kstats and print real pointers 2019-04-04 18:57:06 -07:00
spa.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
space_map.c Restrict kstats and print real pointers 2019-04-04 18:57:06 -07:00
space_reftree.c OpenZFS 7614, 9064 - zfs device evacuation/removal 2018-04-14 12:16:17 -07:00
THIRDPARTYLICENSE.cityhash OpenZFS 8484 - Implement aggregate sum and use for arc counters 2018-06-06 09:35:59 -07:00
THIRDPARTYLICENSE.cityhash.descrip OpenZFS 8484 - Implement aggregate sum and use for arc counters 2018-06-06 09:35:59 -07:00
trace.c OpenZFS 7614, 9064 - zfs device evacuation/removal 2018-04-14 12:16:17 -07:00
txg.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
uberblock.c MMP interval and fail_intervals in uberblock 2019-03-21 12:47:57 -07:00
unique.c Performance optimization of AVL tree comparator functions 2016-08-31 14:35:34 -07:00
vdev_cache.c Update build system and packaging 2018-05-29 16:00:33 -07:00
vdev_disk.c Improve rate at which new zvols are processed 2019-05-04 16:39:10 -07:00
vdev_file.c Use SEEK_{SET,CUR,END} for file seek "whence" 2019-04-25 10:17:27 -07:00
vdev_indirect_births.c Update build system and packaging 2018-05-29 16:00:33 -07:00
vdev_indirect_mapping.c Get rid of space_map_update() for ms_synced_length 2019-02-12 10:38:11 -08:00
vdev_indirect.c Device removal panics on 32-bit systems 2019-05-24 12:17:52 -07:00
vdev_initialize.c Add TRIM support 2019-03-29 09:13:20 -07:00
vdev_label.c panic in removal_remap test on 4K devices 2019-06-13 13:12:39 -07:00
vdev_mirror.c Cleanup nits from ab7615d92 2019-04-14 11:03:06 -07:00
vdev_missing.c OpenZFS 9102 - zfs should be able to initialize storage devices 2019-01-07 10:37:26 -08:00
vdev_queue.c Move write aggregation memory copy out of vq_lock 2019-06-13 13:08:24 -07:00
vdev_raidz_math_aarch64_neon_common.h Linux 5.0 compat: ASM_BUG macro 2019-05-08 10:18:40 -07:00
vdev_raidz_math_aarch64_neon.c codebase style improvements for OpenZFS 6459 port 2017-01-22 13:25:40 -08:00
vdev_raidz_math_aarch64_neonx2.c ABD raidz NEON support 2016-11-29 14:34:33 -08:00
vdev_raidz_math_avx2.c Linux 5.0 compat: ASM_BUG macro 2019-05-08 10:18:40 -07:00
vdev_raidz_math_avx512bw.c Linux 5.0 compat: ASM_BUG macro 2019-05-08 10:18:40 -07:00
vdev_raidz_math_avx512f.c Use cstyle -cpP in make cstyle check 2016-12-12 10:46:26 -08:00
vdev_raidz_math_impl.h codebase style improvements for OpenZFS 6459 port 2017-01-22 13:25:40 -08:00
vdev_raidz_math_scalar.c ABD Vectorized raidz 2016-11-29 14:34:33 -08:00
vdev_raidz_math_sse2.c ABD raidz avx512f support 2016-11-29 14:34:33 -08:00
vdev_raidz_math_ssse3.c Linux 5.0 compat: ASM_BUG macro 2019-05-08 10:18:40 -07:00
vdev_raidz_math.c Fix typo in vdev_raidz_math.c 2019-06-12 13:03:33 -07:00
vdev_raidz.c Add TRIM support 2019-03-29 09:13:20 -07:00
vdev_removal.c panic in removal_remap test on 4K devices 2019-06-13 13:12:39 -07:00
vdev_root.c OpenZFS 9102 - zfs should be able to initialize storage devices 2019-01-07 10:37:26 -08:00
vdev_trim.c Add TRIM support 2019-03-29 09:13:20 -07:00
vdev.c Allow metaslab to be unloaded even when not freed from 2019-06-06 19:10:43 -07:00
zap_leaf.c Off-by-one in zap_leaf_array_create() 2019-01-18 09:58:46 -08:00
zap_micro.c fat zap should prefetch when iterating 2019-06-12 13:13:09 -07:00
zap.c fat zap should prefetch when iterating 2019-06-12 13:13:09 -07:00
zcp_get.c Detect and prevent mixed raw and non-raw sends 2019-03-13 11:00:43 -07:00
zcp_global.c OpenZFS 8600 - ZFS channel programs - snapshot 2018-02-08 15:29:24 -08:00
zcp_iter.c OpenZFS 9337 - zfs get all is slow due to uncached metadata 2018-07-12 10:49:27 -07:00
zcp_synctask.c OpenZFS 9166 - zfs storage pool checkpoint 2018-06-26 10:07:42 -07:00
zcp.c OpenZFS 9424 - ztest failure: "unprotected error in call to Lua API (Invalid value type 'function' for key 'error')" 2018-07-10 21:29:23 -07:00
zfeature.c Consistently captialize GUID for features 2019-04-16 10:01:51 -07:00
zfs_acl.c Update build system and packaging 2018-05-29 16:00:33 -07:00
zfs_byteswap.c Update build system and packaging 2018-05-29 16:00:33 -07:00
zfs_ctldir.c Drop local definition of MOUNT_BUSY 2019-05-24 16:43:23 -07:00
zfs_debug.c Restrict kstats and print real pointers 2019-04-04 18:57:06 -07:00
zfs_dir.c port async unlinked drain from illumos-nexenta 2019-02-12 10:41:15 -08:00
zfs_fm.c Add zpool status -s (slow I/Os) and -p (parseable) 2018-11-08 16:47:24 -08:00
zfs_fuid.c Update build system and packaging 2018-05-29 16:00:33 -07:00
zfs_ioctl.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
zfs_log.c make zil max block size tunable 2019-06-10 11:48:42 -07:00
zfs_onexit.c Update build system and packaging 2018-05-29 16:00:33 -07:00
zfs_ratelimit.c Change checksum & IO delay ratelimit values 2018-03-04 17:34:51 -08:00
zfs_replay.c Use SEEK_{SET,CUR,END} for file seek "whence" 2019-04-25 10:17:27 -07:00
zfs_rlock.c OpenZFS 9689 - zfs range lock code should not be zpl-specific 2018-10-11 10:19:33 -07:00
zfs_sa.c Project Quota on ZFS 2018-02-13 14:54:54 -08:00
zfs_sysfs.c Fix lockdep warning on insmod 2019-06-12 17:15:06 -07:00
zfs_vfsops.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
zfs_vnops.c Update descriptions for vnops 2019-05-25 14:29:10 -07:00
zfs_znode.c Fix integer overflow of ZTOI(zp)->i_generation 2019-06-06 12:59:39 -07:00
zil.c make zil max block size tunable 2019-06-10 11:48:42 -07:00
zio_checksum.c Undo c89 workarounds to match with upstream 2017-11-04 13:25:13 -07:00
zio_compress.c OpenZFS 9403 - assertion failed in arc_buf_destroy() 2018-08-29 11:33:33 -07:00
zio_crypt.c Always call rw_init in zio_crypt_key_unwrap 2019-04-10 15:39:40 -07:00
zio_inject.c Multiple DVA Scrubbing Fix 2019-03-15 14:14:31 -07:00
zio.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
zle.c Fix zle_decompress out of bound access 2018-02-09 10:08:05 -08:00
zpl_ctldir.c RHEL 7.5 compat: FMODE_KABI_ITERATE 2018-05-02 15:01:24 -07:00
zpl_export.c Use cstyle -cpP in make cstyle check 2016-12-12 10:46:26 -08:00
zpl_file.c Fix errant EFAULT during writes (#8719) 2019-05-08 10:04:04 -07:00
zpl_inode.c Fix errant EFAULT during writes (#8719) 2019-05-08 10:04:04 -07:00
zpl_super.c Fix statfs(2) for 32-bit user space 2018-09-24 17:11:25 -07:00
zpl_xattr.c Add missing checks to zpl_xattr_* functions 2018-08-02 14:03:56 -07:00
zrlock.c Update build system and packaging 2018-05-29 16:00:33 -07:00
zthr.c Fix txg_wait_open() load average inflation 2019-04-04 09:44:46 -07:00
zvol.c make zil max block size tunable 2019-06-10 11:48:42 -07:00