c/mirror_zfs
1
0
Fork 0
mirror of https://git.proxmox.com/git/mirror_zfs.git synced 2024-12-26 03:09:34 +03:00
Code Issues Packages Projects Releases Wiki Activity
f4aeb23f52
mirror_zfs/module/zfs
History
Rob Norris f4aeb23f52 ddt: add "flat phys" feature 
Traditional dedup keeps a separate ddt_phys_t "type" for each possible
count of DVAs (that is, copies=) parameter. Each of these are tracked
independently of each other, and have their own set of DVAs. This leads
to an (admittedly rare) situation where you can create as many as six
copies of the data, by changing the copies= parameter between copying.
This is both a waste of storage on disk, but also a waste of space in
the stored DDT entries, since there never needs to be more than three
DVAs to handle all possible values of copies=.

This commit adds a new FDT feature, DDT_FLAG_FLAT. When active, only the
first ddt_phys_t is used. Each time a block is written with the dedup
bit set, this single phys is checked to see if it has enough DVAs to
fulfill the request. If it does, the block is filled with the saved DVAs
as normal. If not, an adjusted write is issued to create as many extra
copies as are needed to fulfill the request, which are then saved into
the entry too.

Because a single phys is no longer an all-or-nothing, but can be
transitioning from fewer to more DVAs, the write path now has to keep a
copy of the previous "known good" DVA set so we can revert to it in case
an error occurs. zio_ddt_write() has been restructured and heavily
commented to make it much easier to see what's happening.

Backwards compatibility is maintained simply by allocating four
ddt_phys_t when the DDT_FLAG_FLAT flag is not set, and updating the phys
selection macros to check the flag. In the old arrangement, each number
of copies gets a whole phys, so it will always have either zero or all
necessary DVAs filled, with no in-between, so the old behaviour
naturally falls out of the new code.

Reviewed-by: Alexander Motin <mav@FreeBSD.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Rob Norris <rob.norris@klarasystems.com>
Co-authored-by: Don Brady <don.brady@klarasystems.com>
Sponsored-by: Klara, Inc.
Sponsored-by: iXsystems, Inc.
Closes #15893
2024-08-16 12:02:39 -07:00
..
abd.c abd: lift ABD zero scan from zio_compress_data() to abd_cmp_zero() 2024-08-09 14:30:26 -07:00
aggsum.c Remove bcopy(), bzero(), bcmp() 2022-03-15 15:13:42 -07:00
arc.c Soften pruning threshold on not evictable metadata 2024-08-08 15:26:35 -07:00
blake3_zfs.c blake3: fix up bogus checksums in face of cpu migration 2023-05-01 17:21:27 -07:00
blkptr.c Remove bcopy(), bzero(), bcmp() 2022-03-15 15:13:42 -07:00
bplist.c Use list_remove_head() where possible. 2023-06-09 10:12:52 -07:00
bpobj.c Provide macros for setting and getting blkptr birth times 2024-03-25 15:01:54 -07:00
bptree.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
bqueue.c Batch enqueue/dequeue for bqueue 2023-01-10 13:39:22 -08:00
brt.c BRT: Skip getting length in brt_entry_lookup() 2024-03-25 17:13:45 -07:00
btree.c Replace P2ALIGN with P2ALIGN_TYPED and delete P2ALIGN. 2024-05-10 08:47:21 -07:00
dataset_kstats.c Fix null ptr deref when renaming a zvol with snaps and snapdev=visible (#16316) 2024-08-15 14:13:18 -07:00
dbuf_stats.c Consider dnode_t allocations in dbuf cache size accounting 2023-11-17 13:25:53 -08:00
dbuf.c Block cloning conditionally destroy ARC buffer 2024-08-01 18:22:43 -07:00
ddt_stats.c ddt: add "flat phys" feature 2024-08-16 12:02:39 -07:00
ddt_zap.c ddt: add "flat phys" feature 2024-08-16 12:02:39 -07:00
ddt.c ddt: add "flat phys" feature 2024-08-16 12:02:39 -07:00
dmu_diff.c Simplify issig(). 2024-05-29 10:49:11 -07:00
dmu_object.c Replace P2ALIGN with P2ALIGN_TYPED and delete P2ALIGN. 2024-05-10 08:47:21 -07:00
dmu_objset.c Simplify issig(). 2024-05-29 10:49:11 -07:00
dmu_recv.c Simplify issig(). 2024-05-29 10:49:11 -07:00
dmu_redact.c Simplify issig(). 2024-05-29 10:49:11 -07:00
dmu_send.c Simplify issig(). 2024-05-29 10:49:11 -07:00
dmu_traverse.c Provide macros for setting and getting blkptr birth times 2024-03-25 15:01:54 -07:00
dmu_tx.c Cleanup DB_DNODE() macros usage 2024-07-29 14:47:01 -07:00
dmu_zfetch.c Small fix to prefetch ranges aggregation 2024-04-09 16:14:04 -07:00
dmu.c dnode: allow storage class to be overridden by object type 2024-07-29 17:05:41 -07:00
dnode_sync.c For db_marker inherit the db pointer for AVL comparision. 2023-12-11 14:42:06 -08:00
dnode.c Linux: Report reclaimable memory to kernel as such (#16385) 2024-07-30 11:40:47 -07:00
dsl_bookmark.c Provide macros for setting and getting blkptr birth times 2024-03-25 15:01:54 -07:00
dsl_crypt.c Allow block cloning across encrypted datasets 2023-12-05 11:03:48 -08:00
dsl_dataset.c head_errlog: fix use-after-free 2024-07-15 09:05:42 -07:00
dsl_deadlist.c Provide macros for setting and getting blkptr birth times 2024-03-25 15:01:54 -07:00
dsl_deleg.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
dsl_destroy.c Provide macros for setting and getting blkptr birth times 2024-03-25 15:01:54 -07:00
dsl_dir.c Fix ENOSPC for extended quota 2023-09-28 14:10:07 -07:00
dsl_pool.c Provide macros for setting and getting blkptr birth times 2024-03-25 15:01:54 -07:00
dsl_prop.c nvpair: Constify string functions 2023-03-14 15:25:50 -07:00
dsl_scan.c ddt: add "flat phys" feature 2024-08-16 12:02:39 -07:00
dsl_synctask.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
dsl_userhold.c nvpair: Constify string functions 2023-03-14 15:25:50 -07:00
edonr_zfs.c Remove unused Edon-R variants 2023-03-14 15:59:58 -07:00
fm.c Use list_remove_head() where possible. 2023-06-09 10:12:52 -07:00
gzip.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
hkdf.c Remove bcopy(), bzero(), bcmp() 2022-03-15 15:13:42 -07:00
lz4_zfs.c Linux: Report reclaimable memory to kernel as such (#16385) 2024-07-30 11:40:47 -07:00
lz4.c lz4: Cherrypick fix for CVE-2021-3520 2022-01-12 16:14:36 -08:00
lzjb.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
metaslab.c Some improvements to metaslabs eviction 2024-05-29 08:53:31 -07:00
mmp.c vdev probe to slow disk can stall mmp write checker 2024-04-29 14:35:53 -07:00
multilist.c L2ARC: Relax locking during write 2024-04-09 16:23:19 -07:00
objlist.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
pathname.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
range_tree.c btree: Implement faster binary search algorithm 2023-05-26 10:03:12 -07:00
refcount.c Switch refcount tracking from lists to AVL-trees. 2023-06-14 08:02:27 -07:00
rrwlock.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
sa.c Fix sa.c to build on FreeBSD again. (#16403) 2024-08-01 13:04:08 -07:00
sha2_zfs.c Add generic implementation handling and SHA2 impl 2023-03-02 13:52:21 -08:00
skein_zfs.c Remove bcopy(), bzero(), bcmp() 2022-03-15 15:13:42 -07:00
spa_checkpoint.c RAID-Z expansion feature 2023-11-08 10:19:41 -08:00
spa_config.c Add mutex_enter_interruptible() for interruptible sleeping IOCTLs 2023-10-26 09:17:40 -07:00
spa_errlog.c Provide macros for setting and getting blkptr birth times 2024-03-25 15:01:54 -07:00
spa_history.c nvpair: Constify string functions 2023-03-14 15:25:50 -07:00
spa_log_spacemap.c Provide macros for setting and getting blkptr birth times 2024-03-25 15:01:54 -07:00
spa_misc.c dnode: allow storage class to be overridden by object type 2024-07-29 17:05:41 -07:00
spa_stats.c Cleanup: Specify unsignedness on things that should not be signed 2022-09-27 16:42:41 -07:00
spa.c Fix incorrect error report on vdev attach/replace 2024-08-15 12:39:44 -07:00
space_map.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
space_reftree.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
THIRDPARTYLICENSE.cityhash OpenZFS 8484 - Implement aggregate sum and use for arc counters 2018-06-06 09:35:59 -07:00
THIRDPARTYLICENSE.cityhash.descrip OpenZFS 8484 - Implement aggregate sum and use for arc counters 2018-06-06 09:35:59 -07:00
txg.c vdev probe to slow disk can stall mmp write checker 2024-04-29 14:35:53 -07:00
uberblock.c Provide macros for setting and getting blkptr birth times 2024-03-25 15:01:54 -07:00
unique.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
vdev_draid_rand.c Distributed Spare (dRAID) Feature 2020-11-13 13:51:51 -08:00
vdev_draid.c zio: rename ZIO_TYPE_IOCTL to ZIO_TYPE_FLUSH 2024-04-11 17:17:23 -07:00
vdev_indirect_births.c Remove bcopy(), bzero(), bcmp() 2022-03-15 15:13:42 -07:00
vdev_indirect_mapping.c Remove bcopy(), bzero(), bcmp() 2022-03-15 15:13:42 -07:00
vdev_indirect.c Don't emit cksum_{actual_expected} in ereport.fs.zfs.checksum events 2023-07-21 11:49:26 -07:00
vdev_initialize.c Add support for parallel pool exports 2024-05-14 08:57:41 -07:00
vdev_label.c Sync AUX label during pool import 2024-08-08 15:16:46 -07:00
vdev_mirror.c Provide macros for setting and getting blkptr birth times 2024-03-25 15:01:54 -07:00
vdev_missing.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
vdev_queue.c Fix accounting error for pending sync IO ops in zpool iostat 2023-11-07 09:06:14 -08:00
vdev_raidz_math_aarch64_neon_common.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
vdev_raidz_math_aarch64_neon.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
vdev_raidz_math_aarch64_neonx2.c Fix Clang 15 compilation errors 2022-11-30 13:46:26 -08:00
vdev_raidz_math_avx2.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
vdev_raidz_math_avx512bw.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
vdev_raidz_math_avx512f.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
vdev_raidz_math_impl.h RAIDZ: Use cache blocking during parity math 2023-10-30 14:54:27 -07:00
vdev_raidz_math_powerpc_altivec_common.h Linux ppc64le ieee128 compat: Do not redefine __asm on external headers 2023-01-13 10:58:58 -08:00
vdev_raidz_math_powerpc_altivec.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
vdev_raidz_math_scalar.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
vdev_raidz_math_sse2.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
vdev_raidz_math_ssse3.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
vdev_raidz_math.c Miscellaneous FreBSD compilation bugfixes 2023-04-06 10:35:02 -07:00
vdev_raidz.c Replace P2ALIGN with P2ALIGN_TYPED and delete P2ALIGN. 2024-05-10 08:47:21 -07:00
vdev_rebuild.c Add support for parallel pool exports 2024-05-14 08:57:41 -07:00
vdev_removal.c nvpair: Constify string functions 2023-03-14 15:25:50 -07:00
vdev_root.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
vdev_trim.c Add support for parallel pool exports 2024-05-14 08:57:41 -07:00
vdev.c Add some missing vdev properties (#16346) 2024-07-23 16:34:09 -07:00
zap_leaf.c ZAP: Some cleanups/micro-optimizations 2024-03-21 16:43:53 -07:00
zap_micro.c ddt: add support for prefetching tables into the ARC 2024-07-26 09:16:18 -07:00
zap.c zap: reuse zap_leaf_t on dbuf reuse after shrink 2024-05-24 18:55:47 -07:00
zcp_get.c nvpair: Constify string functions 2023-03-14 15:25:50 -07:00
zcp_global.c OpenZFS 8600 - ZFS channel programs - snapshot 2018-02-08 15:29:24 -08:00
zcp_iter.c nvpair: Constify string functions 2023-03-14 15:25:50 -07:00
zcp_set.c Support setting user properties in a channel program 2020-02-14 13:41:42 -08:00
zcp_synctask.c Add zfs.sync.snapshot_rename 2022-09-02 13:31:19 -07:00
zcp.c Simplify issig(). 2024-05-29 10:49:11 -07:00
zfeature.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zfs_byteswap.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zfs_chksum.c Add generic implementation handling and SHA2 impl 2023-03-02 13:52:21 -08:00
zfs_fm.c zio: remove io_cmd and DKIOCFLUSHWRITECACHE 2024-04-11 17:17:11 -07:00
zfs_fuid.c Use list_remove_head() where possible. 2023-06-09 10:12:52 -07:00
zfs_impl.c Add generic implementation handling and SHA2 impl 2023-03-02 13:52:21 -08:00
zfs_ioctl.c ddt: add support for prefetching tables into the ARC 2024-07-26 09:16:18 -07:00
zfs_log.c Cleanup DB_DNODE() macros usage 2024-07-29 14:47:01 -07:00
zfs_onexit.c Use list_remove_head() where possible. 2023-06-09 10:12:52 -07:00
zfs_quota.c Revert "Do not persist user/group/project quota zap objects when unneeded" 2023-10-23 09:55:36 -07:00
zfs_ratelimit.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zfs_replay.c FreeBSD: remove support for FreeBSD < 13.0-RELEASE (#16372) 2024-08-05 16:56:45 -07:00
zfs_rlock.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zfs_sa.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zfs_vnops.c Simplify issig(). 2024-05-29 10:49:11 -07:00
zil.c zfs: add bounds checking to zil_parse (#16308) 2024-07-31 17:17:04 -07:00
zio_checksum.c Provide macros for setting and getting blkptr birth times 2024-03-25 15:01:54 -07:00
zio_compress.c ddt: add FDT feature and support for legacy and new on-disk formats 2024-08-16 11:58:59 -07:00
zio_inject.c vdev probe to slow disk can stall mmp write checker 2024-04-29 14:35:53 -07:00
zio.c ddt: add "flat phys" feature 2024-08-16 12:02:39 -07:00
zle.c Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zrlock.c Micro-optimize zrl_remove() 2022-11-29 09:26:03 -08:00
zthr.c Switch from _Noreturn to __attribute__((noreturn)) 2022-03-23 08:51:00 -07:00
zvol.c zvol: ensure device minors are properly cleaned up 2024-08-06 12:08:14 -07:00