Alexander Motin 734eba251d Wire O_DIRECT also to Uncached I/O (#17218) ... Before Direct I/O was implemented, I've implemented lighter version I called Uncached I/O. It uses normal DMU/ARC data path with some optimizations, but evicts data from caches as soon as possible and reasonable. Originally I wired it only to a primarycache property, but now completing the integration all the way up to the VFS. While Direct I/O has the lowest possible memory bandwidth usage, it also has a significant number of limitations. It require I/Os to be page aligned, does not allow speculative prefetch, etc. The Uncached I/O does not have those limitations, but instead require additional memory copy, though still one less than regular cached I/O. As such it should fill the gap in between. Considering this I've disabled annoying EINVAL errors on misaligned requests, adding a tunable for those who wants to test their applications. To pass the information between the layers I had to change a number of APIs. But as side effect upper layers can now control not only the caching, but also speculative prefetch. I haven't wired it to VFS yet, since it require looking on some OS specifics. But while there I've implemented speculative prefetch of indirect blocks for Direct I/O, controllable via all the same mechanisms. Signed-off-by: Alexander Motin <mav@FreeBSD.org> Sponsored by: iXsystems, Inc. Fixes #17027 Reviewed-by: Rob Norris <robn@despairlabs.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>		2025-05-13 14:26:55 -07:00
..
abd.c	Export correct symbols for Lustre Direct I/O	2025-04-24 13:55:21 -04:00
aggsum.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
arc.c	Wire O_DIRECT also to Uncached I/O (#17218)	2025-05-13 14:26:55 -07:00
blake3_zfs.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
blkptr.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
bplist.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
bpobj.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
bptree.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
bqueue.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
brt.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
btree.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
dataset_kstats.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
dbuf_stats.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
dbuf.c	Wire O_DIRECT also to Uncached I/O (#17218)	2025-05-13 14:26:55 -07:00
ddt_log.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
ddt_stats.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
ddt_zap.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
ddt.c	Handle interaction between gang blocks, copies, and FDT.	2025-04-21 11:26:30 -04:00
dmu_diff.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
dmu_direct.c	Wire O_DIRECT also to Uncached I/O (#17218)	2025-05-13 14:26:55 -07:00
dmu_object.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
dmu_objset.c	Wire O_DIRECT also to Uncached I/O (#17218)	2025-05-13 14:26:55 -07:00
dmu_recv.c	Wire O_DIRECT also to Uncached I/O (#17218)	2025-05-13 14:26:55 -07:00
dmu_redact.c	dmu_tx: rename dmu_tx_assign() flags from TXG_* to DMU_TX_* (#17143)	2025-03-18 16:04:22 -07:00
dmu_send.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
dmu_traverse.c	Wire O_DIRECT also to Uncached I/O (#17218)	2025-05-13 14:26:55 -07:00
dmu_tx.c	Wire O_DIRECT also to Uncached I/O (#17218)	2025-05-13 14:26:55 -07:00
dmu_zfetch.c	Wire O_DIRECT also to Uncached I/O (#17218)	2025-05-13 14:26:55 -07:00
dmu.c	Wire O_DIRECT also to Uncached I/O (#17218)	2025-05-13 14:26:55 -07:00
dnode_sync.c	Wire O_DIRECT also to Uncached I/O (#17218)	2025-05-13 14:26:55 -07:00
dnode.c	Wire O_DIRECT also to Uncached I/O (#17218)	2025-05-13 14:26:55 -07:00
dsl_bookmark.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
dsl_crypt.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
dsl_dataset.c	cred: properly pass and test creds on other threads (#17273)	2025-04-29 16:27:48 -07:00
dsl_deadlist.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
dsl_deleg.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
dsl_destroy.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
dsl_dir.c	cred: properly pass and test creds on other threads (#17273)	2025-04-29 16:27:48 -07:00
dsl_pool.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
dsl_prop.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
dsl_scan.c	Fix race between resilver wait and offline/detach	2025-05-02 15:19:24 -07:00
dsl_synctask.c	txg: generalise txg_wait_synced_sig() to txg_wait_synced_flags() (#17284)	2025-05-02 15:29:50 -07:00
dsl_userhold.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
edonr_zfs.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
fm.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
gzip.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
hkdf.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
lz4_zfs.c	SPDX: license tags: BSD-2-Clause	2025-03-13 17:56:46 -07:00
lz4.c	SPDX: license tags: BSD-2-Clause	2025-03-13 17:56:46 -07:00
lzjb.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
metaslab.c	metaslab_alloc: make hint BP and DVA const (#17324)	2025-05-12 10:52:46 -07:00
mmp.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
multilist.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
objlist.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
pathname.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
range_tree.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
refcount.c	Implement allocation size ranges and use for gang leaves (#17111)	2025-05-02 15:32:18 -07:00
rrwlock.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
sa.c	Wire O_DIRECT also to Uncached I/O (#17218)	2025-05-13 14:26:55 -07:00
sha2_zfs.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
skein_zfs.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
spa_checkpoint.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
spa_config.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
spa_errlog.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
spa_history.c	dmu_tx: rename dmu_tx_assign() flags from TXG_* to DMU_TX_* (#17143)	2025-03-18 16:04:22 -07:00
spa_log_spacemap.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
spa_misc.c	Unified allocation throttling (#17020)	2025-03-24 09:25:01 -07:00
spa_stats.c	Wire O_DIRECT also to Uncached I/O (#17218)	2025-05-13 14:26:55 -07:00
spa.c	spa: clear checkpoint information during retry	2025-05-11 12:49:38 -04:00
space_map.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
space_reftree.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
THIRDPARTYLICENSE.cityhash	OpenZFS 8484 - Implement aggregate sum and use for arc counters	2018-06-06 09:35:59 -07:00
THIRDPARTYLICENSE.cityhash.descrip	OpenZFS 8484 - Implement aggregate sum and use for arc counters	2018-06-06 09:35:59 -07:00
txg.c	txg: generalise txg_wait_synced_sig() to txg_wait_synced_flags() (#17284)	2025-05-02 15:29:50 -07:00
uberblock.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
unique.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_draid_rand.c	SPDX: license tags: LicenseRef-OpenZFS-ThirdParty-PublicDomain	2025-03-13 17:57:31 -07:00
vdev_draid.c	Implement allocation size ranges and use for gang leaves (#17111)	2025-05-02 15:32:18 -07:00
vdev_file.c	Implement allocation size ranges and use for gang leaves (#17111)	2025-05-02 15:32:18 -07:00
vdev_indirect_births.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_indirect_mapping.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_indirect.c	Implement allocation size ranges and use for gang leaves (#17111)	2025-05-02 15:32:18 -07:00
vdev_initialize.c	dmu_tx: rename dmu_tx_assign() flags from TXG_* to DMU_TX_* (#17143)	2025-03-18 16:04:22 -07:00
vdev_label.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_mirror.c	Implement allocation size ranges and use for gang leaves (#17111)	2025-05-02 15:32:18 -07:00
vdev_missing.c	Implement allocation size ranges and use for gang leaves (#17111)	2025-05-02 15:32:18 -07:00
vdev_queue.c	Unified allocation throttling (#17020)	2025-03-24 09:25:01 -07:00
vdev_raidz_math_aarch64_neon_common.h	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_aarch64_neon.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_aarch64_neonx2.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_avx2.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_avx512bw.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_avx512f.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_impl.h	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_powerpc_altivec_common.h	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_powerpc_altivec.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_scalar.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_sse2.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_ssse3.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz.c	Implement allocation size ranges and use for gang leaves (#17111)	2025-05-02 15:32:18 -07:00
vdev_rebuild.c	Implement allocation size ranges and use for gang leaves (#17111)	2025-05-02 15:32:18 -07:00
vdev_removal.c	Fix lock reversal on device removal cancel	2025-04-01 09:31:24 -04:00
vdev_root.c	Implement allocation size ranges and use for gang leaves (#17111)	2025-05-02 15:32:18 -07:00
vdev_trim.c	dmu_tx: rename dmu_tx_assign() flags from TXG_* to DMU_TX_* (#17143)	2025-03-18 16:04:22 -07:00
vdev.c	Implement allocation size ranges and use for gang leaves (#17111)	2025-05-02 15:32:18 -07:00
zap_leaf.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zap_micro.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zap.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zcp_get.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zcp_global.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zcp_iter.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zcp_set.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zcp_synctask.c	cred: properly pass and test creds on other threads (#17273)	2025-04-29 16:27:48 -07:00
zcp.c	txg: generalise txg_wait_synced_sig() to txg_wait_synced_flags() (#17284)	2025-05-02 15:29:50 -07:00
zfeature.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zfs_byteswap.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zfs_chksum.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zfs_debug_common.c	nvlist: Add nvlist_snprintf() and zfs_dbgmsg_nvlist()	2025-04-18 09:22:16 -04:00
zfs_fm.c	zed: Ensure spare activation after kernel-initiated device removal	2025-03-28 15:48:38 -04:00
zfs_fuid.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zfs_impl.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zfs_ioctl.c	Show default quotas in zfs userspace tools	2025-04-03 10:36:45 -07:00
zfs_log.c	Wire O_DIRECT also to Uncached I/O (#17218)	2025-05-13 14:26:55 -07:00
zfs_onexit.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zfs_quota.c	Show default quotas in zfs userspace tools	2025-04-03 10:36:45 -07:00
zfs_ratelimit.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zfs_replay.c	dmu_tx: rename dmu_tx_assign() flags from TXG_* to DMU_TX_* (#17143)	2025-03-18 16:04:22 -07:00
zfs_rlock.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zfs_sa.c	dmu_tx: rename dmu_tx_assign() flags from TXG_* to DMU_TX_* (#17143)	2025-03-18 16:04:22 -07:00
zfs_vnops.c	Wire O_DIRECT also to Uncached I/O (#17218)	2025-05-13 14:26:55 -07:00
zfs_znode.c	Add default user/group/project quota properties	2025-04-03 10:35:22 -07:00
zil.c	dmu_tx: rename dmu_tx_assign() flags from TXG_* to DMU_TX_* (#17143)	2025-03-18 16:04:22 -07:00
zio_checksum.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zio_compress.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zio_inject.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zio.c	Implement allocation size ranges and use for gang leaves (#17111)	2025-05-02 15:32:18 -07:00
zle.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zrlock.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zthr.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zvol.c	Wire O_DIRECT also to Uncached I/O (#17218)	2025-05-13 14:26:55 -07:00