2008-11-20 23:01:55 +03:00
|
|
|
/*
|
|
|
|
|
* CDDL HEADER START
|
|
|
|
|
*
|
|
|
|
|
* The contents of this file are subject to the terms of the
|
|
|
|
|
* Common Development and Distribution License (the "License").
|
|
|
|
|
* You may not use this file except in compliance with the License.
|
|
|
|
|
*
|
|
|
|
|
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
|
2022-07-12 00:16:13 +03:00
|
|
|
* or https://opensource.org/licenses/CDDL-1.0.
|
2008-11-20 23:01:55 +03:00
|
|
|
* See the License for the specific language governing permissions
|
|
|
|
|
* and limitations under the License.
|
|
|
|
|
*
|
|
|
|
|
* When distributing Covered Code, include this CDDL HEADER in each
|
|
|
|
|
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
|
|
|
|
|
* If applicable, add the following below this CDDL HEADER, with the
|
|
|
|
|
* fields enclosed by brackets "[]" replaced with your own identifying
|
|
|
|
|
* information: Portions Copyright [yyyy] [name of copyright owner]
|
|
|
|
|
*
|
|
|
|
|
* CDDL HEADER END
|
|
|
|
|
*/
|
|
|
|
|
/*
|
2010-05-29 00:45:14 +04:00
|
|
|
* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
|
2020-04-23 20:06:57 +03:00
|
|
|
* Copyright (c) 2011, 2020 by Delphix. All rights reserved.
|
2013-08-02 00:02:10 +04:00
|
|
|
* Copyright (c) 2013 by Saso Kiselkov. All rights reserved.
|
2016-05-15 18:02:28 +03:00
|
|
|
* Copyright (c) 2013, Joyent, Inc. All rights reserved.
|
2017-01-27 22:46:39 +03:00
|
|
|
* Copyright (c) 2016, Nexenta Systems, Inc. All rights reserved.
|
2015-07-30 17:24:36 +03:00
|
|
|
* Copyright (c) 2015 by Chunwei Chen. All rights reserved.
|
2019-02-12 23:01:08 +03:00
|
|
|
* Copyright (c) 2019 Datto Inc.
|
Add zstd support to zfs
This PR adds two new compression types, based on ZStandard:
- zstd: A basic ZStandard compression algorithm Available compression.
Levels for zstd are zstd-1 through zstd-19, where the compression
increases with every level, but speed decreases.
- zstd-fast: A faster version of the ZStandard compression algorithm
zstd-fast is basically a "negative" level of zstd. The compression
decreases with every level, but speed increases.
Available compression levels for zstd-fast:
- zstd-fast-1 through zstd-fast-10
- zstd-fast-20 through zstd-fast-100 (in increments of 10)
- zstd-fast-500 and zstd-fast-1000
For more information check the man page.
Implementation details:
Rather than treat each level of zstd as a different algorithm (as was
done historically with gzip), the block pointer `enum zio_compress`
value is simply zstd for all levels, including zstd-fast, since they all
use the same decompression function.
The compress= property (a 64bit unsigned integer) uses the lower 7 bits
to store the compression algorithm (matching the number of bits used in
a block pointer, as the 8th bit was borrowed for embedded block
pointers). The upper bits are used to store the compression level.
It is necessary to be able to determine what compression level was used
when later reading a block back, so the concept used in LZ4, where the
first 32bits of the on-disk value are the size of the compressed data
(since the allocation is rounded up to the nearest ashift), was
extended, and we store the version of ZSTD and the level as well as the
compressed size. This value is returned when decompressing a block, so
that if the block needs to be recompressed (L2ARC, nop-write, etc), that
the same parameters will be used to result in the matching checksum.
All of the internal ZFS code ( `arc_buf_hdr_t`, `objset_t`,
`zio_prop_t`, etc.) uses the separated _compress and _complevel
variables. Only the properties ZAP contains the combined/bit-shifted
value. The combined value is split when the compression_changed_cb()
callback is called, and sets both objset members (os_compress and
os_complevel).
The userspace tools all use the combined/bit-shifted value.
Additional notes:
zdb can now also decode the ZSTD compression header (flag -Z) and
inspect the size, version and compression level saved in that header.
For each record, if it is ZSTD compressed, the parameters of the decoded
compression header get printed.
ZSTD is included with all current tests and new tests are added
as-needed.
Per-dataset feature flags now get activated when the property is set.
If a compression algorithm requires a feature flag, zfs activates the
feature when the property is set, rather than waiting for the first
block to be born. This is currently only used by zstd but can be
extended as needed.
Portions-Sponsored-By: The FreeBSD Foundation
Co-authored-by: Allan Jude <allanjude@freebsd.org>
Co-authored-by: Brian Behlendorf <behlendorf1@llnl.gov>
Co-authored-by: Sebastian Gottschall <s.gottschall@dd-wrt.com>
Co-authored-by: Kjeld Schouten-Lebbing <kjeld@schouten-lebbing.nl>
Co-authored-by: Michael Niewöhner <foss@mniewoehner.de>
Signed-off-by: Allan Jude <allan@klarasystems.com>
Signed-off-by: Allan Jude <allanjude@freebsd.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Sebastian Gottschall <s.gottschall@dd-wrt.com>
Signed-off-by: Kjeld Schouten-Lebbing <kjeld@schouten-lebbing.nl>
Signed-off-by: Michael Niewöhner <foss@mniewoehner.de>
Closes #6247
Closes #9024
Closes #10277
Closes #10278
2020-08-18 20:10:17 +03:00
|
|
|
* Copyright (c) 2019, Klara Inc.
|
|
|
|
|
* Copyright (c) 2019, Allan Jude
|
2022-10-20 03:07:51 +03:00
|
|
|
* Copyright (c) 2022 Hewlett Packard Enterprise Development LP.
|
2023-03-10 22:59:53 +03:00
|
|
|
* Copyright (c) 2021, 2022 by Pawel Jakub Dawidek
|
2008-11-20 23:01:55 +03:00
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#include <sys/dmu.h>
|
|
|
|
|
#include <sys/dmu_impl.h>
|
|
|
|
|
#include <sys/dmu_tx.h>
|
|
|
|
|
#include <sys/dbuf.h>
|
|
|
|
|
#include <sys/dnode.h>
|
|
|
|
|
#include <sys/zfs_context.h>
|
|
|
|
|
#include <sys/dmu_objset.h>
|
|
|
|
|
#include <sys/dmu_traverse.h>
|
|
|
|
|
#include <sys/dsl_dataset.h>
|
|
|
|
|
#include <sys/dsl_dir.h>
|
|
|
|
|
#include <sys/dsl_pool.h>
|
|
|
|
|
#include <sys/dsl_synctask.h>
|
|
|
|
|
#include <sys/dsl_prop.h>
|
|
|
|
|
#include <sys/dmu_zfetch.h>
|
|
|
|
|
#include <sys/zfs_ioctl.h>
|
|
|
|
|
#include <sys/zap.h>
|
|
|
|
|
#include <sys/zio_checksum.h>
|
2013-05-10 23:47:54 +04:00
|
|
|
#include <sys/zio_compress.h>
|
2010-05-29 00:45:14 +04:00
|
|
|
#include <sys/sa.h>
|
2014-10-18 19:58:11 +04:00
|
|
|
#include <sys/zfeature.h>
|
2016-07-22 18:52:49 +03:00
|
|
|
#include <sys/abd.h>
|
2023-03-10 22:59:53 +03:00
|
|
|
#include <sys/brt.h>
|
Enable use of DTRACE_PROBE* macros in "spl" module
This change modifies some of the infrastructure for enabling the use of
the DTRACE_PROBE* macros, such that we can use tehm in the "spl" module.
Currently, when the DTRACE_PROBE* macros are used, they get expanded to
create new functions, and these dynamically generated functions become
part of the "zfs" module.
Since the "spl" module does not depend on the "zfs" module, the use of
DTRACE_PROBE* in the "spl" module would result in undefined symbols
being used in the "spl" module. Specifically, DTRACE_PROBE* would turn
into a function call, and the function being called would be a symbol
only contained in the "zfs" module; which results in a linker and/or
runtime error.
Thus, this change adds the necessary logic to the "spl" module, to
mirror the tracing functionality available to the "zfs" module. After
this change, we'll have a "trace_zfs.h" header file which defines the
probes available only to the "zfs" module, and a "trace_spl.h" header
file which defines the probes available only to the "spl" module.
Reviewed by: Brad Lewis <brad.lewis@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Prakash Surya <prakash.surya@delphix.com>
Closes #9525
2019-10-30 21:02:41 +03:00
|
|
|
#include <sys/trace_zfs.h>
|
2021-02-20 09:34:33 +03:00
|
|
|
#include <sys/zfs_racct.h>
|
2017-08-21 18:59:48 +03:00
|
|
|
#include <sys/zfs_rlock.h>
|
2008-11-20 23:01:55 +03:00
|
|
|
#ifdef _KERNEL
|
|
|
|
|
#include <sys/vmsystm.h>
|
2008-12-03 23:09:06 +03:00
|
|
|
#include <sys/zfs_znode.h>
|
2008-11-20 23:01:55 +03:00
|
|
|
#endif
|
|
|
|
|
|
2013-05-10 23:47:54 +04:00
|
|
|
/*
|
|
|
|
|
* Enable/disable nopwrite feature.
|
|
|
|
|
*/
|
2022-01-15 02:37:55 +03:00
|
|
|
static int zfs_nopwrite_enabled = 1;
|
2013-05-10 23:47:54 +04:00
|
|
|
|
2017-02-01 01:44:03 +03:00
|
|
|
/*
|
2019-02-12 23:01:08 +03:00
|
|
|
* Tunable to control percentage of dirtied L1 blocks from frees allowed into
|
|
|
|
|
* one TXG. After this threshold is crossed, additional dirty blocks from frees
|
|
|
|
|
* will wait until the next TXG.
|
2017-02-01 01:44:03 +03:00
|
|
|
* A value of zero will disable this throttle.
|
|
|
|
|
*/
|
Cleanup: 64-bit kernel module parameters should use fixed width types
Various module parameters such as `zfs_arc_max` were originally
`uint64_t` on OpenSolaris/Illumos, but were changed to `unsigned long`
for Linux compatibility because Linux's kernel default module parameter
implementation did not support 64-bit types on 32-bit platforms. This
caused problems when porting OpenZFS to Windows because its LLP64 memory
model made `unsigned long` a 32-bit type on 64-bit, which created the
undesireable situation that parameters that should accept 64-bit values
could not on 64-bit Windows.
Upon inspection, it turns out that the Linux kernel module parameter
interface is extensible, such that we are allowed to define our own
types. Rather than maintaining the original type change via hacks to to
continue shrinking module parameters on 32-bit Linux, we implement
support for 64-bit module parameters on Linux.
After doing a review of all 64-bit kernel parameters (found via the man
page and also proposed changes by Andrew Innes), the kernel module
parameters fell into a few groups:
Parameters that were originally 64-bit on Illumos:
* dbuf_cache_max_bytes
* dbuf_metadata_cache_max_bytes
* l2arc_feed_min_ms
* l2arc_feed_secs
* l2arc_headroom
* l2arc_headroom_boost
* l2arc_write_boost
* l2arc_write_max
* metaslab_aliquot
* metaslab_force_ganging
* zfetch_array_rd_sz
* zfs_arc_max
* zfs_arc_meta_limit
* zfs_arc_meta_min
* zfs_arc_min
* zfs_async_block_max_blocks
* zfs_condense_max_obsolete_bytes
* zfs_condense_min_mapping_bytes
* zfs_deadman_checktime_ms
* zfs_deadman_synctime_ms
* zfs_initialize_chunk_size
* zfs_initialize_value
* zfs_lua_max_instrlimit
* zfs_lua_max_memlimit
* zil_slog_bulk
Parameters that were originally 32-bit on Illumos:
* zfs_per_txg_dirty_frees_percent
Parameters that were originally `ssize_t` on Illumos:
* zfs_immediate_write_sz
Note that `ssize_t` is `int32_t` on 32-bit and `int64_t` on 64-bit. It
has been upgraded to 64-bit.
Parameters that were `long`/`unsigned long` because of Linux/FreeBSD
influence:
* l2arc_rebuild_blocks_min_l2size
* zfs_key_max_salt_uses
* zfs_max_log_walking
* zfs_max_logsm_summary_length
* zfs_metaslab_max_size_cache_sec
* zfs_min_metaslabs_to_flush
* zfs_multihost_interval
* zfs_unflushed_log_block_max
* zfs_unflushed_log_block_min
* zfs_unflushed_log_block_pct
* zfs_unflushed_max_mem_amt
* zfs_unflushed_max_mem_ppm
New parameters that do not exist in Illumos:
* l2arc_trim_ahead
* vdev_file_logical_ashift
* vdev_file_physical_ashift
* zfs_arc_dnode_limit
* zfs_arc_dnode_limit_percent
* zfs_arc_dnode_reduce_percent
* zfs_arc_meta_limit_percent
* zfs_arc_sys_free
* zfs_deadman_ziotime_ms
* zfs_delete_blocks
* zfs_history_output_max
* zfs_livelist_max_entries
* zfs_max_async_dedup_frees
* zfs_max_nvlist_src_size
* zfs_rebuild_max_segment
* zfs_rebuild_vdev_limit
* zfs_unflushed_log_txg_max
* zfs_vdev_max_auto_ashift
* zfs_vdev_min_auto_ashift
* zfs_vnops_read_chunk_size
* zvol_max_discard_blocks
Rather than clutter the lists with commentary, the module parameters
that need comments are repeated below.
A few parameters were defined in Linux/FreeBSD specific code, where the
use of ulong/long is not an issue for portability, so we leave them
alone:
* zfs_delete_blocks
* zfs_key_max_salt_uses
* zvol_max_discard_blocks
The documentation for a few parameters was found to be incorrect:
* zfs_deadman_checktime_ms - incorrectly documented as int
* zfs_delete_blocks - not documented as Linux only
* zfs_history_output_max - incorrectly documented as int
* zfs_vnops_read_chunk_size - incorrectly documented as long
* zvol_max_discard_blocks - incorrectly documented as ulong
The documentation for these has been fixed, alongside the changes to
document the switch to fixed width types.
In addition, several kernel module parameters were percentages or held
ashift values, so being 64-bit never made sense for them. They have been
downgraded to 32-bit:
* vdev_file_logical_ashift
* vdev_file_physical_ashift
* zfs_arc_dnode_limit_percent
* zfs_arc_dnode_reduce_percent
* zfs_arc_meta_limit_percent
* zfs_per_txg_dirty_frees_percent
* zfs_unflushed_log_block_pct
* zfs_vdev_max_auto_ashift
* zfs_vdev_min_auto_ashift
Of special note are `zfs_vdev_max_auto_ashift` and
`zfs_vdev_min_auto_ashift`, which were already defined as `uint64_t`,
and passed to the kernel as `ulong`. This is inherently buggy on big
endian 32-bit Linux, since the values would not be written to the
correct locations. 32-bit FreeBSD was unaffected because its sysctl code
correctly treated this as a `uint64_t`.
Lastly, a code comment suggests that `zfs_arc_sys_free` is
Linux-specific, but there is nothing to indicate to me that it is
Linux-specific. Nothing was done about that.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Reviewed-by: Alexander Motin <mav@FreeBSD.org>
Original-patch-by: Andrew Innes <andrew.c12@gmail.com>
Original-patch-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Closes #13984
Closes #14004
2022-10-03 22:06:54 +03:00
|
|
|
static uint_t zfs_per_txg_dirty_frees_percent = 30;
|
2017-02-01 01:44:03 +03:00
|
|
|
|
2017-03-25 00:28:38 +03:00
|
|
|
/*
|
2021-11-30 21:38:09 +03:00
|
|
|
* Enable/disable forcing txg sync when dirty checking for holes with lseek().
|
|
|
|
|
* By default this is enabled to ensure accurate hole reporting, it can result
|
|
|
|
|
* in a significant performance penalty for lseek(SEEK_HOLE) heavy workloads.
|
|
|
|
|
* Disabling this option will result in holes never being reported in dirty
|
|
|
|
|
* files which is always safe.
|
2017-03-25 00:28:38 +03:00
|
|
|
*/
|
2022-01-15 02:37:55 +03:00
|
|
|
static int zfs_dmu_offset_next_sync = 1;
|
2017-03-25 00:28:38 +03:00
|
|
|
|
2019-06-12 23:13:09 +03:00
|
|
|
/*
|
|
|
|
|
* Limit the amount we can prefetch with one call to this amount. This
|
|
|
|
|
* helps to limit the amount of memory that can be used by prefetching.
|
|
|
|
|
* Larger objects should be prefetched a bit at a time.
|
|
|
|
|
*/
|
2023-07-21 21:51:47 +03:00
|
|
|
#ifdef _ILP32
|
|
|
|
|
uint_t dmu_prefetch_max = 8 * 1024 * 1024;
|
|
|
|
|
#else
|
Cleanup: Specify unsignedness on things that should not be signed
In #13871, zfs_vdev_aggregation_limit_non_rotating and
zfs_vdev_aggregation_limit being signed was pointed out as a possible
reason not to eliminate an unnecessary MAX(unsigned, 0) since the
unsigned value was assigned from them.
There is no reason for these module parameters to be signed and upon
inspection, it was found that there are a number of other module
parameters that are signed, but should not be, so we make them unsigned.
Making them unsigned made it clear that some other variables in the code
should also be unsigned, so we also make those unsigned. This prevents
users from setting negative values that could potentially cause bad
behaviors. It also makes the code slightly easier to understand.
Mostly module parameters that deal with timeouts, limits, bitshifts and
percentages are made unsigned by this. Any that are boolean are left
signed, since whether booleans should be considered signed or unsigned
does not matter.
Making zfs_arc_lotsfree_percent unsigned caused a
`zfs_arc_lotsfree_percent >= 0` check to become redundant, so it was
removed. Removing the check was also necessary to prevent a compiler
error from -Werror=type-limits.
Several end of line comments had to be moved to their own lines because
replacing int with uint_t caused us to exceed the 80 character limit
enforced by cstyle.pl.
The following were kept signed because they are passed to
taskq_create(), which expects signed values and modifying the
OpenSolaris/Illumos DDI is out of scope of this patch:
* metaslab_load_pct
* zfs_sync_taskq_batch_pct
* zfs_zil_clean_taskq_nthr_pct
* zfs_zil_clean_taskq_minalloc
* zfs_zil_clean_taskq_maxalloc
* zfs_arc_prune_task_threads
Also, negative values in those parameters was found to be harmless.
The following were left signed because either negative values make
sense, or more analysis was needed to determine whether negative values
should be disallowed:
* zfs_metaslab_switch_threshold
* zfs_pd_bytes_max
* zfs_livelist_min_percent_shared
zfs_multihost_history was made static to be consistent with other
parameters.
A number of module parameters were marked as signed, but in reality
referenced unsigned variables. upgrade_errlog_limit is one of the
numerous examples. In the case of zfs_vdev_async_read_max_active, it was
already uint32_t, but zdb had an extern int declaration for it.
Interestingly, the documentation in zfs.4 was right for
upgrade_errlog_limit despite the module parameter being wrongly marked,
while the documentation for zfs_vdev_async_read_max_active (and friends)
was wrong. It was also wrong for zstd_abort_size, which was unsigned,
but was documented as signed.
Also, the documentation in zfs.4 incorrectly described the following
parameters as ulong when they were int:
* zfs_arc_meta_adjust_restarts
* zfs_override_estimate_recordsize
They are now uint_t as of this patch and thus the man page has been
updated to describe them as uint.
dbuf_state_index was left alone since it does nothing and perhaps should
be removed in another patch.
If any module parameters were missed, they were not found by `grep -r
'ZFS_MODULE_PARAM' | grep ', INT'`. I did find a few that grep missed,
but only because they were in files that had hits.
This patch intentionally did not attempt to address whether some of
these module parameters should be elevated to 64-bit parameters, because
the length of a long on 32-bit is 32-bit.
Lastly, it was pointed out during review that uint_t is a better match
for these variables than uint32_t because FreeBSD kernel parameter
definitions are designed for uint_t, whose bit width can change in
future memory models. As a result, we change the existing parameters
that are uint32_t to use uint_t.
Reviewed-by: Alexander Motin <mav@FreeBSD.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Neal Gompa <ngompa@datto.com>
Signed-off-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Closes #13875
2022-09-28 02:42:41 +03:00
|
|
|
uint_t dmu_prefetch_max = 8 * SPA_MAXBLOCKSIZE;
|
2023-07-21 21:51:47 +03:00
|
|
|
#endif
|
2019-06-12 23:13:09 +03:00
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
const dmu_object_type_info_t dmu_ot[DMU_OT_NUMTYPES] = {
|
2018-07-10 20:49:50 +03:00
|
|
|
{DMU_BSWAP_UINT8, TRUE, FALSE, FALSE, "unallocated" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, TRUE, FALSE, "object directory" },
|
|
|
|
|
{DMU_BSWAP_UINT64, TRUE, TRUE, FALSE, "object array" },
|
|
|
|
|
{DMU_BSWAP_UINT8, TRUE, FALSE, FALSE, "packed nvlist" },
|
|
|
|
|
{DMU_BSWAP_UINT64, TRUE, FALSE, FALSE, "packed nvlist size" },
|
|
|
|
|
{DMU_BSWAP_UINT64, TRUE, FALSE, FALSE, "bpobj" },
|
|
|
|
|
{DMU_BSWAP_UINT64, TRUE, FALSE, FALSE, "bpobj header" },
|
|
|
|
|
{DMU_BSWAP_UINT64, TRUE, FALSE, FALSE, "SPA space map header" },
|
|
|
|
|
{DMU_BSWAP_UINT64, TRUE, FALSE, FALSE, "SPA space map" },
|
|
|
|
|
{DMU_BSWAP_UINT64, TRUE, FALSE, TRUE, "ZIL intent log" },
|
|
|
|
|
{DMU_BSWAP_DNODE, TRUE, FALSE, TRUE, "DMU dnode" },
|
|
|
|
|
{DMU_BSWAP_OBJSET, TRUE, TRUE, FALSE, "DMU objset" },
|
|
|
|
|
{DMU_BSWAP_UINT64, TRUE, TRUE, FALSE, "DSL directory" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, TRUE, FALSE, "DSL directory child map"},
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, TRUE, FALSE, "DSL dataset snap map" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, TRUE, FALSE, "DSL props" },
|
|
|
|
|
{DMU_BSWAP_UINT64, TRUE, TRUE, FALSE, "DSL dataset" },
|
|
|
|
|
{DMU_BSWAP_ZNODE, TRUE, FALSE, FALSE, "ZFS znode" },
|
|
|
|
|
{DMU_BSWAP_OLDACL, TRUE, FALSE, TRUE, "ZFS V0 ACL" },
|
|
|
|
|
{DMU_BSWAP_UINT8, FALSE, FALSE, TRUE, "ZFS plain file" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, FALSE, TRUE, "ZFS directory" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, FALSE, FALSE, "ZFS master node" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, FALSE, TRUE, "ZFS delete queue" },
|
|
|
|
|
{DMU_BSWAP_UINT8, FALSE, FALSE, TRUE, "zvol object" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, FALSE, FALSE, "zvol prop" },
|
|
|
|
|
{DMU_BSWAP_UINT8, FALSE, FALSE, TRUE, "other uint8[]" },
|
|
|
|
|
{DMU_BSWAP_UINT64, FALSE, FALSE, TRUE, "other uint64[]" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, FALSE, FALSE, "other ZAP" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, FALSE, FALSE, "persistent error log" },
|
|
|
|
|
{DMU_BSWAP_UINT8, TRUE, FALSE, FALSE, "SPA history" },
|
|
|
|
|
{DMU_BSWAP_UINT64, TRUE, FALSE, FALSE, "SPA history offsets" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, TRUE, FALSE, "Pool properties" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, TRUE, FALSE, "DSL permissions" },
|
|
|
|
|
{DMU_BSWAP_ACL, TRUE, FALSE, TRUE, "ZFS ACL" },
|
|
|
|
|
{DMU_BSWAP_UINT8, TRUE, FALSE, TRUE, "ZFS SYSACL" },
|
|
|
|
|
{DMU_BSWAP_UINT8, TRUE, FALSE, TRUE, "FUID table" },
|
|
|
|
|
{DMU_BSWAP_UINT64, TRUE, FALSE, FALSE, "FUID table size" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, TRUE, FALSE, "DSL dataset next clones"},
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, FALSE, FALSE, "scan work queue" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, FALSE, TRUE, "ZFS user/group/project used" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, FALSE, TRUE, "ZFS user/group/project quota"},
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, TRUE, FALSE, "snapshot refcount tags"},
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, FALSE, FALSE, "DDT ZAP algorithm" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, FALSE, FALSE, "DDT statistics" },
|
|
|
|
|
{DMU_BSWAP_UINT8, TRUE, FALSE, TRUE, "System attributes" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, FALSE, TRUE, "SA master node" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, FALSE, TRUE, "SA attr registration" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, FALSE, TRUE, "SA attr layouts" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, FALSE, FALSE, "scan translations" },
|
|
|
|
|
{DMU_BSWAP_UINT8, FALSE, FALSE, TRUE, "deduplicated block" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, TRUE, FALSE, "DSL deadlist map" },
|
|
|
|
|
{DMU_BSWAP_UINT64, TRUE, TRUE, FALSE, "DSL deadlist map hdr" },
|
|
|
|
|
{DMU_BSWAP_ZAP, TRUE, TRUE, FALSE, "DSL dir clones" },
|
|
|
|
|
{DMU_BSWAP_UINT64, TRUE, FALSE, FALSE, "bpobj subobj" }
|
2012-12-14 03:24:15 +04:00
|
|
|
};
|
|
|
|
|
|
2022-10-01 01:34:39 +03:00
|
|
|
dmu_object_byteswap_info_t dmu_ot_byteswap[DMU_BSWAP_NUMFUNCS] = {
|
2012-12-14 03:24:15 +04:00
|
|
|
{ byteswap_uint8_array, "uint8" },
|
|
|
|
|
{ byteswap_uint16_array, "uint16" },
|
|
|
|
|
{ byteswap_uint32_array, "uint32" },
|
|
|
|
|
{ byteswap_uint64_array, "uint64" },
|
|
|
|
|
{ zap_byteswap, "zap" },
|
|
|
|
|
{ dnode_buf_byteswap, "dnode" },
|
|
|
|
|
{ dmu_objset_byteswap, "objset" },
|
|
|
|
|
{ zfs_znode_byteswap, "znode" },
|
|
|
|
|
{ zfs_oldacl_byteswap, "oldacl" },
|
|
|
|
|
{ zfs_acl_byteswap, "acl" }
|
2008-11-20 23:01:55 +03:00
|
|
|
};
|
|
|
|
|
|
2023-08-11 19:04:08 +03:00
|
|
|
int
|
OpenZFS 7004 - dmu_tx_hold_zap() does dnode_hold() 7x on same object
Using a benchmark which has 32 threads creating 2 million files in the
same directory, on a machine with 16 CPU cores, I observed poor
performance. I noticed that dmu_tx_hold_zap() was using about 30% of
all CPU, and doing dnode_hold() 7 times on the same object (the ZAP
object that is being held).
dmu_tx_hold_zap() keeps a hold on the dnode_t the entire time it is
running, in dmu_tx_hold_t:txh_dnode, so it would be nice to use the
dnode_t that we already have in hand, rather than repeatedly calling
dnode_hold(). To do this, we need to pass the dnode_t down through
all the intermediate calls that dmu_tx_hold_zap() makes, making these
routines take the dnode_t* rather than an objset_t* and a uint64_t
object number. In particular, the following routines will need to have
analogous *_by_dnode() variants created:
dmu_buf_hold_noread()
dmu_buf_hold()
zap_lookup()
zap_lookup_norm()
zap_count_write()
zap_lockdir()
zap_count_write()
This can improve performance on the benchmark described above by 100%,
from 30,000 file creations per second to 60,000. (This improvement is on
top of that provided by working around the object allocation issue. Peak
performance of ~90,000 creations per second was observed with 8 CPUs;
adding CPUs past that decreased performance due to lock contention.) The
CPU used by dmu_tx_hold_zap() was reduced by 88%, from 340 CPU-seconds
to 40 CPU-seconds.
Sponsored by: Intel Corp.
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
OpenZFS-issue: https://www.illumos.org/issues/7004
OpenZFS-commit: https://github.com/openzfs/openzfs/pull/109
Closes #4641
Closes #4972
2016-07-21 01:42:13 +03:00
|
|
|
dmu_buf_hold_noread_by_dnode(dnode_t *dn, uint64_t offset,
|
2022-04-19 21:38:30 +03:00
|
|
|
const void *tag, dmu_buf_t **dbp)
|
OpenZFS 7004 - dmu_tx_hold_zap() does dnode_hold() 7x on same object
Using a benchmark which has 32 threads creating 2 million files in the
same directory, on a machine with 16 CPU cores, I observed poor
performance. I noticed that dmu_tx_hold_zap() was using about 30% of
all CPU, and doing dnode_hold() 7 times on the same object (the ZAP
object that is being held).
dmu_tx_hold_zap() keeps a hold on the dnode_t the entire time it is
running, in dmu_tx_hold_t:txh_dnode, so it would be nice to use the
dnode_t that we already have in hand, rather than repeatedly calling
dnode_hold(). To do this, we need to pass the dnode_t down through
all the intermediate calls that dmu_tx_hold_zap() makes, making these
routines take the dnode_t* rather than an objset_t* and a uint64_t
object number. In particular, the following routines will need to have
analogous *_by_dnode() variants created:
dmu_buf_hold_noread()
dmu_buf_hold()
zap_lookup()
zap_lookup_norm()
zap_count_write()
zap_lockdir()
zap_count_write()
This can improve performance on the benchmark described above by 100%,
from 30,000 file creations per second to 60,000. (This improvement is on
top of that provided by working around the object allocation issue. Peak
performance of ~90,000 creations per second was observed with 8 CPUs;
adding CPUs past that decreased performance due to lock contention.) The
CPU used by dmu_tx_hold_zap() was reduced by 88%, from 340 CPU-seconds
to 40 CPU-seconds.
Sponsored by: Intel Corp.
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
OpenZFS-issue: https://www.illumos.org/issues/7004
OpenZFS-commit: https://github.com/openzfs/openzfs/pull/109
Closes #4641
Closes #4972
2016-07-21 01:42:13 +03:00
|
|
|
{
|
|
|
|
|
uint64_t blkid;
|
|
|
|
|
dmu_buf_impl_t *db;
|
|
|
|
|
|
|
|
|
|
rw_enter(&dn->dn_struct_rwlock, RW_READER);
|
2019-07-08 23:18:50 +03:00
|
|
|
blkid = dbuf_whichblock(dn, 0, offset);
|
OpenZFS 7004 - dmu_tx_hold_zap() does dnode_hold() 7x on same object
Using a benchmark which has 32 threads creating 2 million files in the
same directory, on a machine with 16 CPU cores, I observed poor
performance. I noticed that dmu_tx_hold_zap() was using about 30% of
all CPU, and doing dnode_hold() 7 times on the same object (the ZAP
object that is being held).
dmu_tx_hold_zap() keeps a hold on the dnode_t the entire time it is
running, in dmu_tx_hold_t:txh_dnode, so it would be nice to use the
dnode_t that we already have in hand, rather than repeatedly calling
dnode_hold(). To do this, we need to pass the dnode_t down through
all the intermediate calls that dmu_tx_hold_zap() makes, making these
routines take the dnode_t* rather than an objset_t* and a uint64_t
object number. In particular, the following routines will need to have
analogous *_by_dnode() variants created:
dmu_buf_hold_noread()
dmu_buf_hold()
zap_lookup()
zap_lookup_norm()
zap_count_write()
zap_lockdir()
zap_count_write()
This can improve performance on the benchmark described above by 100%,
from 30,000 file creations per second to 60,000. (This improvement is on
top of that provided by working around the object allocation issue. Peak
performance of ~90,000 creations per second was observed with 8 CPUs;
adding CPUs past that decreased performance due to lock contention.) The
CPU used by dmu_tx_hold_zap() was reduced by 88%, from 340 CPU-seconds
to 40 CPU-seconds.
Sponsored by: Intel Corp.
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
OpenZFS-issue: https://www.illumos.org/issues/7004
OpenZFS-commit: https://github.com/openzfs/openzfs/pull/109
Closes #4641
Closes #4972
2016-07-21 01:42:13 +03:00
|
|
|
db = dbuf_hold(dn, blkid, tag);
|
|
|
|
|
rw_exit(&dn->dn_struct_rwlock);
|
|
|
|
|
|
|
|
|
|
if (db == NULL) {
|
|
|
|
|
*dbp = NULL;
|
|
|
|
|
return (SET_ERROR(EIO));
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
*dbp = &db->db;
|
|
|
|
|
return (0);
|
|
|
|
|
}
|
2023-08-11 19:04:08 +03:00
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
int
|
2014-06-06 01:19:08 +04:00
|
|
|
dmu_buf_hold_noread(objset_t *os, uint64_t object, uint64_t offset,
|
2022-04-19 21:38:30 +03:00
|
|
|
const void *tag, dmu_buf_t **dbp)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
|
|
|
|
dnode_t *dn;
|
|
|
|
|
uint64_t blkid;
|
|
|
|
|
dmu_buf_impl_t *db;
|
|
|
|
|
int err;
|
2010-05-29 00:45:14 +04:00
|
|
|
|
|
|
|
|
err = dnode_hold(os, object, FTAG, &dn);
|
2008-11-20 23:01:55 +03:00
|
|
|
if (err)
|
|
|
|
|
return (err);
|
|
|
|
|
rw_enter(&dn->dn_struct_rwlock, RW_READER);
|
2019-07-08 23:18:50 +03:00
|
|
|
blkid = dbuf_whichblock(dn, 0, offset);
|
2008-11-20 23:01:55 +03:00
|
|
|
db = dbuf_hold(dn, blkid, tag);
|
|
|
|
|
rw_exit(&dn->dn_struct_rwlock);
|
2014-06-06 01:19:08 +04:00
|
|
|
dnode_rele(dn, FTAG);
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
if (db == NULL) {
|
2014-06-06 01:19:08 +04:00
|
|
|
*dbp = NULL;
|
|
|
|
|
return (SET_ERROR(EIO));
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
*dbp = &db->db;
|
|
|
|
|
return (err);
|
|
|
|
|
}
|
|
|
|
|
|
OpenZFS 7004 - dmu_tx_hold_zap() does dnode_hold() 7x on same object
Using a benchmark which has 32 threads creating 2 million files in the
same directory, on a machine with 16 CPU cores, I observed poor
performance. I noticed that dmu_tx_hold_zap() was using about 30% of
all CPU, and doing dnode_hold() 7 times on the same object (the ZAP
object that is being held).
dmu_tx_hold_zap() keeps a hold on the dnode_t the entire time it is
running, in dmu_tx_hold_t:txh_dnode, so it would be nice to use the
dnode_t that we already have in hand, rather than repeatedly calling
dnode_hold(). To do this, we need to pass the dnode_t down through
all the intermediate calls that dmu_tx_hold_zap() makes, making these
routines take the dnode_t* rather than an objset_t* and a uint64_t
object number. In particular, the following routines will need to have
analogous *_by_dnode() variants created:
dmu_buf_hold_noread()
dmu_buf_hold()
zap_lookup()
zap_lookup_norm()
zap_count_write()
zap_lockdir()
zap_count_write()
This can improve performance on the benchmark described above by 100%,
from 30,000 file creations per second to 60,000. (This improvement is on
top of that provided by working around the object allocation issue. Peak
performance of ~90,000 creations per second was observed with 8 CPUs;
adding CPUs past that decreased performance due to lock contention.) The
CPU used by dmu_tx_hold_zap() was reduced by 88%, from 340 CPU-seconds
to 40 CPU-seconds.
Sponsored by: Intel Corp.
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
OpenZFS-issue: https://www.illumos.org/issues/7004
OpenZFS-commit: https://github.com/openzfs/openzfs/pull/109
Closes #4641
Closes #4972
2016-07-21 01:42:13 +03:00
|
|
|
int
|
|
|
|
|
dmu_buf_hold_by_dnode(dnode_t *dn, uint64_t offset,
|
2022-04-19 21:38:30 +03:00
|
|
|
const void *tag, dmu_buf_t **dbp, int flags)
|
OpenZFS 7004 - dmu_tx_hold_zap() does dnode_hold() 7x on same object
Using a benchmark which has 32 threads creating 2 million files in the
same directory, on a machine with 16 CPU cores, I observed poor
performance. I noticed that dmu_tx_hold_zap() was using about 30% of
all CPU, and doing dnode_hold() 7 times on the same object (the ZAP
object that is being held).
dmu_tx_hold_zap() keeps a hold on the dnode_t the entire time it is
running, in dmu_tx_hold_t:txh_dnode, so it would be nice to use the
dnode_t that we already have in hand, rather than repeatedly calling
dnode_hold(). To do this, we need to pass the dnode_t down through
all the intermediate calls that dmu_tx_hold_zap() makes, making these
routines take the dnode_t* rather than an objset_t* and a uint64_t
object number. In particular, the following routines will need to have
analogous *_by_dnode() variants created:
dmu_buf_hold_noread()
dmu_buf_hold()
zap_lookup()
zap_lookup_norm()
zap_count_write()
zap_lockdir()
zap_count_write()
This can improve performance on the benchmark described above by 100%,
from 30,000 file creations per second to 60,000. (This improvement is on
top of that provided by working around the object allocation issue. Peak
performance of ~90,000 creations per second was observed with 8 CPUs;
adding CPUs past that decreased performance due to lock contention.) The
CPU used by dmu_tx_hold_zap() was reduced by 88%, from 340 CPU-seconds
to 40 CPU-seconds.
Sponsored by: Intel Corp.
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
OpenZFS-issue: https://www.illumos.org/issues/7004
OpenZFS-commit: https://github.com/openzfs/openzfs/pull/109
Closes #4641
Closes #4972
2016-07-21 01:42:13 +03:00
|
|
|
{
|
|
|
|
|
int err;
|
|
|
|
|
int db_flags = DB_RF_CANFAIL;
|
|
|
|
|
|
|
|
|
|
if (flags & DMU_READ_NO_PREFETCH)
|
|
|
|
|
db_flags |= DB_RF_NOPREFETCH;
|
Native Encryption for ZFS on Linux
This change incorporates three major pieces:
The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
convenient.
The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.
The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
compromised.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #494
Closes #5769
2017-08-14 20:36:48 +03:00
|
|
|
if (flags & DMU_READ_NO_DECRYPT)
|
|
|
|
|
db_flags |= DB_RF_NO_DECRYPT;
|
OpenZFS 7004 - dmu_tx_hold_zap() does dnode_hold() 7x on same object
Using a benchmark which has 32 threads creating 2 million files in the
same directory, on a machine with 16 CPU cores, I observed poor
performance. I noticed that dmu_tx_hold_zap() was using about 30% of
all CPU, and doing dnode_hold() 7 times on the same object (the ZAP
object that is being held).
dmu_tx_hold_zap() keeps a hold on the dnode_t the entire time it is
running, in dmu_tx_hold_t:txh_dnode, so it would be nice to use the
dnode_t that we already have in hand, rather than repeatedly calling
dnode_hold(). To do this, we need to pass the dnode_t down through
all the intermediate calls that dmu_tx_hold_zap() makes, making these
routines take the dnode_t* rather than an objset_t* and a uint64_t
object number. In particular, the following routines will need to have
analogous *_by_dnode() variants created:
dmu_buf_hold_noread()
dmu_buf_hold()
zap_lookup()
zap_lookup_norm()
zap_count_write()
zap_lockdir()
zap_count_write()
This can improve performance on the benchmark described above by 100%,
from 30,000 file creations per second to 60,000. (This improvement is on
top of that provided by working around the object allocation issue. Peak
performance of ~90,000 creations per second was observed with 8 CPUs;
adding CPUs past that decreased performance due to lock contention.) The
CPU used by dmu_tx_hold_zap() was reduced by 88%, from 340 CPU-seconds
to 40 CPU-seconds.
Sponsored by: Intel Corp.
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
OpenZFS-issue: https://www.illumos.org/issues/7004
OpenZFS-commit: https://github.com/openzfs/openzfs/pull/109
Closes #4641
Closes #4972
2016-07-21 01:42:13 +03:00
|
|
|
|
|
|
|
|
err = dmu_buf_hold_noread_by_dnode(dn, offset, tag, dbp);
|
|
|
|
|
if (err == 0) {
|
|
|
|
|
dmu_buf_impl_t *db = (dmu_buf_impl_t *)(*dbp);
|
|
|
|
|
err = dbuf_read(db, NULL, db_flags);
|
|
|
|
|
if (err != 0) {
|
|
|
|
|
dbuf_rele(db, tag);
|
|
|
|
|
*dbp = NULL;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return (err);
|
|
|
|
|
}
|
|
|
|
|
|
2014-06-06 01:19:08 +04:00
|
|
|
int
|
|
|
|
|
dmu_buf_hold(objset_t *os, uint64_t object, uint64_t offset,
|
2022-04-19 21:38:30 +03:00
|
|
|
const void *tag, dmu_buf_t **dbp, int flags)
|
2014-06-06 01:19:08 +04:00
|
|
|
{
|
|
|
|
|
int err;
|
|
|
|
|
int db_flags = DB_RF_CANFAIL;
|
|
|
|
|
|
|
|
|
|
if (flags & DMU_READ_NO_PREFETCH)
|
|
|
|
|
db_flags |= DB_RF_NOPREFETCH;
|
Native Encryption for ZFS on Linux
This change incorporates three major pieces:
The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
convenient.
The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.
The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
compromised.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #494
Closes #5769
2017-08-14 20:36:48 +03:00
|
|
|
if (flags & DMU_READ_NO_DECRYPT)
|
|
|
|
|
db_flags |= DB_RF_NO_DECRYPT;
|
2014-06-06 01:19:08 +04:00
|
|
|
|
|
|
|
|
err = dmu_buf_hold_noread(os, object, offset, tag, dbp);
|
|
|
|
|
if (err == 0) {
|
|
|
|
|
dmu_buf_impl_t *db = (dmu_buf_impl_t *)(*dbp);
|
2010-05-29 00:45:14 +04:00
|
|
|
err = dbuf_read(db, NULL, db_flags);
|
2014-06-06 01:19:08 +04:00
|
|
|
if (err != 0) {
|
2008-11-20 23:01:55 +03:00
|
|
|
dbuf_rele(db, tag);
|
2014-06-06 01:19:08 +04:00
|
|
|
*dbp = NULL;
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return (err);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
int
|
|
|
|
|
dmu_bonus_max(void)
|
|
|
|
|
{
|
Implement large_dnode pool feature
Justification
-------------
This feature adds support for variable length dnodes. Our motivation is
to eliminate the overhead associated with using spill blocks. Spill
blocks are used to store system attribute data (i.e. file metadata) that
does not fit in the dnode's bonus buffer. By allowing a larger bonus
buffer area the use of a spill block can be avoided. Spill blocks
potentially incur an additional read I/O for every dnode in a dnode
block. As a worst case example, reading 32 dnodes from a 16k dnode block
and all of the spill blocks could issue 33 separate reads. Now suppose
those dnodes have size 1024 and therefore don't need spill blocks. Then
the worst case number of blocks read is reduced to from 33 to two--one
per dnode block. In practice spill blocks may tend to be co-located on
disk with the dnode blocks so the reduction in I/O would not be this
drastic. In a badly fragmented pool, however, the improvement could be
significant.
ZFS-on-Linux systems that make heavy use of extended attributes would
benefit from this feature. In particular, ZFS-on-Linux supports the
xattr=sa dataset property which allows file extended attribute data
to be stored in the dnode bonus buffer as an alternative to the
traditional directory-based format. Workloads such as SELinux and the
Lustre distributed filesystem often store enough xattr data to force
spill bocks when xattr=sa is in effect. Large dnodes may therefore
provide a performance benefit to such systems.
Other use cases that may benefit from this feature include files with
large ACLs and symbolic links with long target names. Furthermore,
this feature may be desirable on other platforms in case future
applications or features are developed that could make use of a
larger bonus buffer area.
Implementation
--------------
The size of a dnode may be a multiple of 512 bytes up to the size of
a dnode block (currently 16384 bytes). A dn_extra_slots field was
added to the current on-disk dnode_phys_t structure to describe the
size of the physical dnode on disk. The 8 bits for this field were
taken from the zero filled dn_pad2 field. The field represents how
many "extra" dnode_phys_t slots a dnode consumes in its dnode block.
This convention results in a value of 0 for 512 byte dnodes which
preserves on-disk format compatibility with older software.
Similarly, the in-memory dnode_t structure has a new dn_num_slots field
to represent the total number of dnode_phys_t slots consumed on disk.
Thus dn->dn_num_slots is 1 greater than the corresponding
dnp->dn_extra_slots. This difference in convention was adopted
because, unlike on-disk structures, backward compatibility is not a
concern for in-memory objects, so we used a more natural way to
represent size for a dnode_t.
The default size for newly created dnodes is determined by the value of
a new "dnodesize" dataset property. By default the property is set to
"legacy" which is compatible with older software. Setting the property
to "auto" will allow the filesystem to choose the most suitable dnode
size. Currently this just sets the default dnode size to 1k, but future
code improvements could dynamically choose a size based on observed
workload patterns. Dnodes of varying sizes can coexist within the same
dataset and even within the same dnode block. For example, to enable
automatically-sized dnodes, run
# zfs set dnodesize=auto tank/fish
The user can also specify literal values for the dnodesize property.
These are currently limited to powers of two from 1k to 16k. The
power-of-2 limitation is only for simplicity of the user interface.
Internally the implementation can handle any multiple of 512 up to 16k,
and consumers of the DMU API can specify any legal dnode value.
The size of a new dnode is determined at object allocation time and
stored as a new field in the znode in-memory structure. New DMU
interfaces are added to allow the consumer to specify the dnode size
that a newly allocated object should use. Existing interfaces are
unchanged to avoid having to update every call site and to preserve
compatibility with external consumers such as Lustre. The new
interfaces names are given below. The versions of these functions that
don't take a dnodesize parameter now just call the _dnsize() versions
with a dnodesize of 0, which means use the legacy dnode size.
New DMU interfaces:
dmu_object_alloc_dnsize()
dmu_object_claim_dnsize()
dmu_object_reclaim_dnsize()
New ZAP interfaces:
zap_create_dnsize()
zap_create_norm_dnsize()
zap_create_flags_dnsize()
zap_create_claim_norm_dnsize()
zap_create_link_dnsize()
The constant DN_MAX_BONUSLEN is renamed to DN_OLD_MAX_BONUSLEN. The
spa_maxdnodesize() function should be used to determine the maximum
bonus length for a pool.
These are a few noteworthy changes to key functions:
* The prototype for dnode_hold_impl() now takes a "slots" parameter.
When the DNODE_MUST_BE_FREE flag is set, this parameter is used to
ensure the hole at the specified object offset is large enough to
hold the dnode being created. The slots parameter is also used
to ensure a dnode does not span multiple dnode blocks. In both of
these cases, if a failure occurs, ENOSPC is returned. Keep in mind,
these failure cases are only possible when using DNODE_MUST_BE_FREE.
If the DNODE_MUST_BE_ALLOCATED flag is set, "slots" must be 0.
dnode_hold_impl() will check if the requested dnode is already
consumed as an extra dnode slot by an large dnode, in which case
it returns ENOENT.
* The function dmu_object_alloc() advances to the next dnode block
if dnode_hold_impl() returns an error for a requested object.
This is because the beginning of the next dnode block is the only
location it can safely assume to either be a hole or a valid
starting point for a dnode.
* dnode_next_offset_level() and other functions that iterate
through dnode blocks may no longer use a simple array indexing
scheme. These now use the current dnode's dn_num_slots field to
advance to the next dnode in the block. This is to ensure we
properly skip the current dnode's bonus area and don't interpret it
as a valid dnode.
zdb
---
The zdb command was updated to display a dnode's size under the
"dnsize" column when the object is dumped.
For ZIL create log records, zdb will now display the slot count for
the object.
ztest
-----
Ztest chooses a random dnodesize for every newly created object. The
random distribution is more heavily weighted toward small dnodes to
better simulate real-world datasets.
Unused bonus buffer space is filled with non-zero values computed from
the object number, dataset id, offset, and generation number. This
helps ensure that the dnode traversal code properly skips the interior
regions of large dnodes, and that these interior regions are not
overwritten by data belonging to other dnodes. A new test visits each
object in a dataset. It verifies that the actual dnode size matches what
was stored in the ztest block tag when it was created. It also verifies
that the unused bonus buffer space is filled with the expected data
patterns.
ZFS Test Suite
--------------
Added six new large dnode-specific tests, and integrated the dnodesize
property into existing tests for zfs allow and send/recv.
Send/Receive
------------
ZFS send streams for datasets containing large dnodes cannot be received
on pools that don't support the large_dnode feature. A send stream with
large dnodes sets a DMU_BACKUP_FEATURE_LARGE_DNODE flag which will be
unrecognized by an incompatible receiving pool so that the zfs receive
will fail gracefully.
While not implemented here, it may be possible to generate a
backward-compatible send stream from a dataset containing large
dnodes. The implementation may be tricky, however, because the send
object record for a large dnode would need to be resized to a 512
byte dnode, possibly kicking in a spill block in the process. This
means we would need to construct a new SA layout and possibly
register it in the SA layout object. The SA layout is normally just
sent as an ordinary object record. But if we are constructing new
layouts while generating the send stream we'd have to build the SA
layout object dynamically and send it at the end of the stream.
For sending and receiving between pools that do support large dnodes,
the drr_object send record type is extended with a new field to store
the dnode slot count. This field was repurposed from unused padding
in the structure.
ZIL Replay
----------
The dnode slot count is stored in the uppermost 8 bits of the lr_foid
field. The bits were unused as the object id is currently capped at
48 bits.
Resizing Dnodes
---------------
It should be possible to resize a dnode when it is dirtied if the
current dnodesize dataset property differs from the dnode's size, but
this functionality is not currently implemented. Clearly a dnode can
only grow if there are sufficient contiguous unused slots in the
dnode block, but it should always be possible to shrink a dnode.
Growing dnodes may be useful to reduce fragmentation in a pool with
many spill blocks in use. Shrinking dnodes may be useful to allow
sending a dataset to a pool that doesn't support the large_dnode
feature.
Feature Reference Counting
--------------------------
The reference count for the large_dnode pool feature tracks the
number of datasets that have ever contained a dnode of size larger
than 512 bytes. The first time a large dnode is created in a dataset
the dataset is converted to an extensible dataset. This is a one-way
operation and the only way to decrement the feature count is to
destroy the dataset, even if the dataset no longer contains any large
dnodes. The complexity of reference counting on a per-dnode basis was
too high, so we chose to track it on a per-dataset basis similarly to
the large_block feature.
Signed-off-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #3542
2016-03-17 04:25:34 +03:00
|
|
|
return (DN_OLD_MAX_BONUSLEN);
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
int
|
2010-08-27 01:24:34 +04:00
|
|
|
dmu_set_bonus(dmu_buf_t *db_fake, int newsize, dmu_tx_t *tx)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
2010-08-27 01:24:34 +04:00
|
|
|
dmu_buf_impl_t *db = (dmu_buf_impl_t *)db_fake;
|
|
|
|
|
dnode_t *dn;
|
|
|
|
|
int error;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2010-08-27 01:24:34 +04:00
|
|
|
DB_DNODE_ENTER(db);
|
|
|
|
|
dn = DB_DNODE(db);
|
|
|
|
|
|
|
|
|
|
if (dn->dn_bonus != db) {
|
2013-03-08 22:41:28 +04:00
|
|
|
error = SET_ERROR(EINVAL);
|
2010-08-27 01:24:34 +04:00
|
|
|
} else if (newsize < 0 || newsize > db_fake->db_size) {
|
2013-03-08 22:41:28 +04:00
|
|
|
error = SET_ERROR(EINVAL);
|
2010-08-27 01:24:34 +04:00
|
|
|
} else {
|
|
|
|
|
dnode_setbonuslen(dn, newsize, tx);
|
|
|
|
|
error = 0;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
DB_DNODE_EXIT(db);
|
|
|
|
|
return (error);
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
int
|
2010-08-27 01:24:34 +04:00
|
|
|
dmu_set_bonustype(dmu_buf_t *db_fake, dmu_object_type_t type, dmu_tx_t *tx)
|
2010-05-29 00:45:14 +04:00
|
|
|
{
|
2010-08-27 01:24:34 +04:00
|
|
|
dmu_buf_impl_t *db = (dmu_buf_impl_t *)db_fake;
|
|
|
|
|
dnode_t *dn;
|
|
|
|
|
int error;
|
2010-05-29 00:45:14 +04:00
|
|
|
|
2010-08-27 01:24:34 +04:00
|
|
|
DB_DNODE_ENTER(db);
|
|
|
|
|
dn = DB_DNODE(db);
|
2010-05-29 00:45:14 +04:00
|
|
|
|
2012-12-14 03:24:15 +04:00
|
|
|
if (!DMU_OT_IS_VALID(type)) {
|
2013-03-08 22:41:28 +04:00
|
|
|
error = SET_ERROR(EINVAL);
|
2010-08-27 01:24:34 +04:00
|
|
|
} else if (dn->dn_bonus != db) {
|
2013-03-08 22:41:28 +04:00
|
|
|
error = SET_ERROR(EINVAL);
|
2010-08-27 01:24:34 +04:00
|
|
|
} else {
|
|
|
|
|
dnode_setbonus_type(dn, type, tx);
|
|
|
|
|
error = 0;
|
|
|
|
|
}
|
2010-05-29 00:45:14 +04:00
|
|
|
|
2010-08-27 01:24:34 +04:00
|
|
|
DB_DNODE_EXIT(db);
|
|
|
|
|
return (error);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
dmu_object_type_t
|
|
|
|
|
dmu_get_bonustype(dmu_buf_t *db_fake)
|
|
|
|
|
{
|
|
|
|
|
dmu_buf_impl_t *db = (dmu_buf_impl_t *)db_fake;
|
|
|
|
|
dnode_t *dn;
|
|
|
|
|
dmu_object_type_t type;
|
|
|
|
|
|
|
|
|
|
DB_DNODE_ENTER(db);
|
|
|
|
|
dn = DB_DNODE(db);
|
|
|
|
|
type = dn->dn_bonustype;
|
|
|
|
|
DB_DNODE_EXIT(db);
|
|
|
|
|
|
|
|
|
|
return (type);
|
2010-05-29 00:45:14 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
int
|
|
|
|
|
dmu_rm_spill(objset_t *os, uint64_t object, dmu_tx_t *tx)
|
|
|
|
|
{
|
|
|
|
|
dnode_t *dn;
|
|
|
|
|
int error;
|
|
|
|
|
|
|
|
|
|
error = dnode_hold(os, object, FTAG, &dn);
|
|
|
|
|
dbuf_rm_spill(dn, tx);
|
|
|
|
|
rw_enter(&dn->dn_struct_rwlock, RW_WRITER);
|
|
|
|
|
dnode_rm_spill(dn, tx);
|
|
|
|
|
rw_exit(&dn->dn_struct_rwlock);
|
|
|
|
|
dnode_rele(dn, FTAG);
|
|
|
|
|
return (error);
|
|
|
|
|
}
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
/*
|
2019-01-11 01:37:43 +03:00
|
|
|
* Lookup and hold the bonus buffer for the provided dnode. If the dnode
|
|
|
|
|
* has not yet been allocated a new bonus dbuf a will be allocated.
|
|
|
|
|
* Returns ENOENT, EIO, or 0.
|
2008-11-20 23:01:55 +03:00
|
|
|
*/
|
2022-04-19 21:38:30 +03:00
|
|
|
int dmu_bonus_hold_by_dnode(dnode_t *dn, const void *tag, dmu_buf_t **dbp,
|
2019-01-11 01:37:43 +03:00
|
|
|
uint32_t flags)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
|
|
|
|
dmu_buf_impl_t *db;
|
|
|
|
|
int error;
|
Native Encryption for ZFS on Linux
This change incorporates three major pieces:
The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
convenient.
The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.
The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
compromised.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #494
Closes #5769
2017-08-14 20:36:48 +03:00
|
|
|
uint32_t db_flags = DB_RF_MUST_SUCCEED;
|
|
|
|
|
|
|
|
|
|
if (flags & DMU_READ_NO_PREFETCH)
|
|
|
|
|
db_flags |= DB_RF_NOPREFETCH;
|
|
|
|
|
if (flags & DMU_READ_NO_DECRYPT)
|
|
|
|
|
db_flags |= DB_RF_NO_DECRYPT;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
rw_enter(&dn->dn_struct_rwlock, RW_READER);
|
|
|
|
|
if (dn->dn_bonus == NULL) {
|
2023-02-23 03:33:23 +03:00
|
|
|
if (!rw_tryupgrade(&dn->dn_struct_rwlock)) {
|
|
|
|
|
rw_exit(&dn->dn_struct_rwlock);
|
|
|
|
|
rw_enter(&dn->dn_struct_rwlock, RW_WRITER);
|
|
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
if (dn->dn_bonus == NULL)
|
|
|
|
|
dbuf_create_bonus(dn);
|
|
|
|
|
}
|
|
|
|
|
db = dn->dn_bonus;
|
|
|
|
|
|
|
|
|
|
/* as long as the bonus buf is held, the dnode will be held */
|
2018-09-26 20:29:26 +03:00
|
|
|
if (zfs_refcount_add(&db->db_holds, tag) == 1) {
|
2008-11-20 23:01:55 +03:00
|
|
|
VERIFY(dnode_add_ref(dn, db));
|
2015-03-12 03:03:31 +03:00
|
|
|
atomic_inc_32(&dn->dn_dbufs_count);
|
2010-08-27 01:24:34 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Wait to drop dn_struct_rwlock until after adding the bonus dbuf's
|
|
|
|
|
* hold and incrementing the dbuf count to ensure that dnode_move() sees
|
|
|
|
|
* a dnode hold for every dbuf.
|
|
|
|
|
*/
|
|
|
|
|
rw_exit(&dn->dn_struct_rwlock);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
Native Encryption for ZFS on Linux
This change incorporates three major pieces:
The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
convenient.
The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.
The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
compromised.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #494
Closes #5769
2017-08-14 20:36:48 +03:00
|
|
|
error = dbuf_read(db, NULL, db_flags);
|
|
|
|
|
if (error) {
|
|
|
|
|
dnode_evict_bonus(dn);
|
|
|
|
|
dbuf_rele(db, tag);
|
|
|
|
|
*dbp = NULL;
|
|
|
|
|
return (error);
|
|
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
*dbp = &db->db;
|
|
|
|
|
return (0);
|
|
|
|
|
}
|
|
|
|
|
|
Native Encryption for ZFS on Linux
This change incorporates three major pieces:
The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
convenient.
The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.
The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
compromised.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #494
Closes #5769
2017-08-14 20:36:48 +03:00
|
|
|
int
|
2022-04-19 21:38:30 +03:00
|
|
|
dmu_bonus_hold(objset_t *os, uint64_t object, const void *tag, dmu_buf_t **dbp)
|
Native Encryption for ZFS on Linux
This change incorporates three major pieces:
The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
convenient.
The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.
The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
compromised.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #494
Closes #5769
2017-08-14 20:36:48 +03:00
|
|
|
{
|
2019-01-11 01:37:43 +03:00
|
|
|
dnode_t *dn;
|
|
|
|
|
int error;
|
|
|
|
|
|
|
|
|
|
error = dnode_hold(os, object, FTAG, &dn);
|
|
|
|
|
if (error)
|
|
|
|
|
return (error);
|
|
|
|
|
|
|
|
|
|
error = dmu_bonus_hold_by_dnode(dn, tag, dbp, DMU_READ_NO_PREFETCH);
|
|
|
|
|
dnode_rele(dn, FTAG);
|
|
|
|
|
|
|
|
|
|
return (error);
|
Native Encryption for ZFS on Linux
This change incorporates three major pieces:
The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
convenient.
The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.
The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
compromised.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #494
Closes #5769
2017-08-14 20:36:48 +03:00
|
|
|
}
|
|
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
/*
|
|
|
|
|
* returns ENOENT, EIO, or 0.
|
|
|
|
|
*
|
|
|
|
|
* This interface will allocate a blank spill dbuf when a spill blk
|
|
|
|
|
* doesn't already exist on the dnode.
|
|
|
|
|
*
|
|
|
|
|
* if you only want to find an already existing spill db, then
|
|
|
|
|
* dmu_spill_hold_existing() should be used.
|
|
|
|
|
*/
|
|
|
|
|
int
|
2022-04-19 21:38:30 +03:00
|
|
|
dmu_spill_hold_by_dnode(dnode_t *dn, uint32_t flags, const void *tag,
|
|
|
|
|
dmu_buf_t **dbp)
|
2010-05-29 00:45:14 +04:00
|
|
|
{
|
|
|
|
|
dmu_buf_impl_t *db = NULL;
|
|
|
|
|
int err;
|
|
|
|
|
|
|
|
|
|
if ((flags & DB_RF_HAVESTRUCT) == 0)
|
|
|
|
|
rw_enter(&dn->dn_struct_rwlock, RW_READER);
|
|
|
|
|
|
|
|
|
|
db = dbuf_hold(dn, DMU_SPILL_BLKID, tag);
|
|
|
|
|
|
|
|
|
|
if ((flags & DB_RF_HAVESTRUCT) == 0)
|
|
|
|
|
rw_exit(&dn->dn_struct_rwlock);
|
|
|
|
|
|
2016-11-01 02:23:56 +03:00
|
|
|
if (db == NULL) {
|
|
|
|
|
*dbp = NULL;
|
|
|
|
|
return (SET_ERROR(EIO));
|
|
|
|
|
}
|
2010-08-27 01:24:34 +04:00
|
|
|
err = dbuf_read(db, NULL, flags);
|
|
|
|
|
if (err == 0)
|
|
|
|
|
*dbp = &db->db;
|
2016-11-01 02:23:56 +03:00
|
|
|
else {
|
2010-08-27 01:24:34 +04:00
|
|
|
dbuf_rele(db, tag);
|
2016-11-01 02:23:56 +03:00
|
|
|
*dbp = NULL;
|
|
|
|
|
}
|
2010-05-29 00:45:14 +04:00
|
|
|
return (err);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
int
|
2022-04-19 21:38:30 +03:00
|
|
|
dmu_spill_hold_existing(dmu_buf_t *bonus, const void *tag, dmu_buf_t **dbp)
|
2010-05-29 00:45:14 +04:00
|
|
|
{
|
2010-08-27 01:24:34 +04:00
|
|
|
dmu_buf_impl_t *db = (dmu_buf_impl_t *)bonus;
|
|
|
|
|
dnode_t *dn;
|
2010-05-29 00:45:14 +04:00
|
|
|
int err;
|
|
|
|
|
|
2010-08-27 01:24:34 +04:00
|
|
|
DB_DNODE_ENTER(db);
|
|
|
|
|
dn = DB_DNODE(db);
|
|
|
|
|
|
|
|
|
|
if (spa_version(dn->dn_objset->os_spa) < SPA_VERSION_SA) {
|
2013-03-08 22:41:28 +04:00
|
|
|
err = SET_ERROR(EINVAL);
|
2010-08-27 01:24:34 +04:00
|
|
|
} else {
|
|
|
|
|
rw_enter(&dn->dn_struct_rwlock, RW_READER);
|
|
|
|
|
|
|
|
|
|
if (!dn->dn_have_spill) {
|
2013-03-08 22:41:28 +04:00
|
|
|
err = SET_ERROR(ENOENT);
|
2010-08-27 01:24:34 +04:00
|
|
|
} else {
|
|
|
|
|
err = dmu_spill_hold_by_dnode(dn,
|
|
|
|
|
DB_RF_HAVESTRUCT | DB_RF_CANFAIL, tag, dbp);
|
|
|
|
|
}
|
2010-05-29 00:45:14 +04:00
|
|
|
|
|
|
|
|
rw_exit(&dn->dn_struct_rwlock);
|
|
|
|
|
}
|
2010-08-27 01:24:34 +04:00
|
|
|
|
|
|
|
|
DB_DNODE_EXIT(db);
|
2010-05-29 00:45:14 +04:00
|
|
|
return (err);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
int
|
2022-04-19 21:38:30 +03:00
|
|
|
dmu_spill_hold_by_bonus(dmu_buf_t *bonus, uint32_t flags, const void *tag,
|
2018-06-06 20:16:41 +03:00
|
|
|
dmu_buf_t **dbp)
|
2010-05-29 00:45:14 +04:00
|
|
|
{
|
2010-08-27 01:24:34 +04:00
|
|
|
dmu_buf_impl_t *db = (dmu_buf_impl_t *)bonus;
|
|
|
|
|
dnode_t *dn;
|
|
|
|
|
int err;
|
2018-06-06 20:16:41 +03:00
|
|
|
uint32_t db_flags = DB_RF_CANFAIL;
|
|
|
|
|
|
|
|
|
|
if (flags & DMU_READ_NO_DECRYPT)
|
|
|
|
|
db_flags |= DB_RF_NO_DECRYPT;
|
2010-08-27 01:24:34 +04:00
|
|
|
|
|
|
|
|
DB_DNODE_ENTER(db);
|
|
|
|
|
dn = DB_DNODE(db);
|
2018-06-06 20:16:41 +03:00
|
|
|
err = dmu_spill_hold_by_dnode(dn, db_flags, tag, dbp);
|
2010-08-27 01:24:34 +04:00
|
|
|
DB_DNODE_EXIT(db);
|
|
|
|
|
|
|
|
|
|
return (err);
|
2010-05-29 00:45:14 +04:00
|
|
|
}
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
/*
|
|
|
|
|
* Note: longer-term, we should modify all of the dmu_buf_*() interfaces
|
|
|
|
|
* to take a held dnode rather than <os, object> -- the lookup is wasteful,
|
|
|
|
|
* and can induce severe lock contention when writing to several files
|
|
|
|
|
* whose dnodes are in the same block.
|
|
|
|
|
*/
|
2019-10-11 20:06:18 +03:00
|
|
|
int
|
2009-07-03 02:44:48 +04:00
|
|
|
dmu_buf_hold_array_by_dnode(dnode_t *dn, uint64_t offset, uint64_t length,
|
2022-04-19 21:38:30 +03:00
|
|
|
boolean_t read, const void *tag, int *numbufsp, dmu_buf_t ***dbpp,
|
|
|
|
|
uint32_t flags)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
|
|
|
|
dmu_buf_t **dbp;
|
Split dmu_zfetch() speculation and execution parts
To make better predictions on parallel workloads dmu_zfetch() should
be called as early as possible to reduce possible request reordering.
In particular, it should be called before dmu_buf_hold_array_by_dnode()
calls dbuf_hold(), which may sleep waiting for indirect blocks, waking
up multiple threads same time on completion, that can significantly
reorder the requests, making the stream look like random. But we
should not issue prefetch requests before the on-demand ones, since
they may get to the disks first despite the I/O scheduler, increasing
on-demand request latency.
This patch splits dmu_zfetch() into two functions: dmu_zfetch_prepare()
and dmu_zfetch_run(). The first can be executed as early as needed.
It only updates statistics and makes predictions without issuing any
I/Os. The I/O issuance is handled by dmu_zfetch_run(), which can be
called later when all on-demand I/Os are already issued. It even
tracks the activity of other concurrent threads, issuing the prefetch
only when _all_ on-demand requests are issued.
For many years it was a big problem for storage servers, handling
deeper request queues from their clients, having to either serialize
consequential reads to make ZFS prefetcher usable, or execute the
incoming requests as-is and get almost no prefetch from ZFS, relying
only on deep enough prefetch by the clients. Benefits of those ways
varied, but neither was perfect. With this patch deeper queue
sequential read benchmarks with CrystalDiskMark from Windows via
iSCSI to FreeBSD target show me much better throughput with almost
100% prefetcher hit rate, comparing to almost zero before.
While there, I also removed per-stream zs_lock as useless, completely
covered by parent zf_lock. Also I reused zs_blocks refcount to track
zf_stream linkage of the stream, since I believe previous zs_fetch ==
NULL check in dmu_zfetch_stream_done() was racy.
Delete prefetch streams when they reach ends of files. It saves up
to 1KB of RAM per file, plus reduces searches through the stream list.
Block data prefetch (speculation and indirect block prefetch is still
done since they are cheaper) if all dbufs of the stream are already
in DMU cache. First cache miss immediately fires all the prefetch
that would be done for the stream by that time. It saves some CPU
time if same files within DMU cache capacity are read over and over.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Adam Moss <c@yotes.com>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored-By: iXsystems, Inc.
Closes #11652
2021-03-20 08:56:11 +03:00
|
|
|
zstream_t *zs = NULL;
|
2008-11-20 23:01:55 +03:00
|
|
|
uint64_t blkid, nblks, i;
|
2009-07-03 02:44:48 +04:00
|
|
|
uint32_t dbuf_flags;
|
2008-11-20 23:01:55 +03:00
|
|
|
int err;
|
2020-12-02 20:28:55 +03:00
|
|
|
zio_t *zio = NULL;
|
Split dmu_zfetch() speculation and execution parts
To make better predictions on parallel workloads dmu_zfetch() should
be called as early as possible to reduce possible request reordering.
In particular, it should be called before dmu_buf_hold_array_by_dnode()
calls dbuf_hold(), which may sleep waiting for indirect blocks, waking
up multiple threads same time on completion, that can significantly
reorder the requests, making the stream look like random. But we
should not issue prefetch requests before the on-demand ones, since
they may get to the disks first despite the I/O scheduler, increasing
on-demand request latency.
This patch splits dmu_zfetch() into two functions: dmu_zfetch_prepare()
and dmu_zfetch_run(). The first can be executed as early as needed.
It only updates statistics and makes predictions without issuing any
I/Os. The I/O issuance is handled by dmu_zfetch_run(), which can be
called later when all on-demand I/Os are already issued. It even
tracks the activity of other concurrent threads, issuing the prefetch
only when _all_ on-demand requests are issued.
For many years it was a big problem for storage servers, handling
deeper request queues from their clients, having to either serialize
consequential reads to make ZFS prefetcher usable, or execute the
incoming requests as-is and get almost no prefetch from ZFS, relying
only on deep enough prefetch by the clients. Benefits of those ways
varied, but neither was perfect. With this patch deeper queue
sequential read benchmarks with CrystalDiskMark from Windows via
iSCSI to FreeBSD target show me much better throughput with almost
100% prefetcher hit rate, comparing to almost zero before.
While there, I also removed per-stream zs_lock as useless, completely
covered by parent zf_lock. Also I reused zs_blocks refcount to track
zf_stream linkage of the stream, since I believe previous zs_fetch ==
NULL check in dmu_zfetch_stream_done() was racy.
Delete prefetch streams when they reach ends of files. It saves up
to 1KB of RAM per file, plus reduces searches through the stream list.
Block data prefetch (speculation and indirect block prefetch is still
done since they are cheaper) if all dbufs of the stream are already
in DMU cache. First cache miss immediately fires all the prefetch
that would be done for the stream by that time. It saves some CPU
time if same files within DMU cache capacity are read over and over.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Adam Moss <c@yotes.com>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored-By: iXsystems, Inc.
Closes #11652
2021-03-20 08:56:11 +03:00
|
|
|
boolean_t missed = B_FALSE;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2023-03-10 22:59:53 +03:00
|
|
|
ASSERT(!read || length <= DMU_MAX_ACCESS);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2015-12-27 00:10:31 +03:00
|
|
|
/*
|
|
|
|
|
* Note: We directly notify the prefetch code of this read, so that
|
|
|
|
|
* we can tell it about the multi-block read. dbuf_read() only knows
|
|
|
|
|
* about the one block it is accessing.
|
|
|
|
|
*/
|
|
|
|
|
dbuf_flags = DB_RF_CANFAIL | DB_RF_NEVERWAIT | DB_RF_HAVESTRUCT |
|
|
|
|
|
DB_RF_NOPREFETCH;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2022-07-29 01:52:46 +03:00
|
|
|
if ((flags & DMU_READ_NO_DECRYPT) != 0)
|
|
|
|
|
dbuf_flags |= DB_RF_NO_DECRYPT;
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
rw_enter(&dn->dn_struct_rwlock, RW_READER);
|
|
|
|
|
if (dn->dn_datablkshift) {
|
|
|
|
|
int blkshift = dn->dn_datablkshift;
|
2015-12-27 00:10:31 +03:00
|
|
|
nblks = (P2ROUNDUP(offset + length, 1ULL << blkshift) -
|
|
|
|
|
P2ALIGN(offset, 1ULL << blkshift)) >> blkshift;
|
2008-11-20 23:01:55 +03:00
|
|
|
} else {
|
|
|
|
|
if (offset + length > dn->dn_datablksz) {
|
|
|
|
|
zfs_panic_recover("zfs: accessing past end of object "
|
|
|
|
|
"%llx/%llx (size=%u access=%llu+%llu)",
|
|
|
|
|
(longlong_t)dn->dn_objset->
|
|
|
|
|
os_dsl_dataset->ds_object,
|
|
|
|
|
(longlong_t)dn->dn_object, dn->dn_datablksz,
|
|
|
|
|
(longlong_t)offset, (longlong_t)length);
|
2009-08-18 22:43:27 +04:00
|
|
|
rw_exit(&dn->dn_struct_rwlock);
|
2013-03-08 22:41:28 +04:00
|
|
|
return (SET_ERROR(EIO));
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
nblks = 1;
|
|
|
|
|
}
|
2014-11-21 03:09:39 +03:00
|
|
|
dbp = kmem_zalloc(sizeof (dmu_buf_t *) * nblks, KM_SLEEP);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2020-12-02 20:28:55 +03:00
|
|
|
if (read)
|
|
|
|
|
zio = zio_root(dn->dn_objset->os_spa, NULL, NULL,
|
|
|
|
|
ZIO_FLAG_CANFAIL);
|
2015-12-22 04:31:57 +03:00
|
|
|
blkid = dbuf_whichblock(dn, 0, offset);
|
2023-07-21 21:51:47 +03:00
|
|
|
if ((flags & DMU_READ_NO_PREFETCH) == 0) {
|
Split dmu_zfetch() speculation and execution parts
To make better predictions on parallel workloads dmu_zfetch() should
be called as early as possible to reduce possible request reordering.
In particular, it should be called before dmu_buf_hold_array_by_dnode()
calls dbuf_hold(), which may sleep waiting for indirect blocks, waking
up multiple threads same time on completion, that can significantly
reorder the requests, making the stream look like random. But we
should not issue prefetch requests before the on-demand ones, since
they may get to the disks first despite the I/O scheduler, increasing
on-demand request latency.
This patch splits dmu_zfetch() into two functions: dmu_zfetch_prepare()
and dmu_zfetch_run(). The first can be executed as early as needed.
It only updates statistics and makes predictions without issuing any
I/Os. The I/O issuance is handled by dmu_zfetch_run(), which can be
called later when all on-demand I/Os are already issued. It even
tracks the activity of other concurrent threads, issuing the prefetch
only when _all_ on-demand requests are issued.
For many years it was a big problem for storage servers, handling
deeper request queues from their clients, having to either serialize
consequential reads to make ZFS prefetcher usable, or execute the
incoming requests as-is and get almost no prefetch from ZFS, relying
only on deep enough prefetch by the clients. Benefits of those ways
varied, but neither was perfect. With this patch deeper queue
sequential read benchmarks with CrystalDiskMark from Windows via
iSCSI to FreeBSD target show me much better throughput with almost
100% prefetcher hit rate, comparing to almost zero before.
While there, I also removed per-stream zs_lock as useless, completely
covered by parent zf_lock. Also I reused zs_blocks refcount to track
zf_stream linkage of the stream, since I believe previous zs_fetch ==
NULL check in dmu_zfetch_stream_done() was racy.
Delete prefetch streams when they reach ends of files. It saves up
to 1KB of RAM per file, plus reduces searches through the stream list.
Block data prefetch (speculation and indirect block prefetch is still
done since they are cheaper) if all dbufs of the stream are already
in DMU cache. First cache miss immediately fires all the prefetch
that would be done for the stream by that time. It saves some CPU
time if same files within DMU cache capacity are read over and over.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Adam Moss <c@yotes.com>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored-By: iXsystems, Inc.
Closes #11652
2021-03-20 08:56:11 +03:00
|
|
|
/*
|
|
|
|
|
* Prepare the zfetch before initiating the demand reads, so
|
|
|
|
|
* that if multiple threads block on same indirect block, we
|
|
|
|
|
* base predictions on the original less racy request order.
|
|
|
|
|
*/
|
Implement uncached prefetch
Previously the primarycache property was handled only in the dbuf
layer. Since the speculative prefetcher is implemented in the ARC,
it had to be disabled for uncacheable buffers.
This change gives the ARC knowledge about uncacheable buffers
via arc_read() and arc_write(). So when remove_reference() drops
the last reference on the ARC header, it can either immediately destroy
it, or if it is marked as prefetch, put it into a new arc_uncached state.
That state is scanned every second, evicting stale buffers that were
not demand read.
This change also tracks dbufs that were read from the beginning,
but not to the end. It is assumed that such buffers may receive further
reads, and so they are stored in dbuf cache. If a following
reads reaches the end of the buffer, it is immediately evicted.
Otherwise it will follow regular dbuf cache eviction. Since the dbuf
layer does not know actual file sizes, this logic is not applied to
the final buffer of a dnode.
Since uncacheable buffers should no longer stay in the ARC for long,
this patch also tries to optimize I/O by allocating ARC physical
buffers as linear to allow buffer sharing.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored by: iXsystems, Inc.
Closes #14243
2023-01-05 03:29:54 +03:00
|
|
|
zs = dmu_zfetch_prepare(&dn->dn_zfetch, blkid, nblks, read,
|
|
|
|
|
B_TRUE);
|
Split dmu_zfetch() speculation and execution parts
To make better predictions on parallel workloads dmu_zfetch() should
be called as early as possible to reduce possible request reordering.
In particular, it should be called before dmu_buf_hold_array_by_dnode()
calls dbuf_hold(), which may sleep waiting for indirect blocks, waking
up multiple threads same time on completion, that can significantly
reorder the requests, making the stream look like random. But we
should not issue prefetch requests before the on-demand ones, since
they may get to the disks first despite the I/O scheduler, increasing
on-demand request latency.
This patch splits dmu_zfetch() into two functions: dmu_zfetch_prepare()
and dmu_zfetch_run(). The first can be executed as early as needed.
It only updates statistics and makes predictions without issuing any
I/Os. The I/O issuance is handled by dmu_zfetch_run(), which can be
called later when all on-demand I/Os are already issued. It even
tracks the activity of other concurrent threads, issuing the prefetch
only when _all_ on-demand requests are issued.
For many years it was a big problem for storage servers, handling
deeper request queues from their clients, having to either serialize
consequential reads to make ZFS prefetcher usable, or execute the
incoming requests as-is and get almost no prefetch from ZFS, relying
only on deep enough prefetch by the clients. Benefits of those ways
varied, but neither was perfect. With this patch deeper queue
sequential read benchmarks with CrystalDiskMark from Windows via
iSCSI to FreeBSD target show me much better throughput with almost
100% prefetcher hit rate, comparing to almost zero before.
While there, I also removed per-stream zs_lock as useless, completely
covered by parent zf_lock. Also I reused zs_blocks refcount to track
zf_stream linkage of the stream, since I believe previous zs_fetch ==
NULL check in dmu_zfetch_stream_done() was racy.
Delete prefetch streams when they reach ends of files. It saves up
to 1KB of RAM per file, plus reduces searches through the stream list.
Block data prefetch (speculation and indirect block prefetch is still
done since they are cheaper) if all dbufs of the stream are already
in DMU cache. First cache miss immediately fires all the prefetch
that would be done for the stream by that time. It saves some CPU
time if same files within DMU cache capacity are read over and over.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Adam Moss <c@yotes.com>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored-By: iXsystems, Inc.
Closes #11652
2021-03-20 08:56:11 +03:00
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
for (i = 0; i < nblks; i++) {
|
2015-12-27 00:10:31 +03:00
|
|
|
dmu_buf_impl_t *db = dbuf_hold(dn, blkid + i, tag);
|
2008-11-20 23:01:55 +03:00
|
|
|
if (db == NULL) {
|
Split dmu_zfetch() speculation and execution parts
To make better predictions on parallel workloads dmu_zfetch() should
be called as early as possible to reduce possible request reordering.
In particular, it should be called before dmu_buf_hold_array_by_dnode()
calls dbuf_hold(), which may sleep waiting for indirect blocks, waking
up multiple threads same time on completion, that can significantly
reorder the requests, making the stream look like random. But we
should not issue prefetch requests before the on-demand ones, since
they may get to the disks first despite the I/O scheduler, increasing
on-demand request latency.
This patch splits dmu_zfetch() into two functions: dmu_zfetch_prepare()
and dmu_zfetch_run(). The first can be executed as early as needed.
It only updates statistics and makes predictions without issuing any
I/Os. The I/O issuance is handled by dmu_zfetch_run(), which can be
called later when all on-demand I/Os are already issued. It even
tracks the activity of other concurrent threads, issuing the prefetch
only when _all_ on-demand requests are issued.
For many years it was a big problem for storage servers, handling
deeper request queues from their clients, having to either serialize
consequential reads to make ZFS prefetcher usable, or execute the
incoming requests as-is and get almost no prefetch from ZFS, relying
only on deep enough prefetch by the clients. Benefits of those ways
varied, but neither was perfect. With this patch deeper queue
sequential read benchmarks with CrystalDiskMark from Windows via
iSCSI to FreeBSD target show me much better throughput with almost
100% prefetcher hit rate, comparing to almost zero before.
While there, I also removed per-stream zs_lock as useless, completely
covered by parent zf_lock. Also I reused zs_blocks refcount to track
zf_stream linkage of the stream, since I believe previous zs_fetch ==
NULL check in dmu_zfetch_stream_done() was racy.
Delete prefetch streams when they reach ends of files. It saves up
to 1KB of RAM per file, plus reduces searches through the stream list.
Block data prefetch (speculation and indirect block prefetch is still
done since they are cheaper) if all dbufs of the stream are already
in DMU cache. First cache miss immediately fires all the prefetch
that would be done for the stream by that time. It saves some CPU
time if same files within DMU cache capacity are read over and over.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Adam Moss <c@yotes.com>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored-By: iXsystems, Inc.
Closes #11652
2021-03-20 08:56:11 +03:00
|
|
|
if (zs)
|
|
|
|
|
dmu_zfetch_run(zs, missed, B_TRUE);
|
2008-11-20 23:01:55 +03:00
|
|
|
rw_exit(&dn->dn_struct_rwlock);
|
|
|
|
|
dmu_buf_rele_array(dbp, nblks, tag);
|
2020-12-02 20:28:55 +03:00
|
|
|
if (read)
|
|
|
|
|
zio_nowait(zio);
|
2013-03-08 22:41:28 +04:00
|
|
|
return (SET_ERROR(EIO));
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
2015-12-27 00:10:31 +03:00
|
|
|
|
Split dmu_zfetch() speculation and execution parts
To make better predictions on parallel workloads dmu_zfetch() should
be called as early as possible to reduce possible request reordering.
In particular, it should be called before dmu_buf_hold_array_by_dnode()
calls dbuf_hold(), which may sleep waiting for indirect blocks, waking
up multiple threads same time on completion, that can significantly
reorder the requests, making the stream look like random. But we
should not issue prefetch requests before the on-demand ones, since
they may get to the disks first despite the I/O scheduler, increasing
on-demand request latency.
This patch splits dmu_zfetch() into two functions: dmu_zfetch_prepare()
and dmu_zfetch_run(). The first can be executed as early as needed.
It only updates statistics and makes predictions without issuing any
I/Os. The I/O issuance is handled by dmu_zfetch_run(), which can be
called later when all on-demand I/Os are already issued. It even
tracks the activity of other concurrent threads, issuing the prefetch
only when _all_ on-demand requests are issued.
For many years it was a big problem for storage servers, handling
deeper request queues from their clients, having to either serialize
consequential reads to make ZFS prefetcher usable, or execute the
incoming requests as-is and get almost no prefetch from ZFS, relying
only on deep enough prefetch by the clients. Benefits of those ways
varied, but neither was perfect. With this patch deeper queue
sequential read benchmarks with CrystalDiskMark from Windows via
iSCSI to FreeBSD target show me much better throughput with almost
100% prefetcher hit rate, comparing to almost zero before.
While there, I also removed per-stream zs_lock as useless, completely
covered by parent zf_lock. Also I reused zs_blocks refcount to track
zf_stream linkage of the stream, since I believe previous zs_fetch ==
NULL check in dmu_zfetch_stream_done() was racy.
Delete prefetch streams when they reach ends of files. It saves up
to 1KB of RAM per file, plus reduces searches through the stream list.
Block data prefetch (speculation and indirect block prefetch is still
done since they are cheaper) if all dbufs of the stream are already
in DMU cache. First cache miss immediately fires all the prefetch
that would be done for the stream by that time. It saves some CPU
time if same files within DMU cache capacity are read over and over.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Adam Moss <c@yotes.com>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored-By: iXsystems, Inc.
Closes #11652
2021-03-20 08:56:11 +03:00
|
|
|
/*
|
|
|
|
|
* Initiate async demand data read.
|
|
|
|
|
* We check the db_state after calling dbuf_read() because
|
|
|
|
|
* (1) dbuf_read() may change the state to CACHED due to a
|
|
|
|
|
* hit in the ARC, and (2) on a cache miss, a child will
|
|
|
|
|
* have been added to "zio" but not yet completed, so the
|
|
|
|
|
* state will not yet be CACHED.
|
|
|
|
|
*/
|
|
|
|
|
if (read) {
|
Implement uncached prefetch
Previously the primarycache property was handled only in the dbuf
layer. Since the speculative prefetcher is implemented in the ARC,
it had to be disabled for uncacheable buffers.
This change gives the ARC knowledge about uncacheable buffers
via arc_read() and arc_write(). So when remove_reference() drops
the last reference on the ARC header, it can either immediately destroy
it, or if it is marked as prefetch, put it into a new arc_uncached state.
That state is scanned every second, evicting stale buffers that were
not demand read.
This change also tracks dbufs that were read from the beginning,
but not to the end. It is assumed that such buffers may receive further
reads, and so they are stored in dbuf cache. If a following
reads reaches the end of the buffer, it is immediately evicted.
Otherwise it will follow regular dbuf cache eviction. Since the dbuf
layer does not know actual file sizes, this logic is not applied to
the final buffer of a dnode.
Since uncacheable buffers should no longer stay in the ARC for long,
this patch also tries to optimize I/O by allocating ARC physical
buffers as linear to allow buffer sharing.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored by: iXsystems, Inc.
Closes #14243
2023-01-05 03:29:54 +03:00
|
|
|
if (i == nblks - 1 && blkid + i < dn->dn_maxblkid &&
|
|
|
|
|
offset + length < db->db.db_offset +
|
|
|
|
|
db->db.db_size) {
|
|
|
|
|
if (offset <= db->db.db_offset)
|
|
|
|
|
dbuf_flags |= DB_RF_PARTIAL_FIRST;
|
|
|
|
|
else
|
|
|
|
|
dbuf_flags |= DB_RF_PARTIAL_MORE;
|
|
|
|
|
}
|
2009-07-03 02:44:48 +04:00
|
|
|
(void) dbuf_read(db, zio, dbuf_flags);
|
Split dmu_zfetch() speculation and execution parts
To make better predictions on parallel workloads dmu_zfetch() should
be called as early as possible to reduce possible request reordering.
In particular, it should be called before dmu_buf_hold_array_by_dnode()
calls dbuf_hold(), which may sleep waiting for indirect blocks, waking
up multiple threads same time on completion, that can significantly
reorder the requests, making the stream look like random. But we
should not issue prefetch requests before the on-demand ones, since
they may get to the disks first despite the I/O scheduler, increasing
on-demand request latency.
This patch splits dmu_zfetch() into two functions: dmu_zfetch_prepare()
and dmu_zfetch_run(). The first can be executed as early as needed.
It only updates statistics and makes predictions without issuing any
I/Os. The I/O issuance is handled by dmu_zfetch_run(), which can be
called later when all on-demand I/Os are already issued. It even
tracks the activity of other concurrent threads, issuing the prefetch
only when _all_ on-demand requests are issued.
For many years it was a big problem for storage servers, handling
deeper request queues from their clients, having to either serialize
consequential reads to make ZFS prefetcher usable, or execute the
incoming requests as-is and get almost no prefetch from ZFS, relying
only on deep enough prefetch by the clients. Benefits of those ways
varied, but neither was perfect. With this patch deeper queue
sequential read benchmarks with CrystalDiskMark from Windows via
iSCSI to FreeBSD target show me much better throughput with almost
100% prefetcher hit rate, comparing to almost zero before.
While there, I also removed per-stream zs_lock as useless, completely
covered by parent zf_lock. Also I reused zs_blocks refcount to track
zf_stream linkage of the stream, since I believe previous zs_fetch ==
NULL check in dmu_zfetch_stream_done() was racy.
Delete prefetch streams when they reach ends of files. It saves up
to 1KB of RAM per file, plus reduces searches through the stream list.
Block data prefetch (speculation and indirect block prefetch is still
done since they are cheaper) if all dbufs of the stream are already
in DMU cache. First cache miss immediately fires all the prefetch
that would be done for the stream by that time. It saves some CPU
time if same files within DMU cache capacity are read over and over.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Adam Moss <c@yotes.com>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored-By: iXsystems, Inc.
Closes #11652
2021-03-20 08:56:11 +03:00
|
|
|
if (db->db_state != DB_CACHED)
|
|
|
|
|
missed = B_TRUE;
|
|
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
dbp[i] = &db->db;
|
|
|
|
|
}
|
2015-12-27 00:10:31 +03:00
|
|
|
|
2021-02-20 09:34:33 +03:00
|
|
|
if (!read)
|
|
|
|
|
zfs_racct_write(length, nblks);
|
|
|
|
|
|
Split dmu_zfetch() speculation and execution parts
To make better predictions on parallel workloads dmu_zfetch() should
be called as early as possible to reduce possible request reordering.
In particular, it should be called before dmu_buf_hold_array_by_dnode()
calls dbuf_hold(), which may sleep waiting for indirect blocks, waking
up multiple threads same time on completion, that can significantly
reorder the requests, making the stream look like random. But we
should not issue prefetch requests before the on-demand ones, since
they may get to the disks first despite the I/O scheduler, increasing
on-demand request latency.
This patch splits dmu_zfetch() into two functions: dmu_zfetch_prepare()
and dmu_zfetch_run(). The first can be executed as early as needed.
It only updates statistics and makes predictions without issuing any
I/Os. The I/O issuance is handled by dmu_zfetch_run(), which can be
called later when all on-demand I/Os are already issued. It even
tracks the activity of other concurrent threads, issuing the prefetch
only when _all_ on-demand requests are issued.
For many years it was a big problem for storage servers, handling
deeper request queues from their clients, having to either serialize
consequential reads to make ZFS prefetcher usable, or execute the
incoming requests as-is and get almost no prefetch from ZFS, relying
only on deep enough prefetch by the clients. Benefits of those ways
varied, but neither was perfect. With this patch deeper queue
sequential read benchmarks with CrystalDiskMark from Windows via
iSCSI to FreeBSD target show me much better throughput with almost
100% prefetcher hit rate, comparing to almost zero before.
While there, I also removed per-stream zs_lock as useless, completely
covered by parent zf_lock. Also I reused zs_blocks refcount to track
zf_stream linkage of the stream, since I believe previous zs_fetch ==
NULL check in dmu_zfetch_stream_done() was racy.
Delete prefetch streams when they reach ends of files. It saves up
to 1KB of RAM per file, plus reduces searches through the stream list.
Block data prefetch (speculation and indirect block prefetch is still
done since they are cheaper) if all dbufs of the stream are already
in DMU cache. First cache miss immediately fires all the prefetch
that would be done for the stream by that time. It saves some CPU
time if same files within DMU cache capacity are read over and over.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Adam Moss <c@yotes.com>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored-By: iXsystems, Inc.
Closes #11652
2021-03-20 08:56:11 +03:00
|
|
|
if (zs)
|
|
|
|
|
dmu_zfetch_run(zs, missed, B_TRUE);
|
2008-11-20 23:01:55 +03:00
|
|
|
rw_exit(&dn->dn_struct_rwlock);
|
|
|
|
|
|
|
|
|
|
if (read) {
|
2020-12-02 20:28:55 +03:00
|
|
|
/* wait for async read i/o */
|
|
|
|
|
err = zio_wait(zio);
|
|
|
|
|
if (err) {
|
|
|
|
|
dmu_buf_rele_array(dbp, nblks, tag);
|
|
|
|
|
return (err);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* wait for other io to complete */
|
2008-11-20 23:01:55 +03:00
|
|
|
for (i = 0; i < nblks; i++) {
|
|
|
|
|
dmu_buf_impl_t *db = (dmu_buf_impl_t *)dbp[i];
|
|
|
|
|
mutex_enter(&db->db_mtx);
|
|
|
|
|
while (db->db_state == DB_READ ||
|
|
|
|
|
db->db_state == DB_FILL)
|
|
|
|
|
cv_wait(&db->db_changed, &db->db_mtx);
|
|
|
|
|
if (db->db_state == DB_UNCACHED)
|
2013-03-08 22:41:28 +04:00
|
|
|
err = SET_ERROR(EIO);
|
2008-11-20 23:01:55 +03:00
|
|
|
mutex_exit(&db->db_mtx);
|
|
|
|
|
if (err) {
|
|
|
|
|
dmu_buf_rele_array(dbp, nblks, tag);
|
|
|
|
|
return (err);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
*numbufsp = nblks;
|
|
|
|
|
*dbpp = dbp;
|
|
|
|
|
return (0);
|
|
|
|
|
}
|
|
|
|
|
|
2021-10-13 21:01:01 +03:00
|
|
|
int
|
2008-11-20 23:01:55 +03:00
|
|
|
dmu_buf_hold_array(objset_t *os, uint64_t object, uint64_t offset,
|
2022-04-19 21:38:30 +03:00
|
|
|
uint64_t length, int read, const void *tag, int *numbufsp,
|
|
|
|
|
dmu_buf_t ***dbpp)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
|
|
|
|
dnode_t *dn;
|
|
|
|
|
int err;
|
|
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
err = dnode_hold(os, object, FTAG, &dn);
|
2008-11-20 23:01:55 +03:00
|
|
|
if (err)
|
|
|
|
|
return (err);
|
|
|
|
|
|
|
|
|
|
err = dmu_buf_hold_array_by_dnode(dn, offset, length, read, tag,
|
2009-07-03 02:44:48 +04:00
|
|
|
numbufsp, dbpp, DMU_READ_PREFETCH);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
dnode_rele(dn, FTAG);
|
|
|
|
|
|
|
|
|
|
return (err);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
int
|
2010-08-27 01:24:34 +04:00
|
|
|
dmu_buf_hold_array_by_bonus(dmu_buf_t *db_fake, uint64_t offset,
|
2022-04-19 21:38:30 +03:00
|
|
|
uint64_t length, boolean_t read, const void *tag, int *numbufsp,
|
2015-12-27 00:10:31 +03:00
|
|
|
dmu_buf_t ***dbpp)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
2010-08-27 01:24:34 +04:00
|
|
|
dmu_buf_impl_t *db = (dmu_buf_impl_t *)db_fake;
|
|
|
|
|
dnode_t *dn;
|
2008-11-20 23:01:55 +03:00
|
|
|
int err;
|
|
|
|
|
|
2010-08-27 01:24:34 +04:00
|
|
|
DB_DNODE_ENTER(db);
|
|
|
|
|
dn = DB_DNODE(db);
|
2008-11-20 23:01:55 +03:00
|
|
|
err = dmu_buf_hold_array_by_dnode(dn, offset, length, read, tag,
|
2009-07-03 02:44:48 +04:00
|
|
|
numbufsp, dbpp, DMU_READ_PREFETCH);
|
2010-08-27 01:24:34 +04:00
|
|
|
DB_DNODE_EXIT(db);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
return (err);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
2022-04-19 21:38:30 +03:00
|
|
|
dmu_buf_rele_array(dmu_buf_t **dbp_fake, int numbufs, const void *tag)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
|
|
|
|
int i;
|
|
|
|
|
dmu_buf_impl_t **dbp = (dmu_buf_impl_t **)dbp_fake;
|
|
|
|
|
|
|
|
|
|
if (numbufs == 0)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
for (i = 0; i < numbufs; i++) {
|
|
|
|
|
if (dbp[i])
|
|
|
|
|
dbuf_rele(dbp[i], tag);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
kmem_free(dbp, sizeof (dmu_buf_t *) * numbufs);
|
|
|
|
|
}
|
|
|
|
|
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 07:01:20 +04:00
|
|
|
/*
|
2023-08-07 23:54:41 +03:00
|
|
|
* Issue prefetch I/Os for the given blocks. If level is greater than 0, the
|
2019-09-03 03:56:41 +03:00
|
|
|
* indirect blocks prefetched will be those that point to the blocks containing
|
2023-08-07 23:54:41 +03:00
|
|
|
* the data starting at offset, and continuing to offset + len. If the range
|
|
|
|
|
* it too long, prefetch the first dmu_prefetch_max bytes as requested, while
|
|
|
|
|
* for the rest only a higher level, also fitting within dmu_prefetch_max. It
|
|
|
|
|
* should primarily help random reads, since for long sequential reads there is
|
|
|
|
|
* a speculative prefetcher.
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 07:01:20 +04:00
|
|
|
*
|
Native Encryption for ZFS on Linux
This change incorporates three major pieces:
The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
convenient.
The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.
The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
compromised.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #494
Closes #5769
2017-08-14 20:36:48 +03:00
|
|
|
* Note that if the indirect blocks above the blocks being prefetched are not
|
2023-08-07 23:54:41 +03:00
|
|
|
* in cache, they will be asynchronously read in. Dnode read by dnode_hold()
|
|
|
|
|
* is currently synchronous.
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 07:01:20 +04:00
|
|
|
*/
|
2008-11-20 23:01:55 +03:00
|
|
|
void
|
2015-12-22 04:31:57 +03:00
|
|
|
dmu_prefetch(objset_t *os, uint64_t object, int64_t level, uint64_t offset,
|
|
|
|
|
uint64_t len, zio_priority_t pri)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
|
|
|
|
dnode_t *dn;
|
|
|
|
|
|
2023-08-07 23:54:41 +03:00
|
|
|
if (dmu_prefetch_max == 0 || len == 0) {
|
|
|
|
|
dmu_prefetch_dnode(os, object, pri);
|
2008-11-20 23:01:55 +03:00
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
|
2023-08-07 23:54:41 +03:00
|
|
|
if (dnode_hold(os, object, FTAG, &dn) != 0)
|
2008-11-20 23:01:55 +03:00
|
|
|
return;
|
|
|
|
|
|
2024-03-26 00:58:50 +03:00
|
|
|
dmu_prefetch_by_dnode(dn, level, offset, len, pri);
|
|
|
|
|
|
|
|
|
|
dnode_rele(dn, FTAG);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
|
|
|
|
dmu_prefetch_by_dnode(dnode_t *dn, int64_t level, uint64_t offset,
|
|
|
|
|
uint64_t len, zio_priority_t pri)
|
|
|
|
|
{
|
|
|
|
|
int64_t level2 = level;
|
|
|
|
|
uint64_t start, end, start2, end2;
|
|
|
|
|
|
2015-12-22 04:31:57 +03:00
|
|
|
/*
|
2023-08-07 23:54:41 +03:00
|
|
|
* Depending on len we may do two prefetches: blocks [start, end) at
|
|
|
|
|
* level, and following blocks [start2, end2) at higher level2.
|
2015-12-22 04:31:57 +03:00
|
|
|
*/
|
2019-07-08 23:18:50 +03:00
|
|
|
rw_enter(&dn->dn_struct_rwlock, RW_READER);
|
2023-08-07 23:54:41 +03:00
|
|
|
if (dn->dn_datablkshift != 0) {
|
|
|
|
|
/*
|
|
|
|
|
* The object has multiple blocks. Calculate the full range
|
|
|
|
|
* of blocks [start, end2) and then split it into two parts,
|
|
|
|
|
* so that the first [start, end) fits into dmu_prefetch_max.
|
|
|
|
|
*/
|
|
|
|
|
start = dbuf_whichblock(dn, level, offset);
|
|
|
|
|
end2 = dbuf_whichblock(dn, level, offset + len - 1) + 1;
|
|
|
|
|
uint8_t ibs = dn->dn_indblkshift;
|
|
|
|
|
uint8_t bs = (level == 0) ? dn->dn_datablkshift : ibs;
|
|
|
|
|
uint_t limit = P2ROUNDUP(dmu_prefetch_max, 1 << bs) >> bs;
|
|
|
|
|
start2 = end = MIN(end2, start + limit);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Find level2 where [start2, end2) fits into dmu_prefetch_max.
|
|
|
|
|
*/
|
|
|
|
|
uint8_t ibps = ibs - SPA_BLKPTRSHIFT;
|
|
|
|
|
limit = P2ROUNDUP(dmu_prefetch_max, 1 << ibs) >> ibs;
|
|
|
|
|
do {
|
|
|
|
|
level2++;
|
|
|
|
|
start2 = P2ROUNDUP(start2, 1 << ibps) >> ibps;
|
|
|
|
|
end2 = P2ROUNDUP(end2, 1 << ibps) >> ibps;
|
|
|
|
|
} while (end2 - start2 > limit);
|
2008-11-20 23:01:55 +03:00
|
|
|
} else {
|
2023-08-07 23:54:41 +03:00
|
|
|
/* There is only one block. Prefetch it or nothing. */
|
|
|
|
|
start = start2 = end2 = 0;
|
|
|
|
|
end = start + (level == 0 && offset < dn->dn_datablksz);
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
2023-08-07 23:54:41 +03:00
|
|
|
for (uint64_t i = start; i < end; i++)
|
|
|
|
|
dbuf_prefetch(dn, level, i, pri, 0);
|
|
|
|
|
for (uint64_t i = start2; i < end2; i++)
|
|
|
|
|
dbuf_prefetch(dn, level2, i, pri, 0);
|
2008-11-20 23:01:55 +03:00
|
|
|
rw_exit(&dn->dn_struct_rwlock);
|
|
|
|
|
}
|
|
|
|
|
|
2023-08-07 23:54:41 +03:00
|
|
|
/*
|
|
|
|
|
* Issue prefetch I/Os for the given object's dnode.
|
|
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
dmu_prefetch_dnode(objset_t *os, uint64_t object, zio_priority_t pri)
|
|
|
|
|
{
|
|
|
|
|
if (object == 0 || object >= DN_MAX_OBJECT)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
dnode_t *dn = DMU_META_DNODE(os);
|
|
|
|
|
rw_enter(&dn->dn_struct_rwlock, RW_READER);
|
|
|
|
|
uint64_t blkid = dbuf_whichblock(dn, 0, object * sizeof (dnode_phys_t));
|
|
|
|
|
dbuf_prefetch(dn, 0, blkid, pri, 0);
|
|
|
|
|
rw_exit(&dn->dn_struct_rwlock);
|
|
|
|
|
}
|
|
|
|
|
|
2009-08-18 22:43:27 +04:00
|
|
|
/*
|
|
|
|
|
* Get the next "chunk" of file data to free. We traverse the file from
|
|
|
|
|
* the end so that the file gets shorter over time (if we crashes in the
|
|
|
|
|
* middle, this will leave us in a better state). We find allocated file
|
|
|
|
|
* data by simply searching the allocated level 1 indirects.
|
2013-08-21 08:11:52 +04:00
|
|
|
*
|
|
|
|
|
* On input, *start should be the first offset that does not need to be
|
|
|
|
|
* freed (e.g. "offset + length"). On return, *start will be the first
|
2019-02-12 23:01:08 +03:00
|
|
|
* offset that should be freed and l1blks is set to the number of level 1
|
|
|
|
|
* indirect blocks found within the chunk.
|
2009-08-18 22:43:27 +04:00
|
|
|
*/
|
2008-12-03 23:09:06 +03:00
|
|
|
static int
|
2019-02-12 23:01:08 +03:00
|
|
|
get_next_chunk(dnode_t *dn, uint64_t *start, uint64_t minimum, uint64_t *l1blks)
|
2008-12-03 23:09:06 +03:00
|
|
|
{
|
2019-02-12 23:01:08 +03:00
|
|
|
uint64_t blks;
|
2013-08-21 08:11:52 +04:00
|
|
|
uint64_t maxblks = DMU_MAX_ACCESS >> (dn->dn_indblkshift + 1);
|
|
|
|
|
/* bytes of data covered by a level-1 indirect block */
|
2019-05-29 20:17:25 +03:00
|
|
|
uint64_t iblkrange = (uint64_t)dn->dn_datablksz *
|
|
|
|
|
EPB(dn->dn_indblkshift, SPA_BLKPTRSHIFT);
|
2008-12-03 23:09:06 +03:00
|
|
|
|
2013-08-21 08:11:52 +04:00
|
|
|
ASSERT3U(minimum, <=, *start);
|
2008-12-03 23:09:06 +03:00
|
|
|
|
2019-04-25 20:16:24 +03:00
|
|
|
/*
|
|
|
|
|
* Check if we can free the entire range assuming that all of the
|
|
|
|
|
* L1 blocks in this range have data. If we can, we use this
|
|
|
|
|
* worst case value as an estimate so we can avoid having to look
|
|
|
|
|
* at the object's actual data.
|
|
|
|
|
*/
|
|
|
|
|
uint64_t total_l1blks =
|
|
|
|
|
(roundup(*start, iblkrange) - (minimum / iblkrange * iblkrange)) /
|
|
|
|
|
iblkrange;
|
|
|
|
|
if (total_l1blks <= maxblks) {
|
|
|
|
|
*l1blks = total_l1blks;
|
2013-08-21 08:11:52 +04:00
|
|
|
*start = minimum;
|
2008-12-03 23:09:06 +03:00
|
|
|
return (0);
|
|
|
|
|
}
|
2009-08-18 22:43:27 +04:00
|
|
|
ASSERT(ISP2(iblkrange));
|
2008-12-03 23:09:06 +03:00
|
|
|
|
2019-02-12 23:01:08 +03:00
|
|
|
for (blks = 0; *start > minimum && blks < maxblks; blks++) {
|
2008-12-03 23:09:06 +03:00
|
|
|
int err;
|
|
|
|
|
|
2013-08-21 08:11:52 +04:00
|
|
|
/*
|
|
|
|
|
* dnode_next_offset(BACKWARDS) will find an allocated L1
|
|
|
|
|
* indirect block at or before the input offset. We must
|
|
|
|
|
* decrement *start so that it is at the end of the region
|
|
|
|
|
* to search.
|
|
|
|
|
*/
|
|
|
|
|
(*start)--;
|
2019-04-25 20:16:24 +03:00
|
|
|
|
2008-12-03 23:09:06 +03:00
|
|
|
err = dnode_next_offset(dn,
|
2009-08-18 22:43:27 +04:00
|
|
|
DNODE_FIND_BACKWARDS, start, 2, 1, 0);
|
2008-12-03 23:09:06 +03:00
|
|
|
|
2013-08-21 08:11:52 +04:00
|
|
|
/* if there are no indirect blocks before start, we are done */
|
2009-08-18 22:43:27 +04:00
|
|
|
if (err == ESRCH) {
|
2013-08-21 08:11:52 +04:00
|
|
|
*start = minimum;
|
|
|
|
|
break;
|
|
|
|
|
} else if (err != 0) {
|
2019-02-12 23:01:08 +03:00
|
|
|
*l1blks = blks;
|
2008-12-03 23:09:06 +03:00
|
|
|
return (err);
|
2009-08-18 22:43:27 +04:00
|
|
|
}
|
2008-12-03 23:09:06 +03:00
|
|
|
|
2013-08-21 08:11:52 +04:00
|
|
|
/* set start to the beginning of this L1 indirect */
|
2009-08-18 22:43:27 +04:00
|
|
|
*start = P2ALIGN(*start, iblkrange);
|
2008-12-03 23:09:06 +03:00
|
|
|
}
|
2013-08-21 08:11:52 +04:00
|
|
|
if (*start < minimum)
|
|
|
|
|
*start = minimum;
|
2019-02-12 23:01:08 +03:00
|
|
|
*l1blks = blks;
|
2019-04-25 20:16:24 +03:00
|
|
|
|
2008-12-03 23:09:06 +03:00
|
|
|
return (0);
|
|
|
|
|
}
|
|
|
|
|
|
2017-01-27 22:46:39 +03:00
|
|
|
/*
|
|
|
|
|
* If this objset is of type OST_ZFS return true if vfs's unmounted flag is set,
|
|
|
|
|
* otherwise return false.
|
|
|
|
|
* Used below in dmu_free_long_range_impl() to enable abort when unmounting
|
|
|
|
|
*/
|
|
|
|
|
static boolean_t
|
|
|
|
|
dmu_objset_zfs_unmounting(objset_t *os)
|
|
|
|
|
{
|
|
|
|
|
#ifdef _KERNEL
|
|
|
|
|
if (dmu_objset_type(os) == DMU_OST_ZFS)
|
|
|
|
|
return (zfs_get_vfs_flag_unmounted(os));
|
2021-12-12 18:06:44 +03:00
|
|
|
#else
|
|
|
|
|
(void) os;
|
2017-01-27 22:46:39 +03:00
|
|
|
#endif
|
|
|
|
|
return (B_FALSE);
|
|
|
|
|
}
|
|
|
|
|
|
2008-12-03 23:09:06 +03:00
|
|
|
static int
|
|
|
|
|
dmu_free_long_range_impl(objset_t *os, dnode_t *dn, uint64_t offset,
|
2018-04-17 21:06:54 +03:00
|
|
|
uint64_t length)
|
2008-12-03 23:09:06 +03:00
|
|
|
{
|
2015-07-24 22:08:53 +03:00
|
|
|
uint64_t object_size;
|
2013-08-21 08:11:52 +04:00
|
|
|
int err;
|
2017-02-01 01:44:03 +03:00
|
|
|
uint64_t dirty_frees_threshold;
|
|
|
|
|
dsl_pool_t *dp = dmu_objset_pool(os);
|
2013-08-21 08:11:52 +04:00
|
|
|
|
2015-07-24 22:08:53 +03:00
|
|
|
if (dn == NULL)
|
|
|
|
|
return (SET_ERROR(EINVAL));
|
|
|
|
|
|
|
|
|
|
object_size = (dn->dn_maxblkid + 1) * dn->dn_datablksz;
|
2013-08-21 08:11:52 +04:00
|
|
|
if (offset >= object_size)
|
2008-12-03 23:09:06 +03:00
|
|
|
return (0);
|
|
|
|
|
|
2017-02-01 01:44:03 +03:00
|
|
|
if (zfs_per_txg_dirty_frees_percent <= 100)
|
|
|
|
|
dirty_frees_threshold =
|
|
|
|
|
zfs_per_txg_dirty_frees_percent * zfs_dirty_data_max / 100;
|
|
|
|
|
else
|
2019-02-12 23:01:08 +03:00
|
|
|
dirty_frees_threshold = zfs_dirty_data_max / 20;
|
2017-02-01 01:44:03 +03:00
|
|
|
|
2013-08-21 08:11:52 +04:00
|
|
|
if (length == DMU_OBJECT_END || offset + length > object_size)
|
|
|
|
|
length = object_size - offset;
|
|
|
|
|
|
|
|
|
|
while (length != 0) {
|
2017-02-01 01:44:03 +03:00
|
|
|
uint64_t chunk_end, chunk_begin, chunk_len;
|
2019-02-12 23:01:08 +03:00
|
|
|
uint64_t l1blks;
|
2013-08-21 08:11:52 +04:00
|
|
|
dmu_tx_t *tx;
|
|
|
|
|
|
2017-01-27 22:46:39 +03:00
|
|
|
if (dmu_objset_zfs_unmounting(dn->dn_objset))
|
|
|
|
|
return (SET_ERROR(EINTR));
|
|
|
|
|
|
2013-08-21 08:11:52 +04:00
|
|
|
chunk_end = chunk_begin = offset + length;
|
|
|
|
|
|
|
|
|
|
/* move chunk_begin backwards to the beginning of this chunk */
|
2019-02-12 23:01:08 +03:00
|
|
|
err = get_next_chunk(dn, &chunk_begin, offset, &l1blks);
|
2008-12-03 23:09:06 +03:00
|
|
|
if (err)
|
|
|
|
|
return (err);
|
2013-08-21 08:11:52 +04:00
|
|
|
ASSERT3U(chunk_begin, >=, offset);
|
|
|
|
|
ASSERT3U(chunk_begin, <=, chunk_end);
|
2008-12-03 23:09:06 +03:00
|
|
|
|
2017-02-01 01:44:03 +03:00
|
|
|
chunk_len = chunk_end - chunk_begin;
|
|
|
|
|
|
2008-12-03 23:09:06 +03:00
|
|
|
tx = dmu_tx_create(os);
|
2017-02-01 01:44:03 +03:00
|
|
|
dmu_tx_hold_free(tx, dn->dn_object, chunk_begin, chunk_len);
|
2014-07-07 23:49:36 +04:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Mark this transaction as typically resulting in a net
|
|
|
|
|
* reduction in space used.
|
|
|
|
|
*/
|
|
|
|
|
dmu_tx_mark_netfree(tx);
|
2008-12-03 23:09:06 +03:00
|
|
|
err = dmu_tx_assign(tx, TXG_WAIT);
|
|
|
|
|
if (err) {
|
|
|
|
|
dmu_tx_abort(tx);
|
|
|
|
|
return (err);
|
|
|
|
|
}
|
2017-02-01 01:44:03 +03:00
|
|
|
|
2019-04-25 20:16:24 +03:00
|
|
|
uint64_t txg = dmu_tx_get_txg(tx);
|
|
|
|
|
|
|
|
|
|
mutex_enter(&dp->dp_lock);
|
|
|
|
|
uint64_t long_free_dirty =
|
|
|
|
|
dp->dp_long_free_dirty_pertxg[txg & TXG_MASK];
|
|
|
|
|
mutex_exit(&dp->dp_lock);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* To avoid filling up a TXG with just frees, wait for
|
|
|
|
|
* the next TXG to open before freeing more chunks if
|
|
|
|
|
* we have reached the threshold of frees.
|
|
|
|
|
*/
|
|
|
|
|
if (dirty_frees_threshold != 0 &&
|
|
|
|
|
long_free_dirty >= dirty_frees_threshold) {
|
|
|
|
|
DMU_TX_STAT_BUMP(dmu_tx_dirty_frees_delay);
|
|
|
|
|
dmu_tx_commit(tx);
|
|
|
|
|
txg_wait_open(dp, 0, B_TRUE);
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
|
2019-02-12 23:01:08 +03:00
|
|
|
/*
|
|
|
|
|
* In order to prevent unnecessary write throttling, for each
|
|
|
|
|
* TXG, we track the cumulative size of L1 blocks being dirtied
|
|
|
|
|
* in dnode_free_range() below. We compare this number to a
|
|
|
|
|
* tunable threshold, past which we prevent new L1 dirty freeing
|
|
|
|
|
* blocks from being added into the open TXG. See
|
|
|
|
|
* dmu_free_long_range_impl() for details. The threshold
|
|
|
|
|
* prevents write throttle activation due to dirty freeing L1
|
|
|
|
|
* blocks taking up a large percentage of zfs_dirty_data_max.
|
|
|
|
|
*/
|
2017-02-01 01:44:03 +03:00
|
|
|
mutex_enter(&dp->dp_lock);
|
2019-04-25 20:16:24 +03:00
|
|
|
dp->dp_long_free_dirty_pertxg[txg & TXG_MASK] +=
|
2019-02-12 23:01:08 +03:00
|
|
|
l1blks << dn->dn_indblkshift;
|
2017-02-01 01:44:03 +03:00
|
|
|
mutex_exit(&dp->dp_lock);
|
|
|
|
|
DTRACE_PROBE3(free__long__range,
|
2019-04-25 20:16:24 +03:00
|
|
|
uint64_t, long_free_dirty, uint64_t, chunk_len,
|
|
|
|
|
uint64_t, txg);
|
2017-02-01 01:44:03 +03:00
|
|
|
dnode_free_range(dn, chunk_begin, chunk_len, tx);
|
2017-09-28 18:49:13 +03:00
|
|
|
|
2008-12-03 23:09:06 +03:00
|
|
|
dmu_tx_commit(tx);
|
2013-08-21 08:11:52 +04:00
|
|
|
|
2017-02-01 01:44:03 +03:00
|
|
|
length -= chunk_len;
|
2008-12-03 23:09:06 +03:00
|
|
|
}
|
|
|
|
|
return (0);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
int
|
|
|
|
|
dmu_free_long_range(objset_t *os, uint64_t object,
|
|
|
|
|
uint64_t offset, uint64_t length)
|
|
|
|
|
{
|
|
|
|
|
dnode_t *dn;
|
|
|
|
|
int err;
|
|
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
err = dnode_hold(os, object, FTAG, &dn);
|
2008-12-03 23:09:06 +03:00
|
|
|
if (err != 0)
|
|
|
|
|
return (err);
|
2018-04-17 21:06:54 +03:00
|
|
|
err = dmu_free_long_range_impl(os, dn, offset, length);
|
2013-08-30 13:19:35 +04:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* It is important to zero out the maxblkid when freeing the entire
|
|
|
|
|
* file, so that (a) subsequent calls to dmu_free_long_range_impl()
|
|
|
|
|
* will take the fast path, and (b) dnode_reallocate() can verify
|
|
|
|
|
* that the entire file has been freed.
|
|
|
|
|
*/
|
2013-12-09 22:37:51 +04:00
|
|
|
if (err == 0 && offset == 0 && length == DMU_OBJECT_END)
|
2013-08-30 13:19:35 +04:00
|
|
|
dn->dn_maxblkid = 0;
|
|
|
|
|
|
2008-12-03 23:09:06 +03:00
|
|
|
dnode_rele(dn, FTAG);
|
|
|
|
|
return (err);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
int
|
2018-04-17 21:06:54 +03:00
|
|
|
dmu_free_long_object(objset_t *os, uint64_t object)
|
2008-12-03 23:09:06 +03:00
|
|
|
{
|
|
|
|
|
dmu_tx_t *tx;
|
|
|
|
|
int err;
|
|
|
|
|
|
2013-08-21 08:11:52 +04:00
|
|
|
err = dmu_free_long_range(os, object, 0, DMU_OBJECT_END);
|
2008-12-03 23:09:06 +03:00
|
|
|
if (err != 0)
|
|
|
|
|
return (err);
|
2013-08-21 08:11:52 +04:00
|
|
|
|
|
|
|
|
tx = dmu_tx_create(os);
|
|
|
|
|
dmu_tx_hold_bonus(tx, object);
|
|
|
|
|
dmu_tx_hold_free(tx, object, 0, DMU_OBJECT_END);
|
2014-07-07 23:49:36 +04:00
|
|
|
dmu_tx_mark_netfree(tx);
|
2013-08-21 08:11:52 +04:00
|
|
|
err = dmu_tx_assign(tx, TXG_WAIT);
|
|
|
|
|
if (err == 0) {
|
2021-10-02 21:50:57 +03:00
|
|
|
err = dmu_object_free(os, object, tx);
|
2013-08-21 08:11:52 +04:00
|
|
|
dmu_tx_commit(tx);
|
2008-12-03 23:09:06 +03:00
|
|
|
} else {
|
2013-08-21 08:11:52 +04:00
|
|
|
dmu_tx_abort(tx);
|
2008-12-03 23:09:06 +03:00
|
|
|
}
|
2013-08-21 08:11:52 +04:00
|
|
|
|
2008-12-03 23:09:06 +03:00
|
|
|
return (err);
|
|
|
|
|
}
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
int
|
|
|
|
|
dmu_free_range(objset_t *os, uint64_t object, uint64_t offset,
|
|
|
|
|
uint64_t size, dmu_tx_t *tx)
|
|
|
|
|
{
|
|
|
|
|
dnode_t *dn;
|
2010-05-29 00:45:14 +04:00
|
|
|
int err = dnode_hold(os, object, FTAG, &dn);
|
2008-11-20 23:01:55 +03:00
|
|
|
if (err)
|
|
|
|
|
return (err);
|
|
|
|
|
ASSERT(offset < UINT64_MAX);
|
2017-10-27 02:58:38 +03:00
|
|
|
ASSERT(size == DMU_OBJECT_END || size <= UINT64_MAX - offset);
|
2008-11-20 23:01:55 +03:00
|
|
|
dnode_free_range(dn, offset, size, tx);
|
|
|
|
|
dnode_rele(dn, FTAG);
|
|
|
|
|
return (0);
|
|
|
|
|
}
|
|
|
|
|
|
2017-01-14 01:58:41 +03:00
|
|
|
static int
|
|
|
|
|
dmu_read_impl(dnode_t *dn, uint64_t offset, uint64_t size,
|
2009-07-03 02:44:48 +04:00
|
|
|
void *buf, uint32_t flags)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
|
|
|
|
dmu_buf_t **dbp;
|
2017-01-14 01:58:41 +03:00
|
|
|
int numbufs, err = 0;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Deal with odd block sizes, where there can't be data past the first
|
|
|
|
|
* block. If we ever do the tail block optimization, we will need to
|
|
|
|
|
* handle that here as well.
|
|
|
|
|
*/
|
2009-08-18 22:43:27 +04:00
|
|
|
if (dn->dn_maxblkid == 0) {
|
2015-04-30 15:20:38 +03:00
|
|
|
uint64_t newsz = offset > dn->dn_datablksz ? 0 :
|
2008-11-20 23:01:55 +03:00
|
|
|
MIN(size, dn->dn_datablksz - offset);
|
2022-02-25 16:26:54 +03:00
|
|
|
memset((char *)buf + newsz, 0, size - newsz);
|
2008-11-20 23:01:55 +03:00
|
|
|
size = newsz;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
while (size > 0) {
|
|
|
|
|
uint64_t mylen = MIN(size, DMU_MAX_ACCESS / 2);
|
2009-08-18 22:43:27 +04:00
|
|
|
int i;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* NB: we could do this block-at-a-time, but it's nice
|
|
|
|
|
* to be reading in parallel.
|
|
|
|
|
*/
|
|
|
|
|
err = dmu_buf_hold_array_by_dnode(dn, offset, mylen,
|
2009-07-03 02:44:48 +04:00
|
|
|
TRUE, FTAG, &numbufs, &dbp, flags);
|
2008-11-20 23:01:55 +03:00
|
|
|
if (err)
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
for (i = 0; i < numbufs; i++) {
|
2015-04-30 15:20:38 +03:00
|
|
|
uint64_t tocpy;
|
|
|
|
|
int64_t bufoff;
|
2008-11-20 23:01:55 +03:00
|
|
|
dmu_buf_t *db = dbp[i];
|
|
|
|
|
|
|
|
|
|
ASSERT(size > 0);
|
|
|
|
|
|
|
|
|
|
bufoff = offset - db->db_offset;
|
2015-04-30 15:20:38 +03:00
|
|
|
tocpy = MIN(db->db_size - bufoff, size);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2015-04-30 15:20:38 +03:00
|
|
|
(void) memcpy(buf, (char *)db->db_data + bufoff, tocpy);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
offset += tocpy;
|
|
|
|
|
size -= tocpy;
|
|
|
|
|
buf = (char *)buf + tocpy;
|
|
|
|
|
}
|
|
|
|
|
dmu_buf_rele_array(dbp, numbufs, FTAG);
|
|
|
|
|
}
|
|
|
|
|
return (err);
|
|
|
|
|
}
|
|
|
|
|
|
2017-01-14 01:58:41 +03:00
|
|
|
int
|
|
|
|
|
dmu_read(objset_t *os, uint64_t object, uint64_t offset, uint64_t size,
|
|
|
|
|
void *buf, uint32_t flags)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
2017-01-14 01:58:41 +03:00
|
|
|
dnode_t *dn;
|
|
|
|
|
int err;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2017-01-14 01:58:41 +03:00
|
|
|
err = dnode_hold(os, object, FTAG, &dn);
|
|
|
|
|
if (err != 0)
|
|
|
|
|
return (err);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2017-01-14 01:58:41 +03:00
|
|
|
err = dmu_read_impl(dn, offset, size, buf, flags);
|
|
|
|
|
dnode_rele(dn, FTAG);
|
|
|
|
|
return (err);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
int
|
|
|
|
|
dmu_read_by_dnode(dnode_t *dn, uint64_t offset, uint64_t size, void *buf,
|
|
|
|
|
uint32_t flags)
|
|
|
|
|
{
|
|
|
|
|
return (dmu_read_impl(dn, offset, size, buf, flags));
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
dmu_write_impl(dmu_buf_t **dbp, int numbufs, uint64_t offset, uint64_t size,
|
|
|
|
|
const void *buf, dmu_tx_t *tx)
|
|
|
|
|
{
|
|
|
|
|
int i;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
for (i = 0; i < numbufs; i++) {
|
2015-04-30 15:20:38 +03:00
|
|
|
uint64_t tocpy;
|
|
|
|
|
int64_t bufoff;
|
2008-11-20 23:01:55 +03:00
|
|
|
dmu_buf_t *db = dbp[i];
|
|
|
|
|
|
|
|
|
|
ASSERT(size > 0);
|
|
|
|
|
|
|
|
|
|
bufoff = offset - db->db_offset;
|
2015-04-30 15:20:38 +03:00
|
|
|
tocpy = MIN(db->db_size - bufoff, size);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
ASSERT(i == 0 || i == numbufs-1 || tocpy == db->db_size);
|
|
|
|
|
|
|
|
|
|
if (tocpy == db->db_size)
|
dmu: Allow buffer fills to fail
When ZFS overwrites a whole block, it does not bother to read the
old content from disk. It is a good optimization, but if the buffer
fill fails due to page fault or something else, the buffer ends up
corrupted, neither keeping old content, nor getting the new one.
On FreeBSD this is additionally complicated by page faults being
blocked by VFS layer, always returning EFAULT on attempt to write
from mmap()'ed but not yet cached address range. Normally it is
not a big problem, since after original failure VFS will retry the
write after reading the required data. The problem becomes worse
in specific case when somebody tries to write into a file its own
mmap()'ed content from the same location. In that situation the
only copy of the data is getting corrupted on the page fault and
the following retries only fixate the status quo. Block cloning
makes this issue easier to reproduce, since it does not read the
old data, unlike traditional file copy, that may work by chance.
This patch provides the fill status to dmu_buf_fill_done(), that
in case of error can destroy the corrupted buffer as if no write
happened. One more complication in case of block cloning is that
if error is possible during fill, dmu_buf_will_fill() must read
the data via fall-back to dmu_buf_will_dirty(). It is required
to allow in case of error restoring the buffer to a state after
the cloning, not not before it, that would happen if we just call
dbuf_undirty().
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Rob Norris <robn@despairlabs.com>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored by: iXsystems, Inc.
Closes #15665
2023-12-15 20:51:41 +03:00
|
|
|
dmu_buf_will_fill(db, tx, B_FALSE);
|
2008-11-20 23:01:55 +03:00
|
|
|
else
|
|
|
|
|
dmu_buf_will_dirty(db, tx);
|
|
|
|
|
|
2010-08-26 22:45:02 +04:00
|
|
|
(void) memcpy((char *)db->db_data + bufoff, buf, tocpy);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
if (tocpy == db->db_size)
|
dmu: Allow buffer fills to fail
When ZFS overwrites a whole block, it does not bother to read the
old content from disk. It is a good optimization, but if the buffer
fill fails due to page fault or something else, the buffer ends up
corrupted, neither keeping old content, nor getting the new one.
On FreeBSD this is additionally complicated by page faults being
blocked by VFS layer, always returning EFAULT on attempt to write
from mmap()'ed but not yet cached address range. Normally it is
not a big problem, since after original failure VFS will retry the
write after reading the required data. The problem becomes worse
in specific case when somebody tries to write into a file its own
mmap()'ed content from the same location. In that situation the
only copy of the data is getting corrupted on the page fault and
the following retries only fixate the status quo. Block cloning
makes this issue easier to reproduce, since it does not read the
old data, unlike traditional file copy, that may work by chance.
This patch provides the fill status to dmu_buf_fill_done(), that
in case of error can destroy the corrupted buffer as if no write
happened. One more complication in case of block cloning is that
if error is possible during fill, dmu_buf_will_fill() must read
the data via fall-back to dmu_buf_will_dirty(). It is required
to allow in case of error restoring the buffer to a state after
the cloning, not not before it, that would happen if we just call
dbuf_undirty().
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Rob Norris <robn@despairlabs.com>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored by: iXsystems, Inc.
Closes #15665
2023-12-15 20:51:41 +03:00
|
|
|
dmu_buf_fill_done(db, tx, B_FALSE);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
offset += tocpy;
|
|
|
|
|
size -= tocpy;
|
|
|
|
|
buf = (char *)buf + tocpy;
|
|
|
|
|
}
|
2017-01-14 01:58:41 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
|
|
|
|
dmu_write(objset_t *os, uint64_t object, uint64_t offset, uint64_t size,
|
|
|
|
|
const void *buf, dmu_tx_t *tx)
|
|
|
|
|
{
|
|
|
|
|
dmu_buf_t **dbp;
|
|
|
|
|
int numbufs;
|
|
|
|
|
|
|
|
|
|
if (size == 0)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
VERIFY0(dmu_buf_hold_array(os, object, offset, size,
|
|
|
|
|
FALSE, FTAG, &numbufs, &dbp));
|
|
|
|
|
dmu_write_impl(dbp, numbufs, offset, size, buf, tx);
|
|
|
|
|
dmu_buf_rele_array(dbp, numbufs, FTAG);
|
|
|
|
|
}
|
|
|
|
|
|
dmu_tx_wait() hang likely due to cv_signal() in dsl_pool_dirty_delta()
Even though the bug's writeup (Github issue #9136) is very detailed,
we still don't know exactly how we got to that state, thus I wasn't
able to reproduce the bug. That said, we can make an educated guess
combining the information on filled issue with the code.
From the fact that `dp_dirty_total` was 0 (which is less than
`zfs_dirty_data_max`) we know that there was one thread that set it to
0 and then signaled one of the waiters of `dp_spaceavail_cv` [see
`dsl_pool_dirty_delta()` which is also the only place that
`dp_dirty_total` is changed]. Thus, the only logical explaination
then for the bug being hit is that the waiter that just got awaken
didn't go through `dsl_pool_dirty_data()`. Given that this function
is only called by `dsl_pool_dirty_space()` or `dsl_pool_undirty_space()`
I can only think of two possible ways of the above scenario happening:
[1] The waiter didn't call into any of the two functions - which I
find highly unlikely (i.e. why wait on `dp_spaceavail_cv` to begin
with?).
[2] The waiter did call in one of the above function but it passed 0 as
the space/delta to be dirtied (or undirtied) and then the callee
returned immediately (e.g both `dsl_pool_dirty_space()` and
`dsl_pool_undirty_space()` return immediately when space is 0).
In any case and no matter how we got there, the easy fix would be to
just broadcast to all waiters whenever `dp_dirty_total` hits 0. That
said and given that we've never hit this before, it would make sense
to think more on why the above situation occured.
Attempting to mimic what Prakash was doing in the issue filed, I
created a dataset with `sync=always` and started doing contiguous
writes in a file within that dataset. I observed with DTrace that even
though we update the pool's dirty data accounting when we would dirty
stuff, the accounting wouldn't be decremented incrementally as we were
done with the ZIOs of those writes (the reason being that
`dbuf_write_physdone()` isn't be called as we go through the override
code paths, and thus `dsl_pool_undirty_space()` is never called). As a
result we'd have to wait until we get to `dsl_pool_sync()` where we
zero out all dirty data accounting for the pool and the current TXG's
metadata.
In addition, as Matt noted and I later verified, the same issue would
arise when using dedup.
In both cases (sync & dedup) we shouldn't have to wait until
`dsl_pool_sync()` zeros out the accounting data. According to the
comment in that part of the code, the reasons why we do the zeroing,
have nothing to do with what we observe:
````
/*
* We have written all of the accounted dirty data, so our
* dp_space_towrite should now be zero. However, some seldom-used
* code paths do not adhere to this (e.g. dbuf_undirty(), also
* rounding error in dbuf_write_physdone).
* Shore up the accounting of any dirtied space now.
*/
dsl_pool_undirty_space(dp, dp->dp_dirty_pertxg[txg & TXG_MASK], txg);
````
Ideally what we want to do is to undirty in the accounting exactly what
we dirty (I use the word ideally as we can still have rounding errors).
This would make the behavior of the system more clear and predictable.
Another interesting issue that I observed with DTrace was that we
wouldn't update any of the pool's dirty data accounting whenever we
would dirty and/or undirty MOS data. In addition, every time we would
change the size of a dbuf through `dbuf_new_size()` we wouldn't update
the accounted space dirtied in the appropriate dirty record, so when
ZIOs are done we would undirty less that we dirtied from the pool's
accounting point of view.
For the first two issues observed (sync & dedup) this patch ensures
that we still update the pool's accounting when we undirty data,
regardless of the write being physical or not.
For changes in the MOS, we first ensure to zero out the pool's dirty
data accounting in `dsl_pool_sync()` after we synced the MOS. Then we
can go ahead and enable the update of the pool's dirty data accounting
wheneve we change MOS data.
Another fix is that we now update the accounting explicitly for
counting errors in `dbuf_write_done()`.
Finally, `dbuf_new_size()` updates the accounted space of the
appropriate dirty record correctly now.
The problem is that we still don't know how the bug came up in the
issue filled. That said the issues fixed seem to be very relevant, so
instead of going with the broadcasting solution right away,
I decided to leave this patch as is.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
External-issue: DLPX-47285
Closes #9137
2019-08-16 02:53:53 +03:00
|
|
|
/*
|
|
|
|
|
* Note: Lustre is an external consumer of this interface.
|
|
|
|
|
*/
|
2017-01-14 01:58:41 +03:00
|
|
|
void
|
|
|
|
|
dmu_write_by_dnode(dnode_t *dn, uint64_t offset, uint64_t size,
|
|
|
|
|
const void *buf, dmu_tx_t *tx)
|
|
|
|
|
{
|
|
|
|
|
dmu_buf_t **dbp;
|
|
|
|
|
int numbufs;
|
|
|
|
|
|
|
|
|
|
if (size == 0)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
VERIFY0(dmu_buf_hold_array_by_dnode(dn, offset, size,
|
|
|
|
|
FALSE, FTAG, &numbufs, &dbp, DMU_READ_PREFETCH));
|
|
|
|
|
dmu_write_impl(dbp, numbufs, offset, size, buf, tx);
|
2008-11-20 23:01:55 +03:00
|
|
|
dmu_buf_rele_array(dbp, numbufs, FTAG);
|
|
|
|
|
}
|
|
|
|
|
|
2008-12-03 23:09:06 +03:00
|
|
|
void
|
|
|
|
|
dmu_prealloc(objset_t *os, uint64_t object, uint64_t offset, uint64_t size,
|
|
|
|
|
dmu_tx_t *tx)
|
|
|
|
|
{
|
|
|
|
|
dmu_buf_t **dbp;
|
|
|
|
|
int numbufs, i;
|
|
|
|
|
|
|
|
|
|
if (size == 0)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
VERIFY(0 == dmu_buf_hold_array(os, object, offset, size,
|
|
|
|
|
FALSE, FTAG, &numbufs, &dbp));
|
|
|
|
|
|
|
|
|
|
for (i = 0; i < numbufs; i++) {
|
|
|
|
|
dmu_buf_t *db = dbp[i];
|
|
|
|
|
|
|
|
|
|
dmu_buf_will_not_fill(db, tx);
|
|
|
|
|
}
|
|
|
|
|
dmu_buf_rele_array(dbp, numbufs, FTAG);
|
|
|
|
|
}
|
|
|
|
|
|
2014-06-06 01:19:08 +04:00
|
|
|
void
|
|
|
|
|
dmu_write_embedded(objset_t *os, uint64_t object, uint64_t offset,
|
|
|
|
|
void *data, uint8_t etype, uint8_t comp, int uncompressed_size,
|
|
|
|
|
int compressed_size, int byteorder, dmu_tx_t *tx)
|
|
|
|
|
{
|
|
|
|
|
dmu_buf_t *db;
|
|
|
|
|
|
|
|
|
|
ASSERT3U(etype, <, NUM_BP_EMBEDDED_TYPES);
|
|
|
|
|
ASSERT3U(comp, <, ZIO_COMPRESS_FUNCTIONS);
|
|
|
|
|
VERIFY0(dmu_buf_hold_noread(os, object, offset,
|
|
|
|
|
FTAG, &db));
|
|
|
|
|
|
|
|
|
|
dmu_buf_write_embedded(db,
|
|
|
|
|
data, (bp_embedded_type_t)etype, (enum zio_compress)comp,
|
|
|
|
|
uncompressed_size, compressed_size, byteorder, tx);
|
|
|
|
|
|
|
|
|
|
dmu_buf_rele(db, FTAG);
|
|
|
|
|
}
|
|
|
|
|
|
Implement Redacted Send/Receive
Redacted send/receive allows users to send subsets of their data to
a target system. One possible use case for this feature is to not
transmit sensitive information to a data warehousing, test/dev, or
analytics environment. Another is to save space by not replicating
unimportant data within a given dataset, for example in backup tools
like zrepl.
Redacted send/receive is a three-stage process. First, a clone (or
clones) is made of the snapshot to be sent to the target. In this
clone (or clones), all unnecessary or unwanted data is removed or
modified. This clone is then snapshotted to create the "redaction
snapshot" (or snapshots). Second, the new zfs redact command is used
to create a redaction bookmark. The redaction bookmark stores the
list of blocks in a snapshot that were modified by the redaction
snapshot(s). Finally, the redaction bookmark is passed as a parameter
to zfs send. When sending to the snapshot that was redacted, the
redaction bookmark is used to filter out blocks that contain sensitive
or unwanted information, and those blocks are not included in the send
stream. When sending from the redaction bookmark, the blocks it
contains are considered as candidate blocks in addition to those
blocks in the destination snapshot that were modified since the
creation_txg of the redaction bookmark. This step is necessary to
allow the target to rehydrate data in the case where some blocks are
accidentally or unnecessarily modified in the redaction snapshot.
The changes to bookmarks to enable fast space estimation involve
adding deadlists to bookmarks. There is also logic to manage the
life cycles of these deadlists.
The new size estimation process operates in cases where previously
an accurate estimate could not be provided. In those cases, a send
is performed where no data blocks are read, reducing the runtime
significantly and providing a byte-accurate size estimate.
Reviewed-by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Prashanth Sreenivasa <pks@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Chris Williamson <chris.williamson@delphix.com>
Reviewed-by: Pavel Zhakarov <pavel.zakharov@delphix.com>
Reviewed-by: Sebastien Roy <sebastien.roy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #7958
2019-06-19 19:48:13 +03:00
|
|
|
void
|
|
|
|
|
dmu_redact(objset_t *os, uint64_t object, uint64_t offset, uint64_t size,
|
|
|
|
|
dmu_tx_t *tx)
|
|
|
|
|
{
|
|
|
|
|
int numbufs, i;
|
|
|
|
|
dmu_buf_t **dbp;
|
|
|
|
|
|
|
|
|
|
VERIFY0(dmu_buf_hold_array(os, object, offset, size, FALSE, FTAG,
|
|
|
|
|
&numbufs, &dbp));
|
|
|
|
|
for (i = 0; i < numbufs; i++)
|
|
|
|
|
dmu_buf_redact(dbp[i], tx);
|
|
|
|
|
dmu_buf_rele_array(dbp, numbufs, FTAG);
|
|
|
|
|
}
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
#ifdef _KERNEL
|
2017-06-13 19:18:08 +03:00
|
|
|
int
|
2021-01-21 08:27:30 +03:00
|
|
|
dmu_read_uio_dnode(dnode_t *dn, zfs_uio_t *uio, uint64_t size)
|
2010-12-17 20:14:38 +03:00
|
|
|
{
|
|
|
|
|
dmu_buf_t **dbp;
|
|
|
|
|
int numbufs, i, err;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* NB: we could do this block-at-a-time, but it's nice
|
|
|
|
|
* to be reading in parallel.
|
|
|
|
|
*/
|
2021-01-21 08:27:30 +03:00
|
|
|
err = dmu_buf_hold_array_by_dnode(dn, zfs_uio_offset(uio), size,
|
2015-06-17 00:06:27 +03:00
|
|
|
TRUE, FTAG, &numbufs, &dbp, 0);
|
2010-12-17 20:14:38 +03:00
|
|
|
if (err)
|
|
|
|
|
return (err);
|
|
|
|
|
|
|
|
|
|
for (i = 0; i < numbufs; i++) {
|
2015-04-30 15:20:38 +03:00
|
|
|
uint64_t tocpy;
|
|
|
|
|
int64_t bufoff;
|
2010-12-17 20:14:38 +03:00
|
|
|
dmu_buf_t *db = dbp[i];
|
|
|
|
|
|
|
|
|
|
ASSERT(size > 0);
|
|
|
|
|
|
2021-01-21 08:27:30 +03:00
|
|
|
bufoff = zfs_uio_offset(uio) - db->db_offset;
|
2015-04-30 15:20:38 +03:00
|
|
|
tocpy = MIN(db->db_size - bufoff, size);
|
2010-12-17 20:14:38 +03:00
|
|
|
|
2021-01-21 08:27:30 +03:00
|
|
|
err = zfs_uio_fault_move((char *)db->db_data + bufoff, tocpy,
|
|
|
|
|
UIO_READ, uio);
|
|
|
|
|
|
2010-12-17 20:14:38 +03:00
|
|
|
if (err)
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
size -= tocpy;
|
|
|
|
|
}
|
|
|
|
|
dmu_buf_rele_array(dbp, numbufs, FTAG);
|
|
|
|
|
|
|
|
|
|
return (err);
|
|
|
|
|
}
|
|
|
|
|
|
2015-06-17 00:06:27 +03:00
|
|
|
/*
|
|
|
|
|
* Read 'size' bytes into the uio buffer.
|
|
|
|
|
* From object zdb->db_object.
|
2021-01-21 08:27:30 +03:00
|
|
|
* Starting at zfs_uio_offset(uio).
|
2015-06-17 00:06:27 +03:00
|
|
|
*
|
|
|
|
|
* If the caller already has a dbuf in the target object
|
|
|
|
|
* (e.g. its bonus buffer), this routine is faster than dmu_read_uio(),
|
|
|
|
|
* because we don't have to find the dnode_t for the object.
|
|
|
|
|
*/
|
|
|
|
|
int
|
2021-01-21 08:27:30 +03:00
|
|
|
dmu_read_uio_dbuf(dmu_buf_t *zdb, zfs_uio_t *uio, uint64_t size)
|
2015-06-17 00:06:27 +03:00
|
|
|
{
|
|
|
|
|
dmu_buf_impl_t *db = (dmu_buf_impl_t *)zdb;
|
|
|
|
|
dnode_t *dn;
|
|
|
|
|
int err;
|
|
|
|
|
|
|
|
|
|
if (size == 0)
|
|
|
|
|
return (0);
|
|
|
|
|
|
|
|
|
|
DB_DNODE_ENTER(db);
|
|
|
|
|
dn = DB_DNODE(db);
|
|
|
|
|
err = dmu_read_uio_dnode(dn, uio, size);
|
|
|
|
|
DB_DNODE_EXIT(db);
|
|
|
|
|
|
|
|
|
|
return (err);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Read 'size' bytes into the uio buffer.
|
|
|
|
|
* From the specified object
|
2021-01-21 08:27:30 +03:00
|
|
|
* Starting at offset zfs_uio_offset(uio).
|
2015-06-17 00:06:27 +03:00
|
|
|
*/
|
|
|
|
|
int
|
2021-01-21 08:27:30 +03:00
|
|
|
dmu_read_uio(objset_t *os, uint64_t object, zfs_uio_t *uio, uint64_t size)
|
2015-06-17 00:06:27 +03:00
|
|
|
{
|
|
|
|
|
dnode_t *dn;
|
|
|
|
|
int err;
|
|
|
|
|
|
|
|
|
|
if (size == 0)
|
|
|
|
|
return (0);
|
|
|
|
|
|
|
|
|
|
err = dnode_hold(os, object, FTAG, &dn);
|
|
|
|
|
if (err)
|
|
|
|
|
return (err);
|
|
|
|
|
|
|
|
|
|
err = dmu_read_uio_dnode(dn, uio, size);
|
|
|
|
|
|
|
|
|
|
dnode_rele(dn, FTAG);
|
|
|
|
|
|
|
|
|
|
return (err);
|
|
|
|
|
}
|
|
|
|
|
|
2017-06-13 19:18:08 +03:00
|
|
|
int
|
2021-01-21 08:27:30 +03:00
|
|
|
dmu_write_uio_dnode(dnode_t *dn, zfs_uio_t *uio, uint64_t size, dmu_tx_t *tx)
|
2010-12-17 20:14:38 +03:00
|
|
|
{
|
|
|
|
|
dmu_buf_t **dbp;
|
|
|
|
|
int numbufs;
|
|
|
|
|
int err = 0;
|
|
|
|
|
int i;
|
|
|
|
|
|
2021-01-21 08:27:30 +03:00
|
|
|
err = dmu_buf_hold_array_by_dnode(dn, zfs_uio_offset(uio), size,
|
2010-12-17 20:14:38 +03:00
|
|
|
FALSE, FTAG, &numbufs, &dbp, DMU_READ_PREFETCH);
|
|
|
|
|
if (err)
|
|
|
|
|
return (err);
|
|
|
|
|
|
|
|
|
|
for (i = 0; i < numbufs; i++) {
|
2015-04-30 15:20:38 +03:00
|
|
|
uint64_t tocpy;
|
|
|
|
|
int64_t bufoff;
|
2010-12-17 20:14:38 +03:00
|
|
|
dmu_buf_t *db = dbp[i];
|
|
|
|
|
|
|
|
|
|
ASSERT(size > 0);
|
|
|
|
|
|
dmu: Allow buffer fills to fail
When ZFS overwrites a whole block, it does not bother to read the
old content from disk. It is a good optimization, but if the buffer
fill fails due to page fault or something else, the buffer ends up
corrupted, neither keeping old content, nor getting the new one.
On FreeBSD this is additionally complicated by page faults being
blocked by VFS layer, always returning EFAULT on attempt to write
from mmap()'ed but not yet cached address range. Normally it is
not a big problem, since after original failure VFS will retry the
write after reading the required data. The problem becomes worse
in specific case when somebody tries to write into a file its own
mmap()'ed content from the same location. In that situation the
only copy of the data is getting corrupted on the page fault and
the following retries only fixate the status quo. Block cloning
makes this issue easier to reproduce, since it does not read the
old data, unlike traditional file copy, that may work by chance.
This patch provides the fill status to dmu_buf_fill_done(), that
in case of error can destroy the corrupted buffer as if no write
happened. One more complication in case of block cloning is that
if error is possible during fill, dmu_buf_will_fill() must read
the data via fall-back to dmu_buf_will_dirty(). It is required
to allow in case of error restoring the buffer to a state after
the cloning, not not before it, that would happen if we just call
dbuf_undirty().
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Rob Norris <robn@despairlabs.com>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored by: iXsystems, Inc.
Closes #15665
2023-12-15 20:51:41 +03:00
|
|
|
offset_t off = zfs_uio_offset(uio);
|
|
|
|
|
bufoff = off - db->db_offset;
|
2015-04-30 15:20:38 +03:00
|
|
|
tocpy = MIN(db->db_size - bufoff, size);
|
2010-12-17 20:14:38 +03:00
|
|
|
|
|
|
|
|
ASSERT(i == 0 || i == numbufs-1 || tocpy == db->db_size);
|
|
|
|
|
|
|
|
|
|
if (tocpy == db->db_size)
|
dmu: Allow buffer fills to fail
When ZFS overwrites a whole block, it does not bother to read the
old content from disk. It is a good optimization, but if the buffer
fill fails due to page fault or something else, the buffer ends up
corrupted, neither keeping old content, nor getting the new one.
On FreeBSD this is additionally complicated by page faults being
blocked by VFS layer, always returning EFAULT on attempt to write
from mmap()'ed but not yet cached address range. Normally it is
not a big problem, since after original failure VFS will retry the
write after reading the required data. The problem becomes worse
in specific case when somebody tries to write into a file its own
mmap()'ed content from the same location. In that situation the
only copy of the data is getting corrupted on the page fault and
the following retries only fixate the status quo. Block cloning
makes this issue easier to reproduce, since it does not read the
old data, unlike traditional file copy, that may work by chance.
This patch provides the fill status to dmu_buf_fill_done(), that
in case of error can destroy the corrupted buffer as if no write
happened. One more complication in case of block cloning is that
if error is possible during fill, dmu_buf_will_fill() must read
the data via fall-back to dmu_buf_will_dirty(). It is required
to allow in case of error restoring the buffer to a state after
the cloning, not not before it, that would happen if we just call
dbuf_undirty().
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Rob Norris <robn@despairlabs.com>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored by: iXsystems, Inc.
Closes #15665
2023-12-15 20:51:41 +03:00
|
|
|
dmu_buf_will_fill(db, tx, B_TRUE);
|
2010-12-17 20:14:38 +03:00
|
|
|
else
|
|
|
|
|
dmu_buf_will_dirty(db, tx);
|
|
|
|
|
|
2021-01-21 08:27:30 +03:00
|
|
|
err = zfs_uio_fault_move((char *)db->db_data + bufoff,
|
|
|
|
|
tocpy, UIO_WRITE, uio);
|
|
|
|
|
|
dmu: Allow buffer fills to fail
When ZFS overwrites a whole block, it does not bother to read the
old content from disk. It is a good optimization, but if the buffer
fill fails due to page fault or something else, the buffer ends up
corrupted, neither keeping old content, nor getting the new one.
On FreeBSD this is additionally complicated by page faults being
blocked by VFS layer, always returning EFAULT on attempt to write
from mmap()'ed but not yet cached address range. Normally it is
not a big problem, since after original failure VFS will retry the
write after reading the required data. The problem becomes worse
in specific case when somebody tries to write into a file its own
mmap()'ed content from the same location. In that situation the
only copy of the data is getting corrupted on the page fault and
the following retries only fixate the status quo. Block cloning
makes this issue easier to reproduce, since it does not read the
old data, unlike traditional file copy, that may work by chance.
This patch provides the fill status to dmu_buf_fill_done(), that
in case of error can destroy the corrupted buffer as if no write
happened. One more complication in case of block cloning is that
if error is possible during fill, dmu_buf_will_fill() must read
the data via fall-back to dmu_buf_will_dirty(). It is required
to allow in case of error restoring the buffer to a state after
the cloning, not not before it, that would happen if we just call
dbuf_undirty().
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Rob Norris <robn@despairlabs.com>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored by: iXsystems, Inc.
Closes #15665
2023-12-15 20:51:41 +03:00
|
|
|
if (tocpy == db->db_size && dmu_buf_fill_done(db, tx, err)) {
|
|
|
|
|
/* The fill was reverted. Undo any uio progress. */
|
|
|
|
|
zfs_uio_advance(uio, off - zfs_uio_offset(uio));
|
|
|
|
|
}
|
2010-12-17 20:14:38 +03:00
|
|
|
|
|
|
|
|
if (err)
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
size -= tocpy;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
dmu_buf_rele_array(dbp, numbufs, FTAG);
|
|
|
|
|
return (err);
|
|
|
|
|
}
|
|
|
|
|
|
2015-06-17 00:06:27 +03:00
|
|
|
/*
|
|
|
|
|
* Write 'size' bytes from the uio buffer.
|
|
|
|
|
* To object zdb->db_object.
|
2021-01-21 08:27:30 +03:00
|
|
|
* Starting at offset zfs_uio_offset(uio).
|
2015-06-17 00:06:27 +03:00
|
|
|
*
|
|
|
|
|
* If the caller already has a dbuf in the target object
|
|
|
|
|
* (e.g. its bonus buffer), this routine is faster than dmu_write_uio(),
|
|
|
|
|
* because we don't have to find the dnode_t for the object.
|
|
|
|
|
*/
|
2010-05-29 00:45:14 +04:00
|
|
|
int
|
2021-01-21 08:27:30 +03:00
|
|
|
dmu_write_uio_dbuf(dmu_buf_t *zdb, zfs_uio_t *uio, uint64_t size,
|
2010-05-29 00:45:14 +04:00
|
|
|
dmu_tx_t *tx)
|
|
|
|
|
{
|
2010-08-27 01:24:34 +04:00
|
|
|
dmu_buf_impl_t *db = (dmu_buf_impl_t *)zdb;
|
|
|
|
|
dnode_t *dn;
|
|
|
|
|
int err;
|
|
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
if (size == 0)
|
|
|
|
|
return (0);
|
|
|
|
|
|
2010-08-27 01:24:34 +04:00
|
|
|
DB_DNODE_ENTER(db);
|
|
|
|
|
dn = DB_DNODE(db);
|
|
|
|
|
err = dmu_write_uio_dnode(dn, uio, size, tx);
|
|
|
|
|
DB_DNODE_EXIT(db);
|
|
|
|
|
|
|
|
|
|
return (err);
|
2010-05-29 00:45:14 +04:00
|
|
|
}
|
|
|
|
|
|
2015-06-17 00:06:27 +03:00
|
|
|
/*
|
|
|
|
|
* Write 'size' bytes from the uio buffer.
|
|
|
|
|
* To the specified object.
|
2021-01-21 08:27:30 +03:00
|
|
|
* Starting at offset zfs_uio_offset(uio).
|
2015-06-17 00:06:27 +03:00
|
|
|
*/
|
2010-05-29 00:45:14 +04:00
|
|
|
int
|
2021-01-21 08:27:30 +03:00
|
|
|
dmu_write_uio(objset_t *os, uint64_t object, zfs_uio_t *uio, uint64_t size,
|
2010-05-29 00:45:14 +04:00
|
|
|
dmu_tx_t *tx)
|
|
|
|
|
{
|
|
|
|
|
dnode_t *dn;
|
|
|
|
|
int err;
|
|
|
|
|
|
|
|
|
|
if (size == 0)
|
|
|
|
|
return (0);
|
|
|
|
|
|
|
|
|
|
err = dnode_hold(os, object, FTAG, &dn);
|
|
|
|
|
if (err)
|
|
|
|
|
return (err);
|
|
|
|
|
|
|
|
|
|
err = dmu_write_uio_dnode(dn, uio, size, tx);
|
|
|
|
|
|
|
|
|
|
dnode_rele(dn, FTAG);
|
|
|
|
|
|
|
|
|
|
return (err);
|
|
|
|
|
}
|
2010-12-17 20:14:38 +03:00
|
|
|
#endif /* _KERNEL */
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2009-07-03 02:44:48 +04:00
|
|
|
/*
|
|
|
|
|
* Allocate a loaned anonymous arc buffer.
|
|
|
|
|
*/
|
|
|
|
|
arc_buf_t *
|
|
|
|
|
dmu_request_arcbuf(dmu_buf_t *handle, int size)
|
|
|
|
|
{
|
2010-08-27 01:24:34 +04:00
|
|
|
dmu_buf_impl_t *db = (dmu_buf_impl_t *)handle;
|
2009-07-03 02:44:48 +04:00
|
|
|
|
2016-07-11 20:45:52 +03:00
|
|
|
return (arc_loan_buf(db->db_objset->os_spa, B_FALSE, size));
|
2009-07-03 02:44:48 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Free a loaned arc buffer.
|
|
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
dmu_return_arcbuf(arc_buf_t *buf)
|
|
|
|
|
{
|
|
|
|
|
arc_return_buf(buf, FTAG);
|
2016-06-02 07:04:53 +03:00
|
|
|
arc_buf_destroy(buf, FTAG);
|
2009-07-03 02:44:48 +04:00
|
|
|
}
|
|
|
|
|
|
Improve zfs receive performance with lightweight write
The performance of `zfs receive` can be bottlenecked on the CPU consumed
by the `receive_writer` thread, especially when receiving streams with
small compressed block sizes. Much of the CPU is spent creating and
destroying dbuf's and arc buf's, one for each `WRITE` record in the send
stream.
This commit introduces the concept of "lightweight writes", which allows
`zfs receive` to write to the DMU by providing an ABD, and instantiating
only a new type of `dbuf_dirty_record_t`. The dbuf and arc buf for this
"dirty leaf block" are not instantiated.
Because there is no dbuf with the dirty data, this mechanism doesn't
support reading from "lightweight-dirty" blocks (they would see the
on-disk state rather than the dirty data). Since the dedup-receive code
has been removed, `zfs receive` is write-only, so this works fine.
Because there are no arc bufs for the received data, the received data
is no longer cached in the ARC.
Testing a receive of a stream with average compressed block size of 4KB,
this commit improves performance by 50%, while also reducing CPU usage
by 50% of a CPU. On a per-block basis, CPU consumed by receive_writer()
and dbuf_evict() is now 1/7th (14%) of what it was.
Baseline: 450MB/s, CPU in receive_writer() 40% + dbuf_evict() 35%
New: 670MB/s, CPU in receive_writer() 17% + dbuf_evict() 0%
The code is also restructured in a few ways:
Added a `dr_dnode` field to the dbuf_dirty_record_t. This simplifies
some existing code that no longer needs `DB_DNODE_ENTER()` and related
routines. The new field is needed by the lightweight-type dirty record.
To ensure that the `dr_dnode` field remains valid until the dirty record
is freed, we have to ensure that the `dnode_move()` doesn't relocate the
dnode_t. To do this we keep a hold on the dnode until it's zio's have
completed. This is already done by the user-accounting code
(`userquota_updates_task()`), this commit extends that so that it always
keeps the dnode hold until zio completion (see `dnode_rele_task()`).
`dn_dirty_txg` was previously zeroed when the dnode was synced. This
was not necessary, since its meaning can be "when was this dnode last
dirtied". This change simplifies the new `dnode_rele_task()` code.
Removed some dead code related to `DRR_WRITE_BYREF` (dedup receive).
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Paul Dagnelie <pcd@delphix.com>
Reviewed-by: George Wilson <gwilson@delphix.com>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #11105
2020-12-11 21:26:02 +03:00
|
|
|
/*
|
|
|
|
|
* A "lightweight" write is faster than a regular write (e.g.
|
|
|
|
|
* dmu_write_by_dnode() or dmu_assign_arcbuf_by_dnode()), because it avoids the
|
|
|
|
|
* CPU cost of creating a dmu_buf_impl_t and arc_buf_[hdr_]_t. However, the
|
|
|
|
|
* data can not be read or overwritten until the transaction's txg has been
|
|
|
|
|
* synced. This makes it appropriate for workloads that are known to be
|
|
|
|
|
* (temporarily) write-only, like "zfs receive".
|
|
|
|
|
*
|
|
|
|
|
* A single block is written, starting at the specified offset in bytes. If
|
|
|
|
|
* the call is successful, it returns 0 and the provided abd has been
|
|
|
|
|
* consumed (the caller should not free it).
|
|
|
|
|
*/
|
|
|
|
|
int
|
|
|
|
|
dmu_lightweight_write_by_dnode(dnode_t *dn, uint64_t offset, abd_t *abd,
|
2022-10-27 19:54:54 +03:00
|
|
|
const zio_prop_t *zp, zio_flag_t flags, dmu_tx_t *tx)
|
Improve zfs receive performance with lightweight write
The performance of `zfs receive` can be bottlenecked on the CPU consumed
by the `receive_writer` thread, especially when receiving streams with
small compressed block sizes. Much of the CPU is spent creating and
destroying dbuf's and arc buf's, one for each `WRITE` record in the send
stream.
This commit introduces the concept of "lightweight writes", which allows
`zfs receive` to write to the DMU by providing an ABD, and instantiating
only a new type of `dbuf_dirty_record_t`. The dbuf and arc buf for this
"dirty leaf block" are not instantiated.
Because there is no dbuf with the dirty data, this mechanism doesn't
support reading from "lightweight-dirty" blocks (they would see the
on-disk state rather than the dirty data). Since the dedup-receive code
has been removed, `zfs receive` is write-only, so this works fine.
Because there are no arc bufs for the received data, the received data
is no longer cached in the ARC.
Testing a receive of a stream with average compressed block size of 4KB,
this commit improves performance by 50%, while also reducing CPU usage
by 50% of a CPU. On a per-block basis, CPU consumed by receive_writer()
and dbuf_evict() is now 1/7th (14%) of what it was.
Baseline: 450MB/s, CPU in receive_writer() 40% + dbuf_evict() 35%
New: 670MB/s, CPU in receive_writer() 17% + dbuf_evict() 0%
The code is also restructured in a few ways:
Added a `dr_dnode` field to the dbuf_dirty_record_t. This simplifies
some existing code that no longer needs `DB_DNODE_ENTER()` and related
routines. The new field is needed by the lightweight-type dirty record.
To ensure that the `dr_dnode` field remains valid until the dirty record
is freed, we have to ensure that the `dnode_move()` doesn't relocate the
dnode_t. To do this we keep a hold on the dnode until it's zio's have
completed. This is already done by the user-accounting code
(`userquota_updates_task()`), this commit extends that so that it always
keeps the dnode hold until zio completion (see `dnode_rele_task()`).
`dn_dirty_txg` was previously zeroed when the dnode was synced. This
was not necessary, since its meaning can be "when was this dnode last
dirtied". This change simplifies the new `dnode_rele_task()` code.
Removed some dead code related to `DRR_WRITE_BYREF` (dedup receive).
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Paul Dagnelie <pcd@delphix.com>
Reviewed-by: George Wilson <gwilson@delphix.com>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #11105
2020-12-11 21:26:02 +03:00
|
|
|
{
|
|
|
|
|
dbuf_dirty_record_t *dr =
|
|
|
|
|
dbuf_dirty_lightweight(dn, dbuf_whichblock(dn, 0, offset), tx);
|
|
|
|
|
if (dr == NULL)
|
|
|
|
|
return (SET_ERROR(EIO));
|
|
|
|
|
dr->dt.dll.dr_abd = abd;
|
|
|
|
|
dr->dt.dll.dr_props = *zp;
|
|
|
|
|
dr->dt.dll.dr_flags = flags;
|
|
|
|
|
return (0);
|
|
|
|
|
}
|
|
|
|
|
|
2009-07-03 02:44:48 +04:00
|
|
|
/*
|
|
|
|
|
* When possible directly assign passed loaned arc buffer to a dbuf.
|
|
|
|
|
* If this is not possible copy the contents of passed arc buf via
|
|
|
|
|
* dmu_write().
|
|
|
|
|
*/
|
2019-01-18 02:47:08 +03:00
|
|
|
int
|
2017-09-28 18:49:13 +03:00
|
|
|
dmu_assign_arcbuf_by_dnode(dnode_t *dn, uint64_t offset, arc_buf_t *buf,
|
2009-07-03 02:44:48 +04:00
|
|
|
dmu_tx_t *tx)
|
|
|
|
|
{
|
|
|
|
|
dmu_buf_impl_t *db;
|
2017-09-28 18:49:13 +03:00
|
|
|
objset_t *os = dn->dn_objset;
|
|
|
|
|
uint64_t object = dn->dn_object;
|
2016-07-11 20:45:52 +03:00
|
|
|
uint32_t blksz = (uint32_t)arc_buf_lsize(buf);
|
2009-07-03 02:44:48 +04:00
|
|
|
uint64_t blkid;
|
|
|
|
|
|
|
|
|
|
rw_enter(&dn->dn_struct_rwlock, RW_READER);
|
2015-12-22 04:31:57 +03:00
|
|
|
blkid = dbuf_whichblock(dn, 0, offset);
|
2019-01-18 02:47:08 +03:00
|
|
|
db = dbuf_hold(dn, blkid, FTAG);
|
2023-12-09 03:43:39 +03:00
|
|
|
rw_exit(&dn->dn_struct_rwlock);
|
2019-01-18 02:47:08 +03:00
|
|
|
if (db == NULL)
|
|
|
|
|
return (SET_ERROR(EIO));
|
2009-07-03 02:44:48 +04:00
|
|
|
|
2014-09-13 18:02:18 +04:00
|
|
|
/*
|
Improve zfs receive performance with lightweight write
The performance of `zfs receive` can be bottlenecked on the CPU consumed
by the `receive_writer` thread, especially when receiving streams with
small compressed block sizes. Much of the CPU is spent creating and
destroying dbuf's and arc buf's, one for each `WRITE` record in the send
stream.
This commit introduces the concept of "lightweight writes", which allows
`zfs receive` to write to the DMU by providing an ABD, and instantiating
only a new type of `dbuf_dirty_record_t`. The dbuf and arc buf for this
"dirty leaf block" are not instantiated.
Because there is no dbuf with the dirty data, this mechanism doesn't
support reading from "lightweight-dirty" blocks (they would see the
on-disk state rather than the dirty data). Since the dedup-receive code
has been removed, `zfs receive` is write-only, so this works fine.
Because there are no arc bufs for the received data, the received data
is no longer cached in the ARC.
Testing a receive of a stream with average compressed block size of 4KB,
this commit improves performance by 50%, while also reducing CPU usage
by 50% of a CPU. On a per-block basis, CPU consumed by receive_writer()
and dbuf_evict() is now 1/7th (14%) of what it was.
Baseline: 450MB/s, CPU in receive_writer() 40% + dbuf_evict() 35%
New: 670MB/s, CPU in receive_writer() 17% + dbuf_evict() 0%
The code is also restructured in a few ways:
Added a `dr_dnode` field to the dbuf_dirty_record_t. This simplifies
some existing code that no longer needs `DB_DNODE_ENTER()` and related
routines. The new field is needed by the lightweight-type dirty record.
To ensure that the `dr_dnode` field remains valid until the dirty record
is freed, we have to ensure that the `dnode_move()` doesn't relocate the
dnode_t. To do this we keep a hold on the dnode until it's zio's have
completed. This is already done by the user-accounting code
(`userquota_updates_task()`), this commit extends that so that it always
keeps the dnode hold until zio completion (see `dnode_rele_task()`).
`dn_dirty_txg` was previously zeroed when the dnode was synced. This
was not necessary, since its meaning can be "when was this dnode last
dirtied". This change simplifies the new `dnode_rele_task()` code.
Removed some dead code related to `DRR_WRITE_BYREF` (dedup receive).
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Paul Dagnelie <pcd@delphix.com>
Reviewed-by: George Wilson <gwilson@delphix.com>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #11105
2020-12-11 21:26:02 +03:00
|
|
|
* We can only assign if the offset is aligned and the arc buf is the
|
|
|
|
|
* same size as the dbuf.
|
2014-09-13 18:02:18 +04:00
|
|
|
*/
|
2016-07-11 20:45:52 +03:00
|
|
|
if (offset == db->db.db_offset && blksz == db->db.db_size) {
|
2021-02-20 09:34:33 +03:00
|
|
|
zfs_racct_write(blksz, 1);
|
2009-07-03 02:44:48 +04:00
|
|
|
dbuf_assign_arcbuf(db, buf, tx);
|
|
|
|
|
dbuf_rele(db, FTAG);
|
|
|
|
|
} else {
|
2016-07-11 20:45:52 +03:00
|
|
|
/* compressed bufs must always be assignable to their dbuf */
|
|
|
|
|
ASSERT3U(arc_get_compression(buf), ==, ZIO_COMPRESS_OFF);
|
2016-07-14 00:17:41 +03:00
|
|
|
ASSERT(!(buf->b_flags & ARC_BUF_FLAG_COMPRESSED));
|
2016-07-11 20:45:52 +03:00
|
|
|
|
2009-07-03 02:44:48 +04:00
|
|
|
dbuf_rele(db, FTAG);
|
2010-08-27 01:24:34 +04:00
|
|
|
dmu_write(os, object, offset, blksz, buf->b_data, tx);
|
2009-07-03 02:44:48 +04:00
|
|
|
dmu_return_arcbuf(buf);
|
|
|
|
|
}
|
2019-01-18 02:47:08 +03:00
|
|
|
|
|
|
|
|
return (0);
|
2009-07-03 02:44:48 +04:00
|
|
|
}
|
|
|
|
|
|
2019-01-18 02:47:08 +03:00
|
|
|
int
|
2017-09-28 18:49:13 +03:00
|
|
|
dmu_assign_arcbuf_by_dbuf(dmu_buf_t *handle, uint64_t offset, arc_buf_t *buf,
|
|
|
|
|
dmu_tx_t *tx)
|
|
|
|
|
{
|
2019-01-18 02:47:08 +03:00
|
|
|
int err;
|
2017-09-28 18:49:13 +03:00
|
|
|
dmu_buf_impl_t *dbuf = (dmu_buf_impl_t *)handle;
|
|
|
|
|
|
|
|
|
|
DB_DNODE_ENTER(dbuf);
|
2019-01-18 02:47:08 +03:00
|
|
|
err = dmu_assign_arcbuf_by_dnode(DB_DNODE(dbuf), offset, buf, tx);
|
2017-09-28 18:49:13 +03:00
|
|
|
DB_DNODE_EXIT(dbuf);
|
2019-01-18 02:47:08 +03:00
|
|
|
|
|
|
|
|
return (err);
|
2017-09-28 18:49:13 +03:00
|
|
|
}
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
typedef struct {
|
2010-05-29 00:45:14 +04:00
|
|
|
dbuf_dirty_record_t *dsa_dr;
|
|
|
|
|
dmu_sync_cb_t *dsa_done;
|
|
|
|
|
zgd_t *dsa_zgd;
|
|
|
|
|
dmu_tx_t *dsa_tx;
|
2008-11-20 23:01:55 +03:00
|
|
|
} dmu_sync_arg_t;
|
|
|
|
|
|
2008-12-03 23:09:06 +03:00
|
|
|
static void
|
|
|
|
|
dmu_sync_ready(zio_t *zio, arc_buf_t *buf, void *varg)
|
|
|
|
|
{
|
2021-12-12 18:06:44 +03:00
|
|
|
(void) buf;
|
2010-05-29 00:45:14 +04:00
|
|
|
dmu_sync_arg_t *dsa = varg;
|
|
|
|
|
dmu_buf_t *db = dsa->dsa_zgd->zgd_db;
|
2008-12-03 23:09:06 +03:00
|
|
|
blkptr_t *bp = zio->io_bp;
|
|
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
if (zio->io_error == 0) {
|
|
|
|
|
if (BP_IS_HOLE(bp)) {
|
|
|
|
|
/*
|
|
|
|
|
* A block of zeros may compress to a hole, but the
|
|
|
|
|
* block size still needs to be known for replay.
|
|
|
|
|
*/
|
|
|
|
|
BP_SET_LSIZE(bp, db->db_size);
|
2014-06-06 01:19:08 +04:00
|
|
|
} else if (!BP_IS_EMBEDDED(bp)) {
|
2010-05-29 00:45:14 +04:00
|
|
|
ASSERT(BP_GET_LEVEL(bp) == 0);
|
Native Encryption for ZFS on Linux
This change incorporates three major pieces:
The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
convenient.
The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.
The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
compromised.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #494
Closes #5769
2017-08-14 20:36:48 +03:00
|
|
|
BP_SET_FILL(bp, 1);
|
2010-05-29 00:45:14 +04:00
|
|
|
}
|
2008-12-03 23:09:06 +03:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
static void
|
|
|
|
|
dmu_sync_late_arrival_ready(zio_t *zio)
|
|
|
|
|
{
|
|
|
|
|
dmu_sync_ready(zio, NULL, zio->io_private);
|
|
|
|
|
}
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
static void
|
|
|
|
|
dmu_sync_done(zio_t *zio, arc_buf_t *buf, void *varg)
|
|
|
|
|
{
|
2021-12-12 18:06:44 +03:00
|
|
|
(void) buf;
|
2010-05-29 00:45:14 +04:00
|
|
|
dmu_sync_arg_t *dsa = varg;
|
|
|
|
|
dbuf_dirty_record_t *dr = dsa->dsa_dr;
|
2008-11-20 23:01:55 +03:00
|
|
|
dmu_buf_impl_t *db = dr->dr_dbuf;
|
OpenZFS 9962 - zil_commit should omit cache thrash
As a result of the changes made in 8585, it's possible for an excessive
amount of vdev flush commands to be issued under some workloads.
Specifically, when the workload consists of mostly async write activity,
interspersed with some sync write and/or fsync activity, we can end up
issuing more flush commands to the underlying storage than is actually
necessary. As a result of these flush commands, the write latency and
overall throughput of the pool can be poorly impacted (latency
increases, throughput decreases).
Currently, any time an lwb completes, the vdev(s) written to as a result
of that lwb will be issued a flush command. The intenion is so the data
written to that vdev is on stable storage, prior to communicating to any
waiting threads that their data is safe on disk.
The problem with this scheme, is that sometimes an lwb will not have any
threads waiting for it to complete. This can occur when there's async
activity that gets "converted" to sync requests, as a result of calling
the zil_async_to_sync() function via zil_commit_impl(). When this
occurs, the current code may issue many lwbs that don't have waiters
associated with them, resulting in many flush commands, potentially to
the same vdev(s).
For example, given a pool with a single vdev, and a single fsync() call
that results in 10 lwbs being written out (e.g. due to other async
writes), that will result in 10 flush commands to that single vdev (a
flush issued after each lwb write completes). Ideally, we'd only issue a
single flush command to that vdev, after all 10 lwb writes completed.
Further, and most important as it pertains to this change, since the
flush commands are often very impactful to the performance of the pool's
underlying storage, unnecessarily issuing these flush commands can
poorly impact the performance of the lwb writes themselves. Thus, we
need to avoid issuing flush commands when possible, in order to acheive
the best possible performance out of the pool's underlying storage.
This change attempts to address this problem by changing the ZIL's logic
to only issue a vdev flush command when it detects an lwb that has a
thread waiting for it to complete. When an lwb does not have threads
waiting for it, the responsibility of issuing the flush command to the
vdevs involved with that lwb's write is passed on to the "next" lwb.
It's only once a write for an lwb with waiters completes, do we issue
the vdev flush command(s). As a result, now when we issue the flush(s),
we will issue them to the vdevs involved with that specific lwb's write,
but potentially also to vdevs involved with "previous" lwb writes (i.e.
if the previous lwbs did not have waiters associated with them).
Thus, in our prior example with 10 lwbs, it's only once the last lwb
completes (which will be the lwb containing the waiter for the thread
that called fsync) will we issue the vdev flush command; all of the
other lwbs will find they have no waiters, so they'll pass the
responsibility of the flush to the "next" lwb (until reaching the last
lwb that has the waiter).
Porting Notes:
* Reconciled conflicts with the fastwrite feature.
Authored by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Matt Ahrens <matt@delphix.com>
Reviewed by: Brad Lewis <brad.lewis@delphix.com>
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Approved by: Joshua M. Clulow <josh@sysmgr.org>
Ported-by: Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
OpenZFS-issue: https://www.illumos.org/issues/9962
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/545190c6
Closes #8188
2018-10-24 00:14:27 +03:00
|
|
|
zgd_t *zgd = dsa->dsa_zgd;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Record the vdev(s) backing this blkptr so they can be flushed after
|
|
|
|
|
* the writes for the lwb have completed.
|
|
|
|
|
*/
|
|
|
|
|
if (zio->io_error == 0) {
|
|
|
|
|
zil_lwb_add_block(zgd->zgd_lwb, zgd->zgd_bp);
|
|
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
mutex_enter(&db->db_mtx);
|
|
|
|
|
ASSERT(dr->dt.dl.dr_override_state == DR_IN_DMU_SYNC);
|
2010-05-29 00:45:14 +04:00
|
|
|
if (zio->io_error == 0) {
|
2013-05-10 23:47:54 +04:00
|
|
|
dr->dt.dl.dr_nopwrite = !!(zio->io_flags & ZIO_FLAG_NOPWRITE);
|
|
|
|
|
if (dr->dt.dl.dr_nopwrite) {
|
2017-04-14 22:59:18 +03:00
|
|
|
blkptr_t *bp = zio->io_bp;
|
|
|
|
|
blkptr_t *bp_orig = &zio->io_bp_orig;
|
|
|
|
|
uint8_t chksum = BP_GET_CHECKSUM(bp_orig);
|
2013-05-10 23:47:54 +04:00
|
|
|
|
|
|
|
|
ASSERT(BP_EQUAL(bp, bp_orig));
|
2017-04-14 22:59:18 +03:00
|
|
|
VERIFY(BP_EQUAL(bp, db->db_blkptr));
|
2013-05-10 23:47:54 +04:00
|
|
|
ASSERT(zio->io_prop.zp_compress != ZIO_COMPRESS_OFF);
|
2017-04-14 22:59:18 +03:00
|
|
|
VERIFY(zio_checksum_table[chksum].ci_flags &
|
2016-06-16 01:47:05 +03:00
|
|
|
ZCHECKSUM_FLAG_NOPWRITE);
|
2013-05-10 23:47:54 +04:00
|
|
|
}
|
2010-05-29 00:45:14 +04:00
|
|
|
dr->dt.dl.dr_overridden_by = *zio->io_bp;
|
|
|
|
|
dr->dt.dl.dr_override_state = DR_OVERRIDDEN;
|
|
|
|
|
dr->dt.dl.dr_copies = zio->io_prop.zp_copies;
|
2015-03-27 05:03:22 +03:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Old style holes are filled with all zeros, whereas
|
|
|
|
|
* new-style holes maintain their lsize, type, level,
|
|
|
|
|
* and birth time (see zio_write_compress). While we
|
|
|
|
|
* need to reset the BP_SET_LSIZE() call that happened
|
|
|
|
|
* in dmu_sync_ready for old style holes, we do *not*
|
|
|
|
|
* want to wipe out the information contained in new
|
|
|
|
|
* style holes. Thus, only zero out the block pointer if
|
|
|
|
|
* it's an old style hole.
|
|
|
|
|
*/
|
|
|
|
|
if (BP_IS_HOLE(&dr->dt.dl.dr_overridden_by) &&
|
2024-03-26 01:01:54 +03:00
|
|
|
BP_GET_LOGICAL_BIRTH(&dr->dt.dl.dr_overridden_by) == 0)
|
2010-05-29 00:45:14 +04:00
|
|
|
BP_ZERO(&dr->dt.dl.dr_overridden_by);
|
|
|
|
|
} else {
|
|
|
|
|
dr->dt.dl.dr_override_state = DR_NOT_OVERRIDDEN;
|
|
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
cv_broadcast(&db->db_changed);
|
|
|
|
|
mutex_exit(&db->db_mtx);
|
|
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
dsa->dsa_done(dsa->dsa_zgd, zio->io_error);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
kmem_free(dsa, sizeof (*dsa));
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
dmu_sync_late_arrival_done(zio_t *zio)
|
|
|
|
|
{
|
|
|
|
|
blkptr_t *bp = zio->io_bp;
|
|
|
|
|
dmu_sync_arg_t *dsa = zio->io_private;
|
OpenZFS 9962 - zil_commit should omit cache thrash
As a result of the changes made in 8585, it's possible for an excessive
amount of vdev flush commands to be issued under some workloads.
Specifically, when the workload consists of mostly async write activity,
interspersed with some sync write and/or fsync activity, we can end up
issuing more flush commands to the underlying storage than is actually
necessary. As a result of these flush commands, the write latency and
overall throughput of the pool can be poorly impacted (latency
increases, throughput decreases).
Currently, any time an lwb completes, the vdev(s) written to as a result
of that lwb will be issued a flush command. The intenion is so the data
written to that vdev is on stable storage, prior to communicating to any
waiting threads that their data is safe on disk.
The problem with this scheme, is that sometimes an lwb will not have any
threads waiting for it to complete. This can occur when there's async
activity that gets "converted" to sync requests, as a result of calling
the zil_async_to_sync() function via zil_commit_impl(). When this
occurs, the current code may issue many lwbs that don't have waiters
associated with them, resulting in many flush commands, potentially to
the same vdev(s).
For example, given a pool with a single vdev, and a single fsync() call
that results in 10 lwbs being written out (e.g. due to other async
writes), that will result in 10 flush commands to that single vdev (a
flush issued after each lwb write completes). Ideally, we'd only issue a
single flush command to that vdev, after all 10 lwb writes completed.
Further, and most important as it pertains to this change, since the
flush commands are often very impactful to the performance of the pool's
underlying storage, unnecessarily issuing these flush commands can
poorly impact the performance of the lwb writes themselves. Thus, we
need to avoid issuing flush commands when possible, in order to acheive
the best possible performance out of the pool's underlying storage.
This change attempts to address this problem by changing the ZIL's logic
to only issue a vdev flush command when it detects an lwb that has a
thread waiting for it to complete. When an lwb does not have threads
waiting for it, the responsibility of issuing the flush command to the
vdevs involved with that lwb's write is passed on to the "next" lwb.
It's only once a write for an lwb with waiters completes, do we issue
the vdev flush command(s). As a result, now when we issue the flush(s),
we will issue them to the vdevs involved with that specific lwb's write,
but potentially also to vdevs involved with "previous" lwb writes (i.e.
if the previous lwbs did not have waiters associated with them).
Thus, in our prior example with 10 lwbs, it's only once the last lwb
completes (which will be the lwb containing the waiter for the thread
that called fsync) will we issue the vdev flush command; all of the
other lwbs will find they have no waiters, so they'll pass the
responsibility of the flush to the "next" lwb (until reaching the last
lwb that has the waiter).
Porting Notes:
* Reconciled conflicts with the fastwrite feature.
Authored by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Matt Ahrens <matt@delphix.com>
Reviewed by: Brad Lewis <brad.lewis@delphix.com>
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Approved by: Joshua M. Clulow <josh@sysmgr.org>
Ported-by: Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
OpenZFS-issue: https://www.illumos.org/issues/9962
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/545190c6
Closes #8188
2018-10-24 00:14:27 +03:00
|
|
|
zgd_t *zgd = dsa->dsa_zgd;
|
|
|
|
|
|
|
|
|
|
if (zio->io_error == 0) {
|
|
|
|
|
/*
|
|
|
|
|
* Record the vdev(s) backing this blkptr so they can be
|
|
|
|
|
* flushed after the writes for the lwb have completed.
|
|
|
|
|
*/
|
|
|
|
|
zil_lwb_add_block(zgd->zgd_lwb, zgd->zgd_bp);
|
|
|
|
|
|
|
|
|
|
if (!BP_IS_HOLE(bp)) {
|
2019-12-05 23:37:00 +03:00
|
|
|
blkptr_t *bp_orig __maybe_unused = &zio->io_bp_orig;
|
OpenZFS 9962 - zil_commit should omit cache thrash
As a result of the changes made in 8585, it's possible for an excessive
amount of vdev flush commands to be issued under some workloads.
Specifically, when the workload consists of mostly async write activity,
interspersed with some sync write and/or fsync activity, we can end up
issuing more flush commands to the underlying storage than is actually
necessary. As a result of these flush commands, the write latency and
overall throughput of the pool can be poorly impacted (latency
increases, throughput decreases).
Currently, any time an lwb completes, the vdev(s) written to as a result
of that lwb will be issued a flush command. The intenion is so the data
written to that vdev is on stable storage, prior to communicating to any
waiting threads that their data is safe on disk.
The problem with this scheme, is that sometimes an lwb will not have any
threads waiting for it to complete. This can occur when there's async
activity that gets "converted" to sync requests, as a result of calling
the zil_async_to_sync() function via zil_commit_impl(). When this
occurs, the current code may issue many lwbs that don't have waiters
associated with them, resulting in many flush commands, potentially to
the same vdev(s).
For example, given a pool with a single vdev, and a single fsync() call
that results in 10 lwbs being written out (e.g. due to other async
writes), that will result in 10 flush commands to that single vdev (a
flush issued after each lwb write completes). Ideally, we'd only issue a
single flush command to that vdev, after all 10 lwb writes completed.
Further, and most important as it pertains to this change, since the
flush commands are often very impactful to the performance of the pool's
underlying storage, unnecessarily issuing these flush commands can
poorly impact the performance of the lwb writes themselves. Thus, we
need to avoid issuing flush commands when possible, in order to acheive
the best possible performance out of the pool's underlying storage.
This change attempts to address this problem by changing the ZIL's logic
to only issue a vdev flush command when it detects an lwb that has a
thread waiting for it to complete. When an lwb does not have threads
waiting for it, the responsibility of issuing the flush command to the
vdevs involved with that lwb's write is passed on to the "next" lwb.
It's only once a write for an lwb with waiters completes, do we issue
the vdev flush command(s). As a result, now when we issue the flush(s),
we will issue them to the vdevs involved with that specific lwb's write,
but potentially also to vdevs involved with "previous" lwb writes (i.e.
if the previous lwbs did not have waiters associated with them).
Thus, in our prior example with 10 lwbs, it's only once the last lwb
completes (which will be the lwb containing the waiter for the thread
that called fsync) will we issue the vdev flush command; all of the
other lwbs will find they have no waiters, so they'll pass the
responsibility of the flush to the "next" lwb (until reaching the last
lwb that has the waiter).
Porting Notes:
* Reconciled conflicts with the fastwrite feature.
Authored by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Matt Ahrens <matt@delphix.com>
Reviewed by: Brad Lewis <brad.lewis@delphix.com>
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Approved by: Joshua M. Clulow <josh@sysmgr.org>
Ported-by: Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
OpenZFS-issue: https://www.illumos.org/issues/9962
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/545190c6
Closes #8188
2018-10-24 00:14:27 +03:00
|
|
|
ASSERT(!(zio->io_flags & ZIO_FLAG_NOPWRITE));
|
|
|
|
|
ASSERT(BP_IS_HOLE(bp_orig) || !BP_EQUAL(bp, bp_orig));
|
2024-03-26 01:01:54 +03:00
|
|
|
ASSERT(BP_GET_LOGICAL_BIRTH(zio->io_bp) == zio->io_txg);
|
OpenZFS 9962 - zil_commit should omit cache thrash
As a result of the changes made in 8585, it's possible for an excessive
amount of vdev flush commands to be issued under some workloads.
Specifically, when the workload consists of mostly async write activity,
interspersed with some sync write and/or fsync activity, we can end up
issuing more flush commands to the underlying storage than is actually
necessary. As a result of these flush commands, the write latency and
overall throughput of the pool can be poorly impacted (latency
increases, throughput decreases).
Currently, any time an lwb completes, the vdev(s) written to as a result
of that lwb will be issued a flush command. The intenion is so the data
written to that vdev is on stable storage, prior to communicating to any
waiting threads that their data is safe on disk.
The problem with this scheme, is that sometimes an lwb will not have any
threads waiting for it to complete. This can occur when there's async
activity that gets "converted" to sync requests, as a result of calling
the zil_async_to_sync() function via zil_commit_impl(). When this
occurs, the current code may issue many lwbs that don't have waiters
associated with them, resulting in many flush commands, potentially to
the same vdev(s).
For example, given a pool with a single vdev, and a single fsync() call
that results in 10 lwbs being written out (e.g. due to other async
writes), that will result in 10 flush commands to that single vdev (a
flush issued after each lwb write completes). Ideally, we'd only issue a
single flush command to that vdev, after all 10 lwb writes completed.
Further, and most important as it pertains to this change, since the
flush commands are often very impactful to the performance of the pool's
underlying storage, unnecessarily issuing these flush commands can
poorly impact the performance of the lwb writes themselves. Thus, we
need to avoid issuing flush commands when possible, in order to acheive
the best possible performance out of the pool's underlying storage.
This change attempts to address this problem by changing the ZIL's logic
to only issue a vdev flush command when it detects an lwb that has a
thread waiting for it to complete. When an lwb does not have threads
waiting for it, the responsibility of issuing the flush command to the
vdevs involved with that lwb's write is passed on to the "next" lwb.
It's only once a write for an lwb with waiters completes, do we issue
the vdev flush command(s). As a result, now when we issue the flush(s),
we will issue them to the vdevs involved with that specific lwb's write,
but potentially also to vdevs involved with "previous" lwb writes (i.e.
if the previous lwbs did not have waiters associated with them).
Thus, in our prior example with 10 lwbs, it's only once the last lwb
completes (which will be the lwb containing the waiter for the thread
that called fsync) will we issue the vdev flush command; all of the
other lwbs will find they have no waiters, so they'll pass the
responsibility of the flush to the "next" lwb (until reaching the last
lwb that has the waiter).
Porting Notes:
* Reconciled conflicts with the fastwrite feature.
Authored by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Matt Ahrens <matt@delphix.com>
Reviewed by: Brad Lewis <brad.lewis@delphix.com>
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Approved by: Joshua M. Clulow <josh@sysmgr.org>
Ported-by: Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
OpenZFS-issue: https://www.illumos.org/issues/9962
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/545190c6
Closes #8188
2018-10-24 00:14:27 +03:00
|
|
|
ASSERT(zio->io_txg > spa_syncing_txg(zio->io_spa));
|
|
|
|
|
zio_free(zio->io_spa, zio->io_txg, zio->io_bp);
|
|
|
|
|
}
|
2010-05-29 00:45:14 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
dmu_tx_commit(dsa->dsa_tx);
|
|
|
|
|
|
|
|
|
|
dsa->dsa_done(dsa->dsa_zgd, zio->io_error);
|
|
|
|
|
|
2021-01-20 22:24:37 +03:00
|
|
|
abd_free(zio->io_abd);
|
2010-05-29 00:45:14 +04:00
|
|
|
kmem_free(dsa, sizeof (*dsa));
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static int
|
|
|
|
|
dmu_sync_late_arrival(zio_t *pio, objset_t *os, dmu_sync_cb_t *done, zgd_t *zgd,
|
2014-06-25 22:37:59 +04:00
|
|
|
zio_prop_t *zp, zbookmark_phys_t *zb)
|
2010-05-29 00:45:14 +04:00
|
|
|
{
|
|
|
|
|
dmu_sync_arg_t *dsa;
|
|
|
|
|
dmu_tx_t *tx;
|
2023-08-11 19:04:08 +03:00
|
|
|
int error;
|
|
|
|
|
|
|
|
|
|
error = dbuf_read((dmu_buf_impl_t *)zgd->zgd_db, NULL,
|
|
|
|
|
DB_RF_CANFAIL | DB_RF_NOPREFETCH);
|
|
|
|
|
if (error != 0)
|
|
|
|
|
return (error);
|
2010-05-29 00:45:14 +04:00
|
|
|
|
|
|
|
|
tx = dmu_tx_create(os);
|
|
|
|
|
dmu_tx_hold_space(tx, zgd->zgd_db->db_size);
|
2023-07-27 19:07:09 +03:00
|
|
|
/*
|
|
|
|
|
* This transaction does not produce any dirty data or log blocks, so
|
|
|
|
|
* it should not be throttled. All other cases wait for TXG sync, by
|
|
|
|
|
* which time the log block we are writing will be obsolete, so we can
|
|
|
|
|
* skip waiting and just return error here instead.
|
|
|
|
|
*/
|
|
|
|
|
if (dmu_tx_assign(tx, TXG_NOWAIT | TXG_NOTHROTTLE) != 0) {
|
2010-05-29 00:45:14 +04:00
|
|
|
dmu_tx_abort(tx);
|
2013-03-08 22:41:28 +04:00
|
|
|
/* Make zl_get_data do txg_waited_synced() */
|
|
|
|
|
return (SET_ERROR(EIO));
|
2010-05-29 00:45:14 +04:00
|
|
|
}
|
|
|
|
|
|
OpenZFS 8585 - improve batching done in zil_commit()
Authored by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Brad Lewis <brad.lewis@delphix.com>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Dan McDonald <danmcd@joyent.com>
Ported-by: Prakash Surya <prakash.surya@delphix.com>
Problem
=======
The current implementation of zil_commit() can introduce significant
latency, beyond what is inherent due to the latency of the underlying
storage. The additional latency comes from two main problems:
1. When there's outstanding ZIL blocks being written (i.e. there's
already a "writer thread" in progress), then any new calls to
zil_commit() will block waiting for the currently oustanding ZIL
blocks to complete. The blocks written for each "writer thread" is
coined a "batch", and there can only ever be a single "batch" being
written at a time. When a batch is being written, any new ZIL
transactions will have to wait for the next batch to be written,
which won't occur until the current batch finishes.
As a result, the underlying storage may not be used as efficiently
as possible. While "new" threads enter zil_commit() and are blocked
waiting for the next batch, it's possible that the underlying
storage isn't fully utilized by the current batch of ZIL blocks. In
that case, it'd be better to allow these new threads to generate
(and issue) a new ZIL block, such that it could be serviced by the
underlying storage concurrently with the other ZIL blocks that are
being serviced.
2. Any call to zil_commit() must wait for all ZIL blocks in its "batch"
to complete, prior to zil_commit() returning. The size of any given
batch is proportional to the number of ZIL transaction in the queue
at the time that the batch starts processing the queue; which
doesn't occur until the previous batch completes. Thus, if there's a
lot of transactions in the queue, the batch could be composed of
many ZIL blocks, and each call to zil_commit() will have to wait for
all of these writes to complete (even if the thread calling
zil_commit() only cared about one of the transactions in the batch).
To further complicate the situation, these two issues result in the
following side effect:
3. If a given batch takes longer to complete than normal, this results
in larger batch sizes, which then take longer to complete and
further drive up the latency of zil_commit(). This can occur for a
number of reasons, including (but not limited to): transient changes
in the workload, and storage latency irregularites.
Solution
========
The solution attempted by this change has the following goals:
1. no on-disk changes; maintain current on-disk format.
2. modify the "batch size" to be equal to the "ZIL block size".
3. allow new batches to be generated and issued to disk, while there's
already batches being serviced by the disk.
4. allow zil_commit() to wait for as few ZIL blocks as possible.
5. use as few ZIL blocks as possible, for the same amount of ZIL
transactions, without introducing significant latency to any
individual ZIL transaction. i.e. use fewer, but larger, ZIL blocks.
In theory, with these goals met, the new allgorithm will allow the
following improvements:
1. new ZIL blocks can be generated and issued, while there's already
oustanding ZIL blocks being serviced by the storage.
2. the latency of zil_commit() should be proportional to the underlying
storage latency, rather than the incoming synchronous workload.
Porting Notes
=============
Due to the changes made in commit 119a394ab0, the lifetime of an itx
structure differs than in OpenZFS. Specifically, the itx structure is
kept around until the data associated with the itx is considered to be
safe on disk; this is so that the itx's callback can be called after the
data is committed to stable storage. Since OpenZFS doesn't have this itx
callback mechanism, it's able to destroy the itx structure immediately
after the itx is committed to an lwb (before the lwb is written to
disk).
To support this difference, and to ensure the itx's callbacks can still
be called after the itx's data is on disk, a few changes had to be made:
* A list of itxs was added to the lwb structure. This list contains
all of the itxs that have been committed to the lwb, such that the
callbacks for these itxs can be called from zil_lwb_flush_vdevs_done(),
after the data for the itxs is committed to disk.
* A list of itxs was added on the stack of the zil_process_commit_list()
function; the "nolwb_itxs" list. In some circumstances, an itx may
not be committed to an lwb (e.g. if allocating the "next" ZIL block
on disk fails), so this list is used to keep track of which itxs
fall into this state, such that their callbacks can be called after
the ZIL's writer pipeline is "stalled".
* The logic to actually call the itx's callback was moved into the
zil_itx_destroy() function. Since all consumers of zil_itx_destroy()
were effectively performing the same logic (i.e. if callback is
non-null, call the callback), it seemed like useful code cleanup to
consolidate this logic into a single function.
Additionally, the existing Linux tracepoint infrastructure dealing with
the ZIL's probes and structures had to be updated to reflect these code
changes. Specifically:
* The "zil__cw1" and "zil__cw2" probes were removed, so they had to be
removed from "trace_zil.h" as well.
* Some of the zilog structure's fields were removed, which affected
the tracepoint definitions of the structure.
* New tracepoints had to be added for the following 3 new probes:
* zil__process__commit__itx
* zil__process__normal__itx
* zil__commit__io__error
OpenZFS-issue: https://www.illumos.org/issues/8585
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/5d95a3a
Closes #6566
2017-12-05 20:39:16 +03:00
|
|
|
/*
|
|
|
|
|
* In order to prevent the zgd's lwb from being free'd prior to
|
|
|
|
|
* dmu_sync_late_arrival_done() being called, we have to ensure
|
|
|
|
|
* the lwb's "max txg" takes this tx's txg into account.
|
|
|
|
|
*/
|
|
|
|
|
zil_lwb_add_txg(zgd->zgd_lwb, dmu_tx_get_txg(tx));
|
|
|
|
|
|
2014-11-21 03:09:39 +03:00
|
|
|
dsa = kmem_alloc(sizeof (dmu_sync_arg_t), KM_SLEEP);
|
2010-05-29 00:45:14 +04:00
|
|
|
dsa->dsa_dr = NULL;
|
|
|
|
|
dsa->dsa_done = done;
|
|
|
|
|
dsa->dsa_zgd = zgd;
|
|
|
|
|
dsa->dsa_tx = tx;
|
|
|
|
|
|
2017-04-14 22:59:18 +03:00
|
|
|
/*
|
|
|
|
|
* Since we are currently syncing this txg, it's nontrivial to
|
|
|
|
|
* determine what BP to nopwrite against, so we disable nopwrite.
|
|
|
|
|
*
|
|
|
|
|
* When syncing, the db_blkptr is initially the BP of the previous
|
|
|
|
|
* txg. We can not nopwrite against it because it will be changed
|
|
|
|
|
* (this is similar to the non-late-arrival case where the dbuf is
|
|
|
|
|
* dirty in a future txg).
|
|
|
|
|
*
|
|
|
|
|
* Then dbuf_write_ready() sets bp_blkptr to the location we will write.
|
|
|
|
|
* We can not nopwrite against it because although the BP will not
|
|
|
|
|
* (typically) be changed, the data has not yet been persisted to this
|
|
|
|
|
* location.
|
|
|
|
|
*
|
|
|
|
|
* Finally, when dbuf_write_done() is called, it is theoretically
|
|
|
|
|
* possible to always nopwrite, because the data that was written in
|
|
|
|
|
* this txg is the same data that we are trying to write. However we
|
|
|
|
|
* would need to check that this dbuf is not dirty in any future
|
|
|
|
|
* txg's (as we do in the normal dmu_sync() path). For simplicity, we
|
|
|
|
|
* don't nopwrite in this case.
|
|
|
|
|
*/
|
|
|
|
|
zp->zp_nopwrite = B_FALSE;
|
|
|
|
|
|
2016-07-22 18:52:49 +03:00
|
|
|
zio_nowait(zio_write(pio, os->os_spa, dmu_tx_get_txg(tx), zgd->zgd_bp,
|
|
|
|
|
abd_get_from_buf(zgd->zgd_db->db_data, zgd->zgd_db->db_size),
|
|
|
|
|
zgd->zgd_db->db_size, zgd->zgd_db->db_size, zp,
|
Remove ARC/ZIO physdone callbacks.
Those callbacks were introduced many years ago as part of a bigger
patch to smoothen the write throttling within a txg. They allow to
account completion of individual physical writes within a logical
one, improving cases when some of physical writes complete much
sooner than others, gradually opening the write throttle.
Few years after that ZFS got allocation throttling, working on a
level of logical writes and limiting number of writes queued to
vdevs at any point, and so limiting latency distribution between
the physical writes and especially writes of multiple copies.
The addition of scheduling deadline I proposed in #14925 should
further reduce the latency distribution. Grown memory sizes over
the past 10 years should also reduce importance of the smoothing.
While the use of physdone callback may still in theory provide
some smoother throttling, there are cases where we simply can not
afford it. Since dirty data accounting is protected by pool-wide
lock, in case of 6-wide RAIDZ, for example, it requires us to take
it 8 times per logical block write, creating huge lock contention.
My tests of this patch show radical reduction of the lock spinning
time on workloads when smaller blocks are written to RAIDZ pools,
when each of the disks receives 8-16KB chunks, but the total rate
reaching 100K+ blocks per second. Same time attempts to measure
any write time fluctuations didn't show anything noticeable.
While there, remove also io_child_count/io_parent_count counters.
They are used only for couple assertions that can be avoided.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored by: iXsystems, Inc.
Closes #14948
2023-06-15 20:49:03 +03:00
|
|
|
dmu_sync_late_arrival_ready, NULL, dmu_sync_late_arrival_done,
|
2016-07-22 18:52:49 +03:00
|
|
|
dsa, ZIO_PRIORITY_SYNC_WRITE, ZIO_FLAG_CANFAIL, zb));
|
2010-05-29 00:45:14 +04:00
|
|
|
|
|
|
|
|
return (0);
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Intent log support: sync the block associated with db to disk.
|
|
|
|
|
* N.B. and XXX: the caller is responsible for making sure that the
|
|
|
|
|
* data isn't changing while dmu_sync() is writing it.
|
|
|
|
|
*
|
|
|
|
|
* Return values:
|
|
|
|
|
*
|
2013-05-10 23:47:54 +04:00
|
|
|
* EEXIST: this txg has already been synced, so there's nothing to do.
|
2008-11-20 23:01:55 +03:00
|
|
|
* The caller should not log the write.
|
|
|
|
|
*
|
|
|
|
|
* ENOENT: the block was dbuf_free_range()'d, so there's nothing to do.
|
|
|
|
|
* The caller should not log the write.
|
|
|
|
|
*
|
|
|
|
|
* EALREADY: this block is already in the process of being synced.
|
|
|
|
|
* The caller should track its progress (somehow).
|
|
|
|
|
*
|
2010-05-29 00:45:14 +04:00
|
|
|
* EIO: could not do the I/O.
|
|
|
|
|
* The caller should do a txg_wait_synced().
|
2008-11-20 23:01:55 +03:00
|
|
|
*
|
2010-05-29 00:45:14 +04:00
|
|
|
* 0: the I/O has been initiated.
|
|
|
|
|
* The caller should log this blkptr in the done callback.
|
|
|
|
|
* It is possible that the I/O will fail, in which case
|
|
|
|
|
* the error will be reported to the done callback and
|
|
|
|
|
* propagated to pio from zio_done().
|
2008-11-20 23:01:55 +03:00
|
|
|
*/
|
|
|
|
|
int
|
2010-05-29 00:45:14 +04:00
|
|
|
dmu_sync(zio_t *pio, uint64_t txg, dmu_sync_cb_t *done, zgd_t *zgd)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
2010-05-29 00:45:14 +04:00
|
|
|
dmu_buf_impl_t *db = (dmu_buf_impl_t *)zgd->zgd_db;
|
|
|
|
|
objset_t *os = db->db_objset;
|
|
|
|
|
dsl_dataset_t *ds = os->os_dsl_dataset;
|
2020-02-05 22:07:19 +03:00
|
|
|
dbuf_dirty_record_t *dr, *dr_next;
|
2010-05-29 00:45:14 +04:00
|
|
|
dmu_sync_arg_t *dsa;
|
2014-06-25 22:37:59 +04:00
|
|
|
zbookmark_phys_t zb;
|
2010-05-29 00:45:14 +04:00
|
|
|
zio_prop_t zp;
|
2010-08-27 01:24:34 +04:00
|
|
|
dnode_t *dn;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
ASSERT(pio != NULL);
|
2008-11-20 23:01:55 +03:00
|
|
|
ASSERT(txg != 0);
|
|
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
SET_BOOKMARK(&zb, ds->ds_object,
|
|
|
|
|
db->db.db_object, db->db_level, db->db_blkid);
|
|
|
|
|
|
2010-08-27 01:24:34 +04:00
|
|
|
DB_DNODE_ENTER(db);
|
|
|
|
|
dn = DB_DNODE(db);
|
2017-03-23 19:07:27 +03:00
|
|
|
dmu_write_policy(os, dn, db->db_level, WP_DMU_SYNC, &zp);
|
2010-08-27 01:24:34 +04:00
|
|
|
DB_DNODE_EXIT(db);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
/*
|
2010-05-29 00:45:14 +04:00
|
|
|
* If we're frozen (running ziltest), we always need to generate a bp.
|
2008-11-20 23:01:55 +03:00
|
|
|
*/
|
2010-05-29 00:45:14 +04:00
|
|
|
if (txg > spa_freeze_txg(os->os_spa))
|
|
|
|
|
return (dmu_sync_late_arrival(pio, os, done, zgd, &zp, &zb));
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
/*
|
2010-05-29 00:45:14 +04:00
|
|
|
* Grabbing db_mtx now provides a barrier between dbuf_sync_leaf()
|
|
|
|
|
* and us. If we determine that this txg is not yet syncing,
|
|
|
|
|
* but it begins to sync a moment later, that's OK because the
|
|
|
|
|
* sync thread will block in dbuf_sync_leaf() until we drop db_mtx.
|
2008-11-20 23:01:55 +03:00
|
|
|
*/
|
2010-05-29 00:45:14 +04:00
|
|
|
mutex_enter(&db->db_mtx);
|
|
|
|
|
|
|
|
|
|
if (txg <= spa_last_synced_txg(os->os_spa)) {
|
2008-11-20 23:01:55 +03:00
|
|
|
/*
|
2010-05-29 00:45:14 +04:00
|
|
|
* This txg has already synced. There's nothing to do.
|
2008-11-20 23:01:55 +03:00
|
|
|
*/
|
2010-05-29 00:45:14 +04:00
|
|
|
mutex_exit(&db->db_mtx);
|
2013-03-08 22:41:28 +04:00
|
|
|
return (SET_ERROR(EEXIST));
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
if (txg <= spa_syncing_txg(os->os_spa)) {
|
|
|
|
|
/*
|
|
|
|
|
* This txg is currently syncing, so we can't mess with
|
|
|
|
|
* the dirty record anymore; just write a new log block.
|
|
|
|
|
*/
|
|
|
|
|
mutex_exit(&db->db_mtx);
|
|
|
|
|
return (dmu_sync_late_arrival(pio, os, done, zgd, &zp, &zb));
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
2020-02-05 22:07:19 +03:00
|
|
|
dr = dbuf_find_dirty_eq(db, txg);
|
2010-05-29 00:45:14 +04:00
|
|
|
|
|
|
|
|
if (dr == NULL) {
|
2008-11-20 23:01:55 +03:00
|
|
|
/*
|
2010-05-29 00:45:14 +04:00
|
|
|
* There's no dr for this dbuf, so it must have been freed.
|
2008-11-20 23:01:55 +03:00
|
|
|
* There's no need to log writes to freed blocks, so we're done.
|
|
|
|
|
*/
|
|
|
|
|
mutex_exit(&db->db_mtx);
|
2013-03-08 22:41:28 +04:00
|
|
|
return (SET_ERROR(ENOENT));
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
2020-02-05 22:07:19 +03:00
|
|
|
dr_next = list_next(&db->db_dirty_records, dr);
|
|
|
|
|
ASSERT(dr_next == NULL || dr_next->dr_txg < txg);
|
2013-05-10 23:47:54 +04:00
|
|
|
|
2017-04-14 22:59:18 +03:00
|
|
|
if (db->db_blkptr != NULL) {
|
|
|
|
|
/*
|
|
|
|
|
* We need to fill in zgd_bp with the current blkptr so that
|
|
|
|
|
* the nopwrite code can check if we're writing the same
|
|
|
|
|
* data that's already on disk. We can only nopwrite if we
|
|
|
|
|
* are sure that after making the copy, db_blkptr will not
|
|
|
|
|
* change until our i/o completes. We ensure this by
|
|
|
|
|
* holding the db_mtx, and only allowing nopwrite if the
|
|
|
|
|
* block is not already dirty (see below). This is verified
|
|
|
|
|
* by dmu_sync_done(), which VERIFYs that the db_blkptr has
|
|
|
|
|
* not changed.
|
|
|
|
|
*/
|
|
|
|
|
*zgd->zgd_bp = *db->db_blkptr;
|
|
|
|
|
}
|
|
|
|
|
|
2013-05-10 23:47:54 +04:00
|
|
|
/*
|
2015-04-11 21:35:03 +03:00
|
|
|
* Assume the on-disk data is X, the current syncing data (in
|
|
|
|
|
* txg - 1) is Y, and the current in-memory data is Z (currently
|
|
|
|
|
* in dmu_sync).
|
|
|
|
|
*
|
|
|
|
|
* We usually want to perform a nopwrite if X and Z are the
|
|
|
|
|
* same. However, if Y is different (i.e. the BP is going to
|
|
|
|
|
* change before this write takes effect), then a nopwrite will
|
|
|
|
|
* be incorrect - we would override with X, which could have
|
|
|
|
|
* been freed when Y was written.
|
|
|
|
|
*
|
|
|
|
|
* (Note that this is not a concern when we are nop-writing from
|
|
|
|
|
* syncing context, because X and Y must be identical, because
|
|
|
|
|
* all previous txgs have been synced.)
|
|
|
|
|
*
|
|
|
|
|
* Therefore, we disable nopwrite if the current BP could change
|
|
|
|
|
* before this TXG. There are two ways it could change: by
|
|
|
|
|
* being dirty (dr_next is non-NULL), or by being freed
|
|
|
|
|
* (dnode_block_freed()). This behavior is verified by
|
|
|
|
|
* zio_done(), which VERIFYs that the override BP is identical
|
|
|
|
|
* to the on-disk BP.
|
2013-05-10 23:47:54 +04:00
|
|
|
*/
|
2015-04-11 21:35:03 +03:00
|
|
|
DB_DNODE_ENTER(db);
|
|
|
|
|
dn = DB_DNODE(db);
|
2020-02-05 22:07:19 +03:00
|
|
|
if (dr_next != NULL || dnode_block_freed(dn, db->db_blkid))
|
2013-05-10 23:47:54 +04:00
|
|
|
zp.zp_nopwrite = B_FALSE;
|
2015-04-11 21:35:03 +03:00
|
|
|
DB_DNODE_EXIT(db);
|
2013-05-10 23:47:54 +04:00
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
ASSERT(dr->dr_txg == txg);
|
2010-05-29 00:45:14 +04:00
|
|
|
if (dr->dt.dl.dr_override_state == DR_IN_DMU_SYNC ||
|
|
|
|
|
dr->dt.dl.dr_override_state == DR_OVERRIDDEN) {
|
2008-11-20 23:01:55 +03:00
|
|
|
/*
|
2010-05-29 00:45:14 +04:00
|
|
|
* We have already issued a sync write for this buffer,
|
|
|
|
|
* or this buffer has already been synced. It could not
|
2008-11-20 23:01:55 +03:00
|
|
|
* have been dirtied since, or we would have cleared the state.
|
|
|
|
|
*/
|
|
|
|
|
mutex_exit(&db->db_mtx);
|
2013-03-08 22:41:28 +04:00
|
|
|
return (SET_ERROR(EALREADY));
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
ASSERT(dr->dt.dl.dr_override_state == DR_NOT_OVERRIDDEN);
|
2008-11-20 23:01:55 +03:00
|
|
|
dr->dt.dl.dr_override_state = DR_IN_DMU_SYNC;
|
|
|
|
|
mutex_exit(&db->db_mtx);
|
|
|
|
|
|
2014-11-21 03:09:39 +03:00
|
|
|
dsa = kmem_alloc(sizeof (dmu_sync_arg_t), KM_SLEEP);
|
2010-05-29 00:45:14 +04:00
|
|
|
dsa->dsa_dr = dr;
|
|
|
|
|
dsa->dsa_done = done;
|
|
|
|
|
dsa->dsa_zgd = zgd;
|
|
|
|
|
dsa->dsa_tx = NULL;
|
2008-12-03 23:09:06 +03:00
|
|
|
|
Implement uncached prefetch
Previously the primarycache property was handled only in the dbuf
layer. Since the speculative prefetcher is implemented in the ARC,
it had to be disabled for uncacheable buffers.
This change gives the ARC knowledge about uncacheable buffers
via arc_read() and arc_write(). So when remove_reference() drops
the last reference on the ARC header, it can either immediately destroy
it, or if it is marked as prefetch, put it into a new arc_uncached state.
That state is scanned every second, evicting stale buffers that were
not demand read.
This change also tracks dbufs that were read from the beginning,
but not to the end. It is assumed that such buffers may receive further
reads, and so they are stored in dbuf cache. If a following
reads reaches the end of the buffer, it is immediately evicted.
Otherwise it will follow regular dbuf cache eviction. Since the dbuf
layer does not know actual file sizes, this logic is not applied to
the final buffer of a dnode.
Since uncacheable buffers should no longer stay in the ARC for long,
this patch also tries to optimize I/O by allocating ARC physical
buffers as linear to allow buffer sharing.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored by: iXsystems, Inc.
Closes #14243
2023-01-05 03:29:54 +03:00
|
|
|
zio_nowait(arc_write(pio, os->os_spa, txg, zgd->zgd_bp,
|
|
|
|
|
dr->dt.dl.dr_data, !DBUF_IS_CACHEABLE(db), dbuf_is_l2cacheable(db),
|
Remove ARC/ZIO physdone callbacks.
Those callbacks were introduced many years ago as part of a bigger
patch to smoothen the write throttling within a txg. They allow to
account completion of individual physical writes within a logical
one, improving cases when some of physical writes complete much
sooner than others, gradually opening the write throttle.
Few years after that ZFS got allocation throttling, working on a
level of logical writes and limiting number of writes queued to
vdevs at any point, and so limiting latency distribution between
the physical writes and especially writes of multiple copies.
The addition of scheduling deadline I proposed in #14925 should
further reduce the latency distribution. Grown memory sizes over
the past 10 years should also reduce importance of the smoothing.
While the use of physdone callback may still in theory provide
some smoother throttling, there are cases where we simply can not
afford it. Since dirty data accounting is protected by pool-wide
lock, in case of 6-wide RAIDZ, for example, it requires us to take
it 8 times per logical block write, creating huge lock contention.
My tests of this patch show radical reduction of the lock spinning
time on workloads when smaller blocks are written to RAIDZ pools,
when each of the disks receives 8-16KB chunks, but the total rate
reaching 100K+ blocks per second. Same time attempts to measure
any write time fluctuations didn't show anything noticeable.
While there, remove also io_child_count/io_parent_count counters.
They are used only for couple assertions that can be avoided.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored by: iXsystems, Inc.
Closes #14948
2023-06-15 20:49:03 +03:00
|
|
|
&zp, dmu_sync_ready, NULL, dmu_sync_done, dsa,
|
2016-05-15 18:02:28 +03:00
|
|
|
ZIO_PRIORITY_SYNC_WRITE, ZIO_FLAG_CANFAIL, &zb));
|
2008-12-03 23:09:06 +03:00
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
return (0);
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
Native Encryption for ZFS on Linux
This change incorporates three major pieces:
The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
convenient.
The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.
The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
compromised.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #494
Closes #5769
2017-08-14 20:36:48 +03:00
|
|
|
int
|
|
|
|
|
dmu_object_set_nlevels(objset_t *os, uint64_t object, int nlevels, dmu_tx_t *tx)
|
|
|
|
|
{
|
|
|
|
|
dnode_t *dn;
|
|
|
|
|
int err;
|
|
|
|
|
|
|
|
|
|
err = dnode_hold(os, object, FTAG, &dn);
|
|
|
|
|
if (err)
|
|
|
|
|
return (err);
|
|
|
|
|
err = dnode_set_nlevels(dn, nlevels, tx);
|
|
|
|
|
dnode_rele(dn, FTAG);
|
|
|
|
|
return (err);
|
|
|
|
|
}
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
int
|
|
|
|
|
dmu_object_set_blocksize(objset_t *os, uint64_t object, uint64_t size, int ibs,
|
2017-01-21 00:17:55 +03:00
|
|
|
dmu_tx_t *tx)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
|
|
|
|
dnode_t *dn;
|
|
|
|
|
int err;
|
|
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
err = dnode_hold(os, object, FTAG, &dn);
|
2008-11-20 23:01:55 +03:00
|
|
|
if (err)
|
|
|
|
|
return (err);
|
|
|
|
|
err = dnode_set_blksz(dn, size, ibs, tx);
|
|
|
|
|
dnode_rele(dn, FTAG);
|
|
|
|
|
return (err);
|
|
|
|
|
}
|
|
|
|
|
|
2017-11-08 22:12:59 +03:00
|
|
|
int
|
|
|
|
|
dmu_object_set_maxblkid(objset_t *os, uint64_t object, uint64_t maxblkid,
|
|
|
|
|
dmu_tx_t *tx)
|
|
|
|
|
{
|
|
|
|
|
dnode_t *dn;
|
|
|
|
|
int err;
|
|
|
|
|
|
|
|
|
|
err = dnode_hold(os, object, FTAG, &dn);
|
|
|
|
|
if (err)
|
|
|
|
|
return (err);
|
|
|
|
|
rw_enter(&dn->dn_struct_rwlock, RW_WRITER);
|
2019-03-13 20:52:01 +03:00
|
|
|
dnode_new_blkid(dn, maxblkid, tx, B_FALSE, B_TRUE);
|
2017-11-08 22:12:59 +03:00
|
|
|
rw_exit(&dn->dn_struct_rwlock);
|
|
|
|
|
dnode_rele(dn, FTAG);
|
|
|
|
|
return (0);
|
|
|
|
|
}
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
void
|
|
|
|
|
dmu_object_set_checksum(objset_t *os, uint64_t object, uint8_t checksum,
|
2017-01-21 00:17:55 +03:00
|
|
|
dmu_tx_t *tx)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
|
|
|
|
dnode_t *dn;
|
|
|
|
|
|
2014-06-06 01:19:08 +04:00
|
|
|
/*
|
|
|
|
|
* Send streams include each object's checksum function. This
|
|
|
|
|
* check ensures that the receiving system can understand the
|
|
|
|
|
* checksum function transmitted.
|
|
|
|
|
*/
|
|
|
|
|
ASSERT3U(checksum, <, ZIO_CHECKSUM_LEGACY_FUNCTIONS);
|
|
|
|
|
|
|
|
|
|
VERIFY0(dnode_hold(os, object, FTAG, &dn));
|
|
|
|
|
ASSERT3U(checksum, <, ZIO_CHECKSUM_FUNCTIONS);
|
2008-11-20 23:01:55 +03:00
|
|
|
dn->dn_checksum = checksum;
|
|
|
|
|
dnode_setdirty(dn, tx);
|
|
|
|
|
dnode_rele(dn, FTAG);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
|
|
|
|
dmu_object_set_compress(objset_t *os, uint64_t object, uint8_t compress,
|
2017-01-21 00:17:55 +03:00
|
|
|
dmu_tx_t *tx)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
|
|
|
|
dnode_t *dn;
|
|
|
|
|
|
2014-06-06 01:19:08 +04:00
|
|
|
/*
|
|
|
|
|
* Send streams include each object's compression function. This
|
|
|
|
|
* check ensures that the receiving system can understand the
|
|
|
|
|
* compression function transmitted.
|
|
|
|
|
*/
|
|
|
|
|
ASSERT3U(compress, <, ZIO_COMPRESS_LEGACY_FUNCTIONS);
|
|
|
|
|
|
|
|
|
|
VERIFY0(dnode_hold(os, object, FTAG, &dn));
|
2008-11-20 23:01:55 +03:00
|
|
|
dn->dn_compress = compress;
|
|
|
|
|
dnode_setdirty(dn, tx);
|
|
|
|
|
dnode_rele(dn, FTAG);
|
|
|
|
|
}
|
|
|
|
|
|
2014-05-23 20:21:07 +04:00
|
|
|
/*
|
|
|
|
|
* When the "redundant_metadata" property is set to "most", only indirect
|
|
|
|
|
* blocks of this level and higher will have an additional ditto block.
|
|
|
|
|
*/
|
2022-01-15 02:37:55 +03:00
|
|
|
static const int zfs_redundant_metadata_most_ditto_level = 2;
|
2014-05-23 20:21:07 +04:00
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
void
|
2017-03-23 19:07:27 +03:00
|
|
|
dmu_write_policy(objset_t *os, dnode_t *dn, int level, int wp, zio_prop_t *zp)
|
2010-05-29 00:45:14 +04:00
|
|
|
{
|
|
|
|
|
dmu_object_type_t type = dn ? dn->dn_type : DMU_OT_OBJSET;
|
2012-12-14 03:24:15 +04:00
|
|
|
boolean_t ismd = (level > 0 || DMU_OT_IS_METADATA(type) ||
|
2010-08-27 01:24:34 +04:00
|
|
|
(wp & WP_SPILL));
|
2010-05-29 00:45:14 +04:00
|
|
|
enum zio_checksum checksum = os->os_checksum;
|
|
|
|
|
enum zio_compress compress = os->os_compress;
|
Add zstd support to zfs
This PR adds two new compression types, based on ZStandard:
- zstd: A basic ZStandard compression algorithm Available compression.
Levels for zstd are zstd-1 through zstd-19, where the compression
increases with every level, but speed decreases.
- zstd-fast: A faster version of the ZStandard compression algorithm
zstd-fast is basically a "negative" level of zstd. The compression
decreases with every level, but speed increases.
Available compression levels for zstd-fast:
- zstd-fast-1 through zstd-fast-10
- zstd-fast-20 through zstd-fast-100 (in increments of 10)
- zstd-fast-500 and zstd-fast-1000
For more information check the man page.
Implementation details:
Rather than treat each level of zstd as a different algorithm (as was
done historically with gzip), the block pointer `enum zio_compress`
value is simply zstd for all levels, including zstd-fast, since they all
use the same decompression function.
The compress= property (a 64bit unsigned integer) uses the lower 7 bits
to store the compression algorithm (matching the number of bits used in
a block pointer, as the 8th bit was borrowed for embedded block
pointers). The upper bits are used to store the compression level.
It is necessary to be able to determine what compression level was used
when later reading a block back, so the concept used in LZ4, where the
first 32bits of the on-disk value are the size of the compressed data
(since the allocation is rounded up to the nearest ashift), was
extended, and we store the version of ZSTD and the level as well as the
compressed size. This value is returned when decompressing a block, so
that if the block needs to be recompressed (L2ARC, nop-write, etc), that
the same parameters will be used to result in the matching checksum.
All of the internal ZFS code ( `arc_buf_hdr_t`, `objset_t`,
`zio_prop_t`, etc.) uses the separated _compress and _complevel
variables. Only the properties ZAP contains the combined/bit-shifted
value. The combined value is split when the compression_changed_cb()
callback is called, and sets both objset members (os_compress and
os_complevel).
The userspace tools all use the combined/bit-shifted value.
Additional notes:
zdb can now also decode the ZSTD compression header (flag -Z) and
inspect the size, version and compression level saved in that header.
For each record, if it is ZSTD compressed, the parameters of the decoded
compression header get printed.
ZSTD is included with all current tests and new tests are added
as-needed.
Per-dataset feature flags now get activated when the property is set.
If a compression algorithm requires a feature flag, zfs activates the
feature when the property is set, rather than waiting for the first
block to be born. This is currently only used by zstd but can be
extended as needed.
Portions-Sponsored-By: The FreeBSD Foundation
Co-authored-by: Allan Jude <allanjude@freebsd.org>
Co-authored-by: Brian Behlendorf <behlendorf1@llnl.gov>
Co-authored-by: Sebastian Gottschall <s.gottschall@dd-wrt.com>
Co-authored-by: Kjeld Schouten-Lebbing <kjeld@schouten-lebbing.nl>
Co-authored-by: Michael Niewöhner <foss@mniewoehner.de>
Signed-off-by: Allan Jude <allan@klarasystems.com>
Signed-off-by: Allan Jude <allanjude@freebsd.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Sebastian Gottschall <s.gottschall@dd-wrt.com>
Signed-off-by: Kjeld Schouten-Lebbing <kjeld@schouten-lebbing.nl>
Signed-off-by: Michael Niewöhner <foss@mniewoehner.de>
Closes #6247
Closes #9024
Closes #10277
Closes #10278
2020-08-18 20:10:17 +03:00
|
|
|
uint8_t complevel = os->os_complevel;
|
2010-05-29 00:45:14 +04:00
|
|
|
enum zio_checksum dedup_checksum = os->os_dedup_checksum;
|
2013-05-10 23:47:54 +04:00
|
|
|
boolean_t dedup = B_FALSE;
|
|
|
|
|
boolean_t nopwrite = B_FALSE;
|
2010-05-29 00:45:14 +04:00
|
|
|
boolean_t dedup_verify = os->os_dedup_verify;
|
Native Encryption for ZFS on Linux
This change incorporates three major pieces:
The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
convenient.
The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.
The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
compromised.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #494
Closes #5769
2017-08-14 20:36:48 +03:00
|
|
|
boolean_t encrypt = B_FALSE;
|
2010-05-29 00:45:14 +04:00
|
|
|
int copies = os->os_copies;
|
2017-04-12 00:56:54 +03:00
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
/*
|
2013-05-10 23:47:54 +04:00
|
|
|
* We maintain different write policies for each of the following
|
|
|
|
|
* types of data:
|
|
|
|
|
* 1. metadata
|
|
|
|
|
* 2. preallocated blocks (i.e. level-0 blocks of a dump device)
|
|
|
|
|
* 3. all other level 0 blocks
|
2010-05-29 00:45:14 +04:00
|
|
|
*/
|
|
|
|
|
if (ismd) {
|
2018-02-21 23:28:52 +03:00
|
|
|
/*
|
|
|
|
|
* XXX -- we should design a compression algorithm
|
|
|
|
|
* that specializes in arrays of bps.
|
|
|
|
|
*/
|
|
|
|
|
compress = zio_compress_select(os->os_spa,
|
|
|
|
|
ZIO_COMPRESS_ON, ZIO_COMPRESS_ON);
|
2013-05-10 23:47:54 +04:00
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
/*
|
|
|
|
|
* Metadata always gets checksummed. If the data
|
|
|
|
|
* checksum is multi-bit correctable, and it's not a
|
|
|
|
|
* ZBT-style checksum, then it's suitable for metadata
|
|
|
|
|
* as well. Otherwise, the metadata checksum defaults
|
|
|
|
|
* to fletcher4.
|
|
|
|
|
*/
|
2016-06-16 01:47:05 +03:00
|
|
|
if (!(zio_checksum_table[checksum].ci_flags &
|
|
|
|
|
ZCHECKSUM_FLAG_METADATA) ||
|
|
|
|
|
(zio_checksum_table[checksum].ci_flags &
|
|
|
|
|
ZCHECKSUM_FLAG_EMBEDDED))
|
2010-05-29 00:45:14 +04:00
|
|
|
checksum = ZIO_CHECKSUM_FLETCHER_4;
|
2014-05-23 20:21:07 +04:00
|
|
|
|
2022-10-20 03:07:51 +03:00
|
|
|
switch (os->os_redundant_metadata) {
|
|
|
|
|
case ZFS_REDUNDANT_METADATA_ALL:
|
2014-05-23 20:21:07 +04:00
|
|
|
copies++;
|
2022-10-20 03:07:51 +03:00
|
|
|
break;
|
|
|
|
|
case ZFS_REDUNDANT_METADATA_MOST:
|
|
|
|
|
if (level >= zfs_redundant_metadata_most_ditto_level ||
|
|
|
|
|
DMU_OT_IS_METADATA(type) || (wp & WP_SPILL))
|
|
|
|
|
copies++;
|
|
|
|
|
break;
|
|
|
|
|
case ZFS_REDUNDANT_METADATA_SOME:
|
|
|
|
|
if (DMU_OT_IS_CRITICAL(type))
|
|
|
|
|
copies++;
|
|
|
|
|
break;
|
|
|
|
|
case ZFS_REDUNDANT_METADATA_NONE:
|
|
|
|
|
break;
|
|
|
|
|
}
|
2013-05-10 23:47:54 +04:00
|
|
|
} else if (wp & WP_NOFILL) {
|
|
|
|
|
ASSERT(level == 0);
|
2010-05-29 00:45:14 +04:00
|
|
|
|
|
|
|
|
/*
|
2013-05-10 23:47:54 +04:00
|
|
|
* If we're writing preallocated blocks, we aren't actually
|
|
|
|
|
* writing them so don't set any policy properties. These
|
|
|
|
|
* blocks are currently only used by an external subsystem
|
|
|
|
|
* outside of zfs (i.e. dump) and not written by the zio
|
|
|
|
|
* pipeline.
|
2010-05-29 00:45:14 +04:00
|
|
|
*/
|
2013-05-10 23:47:54 +04:00
|
|
|
compress = ZIO_COMPRESS_OFF;
|
|
|
|
|
checksum = ZIO_CHECKSUM_OFF;
|
2010-05-29 00:45:14 +04:00
|
|
|
} else {
|
2015-07-06 04:55:32 +03:00
|
|
|
compress = zio_compress_select(os->os_spa, dn->dn_compress,
|
|
|
|
|
compress);
|
Add zstd support to zfs
This PR adds two new compression types, based on ZStandard:
- zstd: A basic ZStandard compression algorithm Available compression.
Levels for zstd are zstd-1 through zstd-19, where the compression
increases with every level, but speed decreases.
- zstd-fast: A faster version of the ZStandard compression algorithm
zstd-fast is basically a "negative" level of zstd. The compression
decreases with every level, but speed increases.
Available compression levels for zstd-fast:
- zstd-fast-1 through zstd-fast-10
- zstd-fast-20 through zstd-fast-100 (in increments of 10)
- zstd-fast-500 and zstd-fast-1000
For more information check the man page.
Implementation details:
Rather than treat each level of zstd as a different algorithm (as was
done historically with gzip), the block pointer `enum zio_compress`
value is simply zstd for all levels, including zstd-fast, since they all
use the same decompression function.
The compress= property (a 64bit unsigned integer) uses the lower 7 bits
to store the compression algorithm (matching the number of bits used in
a block pointer, as the 8th bit was borrowed for embedded block
pointers). The upper bits are used to store the compression level.
It is necessary to be able to determine what compression level was used
when later reading a block back, so the concept used in LZ4, where the
first 32bits of the on-disk value are the size of the compressed data
(since the allocation is rounded up to the nearest ashift), was
extended, and we store the version of ZSTD and the level as well as the
compressed size. This value is returned when decompressing a block, so
that if the block needs to be recompressed (L2ARC, nop-write, etc), that
the same parameters will be used to result in the matching checksum.
All of the internal ZFS code ( `arc_buf_hdr_t`, `objset_t`,
`zio_prop_t`, etc.) uses the separated _compress and _complevel
variables. Only the properties ZAP contains the combined/bit-shifted
value. The combined value is split when the compression_changed_cb()
callback is called, and sets both objset members (os_compress and
os_complevel).
The userspace tools all use the combined/bit-shifted value.
Additional notes:
zdb can now also decode the ZSTD compression header (flag -Z) and
inspect the size, version and compression level saved in that header.
For each record, if it is ZSTD compressed, the parameters of the decoded
compression header get printed.
ZSTD is included with all current tests and new tests are added
as-needed.
Per-dataset feature flags now get activated when the property is set.
If a compression algorithm requires a feature flag, zfs activates the
feature when the property is set, rather than waiting for the first
block to be born. This is currently only used by zstd but can be
extended as needed.
Portions-Sponsored-By: The FreeBSD Foundation
Co-authored-by: Allan Jude <allanjude@freebsd.org>
Co-authored-by: Brian Behlendorf <behlendorf1@llnl.gov>
Co-authored-by: Sebastian Gottschall <s.gottschall@dd-wrt.com>
Co-authored-by: Kjeld Schouten-Lebbing <kjeld@schouten-lebbing.nl>
Co-authored-by: Michael Niewöhner <foss@mniewoehner.de>
Signed-off-by: Allan Jude <allan@klarasystems.com>
Signed-off-by: Allan Jude <allanjude@freebsd.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Sebastian Gottschall <s.gottschall@dd-wrt.com>
Signed-off-by: Kjeld Schouten-Lebbing <kjeld@schouten-lebbing.nl>
Signed-off-by: Michael Niewöhner <foss@mniewoehner.de>
Closes #6247
Closes #9024
Closes #10277
Closes #10278
2020-08-18 20:10:17 +03:00
|
|
|
complevel = zio_complevel_select(os->os_spa, compress,
|
|
|
|
|
complevel, complevel);
|
2010-05-29 00:45:14 +04:00
|
|
|
|
2013-05-10 23:47:54 +04:00
|
|
|
checksum = (dedup_checksum == ZIO_CHECKSUM_OFF) ?
|
|
|
|
|
zio_checksum_select(dn->dn_checksum, checksum) :
|
|
|
|
|
dedup_checksum;
|
2010-05-29 00:45:14 +04:00
|
|
|
|
2013-05-10 23:47:54 +04:00
|
|
|
/*
|
|
|
|
|
* Determine dedup setting. If we are in dmu_sync(),
|
|
|
|
|
* we won't actually dedup now because that's all
|
|
|
|
|
* done in syncing context; but we do want to use the
|
2019-09-03 03:56:41 +03:00
|
|
|
* dedup checksum. If the checksum is not strong
|
2013-05-10 23:47:54 +04:00
|
|
|
* enough to ensure unique signatures, force
|
|
|
|
|
* dedup_verify.
|
|
|
|
|
*/
|
|
|
|
|
if (dedup_checksum != ZIO_CHECKSUM_OFF) {
|
|
|
|
|
dedup = (wp & WP_DMU_SYNC) ? B_FALSE : B_TRUE;
|
2016-06-16 01:47:05 +03:00
|
|
|
if (!(zio_checksum_table[checksum].ci_flags &
|
|
|
|
|
ZCHECKSUM_FLAG_DEDUP))
|
2013-05-10 23:47:54 +04:00
|
|
|
dedup_verify = B_TRUE;
|
|
|
|
|
}
|
2010-05-29 00:45:14 +04:00
|
|
|
|
2013-05-10 23:47:54 +04:00
|
|
|
/*
|
2016-06-16 01:47:05 +03:00
|
|
|
* Enable nopwrite if we have secure enough checksum
|
|
|
|
|
* algorithm (see comment in zio_nop_write) and
|
|
|
|
|
* compression is enabled. We don't enable nopwrite if
|
|
|
|
|
* dedup is enabled as the two features are mutually
|
|
|
|
|
* exclusive.
|
2013-05-10 23:47:54 +04:00
|
|
|
*/
|
2016-06-16 01:47:05 +03:00
|
|
|
nopwrite = (!dedup && (zio_checksum_table[checksum].ci_flags &
|
|
|
|
|
ZCHECKSUM_FLAG_NOPWRITE) &&
|
2013-05-10 23:47:54 +04:00
|
|
|
compress != ZIO_COMPRESS_OFF && zfs_nopwrite_enabled);
|
2010-05-29 00:45:14 +04:00
|
|
|
}
|
|
|
|
|
|
Native Encryption for ZFS on Linux
This change incorporates three major pieces:
The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
convenient.
The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.
The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
compromised.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #494
Closes #5769
2017-08-14 20:36:48 +03:00
|
|
|
/*
|
|
|
|
|
* All objects in an encrypted objset are protected from modification
|
|
|
|
|
* via a MAC. Encrypted objects store their IV and salt in the last DVA
|
|
|
|
|
* in the bp, so we cannot use all copies. Encrypted objects are also
|
|
|
|
|
* not subject to nopwrite since writing the same data will still
|
|
|
|
|
* result in a new ciphertext. Only encrypted blocks can be dedup'd
|
|
|
|
|
* to avoid ambiguity in the dedup code since the DDT does not store
|
|
|
|
|
* object types.
|
|
|
|
|
*/
|
|
|
|
|
if (os->os_encrypted && (wp & WP_NOFILL) == 0) {
|
|
|
|
|
encrypt = B_TRUE;
|
|
|
|
|
|
|
|
|
|
if (DMU_OT_IS_ENCRYPTED(type)) {
|
|
|
|
|
copies = MIN(copies, SPA_DVAS_PER_BP - 1);
|
|
|
|
|
nopwrite = B_FALSE;
|
|
|
|
|
} else {
|
|
|
|
|
dedup = B_FALSE;
|
|
|
|
|
}
|
|
|
|
|
|
2017-11-08 22:12:59 +03:00
|
|
|
if (level <= 0 &&
|
|
|
|
|
(type == DMU_OT_DNODE || type == DMU_OT_OBJSET)) {
|
Native Encryption for ZFS on Linux
This change incorporates three major pieces:
The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
convenient.
The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.
The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
compromised.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #494
Closes #5769
2017-08-14 20:36:48 +03:00
|
|
|
compress = ZIO_COMPRESS_EMPTY;
|
2017-11-08 22:12:59 +03:00
|
|
|
}
|
Native Encryption for ZFS on Linux
This change incorporates three major pieces:
The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
convenient.
The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.
The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
compromised.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #494
Closes #5769
2017-08-14 20:36:48 +03:00
|
|
|
}
|
2016-07-11 20:45:52 +03:00
|
|
|
|
Native Encryption for ZFS on Linux
This change incorporates three major pieces:
The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
convenient.
The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.
The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
compromised.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #494
Closes #5769
2017-08-14 20:36:48 +03:00
|
|
|
zp->zp_compress = compress;
|
Add zstd support to zfs
This PR adds two new compression types, based on ZStandard:
- zstd: A basic ZStandard compression algorithm Available compression.
Levels for zstd are zstd-1 through zstd-19, where the compression
increases with every level, but speed decreases.
- zstd-fast: A faster version of the ZStandard compression algorithm
zstd-fast is basically a "negative" level of zstd. The compression
decreases with every level, but speed increases.
Available compression levels for zstd-fast:
- zstd-fast-1 through zstd-fast-10
- zstd-fast-20 through zstd-fast-100 (in increments of 10)
- zstd-fast-500 and zstd-fast-1000
For more information check the man page.
Implementation details:
Rather than treat each level of zstd as a different algorithm (as was
done historically with gzip), the block pointer `enum zio_compress`
value is simply zstd for all levels, including zstd-fast, since they all
use the same decompression function.
The compress= property (a 64bit unsigned integer) uses the lower 7 bits
to store the compression algorithm (matching the number of bits used in
a block pointer, as the 8th bit was borrowed for embedded block
pointers). The upper bits are used to store the compression level.
It is necessary to be able to determine what compression level was used
when later reading a block back, so the concept used in LZ4, where the
first 32bits of the on-disk value are the size of the compressed data
(since the allocation is rounded up to the nearest ashift), was
extended, and we store the version of ZSTD and the level as well as the
compressed size. This value is returned when decompressing a block, so
that if the block needs to be recompressed (L2ARC, nop-write, etc), that
the same parameters will be used to result in the matching checksum.
All of the internal ZFS code ( `arc_buf_hdr_t`, `objset_t`,
`zio_prop_t`, etc.) uses the separated _compress and _complevel
variables. Only the properties ZAP contains the combined/bit-shifted
value. The combined value is split when the compression_changed_cb()
callback is called, and sets both objset members (os_compress and
os_complevel).
The userspace tools all use the combined/bit-shifted value.
Additional notes:
zdb can now also decode the ZSTD compression header (flag -Z) and
inspect the size, version and compression level saved in that header.
For each record, if it is ZSTD compressed, the parameters of the decoded
compression header get printed.
ZSTD is included with all current tests and new tests are added
as-needed.
Per-dataset feature flags now get activated when the property is set.
If a compression algorithm requires a feature flag, zfs activates the
feature when the property is set, rather than waiting for the first
block to be born. This is currently only used by zstd but can be
extended as needed.
Portions-Sponsored-By: The FreeBSD Foundation
Co-authored-by: Allan Jude <allanjude@freebsd.org>
Co-authored-by: Brian Behlendorf <behlendorf1@llnl.gov>
Co-authored-by: Sebastian Gottschall <s.gottschall@dd-wrt.com>
Co-authored-by: Kjeld Schouten-Lebbing <kjeld@schouten-lebbing.nl>
Co-authored-by: Michael Niewöhner <foss@mniewoehner.de>
Signed-off-by: Allan Jude <allan@klarasystems.com>
Signed-off-by: Allan Jude <allanjude@freebsd.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Sebastian Gottschall <s.gottschall@dd-wrt.com>
Signed-off-by: Kjeld Schouten-Lebbing <kjeld@schouten-lebbing.nl>
Signed-off-by: Michael Niewöhner <foss@mniewoehner.de>
Closes #6247
Closes #9024
Closes #10277
Closes #10278
2020-08-18 20:10:17 +03:00
|
|
|
zp->zp_complevel = complevel;
|
Native Encryption for ZFS on Linux
This change incorporates three major pieces:
The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
convenient.
The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.
The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
compromised.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #494
Closes #5769
2017-08-14 20:36:48 +03:00
|
|
|
zp->zp_checksum = checksum;
|
2010-05-29 00:45:14 +04:00
|
|
|
zp->zp_type = (wp & WP_SPILL) ? dn->dn_bonustype : type;
|
|
|
|
|
zp->zp_level = level;
|
2014-05-23 20:21:07 +04:00
|
|
|
zp->zp_copies = MIN(copies, spa_max_replication(os->os_spa));
|
2010-05-29 00:45:14 +04:00
|
|
|
zp->zp_dedup = dedup;
|
|
|
|
|
zp->zp_dedup_verify = dedup && dedup_verify;
|
2013-05-10 23:47:54 +04:00
|
|
|
zp->zp_nopwrite = nopwrite;
|
Native Encryption for ZFS on Linux
This change incorporates three major pieces:
The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
convenient.
The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.
The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
compromised.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #494
Closes #5769
2017-08-14 20:36:48 +03:00
|
|
|
zp->zp_encrypt = encrypt;
|
|
|
|
|
zp->zp_byteorder = ZFS_HOST_BYTEORDER;
|
2022-02-25 16:26:54 +03:00
|
|
|
memset(zp->zp_salt, 0, ZIO_DATA_SALT_LEN);
|
|
|
|
|
memset(zp->zp_iv, 0, ZIO_DATA_IV_LEN);
|
|
|
|
|
memset(zp->zp_mac, 0, ZIO_DATA_MAC_LEN);
|
2018-09-06 04:33:36 +03:00
|
|
|
zp->zp_zpl_smallblk = DMU_OT_IS_FILE(zp->zp_type) ?
|
|
|
|
|
os->os_zpl_special_smallblock : 0;
|
Native Encryption for ZFS on Linux
This change incorporates three major pieces:
The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
convenient.
The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.
The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
compromised.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #494
Closes #5769
2017-08-14 20:36:48 +03:00
|
|
|
|
|
|
|
|
ASSERT3U(zp->zp_compress, !=, ZIO_COMPRESS_INHERIT);
|
2010-05-29 00:45:14 +04:00
|
|
|
}
|
|
|
|
|
|
2017-03-25 00:28:38 +03:00
|
|
|
/*
|
2023-03-28 18:19:03 +03:00
|
|
|
* Reports the location of data and holes in an object. In order to
|
|
|
|
|
* accurately report holes all dirty data must be synced to disk. This
|
|
|
|
|
* causes extremely poor performance when seeking for holes in a dirty file.
|
|
|
|
|
* As a compromise, only provide hole data when the dnode is clean. When
|
|
|
|
|
* a dnode is dirty report the dnode as having no holes by returning EBUSY
|
|
|
|
|
* which is always safe to do.
|
2017-03-25 00:28:38 +03:00
|
|
|
*/
|
2008-11-20 23:01:55 +03:00
|
|
|
int
|
|
|
|
|
dmu_offset_next(objset_t *os, uint64_t object, boolean_t hole, uint64_t *off)
|
|
|
|
|
{
|
|
|
|
|
dnode_t *dn;
|
2023-03-28 18:19:03 +03:00
|
|
|
int restarted = 0, err;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2021-11-08 00:27:44 +03:00
|
|
|
restart:
|
2010-05-29 00:45:14 +04:00
|
|
|
err = dnode_hold(os, object, FTAG, &dn);
|
2008-11-20 23:01:55 +03:00
|
|
|
if (err)
|
|
|
|
|
return (err);
|
2017-03-25 00:28:38 +03:00
|
|
|
|
2021-11-08 00:27:44 +03:00
|
|
|
rw_enter(&dn->dn_struct_rwlock, RW_READER);
|
2017-03-25 00:28:38 +03:00
|
|
|
|
2021-11-08 00:27:44 +03:00
|
|
|
if (dnode_is_dirty(dn)) {
|
|
|
|
|
/*
|
|
|
|
|
* If the zfs_dmu_offset_next_sync module option is enabled
|
2023-03-28 18:19:03 +03:00
|
|
|
* then hole reporting has been requested. Dirty dnodes
|
|
|
|
|
* must be synced to disk to accurately report holes.
|
2021-11-08 00:27:44 +03:00
|
|
|
*
|
2023-03-28 18:19:03 +03:00
|
|
|
* Provided a RL_READER rangelock spanning 0-UINT64_MAX is
|
|
|
|
|
* held by the caller only a single restart will be required.
|
|
|
|
|
* We tolerate callers which do not hold the rangelock by
|
|
|
|
|
* returning EBUSY and not reporting holes after one restart.
|
2021-11-08 00:27:44 +03:00
|
|
|
*/
|
|
|
|
|
if (zfs_dmu_offset_next_sync) {
|
|
|
|
|
rw_exit(&dn->dn_struct_rwlock);
|
|
|
|
|
dnode_rele(dn, FTAG);
|
2023-03-28 18:19:03 +03:00
|
|
|
|
|
|
|
|
if (restarted)
|
|
|
|
|
return (SET_ERROR(EBUSY));
|
|
|
|
|
|
2021-11-08 00:27:44 +03:00
|
|
|
txg_wait_synced(dmu_objset_pool(os), 0);
|
2023-03-28 18:19:03 +03:00
|
|
|
restarted = 1;
|
2021-11-08 00:27:44 +03:00
|
|
|
goto restart;
|
|
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2017-03-25 00:28:38 +03:00
|
|
|
err = SET_ERROR(EBUSY);
|
2021-11-08 00:27:44 +03:00
|
|
|
} else {
|
|
|
|
|
err = dnode_next_offset(dn, DNODE_FIND_HAVELOCK |
|
|
|
|
|
(hole ? DNODE_FIND_HOLE : 0), off, 1, 1, 0);
|
|
|
|
|
}
|
2017-03-25 00:28:38 +03:00
|
|
|
|
2021-11-08 00:27:44 +03:00
|
|
|
rw_exit(&dn->dn_struct_rwlock);
|
2008-11-20 23:01:55 +03:00
|
|
|
dnode_rele(dn, FTAG);
|
|
|
|
|
|
|
|
|
|
return (err);
|
2023-03-10 22:59:53 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
int
|
|
|
|
|
dmu_read_l0_bps(objset_t *os, uint64_t object, uint64_t offset, uint64_t length,
|
2023-05-03 01:46:14 +03:00
|
|
|
blkptr_t *bps, size_t *nbpsp)
|
2023-03-10 22:59:53 +03:00
|
|
|
{
|
|
|
|
|
dmu_buf_t **dbp, *dbuf;
|
|
|
|
|
dmu_buf_impl_t *db;
|
|
|
|
|
blkptr_t *bp;
|
|
|
|
|
int error, numbufs;
|
|
|
|
|
|
|
|
|
|
error = dmu_buf_hold_array(os, object, offset, length, FALSE, FTAG,
|
|
|
|
|
&numbufs, &dbp);
|
|
|
|
|
if (error != 0) {
|
|
|
|
|
if (error == ESRCH) {
|
|
|
|
|
error = SET_ERROR(ENXIO);
|
|
|
|
|
}
|
|
|
|
|
return (error);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
ASSERT3U(numbufs, <=, *nbpsp);
|
|
|
|
|
|
|
|
|
|
for (int i = 0; i < numbufs; i++) {
|
|
|
|
|
dbuf = dbp[i];
|
|
|
|
|
db = (dmu_buf_impl_t *)dbuf;
|
2023-03-24 20:18:35 +03:00
|
|
|
|
|
|
|
|
mutex_enter(&db->db_mtx);
|
2023-03-10 22:59:53 +03:00
|
|
|
|
|
|
|
|
if (!list_is_empty(&db->db_dirty_records)) {
|
|
|
|
|
dbuf_dirty_record_t *dr;
|
|
|
|
|
|
|
|
|
|
dr = list_head(&db->db_dirty_records);
|
|
|
|
|
if (dr->dt.dl.dr_brtwrite) {
|
|
|
|
|
/*
|
|
|
|
|
* This is very special case where we clone a
|
|
|
|
|
* block and in the same transaction group we
|
|
|
|
|
* read its BP (most likely to clone the clone).
|
|
|
|
|
*/
|
|
|
|
|
bp = &dr->dt.dl.dr_overridden_by;
|
|
|
|
|
} else {
|
|
|
|
|
/*
|
|
|
|
|
* The block was modified in the same
|
|
|
|
|
* transaction group.
|
|
|
|
|
*/
|
2023-03-24 20:18:35 +03:00
|
|
|
mutex_exit(&db->db_mtx);
|
2023-03-10 22:59:53 +03:00
|
|
|
error = SET_ERROR(EAGAIN);
|
|
|
|
|
goto out;
|
|
|
|
|
}
|
2023-03-24 20:18:35 +03:00
|
|
|
} else {
|
|
|
|
|
bp = db->db_blkptr;
|
2023-03-10 22:59:53 +03:00
|
|
|
}
|
2023-03-24 20:18:35 +03:00
|
|
|
|
|
|
|
|
mutex_exit(&db->db_mtx);
|
|
|
|
|
|
2023-03-10 22:59:53 +03:00
|
|
|
if (bp == NULL) {
|
|
|
|
|
/*
|
|
|
|
|
* The block was created in this transaction group,
|
|
|
|
|
* so it has no BP yet.
|
|
|
|
|
*/
|
|
|
|
|
error = SET_ERROR(EAGAIN);
|
|
|
|
|
goto out;
|
|
|
|
|
}
|
|
|
|
|
/*
|
|
|
|
|
* Make sure we clone only data blocks.
|
|
|
|
|
*/
|
|
|
|
|
if (BP_IS_METADATA(bp) && !BP_IS_HOLE(bp)) {
|
|
|
|
|
error = SET_ERROR(EINVAL);
|
|
|
|
|
goto out;
|
|
|
|
|
}
|
|
|
|
|
|
2023-12-05 21:58:11 +03:00
|
|
|
/*
|
|
|
|
|
* If the block was allocated in transaction group that is not
|
|
|
|
|
* yet synced, we could clone it, but we couldn't write this
|
|
|
|
|
* operation into ZIL, or it may be impossible to replay, since
|
|
|
|
|
* the block may appear not yet allocated at that point.
|
|
|
|
|
*/
|
2024-03-26 01:01:54 +03:00
|
|
|
if (BP_GET_BIRTH(bp) > spa_freeze_txg(os->os_spa)) {
|
2023-12-05 21:58:11 +03:00
|
|
|
error = SET_ERROR(EINVAL);
|
|
|
|
|
goto out;
|
|
|
|
|
}
|
2024-03-26 01:01:54 +03:00
|
|
|
if (BP_GET_BIRTH(bp) > spa_last_synced_txg(os->os_spa)) {
|
2023-12-05 21:58:11 +03:00
|
|
|
error = SET_ERROR(EAGAIN);
|
|
|
|
|
goto out;
|
|
|
|
|
}
|
|
|
|
|
|
2023-03-10 22:59:53 +03:00
|
|
|
bps[i] = *bp;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
*nbpsp = numbufs;
|
|
|
|
|
out:
|
|
|
|
|
dmu_buf_rele_array(dbp, numbufs, FTAG);
|
|
|
|
|
|
|
|
|
|
return (error);
|
|
|
|
|
}
|
|
|
|
|
|
2023-05-03 00:24:43 +03:00
|
|
|
int
|
2023-03-10 22:59:53 +03:00
|
|
|
dmu_brt_clone(objset_t *os, uint64_t object, uint64_t offset, uint64_t length,
|
2023-11-29 21:51:34 +03:00
|
|
|
dmu_tx_t *tx, const blkptr_t *bps, size_t nbps)
|
2023-03-10 22:59:53 +03:00
|
|
|
{
|
|
|
|
|
spa_t *spa;
|
|
|
|
|
dmu_buf_t **dbp, *dbuf;
|
|
|
|
|
dmu_buf_impl_t *db;
|
|
|
|
|
struct dirty_leaf *dl;
|
|
|
|
|
dbuf_dirty_record_t *dr;
|
|
|
|
|
const blkptr_t *bp;
|
2023-05-03 00:24:43 +03:00
|
|
|
int error = 0, i, numbufs;
|
2023-03-10 22:59:53 +03:00
|
|
|
|
|
|
|
|
spa = os->os_spa;
|
|
|
|
|
|
|
|
|
|
VERIFY0(dmu_buf_hold_array(os, object, offset, length, FALSE, FTAG,
|
|
|
|
|
&numbufs, &dbp));
|
|
|
|
|
ASSERT3U(nbps, ==, numbufs);
|
|
|
|
|
|
2023-05-03 00:24:43 +03:00
|
|
|
/*
|
2023-05-05 02:14:19 +03:00
|
|
|
* Before we start cloning make sure that the dbufs sizes match new BPs
|
2023-05-03 00:24:43 +03:00
|
|
|
* sizes. If they don't, that's a no-go, as we are not able to shrink
|
|
|
|
|
* dbufs.
|
|
|
|
|
*/
|
|
|
|
|
for (i = 0; i < numbufs; i++) {
|
|
|
|
|
dbuf = dbp[i];
|
|
|
|
|
db = (dmu_buf_impl_t *)dbuf;
|
|
|
|
|
bp = &bps[i];
|
|
|
|
|
|
|
|
|
|
ASSERT0(db->db_level);
|
|
|
|
|
ASSERT(db->db_blkid != DMU_BONUS_BLKID);
|
2023-05-10 08:32:30 +03:00
|
|
|
ASSERT(db->db_blkid != DMU_SPILL_BLKID);
|
2023-05-03 00:24:43 +03:00
|
|
|
|
|
|
|
|
if (!BP_IS_HOLE(bp) && BP_GET_LSIZE(bp) != dbuf->db_size) {
|
|
|
|
|
error = SET_ERROR(EXDEV);
|
|
|
|
|
goto out;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
for (i = 0; i < numbufs; i++) {
|
2023-03-10 22:59:53 +03:00
|
|
|
dbuf = dbp[i];
|
|
|
|
|
db = (dmu_buf_impl_t *)dbuf;
|
|
|
|
|
bp = &bps[i];
|
|
|
|
|
|
|
|
|
|
ASSERT0(db->db_level);
|
|
|
|
|
ASSERT(db->db_blkid != DMU_BONUS_BLKID);
|
2023-05-10 08:32:30 +03:00
|
|
|
ASSERT(db->db_blkid != DMU_SPILL_BLKID);
|
2023-03-10 22:59:53 +03:00
|
|
|
ASSERT(BP_IS_HOLE(bp) || dbuf->db_size == BP_GET_LSIZE(bp));
|
|
|
|
|
|
2023-04-30 12:47:09 +03:00
|
|
|
dmu_buf_will_clone(dbuf, tx);
|
2023-03-24 20:18:35 +03:00
|
|
|
|
|
|
|
|
mutex_enter(&db->db_mtx);
|
|
|
|
|
|
2023-03-10 22:59:53 +03:00
|
|
|
dr = list_head(&db->db_dirty_records);
|
2023-03-24 20:18:35 +03:00
|
|
|
VERIFY(dr != NULL);
|
2023-03-10 22:59:53 +03:00
|
|
|
ASSERT3U(dr->dr_txg, ==, tx->tx_txg);
|
|
|
|
|
dl = &dr->dt.dl;
|
|
|
|
|
dl->dr_overridden_by = *bp;
|
|
|
|
|
dl->dr_brtwrite = B_TRUE;
|
|
|
|
|
dl->dr_override_state = DR_OVERRIDDEN;
|
|
|
|
|
if (BP_IS_HOLE(bp)) {
|
2024-03-26 01:01:54 +03:00
|
|
|
BP_SET_LOGICAL_BIRTH(&dl->dr_overridden_by, 0);
|
|
|
|
|
BP_SET_PHYSICAL_BIRTH(&dl->dr_overridden_by, 0);
|
2023-03-10 22:59:53 +03:00
|
|
|
} else {
|
2024-03-26 01:01:54 +03:00
|
|
|
BP_SET_LOGICAL_BIRTH(&dl->dr_overridden_by,
|
|
|
|
|
dr->dr_txg);
|
2023-04-13 02:15:05 +03:00
|
|
|
if (!BP_IS_EMBEDDED(bp)) {
|
2024-03-26 01:01:54 +03:00
|
|
|
BP_SET_PHYSICAL_BIRTH(&dl->dr_overridden_by,
|
|
|
|
|
BP_GET_BIRTH(bp));
|
2023-04-13 02:15:05 +03:00
|
|
|
}
|
2023-03-10 22:59:53 +03:00
|
|
|
}
|
|
|
|
|
|
2023-03-24 20:18:35 +03:00
|
|
|
mutex_exit(&db->db_mtx);
|
|
|
|
|
|
2023-03-10 22:59:53 +03:00
|
|
|
/*
|
|
|
|
|
* When data in embedded into BP there is no need to create
|
|
|
|
|
* BRT entry as there is no data block. Just copy the BP as
|
|
|
|
|
* it contains the data.
|
|
|
|
|
*/
|
2023-11-29 21:51:34 +03:00
|
|
|
if (!BP_IS_HOLE(bp) && !BP_IS_EMBEDDED(bp)) {
|
2023-03-10 22:59:53 +03:00
|
|
|
brt_pending_add(spa, bp, tx);
|
|
|
|
|
}
|
|
|
|
|
}
|
2023-05-03 00:24:43 +03:00
|
|
|
out:
|
2023-03-10 22:59:53 +03:00
|
|
|
dmu_buf_rele_array(dbp, numbufs, FTAG);
|
2023-05-03 00:24:43 +03:00
|
|
|
|
|
|
|
|
return (error);
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
2013-10-03 04:11:19 +04:00
|
|
|
__dmu_object_info_from_dnode(dnode_t *dn, dmu_object_info_t *doi)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
2013-10-03 04:11:19 +04:00
|
|
|
dnode_phys_t *dnp = dn->dn_phys;
|
2010-05-29 00:45:14 +04:00
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
doi->doi_data_block_size = dn->dn_datablksz;
|
|
|
|
|
doi->doi_metadata_block_size = dn->dn_indblkshift ?
|
|
|
|
|
1ULL << dn->dn_indblkshift : 0;
|
2010-05-29 00:45:14 +04:00
|
|
|
doi->doi_type = dn->dn_type;
|
|
|
|
|
doi->doi_bonus_type = dn->dn_bonustype;
|
|
|
|
|
doi->doi_bonus_size = dn->dn_bonuslen;
|
Implement large_dnode pool feature
Justification
-------------
This feature adds support for variable length dnodes. Our motivation is
to eliminate the overhead associated with using spill blocks. Spill
blocks are used to store system attribute data (i.e. file metadata) that
does not fit in the dnode's bonus buffer. By allowing a larger bonus
buffer area the use of a spill block can be avoided. Spill blocks
potentially incur an additional read I/O for every dnode in a dnode
block. As a worst case example, reading 32 dnodes from a 16k dnode block
and all of the spill blocks could issue 33 separate reads. Now suppose
those dnodes have size 1024 and therefore don't need spill blocks. Then
the worst case number of blocks read is reduced to from 33 to two--one
per dnode block. In practice spill blocks may tend to be co-located on
disk with the dnode blocks so the reduction in I/O would not be this
drastic. In a badly fragmented pool, however, the improvement could be
significant.
ZFS-on-Linux systems that make heavy use of extended attributes would
benefit from this feature. In particular, ZFS-on-Linux supports the
xattr=sa dataset property which allows file extended attribute data
to be stored in the dnode bonus buffer as an alternative to the
traditional directory-based format. Workloads such as SELinux and the
Lustre distributed filesystem often store enough xattr data to force
spill bocks when xattr=sa is in effect. Large dnodes may therefore
provide a performance benefit to such systems.
Other use cases that may benefit from this feature include files with
large ACLs and symbolic links with long target names. Furthermore,
this feature may be desirable on other platforms in case future
applications or features are developed that could make use of a
larger bonus buffer area.
Implementation
--------------
The size of a dnode may be a multiple of 512 bytes up to the size of
a dnode block (currently 16384 bytes). A dn_extra_slots field was
added to the current on-disk dnode_phys_t structure to describe the
size of the physical dnode on disk. The 8 bits for this field were
taken from the zero filled dn_pad2 field. The field represents how
many "extra" dnode_phys_t slots a dnode consumes in its dnode block.
This convention results in a value of 0 for 512 byte dnodes which
preserves on-disk format compatibility with older software.
Similarly, the in-memory dnode_t structure has a new dn_num_slots field
to represent the total number of dnode_phys_t slots consumed on disk.
Thus dn->dn_num_slots is 1 greater than the corresponding
dnp->dn_extra_slots. This difference in convention was adopted
because, unlike on-disk structures, backward compatibility is not a
concern for in-memory objects, so we used a more natural way to
represent size for a dnode_t.
The default size for newly created dnodes is determined by the value of
a new "dnodesize" dataset property. By default the property is set to
"legacy" which is compatible with older software. Setting the property
to "auto" will allow the filesystem to choose the most suitable dnode
size. Currently this just sets the default dnode size to 1k, but future
code improvements could dynamically choose a size based on observed
workload patterns. Dnodes of varying sizes can coexist within the same
dataset and even within the same dnode block. For example, to enable
automatically-sized dnodes, run
# zfs set dnodesize=auto tank/fish
The user can also specify literal values for the dnodesize property.
These are currently limited to powers of two from 1k to 16k. The
power-of-2 limitation is only for simplicity of the user interface.
Internally the implementation can handle any multiple of 512 up to 16k,
and consumers of the DMU API can specify any legal dnode value.
The size of a new dnode is determined at object allocation time and
stored as a new field in the znode in-memory structure. New DMU
interfaces are added to allow the consumer to specify the dnode size
that a newly allocated object should use. Existing interfaces are
unchanged to avoid having to update every call site and to preserve
compatibility with external consumers such as Lustre. The new
interfaces names are given below. The versions of these functions that
don't take a dnodesize parameter now just call the _dnsize() versions
with a dnodesize of 0, which means use the legacy dnode size.
New DMU interfaces:
dmu_object_alloc_dnsize()
dmu_object_claim_dnsize()
dmu_object_reclaim_dnsize()
New ZAP interfaces:
zap_create_dnsize()
zap_create_norm_dnsize()
zap_create_flags_dnsize()
zap_create_claim_norm_dnsize()
zap_create_link_dnsize()
The constant DN_MAX_BONUSLEN is renamed to DN_OLD_MAX_BONUSLEN. The
spa_maxdnodesize() function should be used to determine the maximum
bonus length for a pool.
These are a few noteworthy changes to key functions:
* The prototype for dnode_hold_impl() now takes a "slots" parameter.
When the DNODE_MUST_BE_FREE flag is set, this parameter is used to
ensure the hole at the specified object offset is large enough to
hold the dnode being created. The slots parameter is also used
to ensure a dnode does not span multiple dnode blocks. In both of
these cases, if a failure occurs, ENOSPC is returned. Keep in mind,
these failure cases are only possible when using DNODE_MUST_BE_FREE.
If the DNODE_MUST_BE_ALLOCATED flag is set, "slots" must be 0.
dnode_hold_impl() will check if the requested dnode is already
consumed as an extra dnode slot by an large dnode, in which case
it returns ENOENT.
* The function dmu_object_alloc() advances to the next dnode block
if dnode_hold_impl() returns an error for a requested object.
This is because the beginning of the next dnode block is the only
location it can safely assume to either be a hole or a valid
starting point for a dnode.
* dnode_next_offset_level() and other functions that iterate
through dnode blocks may no longer use a simple array indexing
scheme. These now use the current dnode's dn_num_slots field to
advance to the next dnode in the block. This is to ensure we
properly skip the current dnode's bonus area and don't interpret it
as a valid dnode.
zdb
---
The zdb command was updated to display a dnode's size under the
"dnsize" column when the object is dumped.
For ZIL create log records, zdb will now display the slot count for
the object.
ztest
-----
Ztest chooses a random dnodesize for every newly created object. The
random distribution is more heavily weighted toward small dnodes to
better simulate real-world datasets.
Unused bonus buffer space is filled with non-zero values computed from
the object number, dataset id, offset, and generation number. This
helps ensure that the dnode traversal code properly skips the interior
regions of large dnodes, and that these interior regions are not
overwritten by data belonging to other dnodes. A new test visits each
object in a dataset. It verifies that the actual dnode size matches what
was stored in the ztest block tag when it was created. It also verifies
that the unused bonus buffer space is filled with the expected data
patterns.
ZFS Test Suite
--------------
Added six new large dnode-specific tests, and integrated the dnodesize
property into existing tests for zfs allow and send/recv.
Send/Receive
------------
ZFS send streams for datasets containing large dnodes cannot be received
on pools that don't support the large_dnode feature. A send stream with
large dnodes sets a DMU_BACKUP_FEATURE_LARGE_DNODE flag which will be
unrecognized by an incompatible receiving pool so that the zfs receive
will fail gracefully.
While not implemented here, it may be possible to generate a
backward-compatible send stream from a dataset containing large
dnodes. The implementation may be tricky, however, because the send
object record for a large dnode would need to be resized to a 512
byte dnode, possibly kicking in a spill block in the process. This
means we would need to construct a new SA layout and possibly
register it in the SA layout object. The SA layout is normally just
sent as an ordinary object record. But if we are constructing new
layouts while generating the send stream we'd have to build the SA
layout object dynamically and send it at the end of the stream.
For sending and receiving between pools that do support large dnodes,
the drr_object send record type is extended with a new field to store
the dnode slot count. This field was repurposed from unused padding
in the structure.
ZIL Replay
----------
The dnode slot count is stored in the uppermost 8 bits of the lr_foid
field. The bits were unused as the object id is currently capped at
48 bits.
Resizing Dnodes
---------------
It should be possible to resize a dnode when it is dirtied if the
current dnodesize dataset property differs from the dnode's size, but
this functionality is not currently implemented. Clearly a dnode can
only grow if there are sufficient contiguous unused slots in the
dnode block, but it should always be possible to shrink a dnode.
Growing dnodes may be useful to reduce fragmentation in a pool with
many spill blocks in use. Shrinking dnodes may be useful to allow
sending a dataset to a pool that doesn't support the large_dnode
feature.
Feature Reference Counting
--------------------------
The reference count for the large_dnode pool feature tracks the
number of datasets that have ever contained a dnode of size larger
than 512 bytes. The first time a large dnode is created in a dataset
the dataset is converted to an extensible dataset. This is a one-way
operation and the only way to decrement the feature count is to
destroy the dataset, even if the dataset no longer contains any large
dnodes. The complexity of reference counting on a per-dnode basis was
too high, so we chose to track it on a per-dataset basis similarly to
the large_block feature.
Signed-off-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #3542
2016-03-17 04:25:34 +03:00
|
|
|
doi->doi_dnodesize = dn->dn_num_slots << DNODE_SHIFT;
|
2008-11-20 23:01:55 +03:00
|
|
|
doi->doi_indirection = dn->dn_nlevels;
|
|
|
|
|
doi->doi_checksum = dn->dn_checksum;
|
|
|
|
|
doi->doi_compress = dn->dn_compress;
|
2014-09-12 07:28:35 +04:00
|
|
|
doi->doi_nblkptr = dn->dn_nblkptr;
|
2010-05-29 00:45:14 +04:00
|
|
|
doi->doi_physical_blocks_512 = (DN_USED_BYTES(dnp) + 256) >> 9;
|
2013-07-05 23:37:16 +04:00
|
|
|
doi->doi_max_offset = (dn->dn_maxblkid + 1) * dn->dn_datablksz;
|
2010-05-29 00:45:14 +04:00
|
|
|
doi->doi_fill_count = 0;
|
2017-11-04 23:25:13 +03:00
|
|
|
for (int i = 0; i < dnp->dn_nblkptr; i++)
|
2014-06-06 01:19:08 +04:00
|
|
|
doi->doi_fill_count += BP_GET_FILL(&dnp->dn_blkptr[i]);
|
2013-10-03 04:11:19 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
|
|
|
|
dmu_object_info_from_dnode(dnode_t *dn, dmu_object_info_t *doi)
|
|
|
|
|
{
|
|
|
|
|
rw_enter(&dn->dn_struct_rwlock, RW_READER);
|
|
|
|
|
mutex_enter(&dn->dn_mtx);
|
|
|
|
|
|
|
|
|
|
__dmu_object_info_from_dnode(dn, doi);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
mutex_exit(&dn->dn_mtx);
|
|
|
|
|
rw_exit(&dn->dn_struct_rwlock);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Get information on a DMU object.
|
|
|
|
|
* If doi is NULL, just indicates whether the object exists.
|
|
|
|
|
*/
|
|
|
|
|
int
|
|
|
|
|
dmu_object_info(objset_t *os, uint64_t object, dmu_object_info_t *doi)
|
|
|
|
|
{
|
|
|
|
|
dnode_t *dn;
|
2010-05-29 00:45:14 +04:00
|
|
|
int err = dnode_hold(os, object, FTAG, &dn);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
if (err)
|
|
|
|
|
return (err);
|
|
|
|
|
|
|
|
|
|
if (doi != NULL)
|
|
|
|
|
dmu_object_info_from_dnode(dn, doi);
|
|
|
|
|
|
|
|
|
|
dnode_rele(dn, FTAG);
|
|
|
|
|
return (0);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* As above, but faster; can be used when you have a held dbuf in hand.
|
|
|
|
|
*/
|
|
|
|
|
void
|
2010-08-27 01:24:34 +04:00
|
|
|
dmu_object_info_from_db(dmu_buf_t *db_fake, dmu_object_info_t *doi)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
2010-08-27 01:24:34 +04:00
|
|
|
dmu_buf_impl_t *db = (dmu_buf_impl_t *)db_fake;
|
|
|
|
|
|
|
|
|
|
DB_DNODE_ENTER(db);
|
|
|
|
|
dmu_object_info_from_dnode(DB_DNODE(db), doi);
|
|
|
|
|
DB_DNODE_EXIT(db);
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Faster still when you only care about the size.
|
|
|
|
|
*/
|
|
|
|
|
void
|
2010-08-27 01:24:34 +04:00
|
|
|
dmu_object_size_from_db(dmu_buf_t *db_fake, uint32_t *blksize,
|
|
|
|
|
u_longlong_t *nblk512)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
2010-08-27 01:24:34 +04:00
|
|
|
dmu_buf_impl_t *db = (dmu_buf_impl_t *)db_fake;
|
|
|
|
|
dnode_t *dn;
|
|
|
|
|
|
|
|
|
|
DB_DNODE_ENTER(db);
|
|
|
|
|
dn = DB_DNODE(db);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
*blksize = dn->dn_datablksz;
|
Implement large_dnode pool feature
Justification
-------------
This feature adds support for variable length dnodes. Our motivation is
to eliminate the overhead associated with using spill blocks. Spill
blocks are used to store system attribute data (i.e. file metadata) that
does not fit in the dnode's bonus buffer. By allowing a larger bonus
buffer area the use of a spill block can be avoided. Spill blocks
potentially incur an additional read I/O for every dnode in a dnode
block. As a worst case example, reading 32 dnodes from a 16k dnode block
and all of the spill blocks could issue 33 separate reads. Now suppose
those dnodes have size 1024 and therefore don't need spill blocks. Then
the worst case number of blocks read is reduced to from 33 to two--one
per dnode block. In practice spill blocks may tend to be co-located on
disk with the dnode blocks so the reduction in I/O would not be this
drastic. In a badly fragmented pool, however, the improvement could be
significant.
ZFS-on-Linux systems that make heavy use of extended attributes would
benefit from this feature. In particular, ZFS-on-Linux supports the
xattr=sa dataset property which allows file extended attribute data
to be stored in the dnode bonus buffer as an alternative to the
traditional directory-based format. Workloads such as SELinux and the
Lustre distributed filesystem often store enough xattr data to force
spill bocks when xattr=sa is in effect. Large dnodes may therefore
provide a performance benefit to such systems.
Other use cases that may benefit from this feature include files with
large ACLs and symbolic links with long target names. Furthermore,
this feature may be desirable on other platforms in case future
applications or features are developed that could make use of a
larger bonus buffer area.
Implementation
--------------
The size of a dnode may be a multiple of 512 bytes up to the size of
a dnode block (currently 16384 bytes). A dn_extra_slots field was
added to the current on-disk dnode_phys_t structure to describe the
size of the physical dnode on disk. The 8 bits for this field were
taken from the zero filled dn_pad2 field. The field represents how
many "extra" dnode_phys_t slots a dnode consumes in its dnode block.
This convention results in a value of 0 for 512 byte dnodes which
preserves on-disk format compatibility with older software.
Similarly, the in-memory dnode_t structure has a new dn_num_slots field
to represent the total number of dnode_phys_t slots consumed on disk.
Thus dn->dn_num_slots is 1 greater than the corresponding
dnp->dn_extra_slots. This difference in convention was adopted
because, unlike on-disk structures, backward compatibility is not a
concern for in-memory objects, so we used a more natural way to
represent size for a dnode_t.
The default size for newly created dnodes is determined by the value of
a new "dnodesize" dataset property. By default the property is set to
"legacy" which is compatible with older software. Setting the property
to "auto" will allow the filesystem to choose the most suitable dnode
size. Currently this just sets the default dnode size to 1k, but future
code improvements could dynamically choose a size based on observed
workload patterns. Dnodes of varying sizes can coexist within the same
dataset and even within the same dnode block. For example, to enable
automatically-sized dnodes, run
# zfs set dnodesize=auto tank/fish
The user can also specify literal values for the dnodesize property.
These are currently limited to powers of two from 1k to 16k. The
power-of-2 limitation is only for simplicity of the user interface.
Internally the implementation can handle any multiple of 512 up to 16k,
and consumers of the DMU API can specify any legal dnode value.
The size of a new dnode is determined at object allocation time and
stored as a new field in the znode in-memory structure. New DMU
interfaces are added to allow the consumer to specify the dnode size
that a newly allocated object should use. Existing interfaces are
unchanged to avoid having to update every call site and to preserve
compatibility with external consumers such as Lustre. The new
interfaces names are given below. The versions of these functions that
don't take a dnodesize parameter now just call the _dnsize() versions
with a dnodesize of 0, which means use the legacy dnode size.
New DMU interfaces:
dmu_object_alloc_dnsize()
dmu_object_claim_dnsize()
dmu_object_reclaim_dnsize()
New ZAP interfaces:
zap_create_dnsize()
zap_create_norm_dnsize()
zap_create_flags_dnsize()
zap_create_claim_norm_dnsize()
zap_create_link_dnsize()
The constant DN_MAX_BONUSLEN is renamed to DN_OLD_MAX_BONUSLEN. The
spa_maxdnodesize() function should be used to determine the maximum
bonus length for a pool.
These are a few noteworthy changes to key functions:
* The prototype for dnode_hold_impl() now takes a "slots" parameter.
When the DNODE_MUST_BE_FREE flag is set, this parameter is used to
ensure the hole at the specified object offset is large enough to
hold the dnode being created. The slots parameter is also used
to ensure a dnode does not span multiple dnode blocks. In both of
these cases, if a failure occurs, ENOSPC is returned. Keep in mind,
these failure cases are only possible when using DNODE_MUST_BE_FREE.
If the DNODE_MUST_BE_ALLOCATED flag is set, "slots" must be 0.
dnode_hold_impl() will check if the requested dnode is already
consumed as an extra dnode slot by an large dnode, in which case
it returns ENOENT.
* The function dmu_object_alloc() advances to the next dnode block
if dnode_hold_impl() returns an error for a requested object.
This is because the beginning of the next dnode block is the only
location it can safely assume to either be a hole or a valid
starting point for a dnode.
* dnode_next_offset_level() and other functions that iterate
through dnode blocks may no longer use a simple array indexing
scheme. These now use the current dnode's dn_num_slots field to
advance to the next dnode in the block. This is to ensure we
properly skip the current dnode's bonus area and don't interpret it
as a valid dnode.
zdb
---
The zdb command was updated to display a dnode's size under the
"dnsize" column when the object is dumped.
For ZIL create log records, zdb will now display the slot count for
the object.
ztest
-----
Ztest chooses a random dnodesize for every newly created object. The
random distribution is more heavily weighted toward small dnodes to
better simulate real-world datasets.
Unused bonus buffer space is filled with non-zero values computed from
the object number, dataset id, offset, and generation number. This
helps ensure that the dnode traversal code properly skips the interior
regions of large dnodes, and that these interior regions are not
overwritten by data belonging to other dnodes. A new test visits each
object in a dataset. It verifies that the actual dnode size matches what
was stored in the ztest block tag when it was created. It also verifies
that the unused bonus buffer space is filled with the expected data
patterns.
ZFS Test Suite
--------------
Added six new large dnode-specific tests, and integrated the dnodesize
property into existing tests for zfs allow and send/recv.
Send/Receive
------------
ZFS send streams for datasets containing large dnodes cannot be received
on pools that don't support the large_dnode feature. A send stream with
large dnodes sets a DMU_BACKUP_FEATURE_LARGE_DNODE flag which will be
unrecognized by an incompatible receiving pool so that the zfs receive
will fail gracefully.
While not implemented here, it may be possible to generate a
backward-compatible send stream from a dataset containing large
dnodes. The implementation may be tricky, however, because the send
object record for a large dnode would need to be resized to a 512
byte dnode, possibly kicking in a spill block in the process. This
means we would need to construct a new SA layout and possibly
register it in the SA layout object. The SA layout is normally just
sent as an ordinary object record. But if we are constructing new
layouts while generating the send stream we'd have to build the SA
layout object dynamically and send it at the end of the stream.
For sending and receiving between pools that do support large dnodes,
the drr_object send record type is extended with a new field to store
the dnode slot count. This field was repurposed from unused padding
in the structure.
ZIL Replay
----------
The dnode slot count is stored in the uppermost 8 bits of the lr_foid
field. The bits were unused as the object id is currently capped at
48 bits.
Resizing Dnodes
---------------
It should be possible to resize a dnode when it is dirtied if the
current dnodesize dataset property differs from the dnode's size, but
this functionality is not currently implemented. Clearly a dnode can
only grow if there are sufficient contiguous unused slots in the
dnode block, but it should always be possible to shrink a dnode.
Growing dnodes may be useful to reduce fragmentation in a pool with
many spill blocks in use. Shrinking dnodes may be useful to allow
sending a dataset to a pool that doesn't support the large_dnode
feature.
Feature Reference Counting
--------------------------
The reference count for the large_dnode pool feature tracks the
number of datasets that have ever contained a dnode of size larger
than 512 bytes. The first time a large dnode is created in a dataset
the dataset is converted to an extensible dataset. This is a one-way
operation and the only way to decrement the feature count is to
destroy the dataset, even if the dataset no longer contains any large
dnodes. The complexity of reference counting on a per-dnode basis was
too high, so we chose to track it on a per-dataset basis similarly to
the large_block feature.
Signed-off-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #3542
2016-03-17 04:25:34 +03:00
|
|
|
/* add in number of slots used for the dnode itself */
|
2008-11-20 23:01:55 +03:00
|
|
|
*nblk512 = ((DN_USED_BYTES(dn->dn_phys) + SPA_MINBLOCKSIZE/2) >>
|
Implement large_dnode pool feature
Justification
-------------
This feature adds support for variable length dnodes. Our motivation is
to eliminate the overhead associated with using spill blocks. Spill
blocks are used to store system attribute data (i.e. file metadata) that
does not fit in the dnode's bonus buffer. By allowing a larger bonus
buffer area the use of a spill block can be avoided. Spill blocks
potentially incur an additional read I/O for every dnode in a dnode
block. As a worst case example, reading 32 dnodes from a 16k dnode block
and all of the spill blocks could issue 33 separate reads. Now suppose
those dnodes have size 1024 and therefore don't need spill blocks. Then
the worst case number of blocks read is reduced to from 33 to two--one
per dnode block. In practice spill blocks may tend to be co-located on
disk with the dnode blocks so the reduction in I/O would not be this
drastic. In a badly fragmented pool, however, the improvement could be
significant.
ZFS-on-Linux systems that make heavy use of extended attributes would
benefit from this feature. In particular, ZFS-on-Linux supports the
xattr=sa dataset property which allows file extended attribute data
to be stored in the dnode bonus buffer as an alternative to the
traditional directory-based format. Workloads such as SELinux and the
Lustre distributed filesystem often store enough xattr data to force
spill bocks when xattr=sa is in effect. Large dnodes may therefore
provide a performance benefit to such systems.
Other use cases that may benefit from this feature include files with
large ACLs and symbolic links with long target names. Furthermore,
this feature may be desirable on other platforms in case future
applications or features are developed that could make use of a
larger bonus buffer area.
Implementation
--------------
The size of a dnode may be a multiple of 512 bytes up to the size of
a dnode block (currently 16384 bytes). A dn_extra_slots field was
added to the current on-disk dnode_phys_t structure to describe the
size of the physical dnode on disk. The 8 bits for this field were
taken from the zero filled dn_pad2 field. The field represents how
many "extra" dnode_phys_t slots a dnode consumes in its dnode block.
This convention results in a value of 0 for 512 byte dnodes which
preserves on-disk format compatibility with older software.
Similarly, the in-memory dnode_t structure has a new dn_num_slots field
to represent the total number of dnode_phys_t slots consumed on disk.
Thus dn->dn_num_slots is 1 greater than the corresponding
dnp->dn_extra_slots. This difference in convention was adopted
because, unlike on-disk structures, backward compatibility is not a
concern for in-memory objects, so we used a more natural way to
represent size for a dnode_t.
The default size for newly created dnodes is determined by the value of
a new "dnodesize" dataset property. By default the property is set to
"legacy" which is compatible with older software. Setting the property
to "auto" will allow the filesystem to choose the most suitable dnode
size. Currently this just sets the default dnode size to 1k, but future
code improvements could dynamically choose a size based on observed
workload patterns. Dnodes of varying sizes can coexist within the same
dataset and even within the same dnode block. For example, to enable
automatically-sized dnodes, run
# zfs set dnodesize=auto tank/fish
The user can also specify literal values for the dnodesize property.
These are currently limited to powers of two from 1k to 16k. The
power-of-2 limitation is only for simplicity of the user interface.
Internally the implementation can handle any multiple of 512 up to 16k,
and consumers of the DMU API can specify any legal dnode value.
The size of a new dnode is determined at object allocation time and
stored as a new field in the znode in-memory structure. New DMU
interfaces are added to allow the consumer to specify the dnode size
that a newly allocated object should use. Existing interfaces are
unchanged to avoid having to update every call site and to preserve
compatibility with external consumers such as Lustre. The new
interfaces names are given below. The versions of these functions that
don't take a dnodesize parameter now just call the _dnsize() versions
with a dnodesize of 0, which means use the legacy dnode size.
New DMU interfaces:
dmu_object_alloc_dnsize()
dmu_object_claim_dnsize()
dmu_object_reclaim_dnsize()
New ZAP interfaces:
zap_create_dnsize()
zap_create_norm_dnsize()
zap_create_flags_dnsize()
zap_create_claim_norm_dnsize()
zap_create_link_dnsize()
The constant DN_MAX_BONUSLEN is renamed to DN_OLD_MAX_BONUSLEN. The
spa_maxdnodesize() function should be used to determine the maximum
bonus length for a pool.
These are a few noteworthy changes to key functions:
* The prototype for dnode_hold_impl() now takes a "slots" parameter.
When the DNODE_MUST_BE_FREE flag is set, this parameter is used to
ensure the hole at the specified object offset is large enough to
hold the dnode being created. The slots parameter is also used
to ensure a dnode does not span multiple dnode blocks. In both of
these cases, if a failure occurs, ENOSPC is returned. Keep in mind,
these failure cases are only possible when using DNODE_MUST_BE_FREE.
If the DNODE_MUST_BE_ALLOCATED flag is set, "slots" must be 0.
dnode_hold_impl() will check if the requested dnode is already
consumed as an extra dnode slot by an large dnode, in which case
it returns ENOENT.
* The function dmu_object_alloc() advances to the next dnode block
if dnode_hold_impl() returns an error for a requested object.
This is because the beginning of the next dnode block is the only
location it can safely assume to either be a hole or a valid
starting point for a dnode.
* dnode_next_offset_level() and other functions that iterate
through dnode blocks may no longer use a simple array indexing
scheme. These now use the current dnode's dn_num_slots field to
advance to the next dnode in the block. This is to ensure we
properly skip the current dnode's bonus area and don't interpret it
as a valid dnode.
zdb
---
The zdb command was updated to display a dnode's size under the
"dnsize" column when the object is dumped.
For ZIL create log records, zdb will now display the slot count for
the object.
ztest
-----
Ztest chooses a random dnodesize for every newly created object. The
random distribution is more heavily weighted toward small dnodes to
better simulate real-world datasets.
Unused bonus buffer space is filled with non-zero values computed from
the object number, dataset id, offset, and generation number. This
helps ensure that the dnode traversal code properly skips the interior
regions of large dnodes, and that these interior regions are not
overwritten by data belonging to other dnodes. A new test visits each
object in a dataset. It verifies that the actual dnode size matches what
was stored in the ztest block tag when it was created. It also verifies
that the unused bonus buffer space is filled with the expected data
patterns.
ZFS Test Suite
--------------
Added six new large dnode-specific tests, and integrated the dnodesize
property into existing tests for zfs allow and send/recv.
Send/Receive
------------
ZFS send streams for datasets containing large dnodes cannot be received
on pools that don't support the large_dnode feature. A send stream with
large dnodes sets a DMU_BACKUP_FEATURE_LARGE_DNODE flag which will be
unrecognized by an incompatible receiving pool so that the zfs receive
will fail gracefully.
While not implemented here, it may be possible to generate a
backward-compatible send stream from a dataset containing large
dnodes. The implementation may be tricky, however, because the send
object record for a large dnode would need to be resized to a 512
byte dnode, possibly kicking in a spill block in the process. This
means we would need to construct a new SA layout and possibly
register it in the SA layout object. The SA layout is normally just
sent as an ordinary object record. But if we are constructing new
layouts while generating the send stream we'd have to build the SA
layout object dynamically and send it at the end of the stream.
For sending and receiving between pools that do support large dnodes,
the drr_object send record type is extended with a new field to store
the dnode slot count. This field was repurposed from unused padding
in the structure.
ZIL Replay
----------
The dnode slot count is stored in the uppermost 8 bits of the lr_foid
field. The bits were unused as the object id is currently capped at
48 bits.
Resizing Dnodes
---------------
It should be possible to resize a dnode when it is dirtied if the
current dnodesize dataset property differs from the dnode's size, but
this functionality is not currently implemented. Clearly a dnode can
only grow if there are sufficient contiguous unused slots in the
dnode block, but it should always be possible to shrink a dnode.
Growing dnodes may be useful to reduce fragmentation in a pool with
many spill blocks in use. Shrinking dnodes may be useful to allow
sending a dataset to a pool that doesn't support the large_dnode
feature.
Feature Reference Counting
--------------------------
The reference count for the large_dnode pool feature tracks the
number of datasets that have ever contained a dnode of size larger
than 512 bytes. The first time a large dnode is created in a dataset
the dataset is converted to an extensible dataset. This is a one-way
operation and the only way to decrement the feature count is to
destroy the dataset, even if the dataset no longer contains any large
dnodes. The complexity of reference counting on a per-dnode basis was
too high, so we chose to track it on a per-dataset basis similarly to
the large_block feature.
Signed-off-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #3542
2016-03-17 04:25:34 +03:00
|
|
|
SPA_MINBLOCKSHIFT) + dn->dn_num_slots;
|
|
|
|
|
DB_DNODE_EXIT(db);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
|
|
|
|
dmu_object_dnsize_from_db(dmu_buf_t *db_fake, int *dnsize)
|
|
|
|
|
{
|
|
|
|
|
dmu_buf_impl_t *db = (dmu_buf_impl_t *)db_fake;
|
|
|
|
|
dnode_t *dn;
|
|
|
|
|
|
|
|
|
|
DB_DNODE_ENTER(db);
|
|
|
|
|
dn = DB_DNODE(db);
|
|
|
|
|
*dnsize = dn->dn_num_slots << DNODE_SHIFT;
|
2010-08-27 01:24:34 +04:00
|
|
|
DB_DNODE_EXIT(db);
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
|
|
|
|
byteswap_uint64_array(void *vbuf, size_t size)
|
|
|
|
|
{
|
|
|
|
|
uint64_t *buf = vbuf;
|
|
|
|
|
size_t count = size >> 3;
|
|
|
|
|
int i;
|
|
|
|
|
|
|
|
|
|
ASSERT((size & 7) == 0);
|
|
|
|
|
|
|
|
|
|
for (i = 0; i < count; i++)
|
|
|
|
|
buf[i] = BSWAP_64(buf[i]);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
|
|
|
|
byteswap_uint32_array(void *vbuf, size_t size)
|
|
|
|
|
{
|
|
|
|
|
uint32_t *buf = vbuf;
|
|
|
|
|
size_t count = size >> 2;
|
|
|
|
|
int i;
|
|
|
|
|
|
|
|
|
|
ASSERT((size & 3) == 0);
|
|
|
|
|
|
|
|
|
|
for (i = 0; i < count; i++)
|
|
|
|
|
buf[i] = BSWAP_32(buf[i]);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
|
|
|
|
byteswap_uint16_array(void *vbuf, size_t size)
|
|
|
|
|
{
|
|
|
|
|
uint16_t *buf = vbuf;
|
|
|
|
|
size_t count = size >> 1;
|
|
|
|
|
int i;
|
|
|
|
|
|
|
|
|
|
ASSERT((size & 1) == 0);
|
|
|
|
|
|
|
|
|
|
for (i = 0; i < count; i++)
|
|
|
|
|
buf[i] = BSWAP_16(buf[i]);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
|
|
|
|
byteswap_uint8_array(void *vbuf, size_t size)
|
|
|
|
|
{
|
2021-12-12 18:06:44 +03:00
|
|
|
(void) vbuf, (void) size;
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
|
|
|
|
dmu_init(void)
|
|
|
|
|
{
|
2016-07-22 18:52:49 +03:00
|
|
|
abd_init();
|
2010-05-29 00:45:14 +04:00
|
|
|
zfs_dbgmsg_init();
|
2010-08-27 01:24:34 +04:00
|
|
|
sa_cache_init();
|
|
|
|
|
dmu_objset_init();
|
2008-11-20 23:01:55 +03:00
|
|
|
dnode_init();
|
2010-05-29 00:45:14 +04:00
|
|
|
zfetch_init();
|
2012-01-20 22:58:57 +04:00
|
|
|
dmu_tx_init();
|
2008-11-20 23:01:55 +03:00
|
|
|
l2arc_init();
|
2012-12-15 04:13:40 +04:00
|
|
|
arc_init();
|
2016-06-02 07:04:53 +03:00
|
|
|
dbuf_init();
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
|
|
|
|
dmu_fini(void)
|
|
|
|
|
{
|
2013-06-11 21:12:34 +04:00
|
|
|
arc_fini(); /* arc depends on l2arc, so arc must go first */
|
2012-12-15 04:13:40 +04:00
|
|
|
l2arc_fini();
|
2012-01-20 22:58:57 +04:00
|
|
|
dmu_tx_fini();
|
2010-05-29 00:45:14 +04:00
|
|
|
zfetch_fini();
|
2008-11-20 23:01:55 +03:00
|
|
|
dbuf_fini();
|
2010-08-27 01:24:34 +04:00
|
|
|
dnode_fini();
|
|
|
|
|
dmu_objset_fini();
|
2010-05-29 00:45:14 +04:00
|
|
|
sa_cache_fini();
|
|
|
|
|
zfs_dbgmsg_fini();
|
2016-07-22 18:52:49 +03:00
|
|
|
abd_fini();
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
2010-08-26 22:49:16 +04:00
|
|
|
|
|
|
|
|
EXPORT_SYMBOL(dmu_bonus_hold);
|
2019-01-11 01:37:43 +03:00
|
|
|
EXPORT_SYMBOL(dmu_bonus_hold_by_dnode);
|
2012-02-18 00:09:21 +04:00
|
|
|
EXPORT_SYMBOL(dmu_buf_hold_array_by_bonus);
|
|
|
|
|
EXPORT_SYMBOL(dmu_buf_rele_array);
|
2013-08-01 20:39:46 +04:00
|
|
|
EXPORT_SYMBOL(dmu_prefetch);
|
2024-03-26 00:58:50 +03:00
|
|
|
EXPORT_SYMBOL(dmu_prefetch_by_dnode);
|
|
|
|
|
EXPORT_SYMBOL(dmu_prefetch_dnode);
|
2010-08-26 22:49:16 +04:00
|
|
|
EXPORT_SYMBOL(dmu_free_range);
|
2013-08-01 20:39:46 +04:00
|
|
|
EXPORT_SYMBOL(dmu_free_long_range);
|
2013-08-21 08:11:52 +04:00
|
|
|
EXPORT_SYMBOL(dmu_free_long_object);
|
2010-08-26 22:49:16 +04:00
|
|
|
EXPORT_SYMBOL(dmu_read);
|
2017-01-14 01:58:41 +03:00
|
|
|
EXPORT_SYMBOL(dmu_read_by_dnode);
|
2010-08-26 22:49:16 +04:00
|
|
|
EXPORT_SYMBOL(dmu_write);
|
2017-01-14 01:58:41 +03:00
|
|
|
EXPORT_SYMBOL(dmu_write_by_dnode);
|
2013-08-01 20:39:46 +04:00
|
|
|
EXPORT_SYMBOL(dmu_prealloc);
|
2010-08-26 22:49:16 +04:00
|
|
|
EXPORT_SYMBOL(dmu_object_info);
|
|
|
|
|
EXPORT_SYMBOL(dmu_object_info_from_dnode);
|
|
|
|
|
EXPORT_SYMBOL(dmu_object_info_from_db);
|
|
|
|
|
EXPORT_SYMBOL(dmu_object_size_from_db);
|
Implement large_dnode pool feature
Justification
-------------
This feature adds support for variable length dnodes. Our motivation is
to eliminate the overhead associated with using spill blocks. Spill
blocks are used to store system attribute data (i.e. file metadata) that
does not fit in the dnode's bonus buffer. By allowing a larger bonus
buffer area the use of a spill block can be avoided. Spill blocks
potentially incur an additional read I/O for every dnode in a dnode
block. As a worst case example, reading 32 dnodes from a 16k dnode block
and all of the spill blocks could issue 33 separate reads. Now suppose
those dnodes have size 1024 and therefore don't need spill blocks. Then
the worst case number of blocks read is reduced to from 33 to two--one
per dnode block. In practice spill blocks may tend to be co-located on
disk with the dnode blocks so the reduction in I/O would not be this
drastic. In a badly fragmented pool, however, the improvement could be
significant.
ZFS-on-Linux systems that make heavy use of extended attributes would
benefit from this feature. In particular, ZFS-on-Linux supports the
xattr=sa dataset property which allows file extended attribute data
to be stored in the dnode bonus buffer as an alternative to the
traditional directory-based format. Workloads such as SELinux and the
Lustre distributed filesystem often store enough xattr data to force
spill bocks when xattr=sa is in effect. Large dnodes may therefore
provide a performance benefit to such systems.
Other use cases that may benefit from this feature include files with
large ACLs and symbolic links with long target names. Furthermore,
this feature may be desirable on other platforms in case future
applications or features are developed that could make use of a
larger bonus buffer area.
Implementation
--------------
The size of a dnode may be a multiple of 512 bytes up to the size of
a dnode block (currently 16384 bytes). A dn_extra_slots field was
added to the current on-disk dnode_phys_t structure to describe the
size of the physical dnode on disk. The 8 bits for this field were
taken from the zero filled dn_pad2 field. The field represents how
many "extra" dnode_phys_t slots a dnode consumes in its dnode block.
This convention results in a value of 0 for 512 byte dnodes which
preserves on-disk format compatibility with older software.
Similarly, the in-memory dnode_t structure has a new dn_num_slots field
to represent the total number of dnode_phys_t slots consumed on disk.
Thus dn->dn_num_slots is 1 greater than the corresponding
dnp->dn_extra_slots. This difference in convention was adopted
because, unlike on-disk structures, backward compatibility is not a
concern for in-memory objects, so we used a more natural way to
represent size for a dnode_t.
The default size for newly created dnodes is determined by the value of
a new "dnodesize" dataset property. By default the property is set to
"legacy" which is compatible with older software. Setting the property
to "auto" will allow the filesystem to choose the most suitable dnode
size. Currently this just sets the default dnode size to 1k, but future
code improvements could dynamically choose a size based on observed
workload patterns. Dnodes of varying sizes can coexist within the same
dataset and even within the same dnode block. For example, to enable
automatically-sized dnodes, run
# zfs set dnodesize=auto tank/fish
The user can also specify literal values for the dnodesize property.
These are currently limited to powers of two from 1k to 16k. The
power-of-2 limitation is only for simplicity of the user interface.
Internally the implementation can handle any multiple of 512 up to 16k,
and consumers of the DMU API can specify any legal dnode value.
The size of a new dnode is determined at object allocation time and
stored as a new field in the znode in-memory structure. New DMU
interfaces are added to allow the consumer to specify the dnode size
that a newly allocated object should use. Existing interfaces are
unchanged to avoid having to update every call site and to preserve
compatibility with external consumers such as Lustre. The new
interfaces names are given below. The versions of these functions that
don't take a dnodesize parameter now just call the _dnsize() versions
with a dnodesize of 0, which means use the legacy dnode size.
New DMU interfaces:
dmu_object_alloc_dnsize()
dmu_object_claim_dnsize()
dmu_object_reclaim_dnsize()
New ZAP interfaces:
zap_create_dnsize()
zap_create_norm_dnsize()
zap_create_flags_dnsize()
zap_create_claim_norm_dnsize()
zap_create_link_dnsize()
The constant DN_MAX_BONUSLEN is renamed to DN_OLD_MAX_BONUSLEN. The
spa_maxdnodesize() function should be used to determine the maximum
bonus length for a pool.
These are a few noteworthy changes to key functions:
* The prototype for dnode_hold_impl() now takes a "slots" parameter.
When the DNODE_MUST_BE_FREE flag is set, this parameter is used to
ensure the hole at the specified object offset is large enough to
hold the dnode being created. The slots parameter is also used
to ensure a dnode does not span multiple dnode blocks. In both of
these cases, if a failure occurs, ENOSPC is returned. Keep in mind,
these failure cases are only possible when using DNODE_MUST_BE_FREE.
If the DNODE_MUST_BE_ALLOCATED flag is set, "slots" must be 0.
dnode_hold_impl() will check if the requested dnode is already
consumed as an extra dnode slot by an large dnode, in which case
it returns ENOENT.
* The function dmu_object_alloc() advances to the next dnode block
if dnode_hold_impl() returns an error for a requested object.
This is because the beginning of the next dnode block is the only
location it can safely assume to either be a hole or a valid
starting point for a dnode.
* dnode_next_offset_level() and other functions that iterate
through dnode blocks may no longer use a simple array indexing
scheme. These now use the current dnode's dn_num_slots field to
advance to the next dnode in the block. This is to ensure we
properly skip the current dnode's bonus area and don't interpret it
as a valid dnode.
zdb
---
The zdb command was updated to display a dnode's size under the
"dnsize" column when the object is dumped.
For ZIL create log records, zdb will now display the slot count for
the object.
ztest
-----
Ztest chooses a random dnodesize for every newly created object. The
random distribution is more heavily weighted toward small dnodes to
better simulate real-world datasets.
Unused bonus buffer space is filled with non-zero values computed from
the object number, dataset id, offset, and generation number. This
helps ensure that the dnode traversal code properly skips the interior
regions of large dnodes, and that these interior regions are not
overwritten by data belonging to other dnodes. A new test visits each
object in a dataset. It verifies that the actual dnode size matches what
was stored in the ztest block tag when it was created. It also verifies
that the unused bonus buffer space is filled with the expected data
patterns.
ZFS Test Suite
--------------
Added six new large dnode-specific tests, and integrated the dnodesize
property into existing tests for zfs allow and send/recv.
Send/Receive
------------
ZFS send streams for datasets containing large dnodes cannot be received
on pools that don't support the large_dnode feature. A send stream with
large dnodes sets a DMU_BACKUP_FEATURE_LARGE_DNODE flag which will be
unrecognized by an incompatible receiving pool so that the zfs receive
will fail gracefully.
While not implemented here, it may be possible to generate a
backward-compatible send stream from a dataset containing large
dnodes. The implementation may be tricky, however, because the send
object record for a large dnode would need to be resized to a 512
byte dnode, possibly kicking in a spill block in the process. This
means we would need to construct a new SA layout and possibly
register it in the SA layout object. The SA layout is normally just
sent as an ordinary object record. But if we are constructing new
layouts while generating the send stream we'd have to build the SA
layout object dynamically and send it at the end of the stream.
For sending and receiving between pools that do support large dnodes,
the drr_object send record type is extended with a new field to store
the dnode slot count. This field was repurposed from unused padding
in the structure.
ZIL Replay
----------
The dnode slot count is stored in the uppermost 8 bits of the lr_foid
field. The bits were unused as the object id is currently capped at
48 bits.
Resizing Dnodes
---------------
It should be possible to resize a dnode when it is dirtied if the
current dnodesize dataset property differs from the dnode's size, but
this functionality is not currently implemented. Clearly a dnode can
only grow if there are sufficient contiguous unused slots in the
dnode block, but it should always be possible to shrink a dnode.
Growing dnodes may be useful to reduce fragmentation in a pool with
many spill blocks in use. Shrinking dnodes may be useful to allow
sending a dataset to a pool that doesn't support the large_dnode
feature.
Feature Reference Counting
--------------------------
The reference count for the large_dnode pool feature tracks the
number of datasets that have ever contained a dnode of size larger
than 512 bytes. The first time a large dnode is created in a dataset
the dataset is converted to an extensible dataset. This is a one-way
operation and the only way to decrement the feature count is to
destroy the dataset, even if the dataset no longer contains any large
dnodes. The complexity of reference counting on a per-dnode basis was
too high, so we chose to track it on a per-dataset basis similarly to
the large_block feature.
Signed-off-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #3542
2016-03-17 04:25:34 +03:00
|
|
|
EXPORT_SYMBOL(dmu_object_dnsize_from_db);
|
Native Encryption for ZFS on Linux
This change incorporates three major pieces:
The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
convenient.
The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.
The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
compromised.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #494
Closes #5769
2017-08-14 20:36:48 +03:00
|
|
|
EXPORT_SYMBOL(dmu_object_set_nlevels);
|
2010-08-26 22:49:16 +04:00
|
|
|
EXPORT_SYMBOL(dmu_object_set_blocksize);
|
2017-11-08 22:12:59 +03:00
|
|
|
EXPORT_SYMBOL(dmu_object_set_maxblkid);
|
2010-08-26 22:49:16 +04:00
|
|
|
EXPORT_SYMBOL(dmu_object_set_checksum);
|
|
|
|
|
EXPORT_SYMBOL(dmu_object_set_compress);
|
2020-08-25 18:34:41 +03:00
|
|
|
EXPORT_SYMBOL(dmu_offset_next);
|
2013-08-01 20:39:46 +04:00
|
|
|
EXPORT_SYMBOL(dmu_write_policy);
|
|
|
|
|
EXPORT_SYMBOL(dmu_sync);
|
2012-02-10 23:53:09 +04:00
|
|
|
EXPORT_SYMBOL(dmu_request_arcbuf);
|
|
|
|
|
EXPORT_SYMBOL(dmu_return_arcbuf);
|
2017-09-28 18:49:13 +03:00
|
|
|
EXPORT_SYMBOL(dmu_assign_arcbuf_by_dnode);
|
|
|
|
|
EXPORT_SYMBOL(dmu_assign_arcbuf_by_dbuf);
|
2012-02-10 23:53:09 +04:00
|
|
|
EXPORT_SYMBOL(dmu_buf_hold);
|
2010-08-26 22:49:16 +04:00
|
|
|
EXPORT_SYMBOL(dmu_ot);
|
2012-04-28 03:20:31 +04:00
|
|
|
|
2019-09-06 00:49:49 +03:00
|
|
|
ZFS_MODULE_PARAM(zfs, zfs_, nopwrite_enabled, INT, ZMOD_RW,
|
|
|
|
|
"Enable NOP writes");
|
2013-05-10 23:47:54 +04:00
|
|
|
|
Cleanup: 64-bit kernel module parameters should use fixed width types
Various module parameters such as `zfs_arc_max` were originally
`uint64_t` on OpenSolaris/Illumos, but were changed to `unsigned long`
for Linux compatibility because Linux's kernel default module parameter
implementation did not support 64-bit types on 32-bit platforms. This
caused problems when porting OpenZFS to Windows because its LLP64 memory
model made `unsigned long` a 32-bit type on 64-bit, which created the
undesireable situation that parameters that should accept 64-bit values
could not on 64-bit Windows.
Upon inspection, it turns out that the Linux kernel module parameter
interface is extensible, such that we are allowed to define our own
types. Rather than maintaining the original type change via hacks to to
continue shrinking module parameters on 32-bit Linux, we implement
support for 64-bit module parameters on Linux.
After doing a review of all 64-bit kernel parameters (found via the man
page and also proposed changes by Andrew Innes), the kernel module
parameters fell into a few groups:
Parameters that were originally 64-bit on Illumos:
* dbuf_cache_max_bytes
* dbuf_metadata_cache_max_bytes
* l2arc_feed_min_ms
* l2arc_feed_secs
* l2arc_headroom
* l2arc_headroom_boost
* l2arc_write_boost
* l2arc_write_max
* metaslab_aliquot
* metaslab_force_ganging
* zfetch_array_rd_sz
* zfs_arc_max
* zfs_arc_meta_limit
* zfs_arc_meta_min
* zfs_arc_min
* zfs_async_block_max_blocks
* zfs_condense_max_obsolete_bytes
* zfs_condense_min_mapping_bytes
* zfs_deadman_checktime_ms
* zfs_deadman_synctime_ms
* zfs_initialize_chunk_size
* zfs_initialize_value
* zfs_lua_max_instrlimit
* zfs_lua_max_memlimit
* zil_slog_bulk
Parameters that were originally 32-bit on Illumos:
* zfs_per_txg_dirty_frees_percent
Parameters that were originally `ssize_t` on Illumos:
* zfs_immediate_write_sz
Note that `ssize_t` is `int32_t` on 32-bit and `int64_t` on 64-bit. It
has been upgraded to 64-bit.
Parameters that were `long`/`unsigned long` because of Linux/FreeBSD
influence:
* l2arc_rebuild_blocks_min_l2size
* zfs_key_max_salt_uses
* zfs_max_log_walking
* zfs_max_logsm_summary_length
* zfs_metaslab_max_size_cache_sec
* zfs_min_metaslabs_to_flush
* zfs_multihost_interval
* zfs_unflushed_log_block_max
* zfs_unflushed_log_block_min
* zfs_unflushed_log_block_pct
* zfs_unflushed_max_mem_amt
* zfs_unflushed_max_mem_ppm
New parameters that do not exist in Illumos:
* l2arc_trim_ahead
* vdev_file_logical_ashift
* vdev_file_physical_ashift
* zfs_arc_dnode_limit
* zfs_arc_dnode_limit_percent
* zfs_arc_dnode_reduce_percent
* zfs_arc_meta_limit_percent
* zfs_arc_sys_free
* zfs_deadman_ziotime_ms
* zfs_delete_blocks
* zfs_history_output_max
* zfs_livelist_max_entries
* zfs_max_async_dedup_frees
* zfs_max_nvlist_src_size
* zfs_rebuild_max_segment
* zfs_rebuild_vdev_limit
* zfs_unflushed_log_txg_max
* zfs_vdev_max_auto_ashift
* zfs_vdev_min_auto_ashift
* zfs_vnops_read_chunk_size
* zvol_max_discard_blocks
Rather than clutter the lists with commentary, the module parameters
that need comments are repeated below.
A few parameters were defined in Linux/FreeBSD specific code, where the
use of ulong/long is not an issue for portability, so we leave them
alone:
* zfs_delete_blocks
* zfs_key_max_salt_uses
* zvol_max_discard_blocks
The documentation for a few parameters was found to be incorrect:
* zfs_deadman_checktime_ms - incorrectly documented as int
* zfs_delete_blocks - not documented as Linux only
* zfs_history_output_max - incorrectly documented as int
* zfs_vnops_read_chunk_size - incorrectly documented as long
* zvol_max_discard_blocks - incorrectly documented as ulong
The documentation for these has been fixed, alongside the changes to
document the switch to fixed width types.
In addition, several kernel module parameters were percentages or held
ashift values, so being 64-bit never made sense for them. They have been
downgraded to 32-bit:
* vdev_file_logical_ashift
* vdev_file_physical_ashift
* zfs_arc_dnode_limit_percent
* zfs_arc_dnode_reduce_percent
* zfs_arc_meta_limit_percent
* zfs_per_txg_dirty_frees_percent
* zfs_unflushed_log_block_pct
* zfs_vdev_max_auto_ashift
* zfs_vdev_min_auto_ashift
Of special note are `zfs_vdev_max_auto_ashift` and
`zfs_vdev_min_auto_ashift`, which were already defined as `uint64_t`,
and passed to the kernel as `ulong`. This is inherently buggy on big
endian 32-bit Linux, since the values would not be written to the
correct locations. 32-bit FreeBSD was unaffected because its sysctl code
correctly treated this as a `uint64_t`.
Lastly, a code comment suggests that `zfs_arc_sys_free` is
Linux-specific, but there is nothing to indicate to me that it is
Linux-specific. Nothing was done about that.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Reviewed-by: Alexander Motin <mav@FreeBSD.org>
Original-patch-by: Andrew Innes <andrew.c12@gmail.com>
Original-patch-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Closes #13984
Closes #14004
2022-10-03 22:06:54 +03:00
|
|
|
ZFS_MODULE_PARAM(zfs, zfs_, per_txg_dirty_frees_percent, UINT, ZMOD_RW,
|
2019-09-06 00:49:49 +03:00
|
|
|
"Percentage of dirtied blocks from frees in one TXG");
|
2017-03-25 00:28:38 +03:00
|
|
|
|
2019-09-06 00:49:49 +03:00
|
|
|
ZFS_MODULE_PARAM(zfs, zfs_, dmu_offset_next_sync, INT, ZMOD_RW,
|
2017-03-25 00:28:38 +03:00
|
|
|
"Enable forcing txg sync to find holes");
|
|
|
|
|
|
2022-01-21 19:07:15 +03:00
|
|
|
/* CSTYLED */
|
Cleanup: Specify unsignedness on things that should not be signed
In #13871, zfs_vdev_aggregation_limit_non_rotating and
zfs_vdev_aggregation_limit being signed was pointed out as a possible
reason not to eliminate an unnecessary MAX(unsigned, 0) since the
unsigned value was assigned from them.
There is no reason for these module parameters to be signed and upon
inspection, it was found that there are a number of other module
parameters that are signed, but should not be, so we make them unsigned.
Making them unsigned made it clear that some other variables in the code
should also be unsigned, so we also make those unsigned. This prevents
users from setting negative values that could potentially cause bad
behaviors. It also makes the code slightly easier to understand.
Mostly module parameters that deal with timeouts, limits, bitshifts and
percentages are made unsigned by this. Any that are boolean are left
signed, since whether booleans should be considered signed or unsigned
does not matter.
Making zfs_arc_lotsfree_percent unsigned caused a
`zfs_arc_lotsfree_percent >= 0` check to become redundant, so it was
removed. Removing the check was also necessary to prevent a compiler
error from -Werror=type-limits.
Several end of line comments had to be moved to their own lines because
replacing int with uint_t caused us to exceed the 80 character limit
enforced by cstyle.pl.
The following were kept signed because they are passed to
taskq_create(), which expects signed values and modifying the
OpenSolaris/Illumos DDI is out of scope of this patch:
* metaslab_load_pct
* zfs_sync_taskq_batch_pct
* zfs_zil_clean_taskq_nthr_pct
* zfs_zil_clean_taskq_minalloc
* zfs_zil_clean_taskq_maxalloc
* zfs_arc_prune_task_threads
Also, negative values in those parameters was found to be harmless.
The following were left signed because either negative values make
sense, or more analysis was needed to determine whether negative values
should be disallowed:
* zfs_metaslab_switch_threshold
* zfs_pd_bytes_max
* zfs_livelist_min_percent_shared
zfs_multihost_history was made static to be consistent with other
parameters.
A number of module parameters were marked as signed, but in reality
referenced unsigned variables. upgrade_errlog_limit is one of the
numerous examples. In the case of zfs_vdev_async_read_max_active, it was
already uint32_t, but zdb had an extern int declaration for it.
Interestingly, the documentation in zfs.4 was right for
upgrade_errlog_limit despite the module parameter being wrongly marked,
while the documentation for zfs_vdev_async_read_max_active (and friends)
was wrong. It was also wrong for zstd_abort_size, which was unsigned,
but was documented as signed.
Also, the documentation in zfs.4 incorrectly described the following
parameters as ulong when they were int:
* zfs_arc_meta_adjust_restarts
* zfs_override_estimate_recordsize
They are now uint_t as of this patch and thus the man page has been
updated to describe them as uint.
dbuf_state_index was left alone since it does nothing and perhaps should
be removed in another patch.
If any module parameters were missed, they were not found by `grep -r
'ZFS_MODULE_PARAM' | grep ', INT'`. I did find a few that grep missed,
but only because they were in files that had hits.
This patch intentionally did not attempt to address whether some of
these module parameters should be elevated to 64-bit parameters, because
the length of a long on 32-bit is 32-bit.
Lastly, it was pointed out during review that uint_t is a better match
for these variables than uint32_t because FreeBSD kernel parameter
definitions are designed for uint_t, whose bit width can change in
future memory models. As a result, we change the existing parameters
that are uint32_t to use uint_t.
Reviewed-by: Alexander Motin <mav@FreeBSD.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Neal Gompa <ngompa@datto.com>
Signed-off-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Closes #13875
2022-09-28 02:42:41 +03:00
|
|
|
ZFS_MODULE_PARAM(zfs, , dmu_prefetch_max, UINT, ZMOD_RW,
|
2019-06-12 23:13:09 +03:00
|
|
|
"Limit one prefetch call to this size");
|