2008-11-20 23:01:55 +03:00
|
|
|
/*
|
|
|
|
|
* CDDL HEADER START
|
|
|
|
|
*
|
|
|
|
|
* The contents of this file are subject to the terms of the
|
|
|
|
|
* Common Development and Distribution License (the "License").
|
|
|
|
|
* You may not use this file except in compliance with the License.
|
|
|
|
|
*
|
|
|
|
|
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
|
2022-07-12 00:16:13 +03:00
|
|
|
* or https://opensource.org/licenses/CDDL-1.0.
|
2008-11-20 23:01:55 +03:00
|
|
|
* See the License for the specific language governing permissions
|
|
|
|
|
* and limitations under the License.
|
|
|
|
|
*
|
|
|
|
|
* When distributing Covered Code, include this CDDL HEADER in each
|
|
|
|
|
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
|
|
|
|
|
* If applicable, add the following below this CDDL HEADER, with the
|
|
|
|
|
* fields enclosed by brackets "[]" replaced with your own identifying
|
|
|
|
|
* information: Portions Copyright [yyyy] [name of copyright owner]
|
|
|
|
|
*
|
|
|
|
|
* CDDL HEADER END
|
|
|
|
|
*/
|
2012-12-14 03:24:15 +04:00
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
/*
|
2010-05-29 00:45:14 +04:00
|
|
|
* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
|
2020-05-07 19:36:33 +03:00
|
|
|
* Copyright (c) 2012, 2020 by Delphix. All rights reserved.
|
2018-09-06 04:33:36 +03:00
|
|
|
* Copyright (c) 2017, Intel Corporation.
|
2008-11-20 23:01:55 +03:00
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Virtual Device Labels
|
|
|
|
|
* ---------------------
|
|
|
|
|
*
|
|
|
|
|
* The vdev label serves several distinct purposes:
|
|
|
|
|
*
|
|
|
|
|
* 1. Uniquely identify this device as part of a ZFS pool and confirm its
|
|
|
|
|
* identity within the pool.
|
|
|
|
|
*
|
OpenZFS 9486 - reduce memory used by device removal on fragmented pools
Device removal allocates a new location for each allocated segment on
the disk that's being removed. Each allocation results in one entry in
the mapping table, which maps from old location + length to new
location. When a fragmented disk is removed, this can result in a large
number of mapping entries, and thus a large amount of memory consumed by
the mapping table. In the worst real-world cases, we've seen around 1GB
of RAM per 1TB of storage removed.
We can improve on this situation by allocating larger segments, which
span across both allocated and free regions of the device being removed.
By including free regions in the allocation (and thus mapping), we
reduce the number of mapping entries. For example, if we have a 4K
allocation followed by 1K free and then 4K allocated, we would allocate
4+1+4 = 9KB, and then move the entire region (including allocated and
free parts). In this case we used one mapping where previously we would
have used two, but often the ratio is much higher (up to 20:1 in
real-world use). We then need to mark the regions that were free on the
removing device as free in the new locations, and also obsolete in the
mapping entry.
This method preserves the fragmentation of the removing device, rather
than consolidating its allocated space into a small number of chunks
where possible. But it results in drastic reduction of memory used by
the mapping table - around 20x in the most-fragmented cases.
In the most fragmented real-world cases, this reduces memory used by the
mapping from ~1GB to ~50MB of RAM per 1TB of storage removed. Less
fragmented cases will typically also see around 50-100MB of RAM per 1TB
of storage.
Porting notes:
* Add the following as module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* Document the following module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* zfs_condense_min_mapping_bytes
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://illumos.org/issues/9486
OpenZFS-commit: https://github.com/ahrens/illumos/commit/07152e142e44c
External-issue: DLPX-57962
Closes #7536
2018-02-27 02:33:55 +03:00
|
|
|
* 2. Verify that all the devices given in a configuration are present
|
2008-11-20 23:01:55 +03:00
|
|
|
* within the pool.
|
|
|
|
|
*
|
OpenZFS 9486 - reduce memory used by device removal on fragmented pools
Device removal allocates a new location for each allocated segment on
the disk that's being removed. Each allocation results in one entry in
the mapping table, which maps from old location + length to new
location. When a fragmented disk is removed, this can result in a large
number of mapping entries, and thus a large amount of memory consumed by
the mapping table. In the worst real-world cases, we've seen around 1GB
of RAM per 1TB of storage removed.
We can improve on this situation by allocating larger segments, which
span across both allocated and free regions of the device being removed.
By including free regions in the allocation (and thus mapping), we
reduce the number of mapping entries. For example, if we have a 4K
allocation followed by 1K free and then 4K allocated, we would allocate
4+1+4 = 9KB, and then move the entire region (including allocated and
free parts). In this case we used one mapping where previously we would
have used two, but often the ratio is much higher (up to 20:1 in
real-world use). We then need to mark the regions that were free on the
removing device as free in the new locations, and also obsolete in the
mapping entry.
This method preserves the fragmentation of the removing device, rather
than consolidating its allocated space into a small number of chunks
where possible. But it results in drastic reduction of memory used by
the mapping table - around 20x in the most-fragmented cases.
In the most fragmented real-world cases, this reduces memory used by the
mapping from ~1GB to ~50MB of RAM per 1TB of storage removed. Less
fragmented cases will typically also see around 50-100MB of RAM per 1TB
of storage.
Porting notes:
* Add the following as module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* Document the following module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* zfs_condense_min_mapping_bytes
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://illumos.org/issues/9486
OpenZFS-commit: https://github.com/ahrens/illumos/commit/07152e142e44c
External-issue: DLPX-57962
Closes #7536
2018-02-27 02:33:55 +03:00
|
|
|
* 3. Determine the uberblock for the pool.
|
2008-11-20 23:01:55 +03:00
|
|
|
*
|
OpenZFS 9486 - reduce memory used by device removal on fragmented pools
Device removal allocates a new location for each allocated segment on
the disk that's being removed. Each allocation results in one entry in
the mapping table, which maps from old location + length to new
location. When a fragmented disk is removed, this can result in a large
number of mapping entries, and thus a large amount of memory consumed by
the mapping table. In the worst real-world cases, we've seen around 1GB
of RAM per 1TB of storage removed.
We can improve on this situation by allocating larger segments, which
span across both allocated and free regions of the device being removed.
By including free regions in the allocation (and thus mapping), we
reduce the number of mapping entries. For example, if we have a 4K
allocation followed by 1K free and then 4K allocated, we would allocate
4+1+4 = 9KB, and then move the entire region (including allocated and
free parts). In this case we used one mapping where previously we would
have used two, but often the ratio is much higher (up to 20:1 in
real-world use). We then need to mark the regions that were free on the
removing device as free in the new locations, and also obsolete in the
mapping entry.
This method preserves the fragmentation of the removing device, rather
than consolidating its allocated space into a small number of chunks
where possible. But it results in drastic reduction of memory used by
the mapping table - around 20x in the most-fragmented cases.
In the most fragmented real-world cases, this reduces memory used by the
mapping from ~1GB to ~50MB of RAM per 1TB of storage removed. Less
fragmented cases will typically also see around 50-100MB of RAM per 1TB
of storage.
Porting notes:
* Add the following as module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* Document the following module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* zfs_condense_min_mapping_bytes
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://illumos.org/issues/9486
OpenZFS-commit: https://github.com/ahrens/illumos/commit/07152e142e44c
External-issue: DLPX-57962
Closes #7536
2018-02-27 02:33:55 +03:00
|
|
|
* 4. In case of an import operation, determine the configuration of the
|
2008-11-20 23:01:55 +03:00
|
|
|
* toplevel vdev of which it is a part.
|
|
|
|
|
*
|
OpenZFS 9486 - reduce memory used by device removal on fragmented pools
Device removal allocates a new location for each allocated segment on
the disk that's being removed. Each allocation results in one entry in
the mapping table, which maps from old location + length to new
location. When a fragmented disk is removed, this can result in a large
number of mapping entries, and thus a large amount of memory consumed by
the mapping table. In the worst real-world cases, we've seen around 1GB
of RAM per 1TB of storage removed.
We can improve on this situation by allocating larger segments, which
span across both allocated and free regions of the device being removed.
By including free regions in the allocation (and thus mapping), we
reduce the number of mapping entries. For example, if we have a 4K
allocation followed by 1K free and then 4K allocated, we would allocate
4+1+4 = 9KB, and then move the entire region (including allocated and
free parts). In this case we used one mapping where previously we would
have used two, but often the ratio is much higher (up to 20:1 in
real-world use). We then need to mark the regions that were free on the
removing device as free in the new locations, and also obsolete in the
mapping entry.
This method preserves the fragmentation of the removing device, rather
than consolidating its allocated space into a small number of chunks
where possible. But it results in drastic reduction of memory used by
the mapping table - around 20x in the most-fragmented cases.
In the most fragmented real-world cases, this reduces memory used by the
mapping from ~1GB to ~50MB of RAM per 1TB of storage removed. Less
fragmented cases will typically also see around 50-100MB of RAM per 1TB
of storage.
Porting notes:
* Add the following as module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* Document the following module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* zfs_condense_min_mapping_bytes
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://illumos.org/issues/9486
OpenZFS-commit: https://github.com/ahrens/illumos/commit/07152e142e44c
External-issue: DLPX-57962
Closes #7536
2018-02-27 02:33:55 +03:00
|
|
|
* 5. If an import operation cannot find all the devices in the pool,
|
2008-11-20 23:01:55 +03:00
|
|
|
* provide enough information to the administrator to determine which
|
|
|
|
|
* devices are missing.
|
|
|
|
|
*
|
|
|
|
|
* It is important to note that while the kernel is responsible for writing the
|
|
|
|
|
* label, it only consumes the information in the first three cases. The
|
|
|
|
|
* latter information is only consumed in userland when determining the
|
|
|
|
|
* configuration to import a pool.
|
|
|
|
|
*
|
|
|
|
|
*
|
|
|
|
|
* Label Organization
|
|
|
|
|
* ------------------
|
|
|
|
|
*
|
|
|
|
|
* Before describing the contents of the label, it's important to understand how
|
|
|
|
|
* the labels are written and updated with respect to the uberblock.
|
|
|
|
|
*
|
|
|
|
|
* When the pool configuration is altered, either because it was newly created
|
|
|
|
|
* or a device was added, we want to update all the labels such that we can deal
|
|
|
|
|
* with fatal failure at any point. To this end, each disk has two labels which
|
|
|
|
|
* are updated before and after the uberblock is synced. Assuming we have
|
|
|
|
|
* labels and an uberblock with the following transaction groups:
|
|
|
|
|
*
|
|
|
|
|
* L1 UB L2
|
|
|
|
|
* +------+ +------+ +------+
|
|
|
|
|
* | | | | | |
|
|
|
|
|
* | t10 | | t10 | | t10 |
|
|
|
|
|
* | | | | | |
|
|
|
|
|
* +------+ +------+ +------+
|
|
|
|
|
*
|
|
|
|
|
* In this stable state, the labels and the uberblock were all updated within
|
|
|
|
|
* the same transaction group (10). Each label is mirrored and checksummed, so
|
|
|
|
|
* that we can detect when we fail partway through writing the label.
|
|
|
|
|
*
|
|
|
|
|
* In order to identify which labels are valid, the labels are written in the
|
|
|
|
|
* following manner:
|
|
|
|
|
*
|
OpenZFS 9486 - reduce memory used by device removal on fragmented pools
Device removal allocates a new location for each allocated segment on
the disk that's being removed. Each allocation results in one entry in
the mapping table, which maps from old location + length to new
location. When a fragmented disk is removed, this can result in a large
number of mapping entries, and thus a large amount of memory consumed by
the mapping table. In the worst real-world cases, we've seen around 1GB
of RAM per 1TB of storage removed.
We can improve on this situation by allocating larger segments, which
span across both allocated and free regions of the device being removed.
By including free regions in the allocation (and thus mapping), we
reduce the number of mapping entries. For example, if we have a 4K
allocation followed by 1K free and then 4K allocated, we would allocate
4+1+4 = 9KB, and then move the entire region (including allocated and
free parts). In this case we used one mapping where previously we would
have used two, but often the ratio is much higher (up to 20:1 in
real-world use). We then need to mark the regions that were free on the
removing device as free in the new locations, and also obsolete in the
mapping entry.
This method preserves the fragmentation of the removing device, rather
than consolidating its allocated space into a small number of chunks
where possible. But it results in drastic reduction of memory used by
the mapping table - around 20x in the most-fragmented cases.
In the most fragmented real-world cases, this reduces memory used by the
mapping from ~1GB to ~50MB of RAM per 1TB of storage removed. Less
fragmented cases will typically also see around 50-100MB of RAM per 1TB
of storage.
Porting notes:
* Add the following as module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* Document the following module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* zfs_condense_min_mapping_bytes
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://illumos.org/issues/9486
OpenZFS-commit: https://github.com/ahrens/illumos/commit/07152e142e44c
External-issue: DLPX-57962
Closes #7536
2018-02-27 02:33:55 +03:00
|
|
|
* 1. For each vdev, update 'L1' to the new label
|
|
|
|
|
* 2. Update the uberblock
|
|
|
|
|
* 3. For each vdev, update 'L2' to the new label
|
2008-11-20 23:01:55 +03:00
|
|
|
*
|
|
|
|
|
* Given arbitrary failure, we can determine the correct label to use based on
|
|
|
|
|
* the transaction group. If we fail after updating L1 but before updating the
|
|
|
|
|
* UB, we will notice that L1's transaction group is greater than the uberblock,
|
|
|
|
|
* so L2 must be valid. If we fail after writing the uberblock but before
|
|
|
|
|
* writing L2, we will notice that L2's transaction group is less than L1, and
|
|
|
|
|
* therefore L1 is valid.
|
|
|
|
|
*
|
|
|
|
|
* Another added complexity is that not every label is updated when the config
|
|
|
|
|
* is synced. If we add a single device, we do not want to have to re-write
|
|
|
|
|
* every label for every device in the pool. This means that both L1 and L2 may
|
|
|
|
|
* be older than the pool uberblock, because the necessary information is stored
|
|
|
|
|
* on another vdev.
|
|
|
|
|
*
|
|
|
|
|
*
|
|
|
|
|
* On-disk Format
|
|
|
|
|
* --------------
|
|
|
|
|
*
|
|
|
|
|
* The vdev label consists of two distinct parts, and is wrapped within the
|
|
|
|
|
* vdev_label_t structure. The label includes 8k of padding to permit legacy
|
|
|
|
|
* VTOC disk labels, but is otherwise ignored.
|
|
|
|
|
*
|
|
|
|
|
* The first half of the label is a packed nvlist which contains pool wide
|
|
|
|
|
* properties, per-vdev properties, and configuration information. It is
|
|
|
|
|
* described in more detail below.
|
|
|
|
|
*
|
|
|
|
|
* The latter half of the label consists of a redundant array of uberblocks.
|
|
|
|
|
* These uberblocks are updated whenever a transaction group is committed,
|
|
|
|
|
* or when the configuration is updated. When a pool is loaded, we scan each
|
|
|
|
|
* vdev for the 'best' uberblock.
|
|
|
|
|
*
|
|
|
|
|
*
|
|
|
|
|
* Configuration Information
|
|
|
|
|
* -------------------------
|
|
|
|
|
*
|
|
|
|
|
* The nvlist describing the pool and vdev contains the following elements:
|
|
|
|
|
*
|
OpenZFS 9486 - reduce memory used by device removal on fragmented pools
Device removal allocates a new location for each allocated segment on
the disk that's being removed. Each allocation results in one entry in
the mapping table, which maps from old location + length to new
location. When a fragmented disk is removed, this can result in a large
number of mapping entries, and thus a large amount of memory consumed by
the mapping table. In the worst real-world cases, we've seen around 1GB
of RAM per 1TB of storage removed.
We can improve on this situation by allocating larger segments, which
span across both allocated and free regions of the device being removed.
By including free regions in the allocation (and thus mapping), we
reduce the number of mapping entries. For example, if we have a 4K
allocation followed by 1K free and then 4K allocated, we would allocate
4+1+4 = 9KB, and then move the entire region (including allocated and
free parts). In this case we used one mapping where previously we would
have used two, but often the ratio is much higher (up to 20:1 in
real-world use). We then need to mark the regions that were free on the
removing device as free in the new locations, and also obsolete in the
mapping entry.
This method preserves the fragmentation of the removing device, rather
than consolidating its allocated space into a small number of chunks
where possible. But it results in drastic reduction of memory used by
the mapping table - around 20x in the most-fragmented cases.
In the most fragmented real-world cases, this reduces memory used by the
mapping from ~1GB to ~50MB of RAM per 1TB of storage removed. Less
fragmented cases will typically also see around 50-100MB of RAM per 1TB
of storage.
Porting notes:
* Add the following as module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* Document the following module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* zfs_condense_min_mapping_bytes
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://illumos.org/issues/9486
OpenZFS-commit: https://github.com/ahrens/illumos/commit/07152e142e44c
External-issue: DLPX-57962
Closes #7536
2018-02-27 02:33:55 +03:00
|
|
|
* version ZFS on-disk version
|
|
|
|
|
* name Pool name
|
|
|
|
|
* state Pool state
|
|
|
|
|
* txg Transaction group in which this label was written
|
|
|
|
|
* pool_guid Unique identifier for this pool
|
|
|
|
|
* vdev_tree An nvlist describing vdev tree.
|
2012-12-14 03:24:15 +04:00
|
|
|
* features_for_read
|
|
|
|
|
* An nvlist of the features necessary for reading the MOS.
|
2008-11-20 23:01:55 +03:00
|
|
|
*
|
|
|
|
|
* Each leaf device label also contains the following:
|
|
|
|
|
*
|
OpenZFS 9486 - reduce memory used by device removal on fragmented pools
Device removal allocates a new location for each allocated segment on
the disk that's being removed. Each allocation results in one entry in
the mapping table, which maps from old location + length to new
location. When a fragmented disk is removed, this can result in a large
number of mapping entries, and thus a large amount of memory consumed by
the mapping table. In the worst real-world cases, we've seen around 1GB
of RAM per 1TB of storage removed.
We can improve on this situation by allocating larger segments, which
span across both allocated and free regions of the device being removed.
By including free regions in the allocation (and thus mapping), we
reduce the number of mapping entries. For example, if we have a 4K
allocation followed by 1K free and then 4K allocated, we would allocate
4+1+4 = 9KB, and then move the entire region (including allocated and
free parts). In this case we used one mapping where previously we would
have used two, but often the ratio is much higher (up to 20:1 in
real-world use). We then need to mark the regions that were free on the
removing device as free in the new locations, and also obsolete in the
mapping entry.
This method preserves the fragmentation of the removing device, rather
than consolidating its allocated space into a small number of chunks
where possible. But it results in drastic reduction of memory used by
the mapping table - around 20x in the most-fragmented cases.
In the most fragmented real-world cases, this reduces memory used by the
mapping from ~1GB to ~50MB of RAM per 1TB of storage removed. Less
fragmented cases will typically also see around 50-100MB of RAM per 1TB
of storage.
Porting notes:
* Add the following as module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* Document the following module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* zfs_condense_min_mapping_bytes
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://illumos.org/issues/9486
OpenZFS-commit: https://github.com/ahrens/illumos/commit/07152e142e44c
External-issue: DLPX-57962
Closes #7536
2018-02-27 02:33:55 +03:00
|
|
|
* top_guid Unique ID for top-level vdev in which this is contained
|
|
|
|
|
* guid Unique ID for the leaf vdev
|
2008-11-20 23:01:55 +03:00
|
|
|
*
|
|
|
|
|
* The 'vs' configuration follows the format described in 'spa_config.c'.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#include <sys/zfs_context.h>
|
|
|
|
|
#include <sys/spa.h>
|
|
|
|
|
#include <sys/spa_impl.h>
|
|
|
|
|
#include <sys/dmu.h>
|
|
|
|
|
#include <sys/zap.h>
|
|
|
|
|
#include <sys/vdev.h>
|
|
|
|
|
#include <sys/vdev_impl.h>
|
RAID-Z expansion feature
This feature allows disks to be added one at a time to a RAID-Z group,
expanding its capacity incrementally. This feature is especially useful
for small pools (typically with only one RAID-Z group), where there
isn't sufficient hardware to add capacity by adding a whole new RAID-Z
group (typically doubling the number of disks).
== Initiating expansion ==
A new device (disk) can be attached to an existing RAIDZ vdev, by
running `zpool attach POOL raidzP-N NEW_DEVICE`, e.g. `zpool attach tank
raidz2-0 sda`. The new device will become part of the RAIDZ group. A
"raidz expansion" will be initiated, and the new device will contribute
additional space to the RAIDZ group once the expansion completes.
The `feature@raidz_expansion` on-disk feature flag must be `enabled` to
initiate an expansion, and it remains `active` for the life of the pool.
In other words, pools with expanded RAIDZ vdevs can not be imported by
older releases of the ZFS software.
== During expansion ==
The expansion entails reading all allocated space from existing disks in
the RAIDZ group, and rewriting it to the new disks in the RAIDZ group
(including the newly added device).
The expansion progress can be monitored with `zpool status`.
Data redundancy is maintained during (and after) the expansion. If a
disk fails while the expansion is in progress, the expansion pauses
until the health of the RAIDZ vdev is restored (e.g. by replacing the
failed disk and waiting for reconstruction to complete).
The pool remains accessible during expansion. Following a reboot or
export/import, the expansion resumes where it left off.
== After expansion ==
When the expansion completes, the additional space is available for use,
and is reflected in the `available` zfs property (as seen in `zfs list`,
`df`, etc).
Expansion does not change the number of failures that can be tolerated
without data loss (e.g. a RAIDZ2 is still a RAIDZ2 even after
expansion).
A RAIDZ vdev can be expanded multiple times.
After the expansion completes, old blocks remain with their old
data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but
distributed among the larger set of disks. New blocks will be written
with the new data-to-parity ratio (e.g. a 5-wide RAIDZ2 which has been
expanded once to 6-wide, has 4 data to 2 parity). However, the RAIDZ
vdev's "assumed parity ratio" does not change, so slightly less space
than is expected may be reported for newly-written blocks, according to
`zfs list`, `df`, `ls -s`, and similar tools.
Sponsored-by: The FreeBSD Foundation
Sponsored-by: iXsystems, Inc.
Sponsored-by: vStack
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Mark Maybee <mark.maybee@delphix.com>
Authored-by: Matthew Ahrens <mahrens@delphix.com>
Contributions-by: Fedor Uporov <fuporov.vstack@gmail.com>
Contributions-by: Stuart Maybee <stuart.maybee@comcast.net>
Contributions-by: Thorsten Behrens <tbehrens@outlook.com>
Contributions-by: Fmstrat <nospam@nowsci.com>
Contributions-by: Don Brady <dev.fs.zfs@gmail.com>
Signed-off-by: Don Brady <dev.fs.zfs@gmail.com>
Closes #15022
2023-11-08 21:19:41 +03:00
|
|
|
#include <sys/vdev_raidz.h>
|
Distributed Spare (dRAID) Feature
This patch adds a new top-level vdev type called dRAID, which stands
for Distributed parity RAID. This pool configuration allows all dRAID
vdevs to participate when rebuilding to a distributed hot spare device.
This can substantially reduce the total time required to restore full
parity to pool with a failed device.
A dRAID pool can be created using the new top-level `draid` type.
Like `raidz`, the desired redundancy is specified after the type:
`draid[1,2,3]`. No additional information is required to create the
pool and reasonable default values will be chosen based on the number
of child vdevs in the dRAID vdev.
zpool create <pool> draid[1,2,3] <vdevs...>
Unlike raidz, additional optional dRAID configuration values can be
provided as part of the draid type as colon separated values. This
allows administrators to fully specify a layout for either performance
or capacity reasons. The supported options include:
zpool create <pool> \
draid[<parity>][:<data>d][:<children>c][:<spares>s] \
<vdevs...>
- draid[parity] - Parity level (default 1)
- draid[:<data>d] - Data devices per group (default 8)
- draid[:<children>c] - Expected number of child vdevs
- draid[:<spares>s] - Distributed hot spares (default 0)
Abbreviated example `zpool status` output for a 68 disk dRAID pool
with two distributed spares using special allocation classes.
```
pool: tank
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
slag7 ONLINE 0 0 0
draid2:8d:68c:2s-0 ONLINE 0 0 0
L0 ONLINE 0 0 0
L1 ONLINE 0 0 0
...
U25 ONLINE 0 0 0
U26 ONLINE 0 0 0
spare-53 ONLINE 0 0 0
U27 ONLINE 0 0 0
draid2-0-0 ONLINE 0 0 0
U28 ONLINE 0 0 0
U29 ONLINE 0 0 0
...
U42 ONLINE 0 0 0
U43 ONLINE 0 0 0
special
mirror-1 ONLINE 0 0 0
L5 ONLINE 0 0 0
U5 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
L6 ONLINE 0 0 0
U6 ONLINE 0 0 0
spares
draid2-0-0 INUSE currently in use
draid2-0-1 AVAIL
```
When adding test coverage for the new dRAID vdev type the following
options were added to the ztest command. These options are leverages
by zloop.sh to test a wide range of dRAID configurations.
-K draid|raidz|random - kind of RAID to test
-D <value> - dRAID data drives per group
-S <value> - dRAID distributed hot spares
-R <value> - RAID parity (raidz or dRAID)
The zpool_create, zpool_import, redundancy, replacement and fault
test groups have all been updated provide test coverage for the
dRAID feature.
Co-authored-by: Isaac Huang <he.huang@intel.com>
Co-authored-by: Mark Maybee <mmaybee@cray.com>
Co-authored-by: Don Brady <don.brady@delphix.com>
Co-authored-by: Matthew Ahrens <mahrens@delphix.com>
Co-authored-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Mark Maybee <mmaybee@cray.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #10102
2020-11-14 00:51:51 +03:00
|
|
|
#include <sys/vdev_draid.h>
|
2008-11-20 23:01:55 +03:00
|
|
|
#include <sys/uberblock_impl.h>
|
|
|
|
|
#include <sys/metaslab.h>
|
OpenZFS 7614, 9064 - zfs device evacuation/removal
OpenZFS 7614 - zfs device evacuation/removal
OpenZFS 9064 - remove_mirror should wait for device removal to complete
This project allows top-level vdevs to be removed from the storage pool
with "zpool remove", reducing the total amount of storage in the pool.
This operation copies all allocated regions of the device to be removed
onto other devices, recording the mapping from old to new location.
After the removal is complete, read and free operations to the removed
(now "indirect") vdev must be remapped and performed at the new location
on disk. The indirect mapping table is kept in memory whenever the pool
is loaded, so there is minimal performance overhead when doing operations
on the indirect vdev.
The size of the in-memory mapping table will be reduced when its entries
become "obsolete" because they are no longer used by any block pointers
in the pool. An entry becomes obsolete when all the blocks that use
it are freed. An entry can also become obsolete when all the snapshots
that reference it are deleted, and the block pointers that reference it
have been "remapped" in all filesystems/zvols (and clones). Whenever an
indirect block is written, all the block pointers in it will be "remapped"
to their new (concrete) locations if possible. This process can be
accelerated by using the "zfs remap" command to proactively rewrite all
indirect blocks that reference indirect (removed) vdevs.
Note that when a device is removed, we do not verify the checksum of
the data that is copied. This makes the process much faster, but if it
were used on redundant vdevs (i.e. mirror or raidz vdevs), it would be
possible to copy the wrong data, when we have the correct data on e.g.
the other side of the mirror.
At the moment, only mirrors and simple top-level vdevs can be removed
and no removal is allowed if any of the top-level vdevs are raidz.
Porting Notes:
* Avoid zero-sized kmem_alloc() in vdev_compact_children().
The device evacuation code adds a dependency that
vdev_compact_children() be able to properly empty the vdev_child
array by setting it to NULL and zeroing vdev_children. Under Linux,
kmem_alloc() and related functions return a sentinel pointer rather
than NULL for zero-sized allocations.
* Remove comment regarding "mpt" driver where zfs_remove_max_segment
is initialized to SPA_MAXBLOCKSIZE.
Change zfs_condense_indirect_commit_entry_delay_ticks to
zfs_condense_indirect_commit_entry_delay_ms for consistency with
most other tunables in which delays are specified in ms.
* ZTS changes:
Use set_tunable rather than mdb
Use zpool sync as appropriate
Use sync_pool instead of sync
Kill jobs during test_removal_with_operation to allow unmount/export
Don't add non-disk names such as "mirror" or "raidz" to $DISKS
Use $TEST_BASE_DIR instead of /tmp
Increase HZ from 100 to 1000 which is more common on Linux
removal_multiple_indirection.ksh
Reduce iterations in order to not time out on the code
coverage builders.
removal_resume_export:
Functionally, the test case is correct but there exists a race
where the kernel thread hasn't been fully started yet and is
not visible. Wait for up to 1 second for the removal thread
to be started before giving up on it. Also, increase the
amount of data copied in order that the removal not finish
before the export has a chance to fail.
* MMP compatibility, the concept of concrete versus non-concrete devices
has slightly changed the semantics of vdev_writeable(). Update
mmp_random_leaf_impl() accordingly.
* Updated dbuf_remap() to handle the org.zfsonlinux:large_dnode pool
feature which is not supported by OpenZFS.
* Added support for new vdev removal tracepoints.
* Test cases removal_with_zdb and removal_condense_export have been
intentionally disabled. When run manually they pass as intended,
but when running in the automated test environment they produce
unreliable results on the latest Fedora release.
They may work better once the upstream pool import refectoring is
merged into ZoL at which point they will be re-enabled.
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Alex Reece <alex@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Richard Laager <rlaager@wiktel.com>
Reviewed by: Tim Chase <tim@chase2k.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Garrett D'Amore <garrett@damore.org>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://www.illumos.org/issues/7614
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/f539f1eb
Closes #6900
2016-09-22 19:30:13 +03:00
|
|
|
#include <sys/metaslab_impl.h>
|
2008-11-20 23:01:55 +03:00
|
|
|
#include <sys/zio.h>
|
2010-05-29 00:45:14 +04:00
|
|
|
#include <sys/dsl_scan.h>
|
2016-07-22 18:52:49 +03:00
|
|
|
#include <sys/abd.h>
|
2008-11-20 23:01:55 +03:00
|
|
|
#include <sys/fs/zfs.h>
|
2020-09-16 01:42:27 +03:00
|
|
|
#include <sys/byteorder.h>
|
|
|
|
|
#include <sys/zfs_bootenv.h>
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Basic routines to read and write from a vdev label.
|
|
|
|
|
* Used throughout the rest of this file.
|
|
|
|
|
*/
|
|
|
|
|
uint64_t
|
|
|
|
|
vdev_label_offset(uint64_t psize, int l, uint64_t offset)
|
|
|
|
|
{
|
|
|
|
|
ASSERT(offset < sizeof (vdev_label_t));
|
|
|
|
|
ASSERT(P2PHASE_TYPED(psize, sizeof (vdev_label_t), uint64_t) == 0);
|
|
|
|
|
|
|
|
|
|
return (offset + l * sizeof (vdev_label_t) + (l < VDEV_LABELS / 2 ?
|
|
|
|
|
0 : psize - VDEV_LABELS * sizeof (vdev_label_t)));
|
|
|
|
|
}
|
|
|
|
|
|
2008-12-03 23:09:06 +03:00
|
|
|
/*
|
|
|
|
|
* Returns back the vdev label associated with the passed in offset.
|
|
|
|
|
*/
|
|
|
|
|
int
|
|
|
|
|
vdev_label_number(uint64_t psize, uint64_t offset)
|
|
|
|
|
{
|
|
|
|
|
int l;
|
|
|
|
|
|
|
|
|
|
if (offset >= psize - VDEV_LABEL_END_SIZE) {
|
|
|
|
|
offset -= psize - VDEV_LABEL_END_SIZE;
|
|
|
|
|
offset += (VDEV_LABELS / 2) * sizeof (vdev_label_t);
|
|
|
|
|
}
|
|
|
|
|
l = offset / sizeof (vdev_label_t);
|
|
|
|
|
return (l < VDEV_LABELS ? l : -1);
|
|
|
|
|
}
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
static void
|
2016-07-22 18:52:49 +03:00
|
|
|
vdev_label_read(zio_t *zio, vdev_t *vd, int l, abd_t *buf, uint64_t offset,
|
2017-01-12 20:42:11 +03:00
|
|
|
uint64_t size, zio_done_func_t *done, void *private, int flags)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
Correct lock ASSERTs in vdev_label_read/write
The existing assertions in vdev_label_read() and vdev_label_write(),
testing which config locks are held, are incorrect. The assertions
test for locks which exceed what is required for safety.
Both vdev_label_{read,write}() are changed to assert SCL_STATE is held
as RW_READER or RW_WRITER. This is safe because:
Changes to the vdev tree occur under SCL_ALL as RW_WRITER, via
spa_vdev_enter() and spa_vdev_exit().
Changes to vdev state occur under SCL_STATE_ALL as RW_WRITER, via
spa_vdev_state_enter() and spa_vdev_state_exit().
Therefore, the new assertions guarantee that the vdev cannot change
out from under a zio, and I/O to a specified leaf vdev's label is
safe.
Furthermore, this is consistent with the SPA locking discussion in
spa_misc.c, "For any zio operation that takes an explicit vdev_t
argument ... zio_read_phys(), or zio_write_phys() ... SCL_STATE as
reader suffices."
Reviewed-by: Chunwei Chen <david.chen@osnexus.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #5983
2017-04-22 00:26:43 +03:00
|
|
|
ASSERT(
|
|
|
|
|
spa_config_held(zio->io_spa, SCL_STATE, RW_READER) == SCL_STATE ||
|
|
|
|
|
spa_config_held(zio->io_spa, SCL_STATE, RW_WRITER) == SCL_STATE);
|
2008-12-03 23:09:06 +03:00
|
|
|
ASSERT(flags & ZIO_FLAG_CONFIG_WRITER);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
zio_nowait(zio_read_phys(zio, vd,
|
|
|
|
|
vdev_label_offset(vd->vdev_psize, l, offset),
|
|
|
|
|
size, buf, ZIO_CHECKSUM_LABEL, done, private,
|
2008-12-03 23:09:06 +03:00
|
|
|
ZIO_PRIORITY_SYNC_READ, flags, B_TRUE));
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
Multi-modifier protection (MMP)
Add multihost=on|off pool property to control MMP. When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp. Property defaults to off.
During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock. Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".
Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval. The period is specified in milliseconds. The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.
Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path. Abbreviated
output below.
$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg timestamp mmp_delay vdev_guid vdev_label vdev_path
20468 261337 250274925 68396651780 3 /dev/sda
20468 261339 252023374 6267402363293 1 /dev/sdc
20468 261340 252000858 6698080955233 1 /dev/sdx
20468 261341 251980635 783892869810 2 /dev/sdy
20468 261342 253385953 8923255792467 3 /dev/sdd
20468 261344 253336622 042125143176 0 /dev/sdab
20468 261345 253310522 1200778101278 2 /dev/sde
20468 261346 253286429 0950576198362 2 /dev/sdt
20468 261347 253261545 96209817917 3 /dev/sds
20468 261349 253238188 8555725937673 3 /dev/sdb
Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.
When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test. For example, the
pool is exported to run zdb and then imported again. Add a new ztest
function, "-M", to alter ztest behavior to prevent this.
Add new tests to verify the new functionality. Tests provided by
Giuseppe Di Natale.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #745
Closes #6279
2017-07-08 06:20:35 +03:00
|
|
|
void
|
2016-07-22 18:52:49 +03:00
|
|
|
vdev_label_write(zio_t *zio, vdev_t *vd, int l, abd_t *buf, uint64_t offset,
|
2017-01-12 20:42:11 +03:00
|
|
|
uint64_t size, zio_done_func_t *done, void *private, int flags)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
Correct lock ASSERTs in vdev_label_read/write
The existing assertions in vdev_label_read() and vdev_label_write(),
testing which config locks are held, are incorrect. The assertions
test for locks which exceed what is required for safety.
Both vdev_label_{read,write}() are changed to assert SCL_STATE is held
as RW_READER or RW_WRITER. This is safe because:
Changes to the vdev tree occur under SCL_ALL as RW_WRITER, via
spa_vdev_enter() and spa_vdev_exit().
Changes to vdev state occur under SCL_STATE_ALL as RW_WRITER, via
spa_vdev_state_enter() and spa_vdev_state_exit().
Therefore, the new assertions guarantee that the vdev cannot change
out from under a zio, and I/O to a specified leaf vdev's label is
safe.
Furthermore, this is consistent with the SPA locking discussion in
spa_misc.c, "For any zio operation that takes an explicit vdev_t
argument ... zio_read_phys(), or zio_write_phys() ... SCL_STATE as
reader suffices."
Reviewed-by: Chunwei Chen <david.chen@osnexus.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #5983
2017-04-22 00:26:43 +03:00
|
|
|
ASSERT(
|
|
|
|
|
spa_config_held(zio->io_spa, SCL_STATE, RW_READER) == SCL_STATE ||
|
|
|
|
|
spa_config_held(zio->io_spa, SCL_STATE, RW_WRITER) == SCL_STATE);
|
2008-12-03 23:09:06 +03:00
|
|
|
ASSERT(flags & ZIO_FLAG_CONFIG_WRITER);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
zio_nowait(zio_write_phys(zio, vd,
|
|
|
|
|
vdev_label_offset(vd->vdev_psize, l, offset),
|
|
|
|
|
size, buf, ZIO_CHECKSUM_LABEL, done, private,
|
|
|
|
|
ZIO_PRIORITY_SYNC_WRITE, flags, B_TRUE));
|
|
|
|
|
}
|
|
|
|
|
|
2016-02-29 21:05:23 +03:00
|
|
|
/*
|
|
|
|
|
* Generate the nvlist representing this vdev's stats
|
|
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
vdev_config_generate_stats(vdev_t *vd, nvlist_t *nv)
|
|
|
|
|
{
|
|
|
|
|
nvlist_t *nvx;
|
|
|
|
|
vdev_stat_t *vs;
|
|
|
|
|
vdev_stat_ex_t *vsx;
|
|
|
|
|
|
|
|
|
|
vs = kmem_alloc(sizeof (*vs), KM_SLEEP);
|
|
|
|
|
vsx = kmem_alloc(sizeof (*vsx), KM_SLEEP);
|
|
|
|
|
|
|
|
|
|
vdev_get_stats_ex(vd, vs, vsx);
|
|
|
|
|
fnvlist_add_uint64_array(nv, ZPOOL_CONFIG_VDEV_STATS,
|
|
|
|
|
(uint64_t *)vs, sizeof (*vs) / sizeof (uint64_t));
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Add extended stats into a special extended stats nvlist. This keeps
|
|
|
|
|
* all the extended stats nicely grouped together. The extended stats
|
|
|
|
|
* nvlist is then added to the main nvlist.
|
|
|
|
|
*/
|
|
|
|
|
nvx = fnvlist_alloc();
|
|
|
|
|
|
|
|
|
|
/* ZIOs in flight to disk */
|
|
|
|
|
fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_SYNC_R_ACTIVE_QUEUE,
|
|
|
|
|
vsx->vsx_active_queue[ZIO_PRIORITY_SYNC_READ]);
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_SYNC_W_ACTIVE_QUEUE,
|
|
|
|
|
vsx->vsx_active_queue[ZIO_PRIORITY_SYNC_WRITE]);
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_ASYNC_R_ACTIVE_QUEUE,
|
|
|
|
|
vsx->vsx_active_queue[ZIO_PRIORITY_ASYNC_READ]);
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_ASYNC_W_ACTIVE_QUEUE,
|
|
|
|
|
vsx->vsx_active_queue[ZIO_PRIORITY_ASYNC_WRITE]);
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_SCRUB_ACTIVE_QUEUE,
|
|
|
|
|
vsx->vsx_active_queue[ZIO_PRIORITY_SCRUB]);
|
|
|
|
|
|
2019-03-29 19:13:20 +03:00
|
|
|
fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_TRIM_ACTIVE_QUEUE,
|
|
|
|
|
vsx->vsx_active_queue[ZIO_PRIORITY_TRIM]);
|
|
|
|
|
|
2021-08-26 21:26:49 +03:00
|
|
|
fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_REBUILD_ACTIVE_QUEUE,
|
|
|
|
|
vsx->vsx_active_queue[ZIO_PRIORITY_REBUILD]);
|
|
|
|
|
|
2016-02-29 21:05:23 +03:00
|
|
|
/* ZIOs pending */
|
|
|
|
|
fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_SYNC_R_PEND_QUEUE,
|
|
|
|
|
vsx->vsx_pend_queue[ZIO_PRIORITY_SYNC_READ]);
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_SYNC_W_PEND_QUEUE,
|
|
|
|
|
vsx->vsx_pend_queue[ZIO_PRIORITY_SYNC_WRITE]);
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_ASYNC_R_PEND_QUEUE,
|
|
|
|
|
vsx->vsx_pend_queue[ZIO_PRIORITY_ASYNC_READ]);
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_ASYNC_W_PEND_QUEUE,
|
|
|
|
|
vsx->vsx_pend_queue[ZIO_PRIORITY_ASYNC_WRITE]);
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_SCRUB_PEND_QUEUE,
|
|
|
|
|
vsx->vsx_pend_queue[ZIO_PRIORITY_SCRUB]);
|
|
|
|
|
|
2019-03-29 19:13:20 +03:00
|
|
|
fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_TRIM_PEND_QUEUE,
|
|
|
|
|
vsx->vsx_pend_queue[ZIO_PRIORITY_TRIM]);
|
|
|
|
|
|
2021-08-26 21:26:49 +03:00
|
|
|
fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_REBUILD_PEND_QUEUE,
|
|
|
|
|
vsx->vsx_pend_queue[ZIO_PRIORITY_REBUILD]);
|
|
|
|
|
|
2016-02-29 21:05:23 +03:00
|
|
|
/* Histograms */
|
|
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_TOT_R_LAT_HISTO,
|
|
|
|
|
vsx->vsx_total_histo[ZIO_TYPE_READ],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_total_histo[ZIO_TYPE_READ]));
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_TOT_W_LAT_HISTO,
|
|
|
|
|
vsx->vsx_total_histo[ZIO_TYPE_WRITE],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_total_histo[ZIO_TYPE_WRITE]));
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_DISK_R_LAT_HISTO,
|
|
|
|
|
vsx->vsx_disk_histo[ZIO_TYPE_READ],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_disk_histo[ZIO_TYPE_READ]));
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_DISK_W_LAT_HISTO,
|
|
|
|
|
vsx->vsx_disk_histo[ZIO_TYPE_WRITE],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_disk_histo[ZIO_TYPE_WRITE]));
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_SYNC_R_LAT_HISTO,
|
|
|
|
|
vsx->vsx_queue_histo[ZIO_PRIORITY_SYNC_READ],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_queue_histo[ZIO_PRIORITY_SYNC_READ]));
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_SYNC_W_LAT_HISTO,
|
|
|
|
|
vsx->vsx_queue_histo[ZIO_PRIORITY_SYNC_WRITE],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_queue_histo[ZIO_PRIORITY_SYNC_WRITE]));
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_ASYNC_R_LAT_HISTO,
|
|
|
|
|
vsx->vsx_queue_histo[ZIO_PRIORITY_ASYNC_READ],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_queue_histo[ZIO_PRIORITY_ASYNC_READ]));
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_ASYNC_W_LAT_HISTO,
|
|
|
|
|
vsx->vsx_queue_histo[ZIO_PRIORITY_ASYNC_WRITE],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_queue_histo[ZIO_PRIORITY_ASYNC_WRITE]));
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_SCRUB_LAT_HISTO,
|
|
|
|
|
vsx->vsx_queue_histo[ZIO_PRIORITY_SCRUB],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_queue_histo[ZIO_PRIORITY_SCRUB]));
|
|
|
|
|
|
2019-03-29 19:13:20 +03:00
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_TRIM_LAT_HISTO,
|
|
|
|
|
vsx->vsx_queue_histo[ZIO_PRIORITY_TRIM],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_queue_histo[ZIO_PRIORITY_TRIM]));
|
|
|
|
|
|
2021-08-26 21:26:49 +03:00
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_REBUILD_LAT_HISTO,
|
|
|
|
|
vsx->vsx_queue_histo[ZIO_PRIORITY_REBUILD],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_queue_histo[ZIO_PRIORITY_REBUILD]));
|
|
|
|
|
|
2016-05-26 00:21:35 +03:00
|
|
|
/* Request sizes */
|
|
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_SYNC_IND_R_HISTO,
|
|
|
|
|
vsx->vsx_ind_histo[ZIO_PRIORITY_SYNC_READ],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_ind_histo[ZIO_PRIORITY_SYNC_READ]));
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_SYNC_IND_W_HISTO,
|
|
|
|
|
vsx->vsx_ind_histo[ZIO_PRIORITY_SYNC_WRITE],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_ind_histo[ZIO_PRIORITY_SYNC_WRITE]));
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_ASYNC_IND_R_HISTO,
|
|
|
|
|
vsx->vsx_ind_histo[ZIO_PRIORITY_ASYNC_READ],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_ind_histo[ZIO_PRIORITY_ASYNC_READ]));
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_ASYNC_IND_W_HISTO,
|
|
|
|
|
vsx->vsx_ind_histo[ZIO_PRIORITY_ASYNC_WRITE],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_ind_histo[ZIO_PRIORITY_ASYNC_WRITE]));
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_IND_SCRUB_HISTO,
|
|
|
|
|
vsx->vsx_ind_histo[ZIO_PRIORITY_SCRUB],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_ind_histo[ZIO_PRIORITY_SCRUB]));
|
|
|
|
|
|
2019-03-29 19:13:20 +03:00
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_IND_TRIM_HISTO,
|
|
|
|
|
vsx->vsx_ind_histo[ZIO_PRIORITY_TRIM],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_ind_histo[ZIO_PRIORITY_TRIM]));
|
|
|
|
|
|
2021-08-26 21:26:49 +03:00
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_IND_REBUILD_HISTO,
|
|
|
|
|
vsx->vsx_ind_histo[ZIO_PRIORITY_REBUILD],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_ind_histo[ZIO_PRIORITY_REBUILD]));
|
|
|
|
|
|
2016-05-26 00:21:35 +03:00
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_SYNC_AGG_R_HISTO,
|
|
|
|
|
vsx->vsx_agg_histo[ZIO_PRIORITY_SYNC_READ],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_agg_histo[ZIO_PRIORITY_SYNC_READ]));
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_SYNC_AGG_W_HISTO,
|
|
|
|
|
vsx->vsx_agg_histo[ZIO_PRIORITY_SYNC_WRITE],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_agg_histo[ZIO_PRIORITY_SYNC_WRITE]));
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_ASYNC_AGG_R_HISTO,
|
|
|
|
|
vsx->vsx_agg_histo[ZIO_PRIORITY_ASYNC_READ],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_agg_histo[ZIO_PRIORITY_ASYNC_READ]));
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_ASYNC_AGG_W_HISTO,
|
|
|
|
|
vsx->vsx_agg_histo[ZIO_PRIORITY_ASYNC_WRITE],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_agg_histo[ZIO_PRIORITY_ASYNC_WRITE]));
|
|
|
|
|
|
|
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_AGG_SCRUB_HISTO,
|
|
|
|
|
vsx->vsx_agg_histo[ZIO_PRIORITY_SCRUB],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_agg_histo[ZIO_PRIORITY_SCRUB]));
|
|
|
|
|
|
2019-03-29 19:13:20 +03:00
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_AGG_TRIM_HISTO,
|
|
|
|
|
vsx->vsx_agg_histo[ZIO_PRIORITY_TRIM],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_agg_histo[ZIO_PRIORITY_TRIM]));
|
|
|
|
|
|
2021-08-26 21:26:49 +03:00
|
|
|
fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_AGG_REBUILD_HISTO,
|
|
|
|
|
vsx->vsx_agg_histo[ZIO_PRIORITY_REBUILD],
|
|
|
|
|
ARRAY_SIZE(vsx->vsx_agg_histo[ZIO_PRIORITY_REBUILD]));
|
|
|
|
|
|
2018-11-09 03:47:24 +03:00
|
|
|
/* IO delays */
|
|
|
|
|
fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_SLOW_IOS, vs->vs_slow_ios);
|
|
|
|
|
|
Adding Direct IO Support
Adding O_DIRECT support to ZFS to bypass the ARC for writes/reads.
O_DIRECT support in ZFS will always ensure there is coherency between
buffered and O_DIRECT IO requests. This ensures that all IO requests,
whether buffered or direct, will see the same file contents at all
times. Just as in other FS's , O_DIRECT does not imply O_SYNC. While
data is written directly to VDEV disks, metadata will not be synced
until the associated TXG is synced.
For both O_DIRECT read and write request the offset and request sizes,
at a minimum, must be PAGE_SIZE aligned. In the event they are not,
then EINVAL is returned unless the direct property is set to always (see
below).
For O_DIRECT writes:
The request also must be block aligned (recordsize) or the write
request will take the normal (buffered) write path. In the event that
request is block aligned and a cached copy of the buffer in the ARC,
then it will be discarded from the ARC forcing all further reads to
retrieve the data from disk.
For O_DIRECT reads:
The only alignment restrictions are PAGE_SIZE alignment. In the event
that the requested data is in buffered (in the ARC) it will just be
copied from the ARC into the user buffer.
For both O_DIRECT writes and reads the O_DIRECT flag will be ignored in
the event that file contents are mmap'ed. In this case, all requests
that are at least PAGE_SIZE aligned will just fall back to the buffered
paths. If the request however is not PAGE_SIZE aligned, EINVAL will
be returned as always regardless if the file's contents are mmap'ed.
Since O_DIRECT writes go through the normal ZIO pipeline, the
following operations are supported just as with normal buffered writes:
Checksum
Compression
Encryption
Erasure Coding
There is one caveat for the data integrity of O_DIRECT writes that is
distinct for each of the OS's supported by ZFS.
FreeBSD - FreeBSD is able to place user pages under write protection so
any data in the user buffers and written directly down to the
VDEV disks is guaranteed to not change. There is no concern
with data integrity and O_DIRECT writes.
Linux - Linux is not able to place anonymous user pages under write
protection. Because of this, if the user decides to manipulate
the page contents while the write operation is occurring, data
integrity can not be guaranteed. However, there is a module
parameter `zfs_vdev_direct_write_verify` that controls the
if a O_DIRECT writes that can occur to a top-level VDEV before
a checksum verify is run before the contents of the I/O buffer
are committed to disk. In the event of a checksum verification
failure the write will return EIO. The number of O_DIRECT write
checksum verification errors can be observed by doing
`zpool status -d`, which will list all verification errors that
have occurred on a top-level VDEV. Along with `zpool status`, a
ZED event will be issues as `dio_verify` when a checksum
verification error occurs.
ZVOLs and dedup is not currently supported with Direct I/O.
A new dataset property `direct` has been added with the following 3
allowable values:
disabled - Accepts O_DIRECT flag, but silently ignores it and treats
the request as a buffered IO request.
standard - Follows the alignment restrictions outlined above for
write/read IO requests when the O_DIRECT flag is used.
always - Treats every write/read IO request as though it passed
O_DIRECT and will do O_DIRECT if the alignment restrictions
are met otherwise will redirect through the ARC. This
property will not allow a request to fail.
There is also a module parameter zfs_dio_enabled that can be used to
force all reads and writes through the ARC. By setting this module
parameter to 0, it mimics as if the direct dataset property is set to
disabled.
Reviewed-by: Brian Behlendorf <behlendorf@llnl.gov>
Reviewed-by: Alexander Motin <mav@FreeBSD.org>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Atkinson <batkinson@lanl.gov>
Co-authored-by: Mark Maybee <mark.maybee@delphix.com>
Co-authored-by: Matt Macy <mmacy@FreeBSD.org>
Co-authored-by: Brian Behlendorf <behlendorf@llnl.gov>
Closes #10018
2024-09-14 23:47:59 +03:00
|
|
|
/* Direct I/O write verify errors */
|
|
|
|
|
fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_DIO_VERIFY_ERRORS,
|
|
|
|
|
vs->vs_dio_verify_errors);
|
|
|
|
|
|
2016-02-29 21:05:23 +03:00
|
|
|
/* Add extended stats nvlist to main nvlist */
|
|
|
|
|
fnvlist_add_nvlist(nv, ZPOOL_CONFIG_VDEV_STATS_EX, nvx);
|
|
|
|
|
|
2016-05-28 01:48:02 +03:00
|
|
|
fnvlist_free(nvx);
|
2018-11-12 05:09:00 +03:00
|
|
|
kmem_free(vs, sizeof (*vs));
|
2016-02-29 21:05:23 +03:00
|
|
|
kmem_free(vsx, sizeof (*vsx));
|
|
|
|
|
}
|
|
|
|
|
|
2016-12-17 01:11:29 +03:00
|
|
|
static void
|
|
|
|
|
root_vdev_actions_getprogress(vdev_t *vd, nvlist_t *nvl)
|
|
|
|
|
{
|
|
|
|
|
spa_t *spa = vd->vdev_spa;
|
|
|
|
|
|
|
|
|
|
if (vd != spa->spa_root_vdev)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
/* provide either current or previous scan information */
|
|
|
|
|
pool_scan_stat_t ps;
|
|
|
|
|
if (spa_scan_get_stats(spa, &ps) == 0) {
|
|
|
|
|
fnvlist_add_uint64_array(nvl,
|
|
|
|
|
ZPOOL_CONFIG_SCAN_STATS, (uint64_t *)&ps,
|
|
|
|
|
sizeof (pool_scan_stat_t) / sizeof (uint64_t));
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
pool_removal_stat_t prs;
|
|
|
|
|
if (spa_removal_get_stats(spa, &prs) == 0) {
|
|
|
|
|
fnvlist_add_uint64_array(nvl,
|
|
|
|
|
ZPOOL_CONFIG_REMOVAL_STATS, (uint64_t *)&prs,
|
|
|
|
|
sizeof (prs) / sizeof (uint64_t));
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
pool_checkpoint_stat_t pcs;
|
|
|
|
|
if (spa_checkpoint_get_stats(spa, &pcs) == 0) {
|
|
|
|
|
fnvlist_add_uint64_array(nvl,
|
|
|
|
|
ZPOOL_CONFIG_CHECKPOINT_STATS, (uint64_t *)&pcs,
|
|
|
|
|
sizeof (pcs) / sizeof (uint64_t));
|
|
|
|
|
}
|
RAID-Z expansion feature
This feature allows disks to be added one at a time to a RAID-Z group,
expanding its capacity incrementally. This feature is especially useful
for small pools (typically with only one RAID-Z group), where there
isn't sufficient hardware to add capacity by adding a whole new RAID-Z
group (typically doubling the number of disks).
== Initiating expansion ==
A new device (disk) can be attached to an existing RAIDZ vdev, by
running `zpool attach POOL raidzP-N NEW_DEVICE`, e.g. `zpool attach tank
raidz2-0 sda`. The new device will become part of the RAIDZ group. A
"raidz expansion" will be initiated, and the new device will contribute
additional space to the RAIDZ group once the expansion completes.
The `feature@raidz_expansion` on-disk feature flag must be `enabled` to
initiate an expansion, and it remains `active` for the life of the pool.
In other words, pools with expanded RAIDZ vdevs can not be imported by
older releases of the ZFS software.
== During expansion ==
The expansion entails reading all allocated space from existing disks in
the RAIDZ group, and rewriting it to the new disks in the RAIDZ group
(including the newly added device).
The expansion progress can be monitored with `zpool status`.
Data redundancy is maintained during (and after) the expansion. If a
disk fails while the expansion is in progress, the expansion pauses
until the health of the RAIDZ vdev is restored (e.g. by replacing the
failed disk and waiting for reconstruction to complete).
The pool remains accessible during expansion. Following a reboot or
export/import, the expansion resumes where it left off.
== After expansion ==
When the expansion completes, the additional space is available for use,
and is reflected in the `available` zfs property (as seen in `zfs list`,
`df`, etc).
Expansion does not change the number of failures that can be tolerated
without data loss (e.g. a RAIDZ2 is still a RAIDZ2 even after
expansion).
A RAIDZ vdev can be expanded multiple times.
After the expansion completes, old blocks remain with their old
data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but
distributed among the larger set of disks. New blocks will be written
with the new data-to-parity ratio (e.g. a 5-wide RAIDZ2 which has been
expanded once to 6-wide, has 4 data to 2 parity). However, the RAIDZ
vdev's "assumed parity ratio" does not change, so slightly less space
than is expected may be reported for newly-written blocks, according to
`zfs list`, `df`, `ls -s`, and similar tools.
Sponsored-by: The FreeBSD Foundation
Sponsored-by: iXsystems, Inc.
Sponsored-by: vStack
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Mark Maybee <mark.maybee@delphix.com>
Authored-by: Matthew Ahrens <mahrens@delphix.com>
Contributions-by: Fedor Uporov <fuporov.vstack@gmail.com>
Contributions-by: Stuart Maybee <stuart.maybee@comcast.net>
Contributions-by: Thorsten Behrens <tbehrens@outlook.com>
Contributions-by: Fmstrat <nospam@nowsci.com>
Contributions-by: Don Brady <dev.fs.zfs@gmail.com>
Signed-off-by: Don Brady <dev.fs.zfs@gmail.com>
Closes #15022
2023-11-08 21:19:41 +03:00
|
|
|
|
|
|
|
|
pool_raidz_expand_stat_t pres;
|
|
|
|
|
if (spa_raidz_expand_get_stats(spa, &pres) == 0) {
|
|
|
|
|
fnvlist_add_uint64_array(nvl,
|
|
|
|
|
ZPOOL_CONFIG_RAIDZ_EXPAND_STATS, (uint64_t *)&pres,
|
|
|
|
|
sizeof (pres) / sizeof (uint64_t));
|
|
|
|
|
}
|
2016-12-17 01:11:29 +03:00
|
|
|
}
|
|
|
|
|
|
2020-07-03 21:05:50 +03:00
|
|
|
static void
|
|
|
|
|
top_vdev_actions_getprogress(vdev_t *vd, nvlist_t *nvl)
|
|
|
|
|
{
|
|
|
|
|
if (vd == vd->vdev_top) {
|
|
|
|
|
vdev_rebuild_stat_t vrs;
|
|
|
|
|
if (vdev_rebuild_get_stats(vd, &vrs) == 0) {
|
|
|
|
|
fnvlist_add_uint64_array(nvl,
|
|
|
|
|
ZPOOL_CONFIG_REBUILD_STATS, (uint64_t *)&vrs,
|
|
|
|
|
sizeof (vrs) / sizeof (uint64_t));
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
/*
|
|
|
|
|
* Generate the nvlist representing this vdev's config.
|
|
|
|
|
*/
|
|
|
|
|
nvlist_t *
|
|
|
|
|
vdev_config_generate(spa_t *spa, vdev_t *vd, boolean_t getstats,
|
2010-05-29 00:45:14 +04:00
|
|
|
vdev_config_flag_t flags)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
|
|
|
|
nvlist_t *nv = NULL;
|
OpenZFS 7614, 9064 - zfs device evacuation/removal
OpenZFS 7614 - zfs device evacuation/removal
OpenZFS 9064 - remove_mirror should wait for device removal to complete
This project allows top-level vdevs to be removed from the storage pool
with "zpool remove", reducing the total amount of storage in the pool.
This operation copies all allocated regions of the device to be removed
onto other devices, recording the mapping from old to new location.
After the removal is complete, read and free operations to the removed
(now "indirect") vdev must be remapped and performed at the new location
on disk. The indirect mapping table is kept in memory whenever the pool
is loaded, so there is minimal performance overhead when doing operations
on the indirect vdev.
The size of the in-memory mapping table will be reduced when its entries
become "obsolete" because they are no longer used by any block pointers
in the pool. An entry becomes obsolete when all the blocks that use
it are freed. An entry can also become obsolete when all the snapshots
that reference it are deleted, and the block pointers that reference it
have been "remapped" in all filesystems/zvols (and clones). Whenever an
indirect block is written, all the block pointers in it will be "remapped"
to their new (concrete) locations if possible. This process can be
accelerated by using the "zfs remap" command to proactively rewrite all
indirect blocks that reference indirect (removed) vdevs.
Note that when a device is removed, we do not verify the checksum of
the data that is copied. This makes the process much faster, but if it
were used on redundant vdevs (i.e. mirror or raidz vdevs), it would be
possible to copy the wrong data, when we have the correct data on e.g.
the other side of the mirror.
At the moment, only mirrors and simple top-level vdevs can be removed
and no removal is allowed if any of the top-level vdevs are raidz.
Porting Notes:
* Avoid zero-sized kmem_alloc() in vdev_compact_children().
The device evacuation code adds a dependency that
vdev_compact_children() be able to properly empty the vdev_child
array by setting it to NULL and zeroing vdev_children. Under Linux,
kmem_alloc() and related functions return a sentinel pointer rather
than NULL for zero-sized allocations.
* Remove comment regarding "mpt" driver where zfs_remove_max_segment
is initialized to SPA_MAXBLOCKSIZE.
Change zfs_condense_indirect_commit_entry_delay_ticks to
zfs_condense_indirect_commit_entry_delay_ms for consistency with
most other tunables in which delays are specified in ms.
* ZTS changes:
Use set_tunable rather than mdb
Use zpool sync as appropriate
Use sync_pool instead of sync
Kill jobs during test_removal_with_operation to allow unmount/export
Don't add non-disk names such as "mirror" or "raidz" to $DISKS
Use $TEST_BASE_DIR instead of /tmp
Increase HZ from 100 to 1000 which is more common on Linux
removal_multiple_indirection.ksh
Reduce iterations in order to not time out on the code
coverage builders.
removal_resume_export:
Functionally, the test case is correct but there exists a race
where the kernel thread hasn't been fully started yet and is
not visible. Wait for up to 1 second for the removal thread
to be started before giving up on it. Also, increase the
amount of data copied in order that the removal not finish
before the export has a chance to fail.
* MMP compatibility, the concept of concrete versus non-concrete devices
has slightly changed the semantics of vdev_writeable(). Update
mmp_random_leaf_impl() accordingly.
* Updated dbuf_remap() to handle the org.zfsonlinux:large_dnode pool
feature which is not supported by OpenZFS.
* Added support for new vdev removal tracepoints.
* Test cases removal_with_zdb and removal_condense_export have been
intentionally disabled. When run manually they pass as intended,
but when running in the automated test environment they produce
unreliable results on the latest Fedora release.
They may work better once the upstream pool import refectoring is
merged into ZoL at which point they will be re-enabled.
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Alex Reece <alex@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Richard Laager <rlaager@wiktel.com>
Reviewed by: Tim Chase <tim@chase2k.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Garrett D'Amore <garrett@damore.org>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://www.illumos.org/issues/7614
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/f539f1eb
Closes #6900
2016-09-22 19:30:13 +03:00
|
|
|
vdev_indirect_config_t *vic = &vd->vdev_indirect_config;
|
|
|
|
|
|
2014-11-21 03:09:39 +03:00
|
|
|
nv = fnvlist_alloc();
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_string(nv, ZPOOL_CONFIG_TYPE, vd->vdev_ops->vdev_op_type);
|
2010-05-29 00:45:14 +04:00
|
|
|
if (!(flags & (VDEV_CONFIG_SPARE | VDEV_CONFIG_L2CACHE)))
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_ID, vd->vdev_id);
|
|
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_GUID, vd->vdev_guid);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
if (vd->vdev_path != NULL)
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_string(nv, ZPOOL_CONFIG_PATH, vd->vdev_path);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
if (vd->vdev_devid != NULL)
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_string(nv, ZPOOL_CONFIG_DEVID, vd->vdev_devid);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
if (vd->vdev_physpath != NULL)
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_string(nv, ZPOOL_CONFIG_PHYS_PATH,
|
|
|
|
|
vd->vdev_physpath);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2016-10-24 20:45:59 +03:00
|
|
|
if (vd->vdev_enc_sysfs_path != NULL)
|
|
|
|
|
fnvlist_add_string(nv, ZPOOL_CONFIG_VDEV_ENC_SYSFS_PATH,
|
|
|
|
|
vd->vdev_enc_sysfs_path);
|
|
|
|
|
|
2009-07-03 02:44:48 +04:00
|
|
|
if (vd->vdev_fru != NULL)
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_string(nv, ZPOOL_CONFIG_FRU, vd->vdev_fru);
|
2009-07-03 02:44:48 +04:00
|
|
|
|
Distributed Spare (dRAID) Feature
This patch adds a new top-level vdev type called dRAID, which stands
for Distributed parity RAID. This pool configuration allows all dRAID
vdevs to participate when rebuilding to a distributed hot spare device.
This can substantially reduce the total time required to restore full
parity to pool with a failed device.
A dRAID pool can be created using the new top-level `draid` type.
Like `raidz`, the desired redundancy is specified after the type:
`draid[1,2,3]`. No additional information is required to create the
pool and reasonable default values will be chosen based on the number
of child vdevs in the dRAID vdev.
zpool create <pool> draid[1,2,3] <vdevs...>
Unlike raidz, additional optional dRAID configuration values can be
provided as part of the draid type as colon separated values. This
allows administrators to fully specify a layout for either performance
or capacity reasons. The supported options include:
zpool create <pool> \
draid[<parity>][:<data>d][:<children>c][:<spares>s] \
<vdevs...>
- draid[parity] - Parity level (default 1)
- draid[:<data>d] - Data devices per group (default 8)
- draid[:<children>c] - Expected number of child vdevs
- draid[:<spares>s] - Distributed hot spares (default 0)
Abbreviated example `zpool status` output for a 68 disk dRAID pool
with two distributed spares using special allocation classes.
```
pool: tank
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
slag7 ONLINE 0 0 0
draid2:8d:68c:2s-0 ONLINE 0 0 0
L0 ONLINE 0 0 0
L1 ONLINE 0 0 0
...
U25 ONLINE 0 0 0
U26 ONLINE 0 0 0
spare-53 ONLINE 0 0 0
U27 ONLINE 0 0 0
draid2-0-0 ONLINE 0 0 0
U28 ONLINE 0 0 0
U29 ONLINE 0 0 0
...
U42 ONLINE 0 0 0
U43 ONLINE 0 0 0
special
mirror-1 ONLINE 0 0 0
L5 ONLINE 0 0 0
U5 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
L6 ONLINE 0 0 0
U6 ONLINE 0 0 0
spares
draid2-0-0 INUSE currently in use
draid2-0-1 AVAIL
```
When adding test coverage for the new dRAID vdev type the following
options were added to the ztest command. These options are leverages
by zloop.sh to test a wide range of dRAID configurations.
-K draid|raidz|random - kind of RAID to test
-D <value> - dRAID data drives per group
-S <value> - dRAID distributed hot spares
-R <value> - RAID parity (raidz or dRAID)
The zpool_create, zpool_import, redundancy, replacement and fault
test groups have all been updated provide test coverage for the
dRAID feature.
Co-authored-by: Isaac Huang <he.huang@intel.com>
Co-authored-by: Mark Maybee <mmaybee@cray.com>
Co-authored-by: Don Brady <don.brady@delphix.com>
Co-authored-by: Matthew Ahrens <mahrens@delphix.com>
Co-authored-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Mark Maybee <mmaybee@cray.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #10102
2020-11-14 00:51:51 +03:00
|
|
|
if (vd->vdev_ops->vdev_op_config_generate != NULL)
|
|
|
|
|
vd->vdev_ops->vdev_op_config_generate(vd, nv);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
Distributed Spare (dRAID) Feature
This patch adds a new top-level vdev type called dRAID, which stands
for Distributed parity RAID. This pool configuration allows all dRAID
vdevs to participate when rebuilding to a distributed hot spare device.
This can substantially reduce the total time required to restore full
parity to pool with a failed device.
A dRAID pool can be created using the new top-level `draid` type.
Like `raidz`, the desired redundancy is specified after the type:
`draid[1,2,3]`. No additional information is required to create the
pool and reasonable default values will be chosen based on the number
of child vdevs in the dRAID vdev.
zpool create <pool> draid[1,2,3] <vdevs...>
Unlike raidz, additional optional dRAID configuration values can be
provided as part of the draid type as colon separated values. This
allows administrators to fully specify a layout for either performance
or capacity reasons. The supported options include:
zpool create <pool> \
draid[<parity>][:<data>d][:<children>c][:<spares>s] \
<vdevs...>
- draid[parity] - Parity level (default 1)
- draid[:<data>d] - Data devices per group (default 8)
- draid[:<children>c] - Expected number of child vdevs
- draid[:<spares>s] - Distributed hot spares (default 0)
Abbreviated example `zpool status` output for a 68 disk dRAID pool
with two distributed spares using special allocation classes.
```
pool: tank
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
slag7 ONLINE 0 0 0
draid2:8d:68c:2s-0 ONLINE 0 0 0
L0 ONLINE 0 0 0
L1 ONLINE 0 0 0
...
U25 ONLINE 0 0 0
U26 ONLINE 0 0 0
spare-53 ONLINE 0 0 0
U27 ONLINE 0 0 0
draid2-0-0 ONLINE 0 0 0
U28 ONLINE 0 0 0
U29 ONLINE 0 0 0
...
U42 ONLINE 0 0 0
U43 ONLINE 0 0 0
special
mirror-1 ONLINE 0 0 0
L5 ONLINE 0 0 0
U5 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
L6 ONLINE 0 0 0
U6 ONLINE 0 0 0
spares
draid2-0-0 INUSE currently in use
draid2-0-1 AVAIL
```
When adding test coverage for the new dRAID vdev type the following
options were added to the ztest command. These options are leverages
by zloop.sh to test a wide range of dRAID configurations.
-K draid|raidz|random - kind of RAID to test
-D <value> - dRAID data drives per group
-S <value> - dRAID distributed hot spares
-R <value> - RAID parity (raidz or dRAID)
The zpool_create, zpool_import, redundancy, replacement and fault
test groups have all been updated provide test coverage for the
dRAID feature.
Co-authored-by: Isaac Huang <he.huang@intel.com>
Co-authored-by: Mark Maybee <mmaybee@cray.com>
Co-authored-by: Don Brady <don.brady@delphix.com>
Co-authored-by: Matthew Ahrens <mahrens@delphix.com>
Co-authored-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Mark Maybee <mmaybee@cray.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #10102
2020-11-14 00:51:51 +03:00
|
|
|
if (vd->vdev_wholedisk != -1ULL) {
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_WHOLE_DISK,
|
|
|
|
|
vd->vdev_wholedisk);
|
Distributed Spare (dRAID) Feature
This patch adds a new top-level vdev type called dRAID, which stands
for Distributed parity RAID. This pool configuration allows all dRAID
vdevs to participate when rebuilding to a distributed hot spare device.
This can substantially reduce the total time required to restore full
parity to pool with a failed device.
A dRAID pool can be created using the new top-level `draid` type.
Like `raidz`, the desired redundancy is specified after the type:
`draid[1,2,3]`. No additional information is required to create the
pool and reasonable default values will be chosen based on the number
of child vdevs in the dRAID vdev.
zpool create <pool> draid[1,2,3] <vdevs...>
Unlike raidz, additional optional dRAID configuration values can be
provided as part of the draid type as colon separated values. This
allows administrators to fully specify a layout for either performance
or capacity reasons. The supported options include:
zpool create <pool> \
draid[<parity>][:<data>d][:<children>c][:<spares>s] \
<vdevs...>
- draid[parity] - Parity level (default 1)
- draid[:<data>d] - Data devices per group (default 8)
- draid[:<children>c] - Expected number of child vdevs
- draid[:<spares>s] - Distributed hot spares (default 0)
Abbreviated example `zpool status` output for a 68 disk dRAID pool
with two distributed spares using special allocation classes.
```
pool: tank
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
slag7 ONLINE 0 0 0
draid2:8d:68c:2s-0 ONLINE 0 0 0
L0 ONLINE 0 0 0
L1 ONLINE 0 0 0
...
U25 ONLINE 0 0 0
U26 ONLINE 0 0 0
spare-53 ONLINE 0 0 0
U27 ONLINE 0 0 0
draid2-0-0 ONLINE 0 0 0
U28 ONLINE 0 0 0
U29 ONLINE 0 0 0
...
U42 ONLINE 0 0 0
U43 ONLINE 0 0 0
special
mirror-1 ONLINE 0 0 0
L5 ONLINE 0 0 0
U5 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
L6 ONLINE 0 0 0
U6 ONLINE 0 0 0
spares
draid2-0-0 INUSE currently in use
draid2-0-1 AVAIL
```
When adding test coverage for the new dRAID vdev type the following
options were added to the ztest command. These options are leverages
by zloop.sh to test a wide range of dRAID configurations.
-K draid|raidz|random - kind of RAID to test
-D <value> - dRAID data drives per group
-S <value> - dRAID distributed hot spares
-R <value> - RAID parity (raidz or dRAID)
The zpool_create, zpool_import, redundancy, replacement and fault
test groups have all been updated provide test coverage for the
dRAID feature.
Co-authored-by: Isaac Huang <he.huang@intel.com>
Co-authored-by: Mark Maybee <mmaybee@cray.com>
Co-authored-by: Don Brady <don.brady@delphix.com>
Co-authored-by: Matthew Ahrens <mahrens@delphix.com>
Co-authored-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Mark Maybee <mmaybee@cray.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #10102
2020-11-14 00:51:51 +03:00
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
|
OpenZFS 9075 - Improve ZFS pool import/load process and corrupted pool recovery
Some work has been done lately to improve the debugability of the ZFS pool
load (and import) process. This includes:
7638 Refactor spa_load_impl into several functions
8961 SPA load/import should tell us why it failed
7277 zdb should be able to print zfs_dbgmsg's
To iterate on top of that, there's a few changes that were made to make the
import process more resilient and crash free. One of the first tasks during the
pool load process is to parse a config provided from userland that describes
what devices the pool is composed of. A vdev tree is generated from that config,
and then all the vdevs are opened.
The Meta Object Set (MOS) of the pool is accessed, and several metadata objects
that are necessary to load the pool are read. The exact configuration of the
pool is also stored inside the MOS. Since the configuration provided from
userland is external and might not accurately describe the vdev tree
of the pool at the txg that is being loaded, it cannot be relied upon to safely
operate the pool. For that reason, the configuration in the MOS is read early
on. In the past, the two configurations were compared together and if there was
a mismatch then the load process was aborted and an error was returned.
The latter was a good way to ensure a pool does not get corrupted, however it
made the pool load process needlessly fragile in cases where the vdev
configuration changed or the userland configuration was outdated. Since the MOS
is stored in 3 copies, the configuration provided by userland doesn't have to be
perfect in order to read its contents. Hence, a new approach has been adopted:
The pool is first opened with the untrusted userland configuration just so that
the real configuration can be read from the MOS. The trusted MOS configuration
is then used to generate a new vdev tree and the pool is re-opened.
When the pool is opened with an untrusted configuration, writes are disabled
to avoid accidentally damaging it. During reads, some sanity checks are
performed on block pointers to see if each DVA points to a known vdev;
when the configuration is untrusted, instead of panicking the system if those
checks fail we simply avoid issuing reads to the invalid DVAs.
This new two-step pool load process now allows rewinding pools accross
vdev tree changes such as device replacement, addition, etc. Loading a pool
from an external config file in a clustering environment also becomes much
safer now since the pool will import even if the config is outdated and didn't,
for instance, register a recent device addition.
With this code in place, it became relatively easy to implement a
long-sought-after feature: the ability to import a pool with missing top level
(i.e. non-redundant) devices. Note that since this almost guarantees some loss
of data, this feature is for now restricted to a read-only import.
Porting notes (ZTS):
* Fix 'make dist' target in zpool_import
* The maximum path length allowed by tar is 99 characters. Several
of the new test cases exceeded this limit resulting in them not
being included in the tarball. Shorten the names slightly.
* Set/get tunables using accessor functions.
* Get last synced txg via the "zfs_txg_history" mechanism.
* Clear zinject handlers in cleanup for import_cache_device_replaced
and import_rewind_device_replaced in order that the zpool can be
exported if there is an error.
* Increase FILESIZE to 8G in zfs-test.sh to allow for a larger
ext4 file system to be created on ZFS_DISK2. Also, there's
no need to partition ZFS_DISK2 at all. The partitioning had
already been disabled for multipath devices. Among other things,
the partitioning steals some space from the ext4 file system,
makes it difficult to accurately calculate the paramters to
parted and can make some of the tests fail.
* Increase FS_SIZE and FILE_SIZE in the zpool_import test
configuration now that FILESIZE is larger.
* Write more data in order that device evacuation take lonnger in
a couple tests.
* Use mkdir -p to avoid errors when the directory already exists.
* Remove use of sudo in import_rewind_config_changed.
Authored by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Andrew Stormont <andyjstormont@gmail.com>
Approved by: Hans Rosenfeld <rosenfeld@grumpf.hope-2000.org>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://illumos.org/issues/9075
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/619c0123
Closes #7459
2016-07-22 17:39:36 +03:00
|
|
|
if (vd->vdev_not_present && !(flags & VDEV_CONFIG_MISSING))
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_NOT_PRESENT, 1);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
if (vd->vdev_isspare)
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_IS_SPARE, 1);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2023-06-14 18:01:17 +03:00
|
|
|
if (flags & VDEV_CONFIG_L2CACHE)
|
|
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_ASHIFT, vd->vdev_ashift);
|
|
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
if (!(flags & (VDEV_CONFIG_SPARE | VDEV_CONFIG_L2CACHE)) &&
|
|
|
|
|
vd == vd->vdev_top) {
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_METASLAB_ARRAY,
|
|
|
|
|
vd->vdev_ms_array);
|
|
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_METASLAB_SHIFT,
|
|
|
|
|
vd->vdev_ms_shift);
|
|
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_ASHIFT, vd->vdev_ashift);
|
|
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_ASIZE,
|
|
|
|
|
vd->vdev_asize);
|
|
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_IS_LOG, vd->vdev_islog);
|
2021-11-30 17:46:25 +03:00
|
|
|
if (vd->vdev_noalloc) {
|
|
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_NONALLOCATING,
|
|
|
|
|
vd->vdev_noalloc);
|
|
|
|
|
}
|
panic loop when removing slog device
There is a window in the slog removal code where a panic loop could
ensue if the system crashes during that operation. The original design
of slog removal did not persisted any state because the removal happened
synchronously. This was changed by a later commit which persisted the
vdev_removing flag and exposed this bug. If a slog removal is in
progress and happens to crash after persisting the vdev_removing flag to
the label but before the vdev is removed from the spa config, then the
pool will continue to panic on import. Here's a sample of the panic:
[ 134.387411] VERIFY0(0 == dmu_buf_hold_array(os, object, offset, size,
FALSE, FTAG, &numbufs, &dbp)) failed (0 == 22)
[ 134.393865] PANIC at dmu.c:1135:dmu_write()
[ 134.396035] Kernel panic - not syncing: VERIFY0(0 ==
dmu_buf_hold_array(os, object, offset, size, FALSE, FTAG, &numbufs,
&dbp)) failed (0 == 22)
[ 134.397857] CPU: 2 PID: 5914 Comm: txg_sync Kdump: loaded Tainted:
P OE 5.4.0-1100-dx2023020205-b3751f8c2-azure #106
[ 134.407938] Hardware name: Microsoft Corporation Virtual
Machine/Virtual Machine, BIOS 090008 12/07/2018
[ 134.407938] Call Trace:
[ 134.407938] dump_stack+0x57/0x6d
[ 134.407938] panic+0xfb/0x2d7
[ 134.407938] spl_panic+0xcf/0x102 [spl]
[ 134.407938] ? traverse_impl+0x1ca/0x420 [zfs]
[ 134.407938] ? dmu_object_alloc_impl+0x3b4/0x3c0 [zfs]
[ 134.407938] ? dnode_hold+0x1b/0x20 [zfs]
[ 134.407938] dmu_write+0xc3/0xd0 [zfs]
[ 134.407938] ? space_map_alloc+0x55/0x80 [zfs]
[ 134.407938] metaslab_sync+0x61a/0x830 [zfs]
[ 134.407938] ? queued_spin_unlock+0x9/0x10 [zfs]
[ 134.407938] vdev_sync+0x72/0x190 [zfs]
[ 134.407938] spa_sync_iterate_to_convergence+0x160/0x250 [zfs]
[ 134.407938] spa_sync+0x2f7/0x670 [zfs]
[ 134.407938] txg_sync_thread+0x22d/0x2d0 [zfs]
[ 134.407938] ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
[ 134.407938] thread_generic_wrapper+0x83/0xa0 [spl]
[ 134.407938] kthread+0x104/0x140
[ 134.407938] ? kasan_check_write.constprop.0+0x10/0x10 [spl]
[ 134.407938] ? kthread_park+0x90/0x90
[ 134.457802] ret_from_fork+0x1f/0x40
This change no longer persists the vdev_removing flag when removing slog
devices and also cleans up some code that was added which is not used.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Mark Maybee <mark.maybee@delphix.com>
Signed-off-by: George Wilson <gwilson@delphix.com>
Closes #14652
2023-03-24 20:27:07 +03:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Slog devices are removed synchronously so don't
|
|
|
|
|
* persist the vdev_removing flag to the label.
|
|
|
|
|
*/
|
|
|
|
|
if (vd->vdev_removing && !vd->vdev_islog) {
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_REMOVING,
|
|
|
|
|
vd->vdev_removing);
|
OpenZFS 7614, 9064 - zfs device evacuation/removal
OpenZFS 7614 - zfs device evacuation/removal
OpenZFS 9064 - remove_mirror should wait for device removal to complete
This project allows top-level vdevs to be removed from the storage pool
with "zpool remove", reducing the total amount of storage in the pool.
This operation copies all allocated regions of the device to be removed
onto other devices, recording the mapping from old to new location.
After the removal is complete, read and free operations to the removed
(now "indirect") vdev must be remapped and performed at the new location
on disk. The indirect mapping table is kept in memory whenever the pool
is loaded, so there is minimal performance overhead when doing operations
on the indirect vdev.
The size of the in-memory mapping table will be reduced when its entries
become "obsolete" because they are no longer used by any block pointers
in the pool. An entry becomes obsolete when all the blocks that use
it are freed. An entry can also become obsolete when all the snapshots
that reference it are deleted, and the block pointers that reference it
have been "remapped" in all filesystems/zvols (and clones). Whenever an
indirect block is written, all the block pointers in it will be "remapped"
to their new (concrete) locations if possible. This process can be
accelerated by using the "zfs remap" command to proactively rewrite all
indirect blocks that reference indirect (removed) vdevs.
Note that when a device is removed, we do not verify the checksum of
the data that is copied. This makes the process much faster, but if it
were used on redundant vdevs (i.e. mirror or raidz vdevs), it would be
possible to copy the wrong data, when we have the correct data on e.g.
the other side of the mirror.
At the moment, only mirrors and simple top-level vdevs can be removed
and no removal is allowed if any of the top-level vdevs are raidz.
Porting Notes:
* Avoid zero-sized kmem_alloc() in vdev_compact_children().
The device evacuation code adds a dependency that
vdev_compact_children() be able to properly empty the vdev_child
array by setting it to NULL and zeroing vdev_children. Under Linux,
kmem_alloc() and related functions return a sentinel pointer rather
than NULL for zero-sized allocations.
* Remove comment regarding "mpt" driver where zfs_remove_max_segment
is initialized to SPA_MAXBLOCKSIZE.
Change zfs_condense_indirect_commit_entry_delay_ticks to
zfs_condense_indirect_commit_entry_delay_ms for consistency with
most other tunables in which delays are specified in ms.
* ZTS changes:
Use set_tunable rather than mdb
Use zpool sync as appropriate
Use sync_pool instead of sync
Kill jobs during test_removal_with_operation to allow unmount/export
Don't add non-disk names such as "mirror" or "raidz" to $DISKS
Use $TEST_BASE_DIR instead of /tmp
Increase HZ from 100 to 1000 which is more common on Linux
removal_multiple_indirection.ksh
Reduce iterations in order to not time out on the code
coverage builders.
removal_resume_export:
Functionally, the test case is correct but there exists a race
where the kernel thread hasn't been fully started yet and is
not visible. Wait for up to 1 second for the removal thread
to be started before giving up on it. Also, increase the
amount of data copied in order that the removal not finish
before the export has a chance to fail.
* MMP compatibility, the concept of concrete versus non-concrete devices
has slightly changed the semantics of vdev_writeable(). Update
mmp_random_leaf_impl() accordingly.
* Updated dbuf_remap() to handle the org.zfsonlinux:large_dnode pool
feature which is not supported by OpenZFS.
* Added support for new vdev removal tracepoints.
* Test cases removal_with_zdb and removal_condense_export have been
intentionally disabled. When run manually they pass as intended,
but when running in the automated test environment they produce
unreliable results on the latest Fedora release.
They may work better once the upstream pool import refectoring is
merged into ZoL at which point they will be re-enabled.
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Alex Reece <alex@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Richard Laager <rlaager@wiktel.com>
Reviewed by: Tim Chase <tim@chase2k.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Garrett D'Amore <garrett@damore.org>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://www.illumos.org/issues/7614
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/f539f1eb
Closes #6900
2016-09-22 19:30:13 +03:00
|
|
|
}
|
2018-09-06 04:33:36 +03:00
|
|
|
|
|
|
|
|
/* zpool command expects alloc class data */
|
|
|
|
|
if (getstats && vd->vdev_alloc_bias != VDEV_BIAS_NONE) {
|
|
|
|
|
const char *bias = NULL;
|
|
|
|
|
|
|
|
|
|
switch (vd->vdev_alloc_bias) {
|
|
|
|
|
case VDEV_BIAS_LOG:
|
|
|
|
|
bias = VDEV_ALLOC_BIAS_LOG;
|
|
|
|
|
break;
|
|
|
|
|
case VDEV_BIAS_SPECIAL:
|
|
|
|
|
bias = VDEV_ALLOC_BIAS_SPECIAL;
|
|
|
|
|
break;
|
|
|
|
|
case VDEV_BIAS_DEDUP:
|
|
|
|
|
bias = VDEV_ALLOC_BIAS_DEDUP;
|
|
|
|
|
break;
|
|
|
|
|
default:
|
|
|
|
|
ASSERT3U(vd->vdev_alloc_bias, ==,
|
|
|
|
|
VDEV_BIAS_NONE);
|
|
|
|
|
}
|
|
|
|
|
fnvlist_add_string(nv, ZPOOL_CONFIG_ALLOCATION_BIAS,
|
|
|
|
|
bias);
|
|
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
Illumos #4101, #4102, #4103, #4105, #4106
4101 metaslab_debug should allow for fine-grained control
4102 space_maps should store more information about themselves
4103 space map object blocksize should be increased
4105 removing a mirrored log device results in a leaked object
4106 asynchronously load metaslab
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Sebastien Roy <seb@delphix.com>
Approved by: Garrett D'Amore <garrett@damore.org>
Prior to this patch, space_maps were preferred solely based on the
amount of free space left in each. Unfortunately, this heuristic didn't
contain any information about the make-up of that free space, which
meant we could keep preferring and loading a highly fragmented space map
that wouldn't actually have enough contiguous space to satisfy the
allocation; then unloading that space_map and repeating the process.
This change modifies the space_map's to store additional information
about the contiguous space in the space_map, so that we can use this
information to make a better decision about which space_map to load.
This requires reallocating all space_map objects to increase their
bonus buffer size sizes enough to fit the new metadata.
The above feature can be enabled via a new feature flag introduced by
this change: com.delphix:spacemap_histogram
In addition to the above, this patch allows the space_map block size to
be increase. Currently the block size is set to be 4K in size, which has
certain implications including the following:
* 4K sector devices will not see any compression benefit
* large space_maps require more metadata on-disk
* large space_maps require more time to load (typically random reads)
Now the space_map block size can adjust as needed up to the maximum size
set via the space_map_max_blksz variable.
A bug was fixed which resulted in potentially leaking an object when
removing a mirrored log device. The previous logic for vdev_remove() did
not deal with removing top-level vdevs that are interior vdevs (i.e.
mirror) correctly. The problem would occur when removing a mirrored log
device, and result in the DTL space map object being leaked; because
top-level vdevs don't have DTL space map objects associated with them.
References:
https://www.illumos.org/issues/4101
https://www.illumos.org/issues/4102
https://www.illumos.org/issues/4103
https://www.illumos.org/issues/4105
https://www.illumos.org/issues/4106
https://github.com/illumos/illumos-gate/commit/0713e23
Porting notes:
A handful of kmem_alloc() calls were converted to kmem_zalloc(). Also,
the KM_PUSHPAGE and TQ_PUSHPAGE flags were used as necessary.
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Prakash Surya <surya1@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2488
2013-10-02 01:25:53 +04:00
|
|
|
if (vd->vdev_dtl_sm != NULL) {
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_DTL,
|
Illumos #4101, #4102, #4103, #4105, #4106
4101 metaslab_debug should allow for fine-grained control
4102 space_maps should store more information about themselves
4103 space map object blocksize should be increased
4105 removing a mirrored log device results in a leaked object
4106 asynchronously load metaslab
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Sebastien Roy <seb@delphix.com>
Approved by: Garrett D'Amore <garrett@damore.org>
Prior to this patch, space_maps were preferred solely based on the
amount of free space left in each. Unfortunately, this heuristic didn't
contain any information about the make-up of that free space, which
meant we could keep preferring and loading a highly fragmented space map
that wouldn't actually have enough contiguous space to satisfy the
allocation; then unloading that space_map and repeating the process.
This change modifies the space_map's to store additional information
about the contiguous space in the space_map, so that we can use this
information to make a better decision about which space_map to load.
This requires reallocating all space_map objects to increase their
bonus buffer size sizes enough to fit the new metadata.
The above feature can be enabled via a new feature flag introduced by
this change: com.delphix:spacemap_histogram
In addition to the above, this patch allows the space_map block size to
be increase. Currently the block size is set to be 4K in size, which has
certain implications including the following:
* 4K sector devices will not see any compression benefit
* large space_maps require more metadata on-disk
* large space_maps require more time to load (typically random reads)
Now the space_map block size can adjust as needed up to the maximum size
set via the space_map_max_blksz variable.
A bug was fixed which resulted in potentially leaking an object when
removing a mirrored log device. The previous logic for vdev_remove() did
not deal with removing top-level vdevs that are interior vdevs (i.e.
mirror) correctly. The problem would occur when removing a mirrored log
device, and result in the DTL space map object being leaked; because
top-level vdevs don't have DTL space map objects associated with them.
References:
https://www.illumos.org/issues/4101
https://www.illumos.org/issues/4102
https://www.illumos.org/issues/4103
https://www.illumos.org/issues/4105
https://www.illumos.org/issues/4106
https://github.com/illumos/illumos-gate/commit/0713e23
Porting notes:
A handful of kmem_alloc() calls were converted to kmem_zalloc(). Also,
the KM_PUSHPAGE and TQ_PUSHPAGE flags were used as necessary.
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Prakash Surya <surya1@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2488
2013-10-02 01:25:53 +04:00
|
|
|
space_map_object(vd->vdev_dtl_sm));
|
|
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
|
OpenZFS 7614, 9064 - zfs device evacuation/removal
OpenZFS 7614 - zfs device evacuation/removal
OpenZFS 9064 - remove_mirror should wait for device removal to complete
This project allows top-level vdevs to be removed from the storage pool
with "zpool remove", reducing the total amount of storage in the pool.
This operation copies all allocated regions of the device to be removed
onto other devices, recording the mapping from old to new location.
After the removal is complete, read and free operations to the removed
(now "indirect") vdev must be remapped and performed at the new location
on disk. The indirect mapping table is kept in memory whenever the pool
is loaded, so there is minimal performance overhead when doing operations
on the indirect vdev.
The size of the in-memory mapping table will be reduced when its entries
become "obsolete" because they are no longer used by any block pointers
in the pool. An entry becomes obsolete when all the blocks that use
it are freed. An entry can also become obsolete when all the snapshots
that reference it are deleted, and the block pointers that reference it
have been "remapped" in all filesystems/zvols (and clones). Whenever an
indirect block is written, all the block pointers in it will be "remapped"
to their new (concrete) locations if possible. This process can be
accelerated by using the "zfs remap" command to proactively rewrite all
indirect blocks that reference indirect (removed) vdevs.
Note that when a device is removed, we do not verify the checksum of
the data that is copied. This makes the process much faster, but if it
were used on redundant vdevs (i.e. mirror or raidz vdevs), it would be
possible to copy the wrong data, when we have the correct data on e.g.
the other side of the mirror.
At the moment, only mirrors and simple top-level vdevs can be removed
and no removal is allowed if any of the top-level vdevs are raidz.
Porting Notes:
* Avoid zero-sized kmem_alloc() in vdev_compact_children().
The device evacuation code adds a dependency that
vdev_compact_children() be able to properly empty the vdev_child
array by setting it to NULL and zeroing vdev_children. Under Linux,
kmem_alloc() and related functions return a sentinel pointer rather
than NULL for zero-sized allocations.
* Remove comment regarding "mpt" driver where zfs_remove_max_segment
is initialized to SPA_MAXBLOCKSIZE.
Change zfs_condense_indirect_commit_entry_delay_ticks to
zfs_condense_indirect_commit_entry_delay_ms for consistency with
most other tunables in which delays are specified in ms.
* ZTS changes:
Use set_tunable rather than mdb
Use zpool sync as appropriate
Use sync_pool instead of sync
Kill jobs during test_removal_with_operation to allow unmount/export
Don't add non-disk names such as "mirror" or "raidz" to $DISKS
Use $TEST_BASE_DIR instead of /tmp
Increase HZ from 100 to 1000 which is more common on Linux
removal_multiple_indirection.ksh
Reduce iterations in order to not time out on the code
coverage builders.
removal_resume_export:
Functionally, the test case is correct but there exists a race
where the kernel thread hasn't been fully started yet and is
not visible. Wait for up to 1 second for the removal thread
to be started before giving up on it. Also, increase the
amount of data copied in order that the removal not finish
before the export has a chance to fail.
* MMP compatibility, the concept of concrete versus non-concrete devices
has slightly changed the semantics of vdev_writeable(). Update
mmp_random_leaf_impl() accordingly.
* Updated dbuf_remap() to handle the org.zfsonlinux:large_dnode pool
feature which is not supported by OpenZFS.
* Added support for new vdev removal tracepoints.
* Test cases removal_with_zdb and removal_condense_export have been
intentionally disabled. When run manually they pass as intended,
but when running in the automated test environment they produce
unreliable results on the latest Fedora release.
They may work better once the upstream pool import refectoring is
merged into ZoL at which point they will be re-enabled.
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Alex Reece <alex@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Richard Laager <rlaager@wiktel.com>
Reviewed by: Tim Chase <tim@chase2k.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Garrett D'Amore <garrett@damore.org>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://www.illumos.org/issues/7614
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/f539f1eb
Closes #6900
2016-09-22 19:30:13 +03:00
|
|
|
if (vic->vic_mapping_object != 0) {
|
|
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_INDIRECT_OBJECT,
|
|
|
|
|
vic->vic_mapping_object);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (vic->vic_births_object != 0) {
|
|
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_INDIRECT_BIRTHS,
|
|
|
|
|
vic->vic_births_object);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (vic->vic_prev_indirect_vdev != UINT64_MAX) {
|
|
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_PREV_INDIRECT_VDEV,
|
|
|
|
|
vic->vic_prev_indirect_vdev);
|
|
|
|
|
}
|
|
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
if (vd->vdev_crtxg)
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_CREATE_TXG, vd->vdev_crtxg);
|
2010-05-29 00:45:14 +04:00
|
|
|
|
2018-09-19 00:45:52 +03:00
|
|
|
if (vd->vdev_expansion_time)
|
|
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_EXPANSION_TIME,
|
|
|
|
|
vd->vdev_expansion_time);
|
|
|
|
|
|
2016-04-11 23:16:57 +03:00
|
|
|
if (flags & VDEV_CONFIG_MOS) {
|
|
|
|
|
if (vd->vdev_leaf_zap != 0) {
|
|
|
|
|
ASSERT(vd->vdev_ops->vdev_op_leaf);
|
|
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_VDEV_LEAF_ZAP,
|
|
|
|
|
vd->vdev_leaf_zap);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (vd->vdev_top_zap != 0) {
|
|
|
|
|
ASSERT(vd == vd->vdev_top);
|
|
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_VDEV_TOP_ZAP,
|
|
|
|
|
vd->vdev_top_zap);
|
|
|
|
|
}
|
2018-10-19 07:06:18 +03:00
|
|
|
|
2023-04-20 20:07:56 +03:00
|
|
|
if (vd->vdev_ops == &vdev_root_ops && vd->vdev_root_zap != 0 &&
|
|
|
|
|
spa_feature_is_active(vd->vdev_spa, SPA_FEATURE_AVZ_V2)) {
|
|
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_VDEV_ROOT_ZAP,
|
|
|
|
|
vd->vdev_root_zap);
|
|
|
|
|
}
|
|
|
|
|
|
2018-10-19 07:06:18 +03:00
|
|
|
if (vd->vdev_resilver_deferred) {
|
|
|
|
|
ASSERT(vd->vdev_ops->vdev_op_leaf);
|
|
|
|
|
ASSERT(spa->spa_resilver_deferred);
|
|
|
|
|
fnvlist_add_boolean(nv, ZPOOL_CONFIG_RESILVER_DEFER);
|
|
|
|
|
}
|
2016-04-11 23:16:57 +03:00
|
|
|
}
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
if (getstats) {
|
2016-02-29 21:05:23 +03:00
|
|
|
vdev_config_generate_stats(vd, nv);
|
2010-05-29 00:45:14 +04:00
|
|
|
|
2016-12-17 01:11:29 +03:00
|
|
|
root_vdev_actions_getprogress(vd, nv);
|
2020-07-03 21:05:50 +03:00
|
|
|
top_vdev_actions_getprogress(vd, nv);
|
OpenZFS 7614, 9064 - zfs device evacuation/removal
OpenZFS 7614 - zfs device evacuation/removal
OpenZFS 9064 - remove_mirror should wait for device removal to complete
This project allows top-level vdevs to be removed from the storage pool
with "zpool remove", reducing the total amount of storage in the pool.
This operation copies all allocated regions of the device to be removed
onto other devices, recording the mapping from old to new location.
After the removal is complete, read and free operations to the removed
(now "indirect") vdev must be remapped and performed at the new location
on disk. The indirect mapping table is kept in memory whenever the pool
is loaded, so there is minimal performance overhead when doing operations
on the indirect vdev.
The size of the in-memory mapping table will be reduced when its entries
become "obsolete" because they are no longer used by any block pointers
in the pool. An entry becomes obsolete when all the blocks that use
it are freed. An entry can also become obsolete when all the snapshots
that reference it are deleted, and the block pointers that reference it
have been "remapped" in all filesystems/zvols (and clones). Whenever an
indirect block is written, all the block pointers in it will be "remapped"
to their new (concrete) locations if possible. This process can be
accelerated by using the "zfs remap" command to proactively rewrite all
indirect blocks that reference indirect (removed) vdevs.
Note that when a device is removed, we do not verify the checksum of
the data that is copied. This makes the process much faster, but if it
were used on redundant vdevs (i.e. mirror or raidz vdevs), it would be
possible to copy the wrong data, when we have the correct data on e.g.
the other side of the mirror.
At the moment, only mirrors and simple top-level vdevs can be removed
and no removal is allowed if any of the top-level vdevs are raidz.
Porting Notes:
* Avoid zero-sized kmem_alloc() in vdev_compact_children().
The device evacuation code adds a dependency that
vdev_compact_children() be able to properly empty the vdev_child
array by setting it to NULL and zeroing vdev_children. Under Linux,
kmem_alloc() and related functions return a sentinel pointer rather
than NULL for zero-sized allocations.
* Remove comment regarding "mpt" driver where zfs_remove_max_segment
is initialized to SPA_MAXBLOCKSIZE.
Change zfs_condense_indirect_commit_entry_delay_ticks to
zfs_condense_indirect_commit_entry_delay_ms for consistency with
most other tunables in which delays are specified in ms.
* ZTS changes:
Use set_tunable rather than mdb
Use zpool sync as appropriate
Use sync_pool instead of sync
Kill jobs during test_removal_with_operation to allow unmount/export
Don't add non-disk names such as "mirror" or "raidz" to $DISKS
Use $TEST_BASE_DIR instead of /tmp
Increase HZ from 100 to 1000 which is more common on Linux
removal_multiple_indirection.ksh
Reduce iterations in order to not time out on the code
coverage builders.
removal_resume_export:
Functionally, the test case is correct but there exists a race
where the kernel thread hasn't been fully started yet and is
not visible. Wait for up to 1 second for the removal thread
to be started before giving up on it. Also, increase the
amount of data copied in order that the removal not finish
before the export has a chance to fail.
* MMP compatibility, the concept of concrete versus non-concrete devices
has slightly changed the semantics of vdev_writeable(). Update
mmp_random_leaf_impl() accordingly.
* Updated dbuf_remap() to handle the org.zfsonlinux:large_dnode pool
feature which is not supported by OpenZFS.
* Added support for new vdev removal tracepoints.
* Test cases removal_with_zdb and removal_condense_export have been
intentionally disabled. When run manually they pass as intended,
but when running in the automated test environment they produce
unreliable results on the latest Fedora release.
They may work better once the upstream pool import refectoring is
merged into ZoL at which point they will be re-enabled.
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Alex Reece <alex@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Richard Laager <rlaager@wiktel.com>
Reviewed by: Tim Chase <tim@chase2k.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Garrett D'Amore <garrett@damore.org>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://www.illumos.org/issues/7614
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/f539f1eb
Closes #6900
2016-09-22 19:30:13 +03:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Note: this can be called from open context
|
|
|
|
|
* (spa_get_stats()), so we need the rwlock to prevent
|
|
|
|
|
* the mapping from being changed by condensing.
|
|
|
|
|
*/
|
|
|
|
|
rw_enter(&vd->vdev_indirect_rwlock, RW_READER);
|
|
|
|
|
if (vd->vdev_indirect_mapping != NULL) {
|
|
|
|
|
ASSERT(vd->vdev_indirect_births != NULL);
|
|
|
|
|
vdev_indirect_mapping_t *vim =
|
|
|
|
|
vd->vdev_indirect_mapping;
|
|
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_INDIRECT_SIZE,
|
|
|
|
|
vdev_indirect_mapping_size(vim));
|
|
|
|
|
}
|
|
|
|
|
rw_exit(&vd->vdev_indirect_rwlock);
|
|
|
|
|
if (vd->vdev_mg != NULL &&
|
|
|
|
|
vd->vdev_mg->mg_fragmentation != ZFS_FRAG_INVALID) {
|
|
|
|
|
/*
|
|
|
|
|
* Compute approximately how much memory would be used
|
|
|
|
|
* for the indirect mapping if this device were to
|
|
|
|
|
* be removed.
|
|
|
|
|
*
|
|
|
|
|
* Note: If the frag metric is invalid, then not
|
|
|
|
|
* enough metaslabs have been converted to have
|
|
|
|
|
* histograms.
|
|
|
|
|
*/
|
|
|
|
|
uint64_t seg_count = 0;
|
OpenZFS 9486 - reduce memory used by device removal on fragmented pools
Device removal allocates a new location for each allocated segment on
the disk that's being removed. Each allocation results in one entry in
the mapping table, which maps from old location + length to new
location. When a fragmented disk is removed, this can result in a large
number of mapping entries, and thus a large amount of memory consumed by
the mapping table. In the worst real-world cases, we've seen around 1GB
of RAM per 1TB of storage removed.
We can improve on this situation by allocating larger segments, which
span across both allocated and free regions of the device being removed.
By including free regions in the allocation (and thus mapping), we
reduce the number of mapping entries. For example, if we have a 4K
allocation followed by 1K free and then 4K allocated, we would allocate
4+1+4 = 9KB, and then move the entire region (including allocated and
free parts). In this case we used one mapping where previously we would
have used two, but often the ratio is much higher (up to 20:1 in
real-world use). We then need to mark the regions that were free on the
removing device as free in the new locations, and also obsolete in the
mapping entry.
This method preserves the fragmentation of the removing device, rather
than consolidating its allocated space into a small number of chunks
where possible. But it results in drastic reduction of memory used by
the mapping table - around 20x in the most-fragmented cases.
In the most fragmented real-world cases, this reduces memory used by the
mapping from ~1GB to ~50MB of RAM per 1TB of storage removed. Less
fragmented cases will typically also see around 50-100MB of RAM per 1TB
of storage.
Porting notes:
* Add the following as module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* Document the following module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* zfs_condense_min_mapping_bytes
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://illumos.org/issues/9486
OpenZFS-commit: https://github.com/ahrens/illumos/commit/07152e142e44c
External-issue: DLPX-57962
Closes #7536
2018-02-27 02:33:55 +03:00
|
|
|
uint64_t to_alloc = vd->vdev_stat.vs_alloc;
|
OpenZFS 7614, 9064 - zfs device evacuation/removal
OpenZFS 7614 - zfs device evacuation/removal
OpenZFS 9064 - remove_mirror should wait for device removal to complete
This project allows top-level vdevs to be removed from the storage pool
with "zpool remove", reducing the total amount of storage in the pool.
This operation copies all allocated regions of the device to be removed
onto other devices, recording the mapping from old to new location.
After the removal is complete, read and free operations to the removed
(now "indirect") vdev must be remapped and performed at the new location
on disk. The indirect mapping table is kept in memory whenever the pool
is loaded, so there is minimal performance overhead when doing operations
on the indirect vdev.
The size of the in-memory mapping table will be reduced when its entries
become "obsolete" because they are no longer used by any block pointers
in the pool. An entry becomes obsolete when all the blocks that use
it are freed. An entry can also become obsolete when all the snapshots
that reference it are deleted, and the block pointers that reference it
have been "remapped" in all filesystems/zvols (and clones). Whenever an
indirect block is written, all the block pointers in it will be "remapped"
to their new (concrete) locations if possible. This process can be
accelerated by using the "zfs remap" command to proactively rewrite all
indirect blocks that reference indirect (removed) vdevs.
Note that when a device is removed, we do not verify the checksum of
the data that is copied. This makes the process much faster, but if it
were used on redundant vdevs (i.e. mirror or raidz vdevs), it would be
possible to copy the wrong data, when we have the correct data on e.g.
the other side of the mirror.
At the moment, only mirrors and simple top-level vdevs can be removed
and no removal is allowed if any of the top-level vdevs are raidz.
Porting Notes:
* Avoid zero-sized kmem_alloc() in vdev_compact_children().
The device evacuation code adds a dependency that
vdev_compact_children() be able to properly empty the vdev_child
array by setting it to NULL and zeroing vdev_children. Under Linux,
kmem_alloc() and related functions return a sentinel pointer rather
than NULL for zero-sized allocations.
* Remove comment regarding "mpt" driver where zfs_remove_max_segment
is initialized to SPA_MAXBLOCKSIZE.
Change zfs_condense_indirect_commit_entry_delay_ticks to
zfs_condense_indirect_commit_entry_delay_ms for consistency with
most other tunables in which delays are specified in ms.
* ZTS changes:
Use set_tunable rather than mdb
Use zpool sync as appropriate
Use sync_pool instead of sync
Kill jobs during test_removal_with_operation to allow unmount/export
Don't add non-disk names such as "mirror" or "raidz" to $DISKS
Use $TEST_BASE_DIR instead of /tmp
Increase HZ from 100 to 1000 which is more common on Linux
removal_multiple_indirection.ksh
Reduce iterations in order to not time out on the code
coverage builders.
removal_resume_export:
Functionally, the test case is correct but there exists a race
where the kernel thread hasn't been fully started yet and is
not visible. Wait for up to 1 second for the removal thread
to be started before giving up on it. Also, increase the
amount of data copied in order that the removal not finish
before the export has a chance to fail.
* MMP compatibility, the concept of concrete versus non-concrete devices
has slightly changed the semantics of vdev_writeable(). Update
mmp_random_leaf_impl() accordingly.
* Updated dbuf_remap() to handle the org.zfsonlinux:large_dnode pool
feature which is not supported by OpenZFS.
* Added support for new vdev removal tracepoints.
* Test cases removal_with_zdb and removal_condense_export have been
intentionally disabled. When run manually they pass as intended,
but when running in the automated test environment they produce
unreliable results on the latest Fedora release.
They may work better once the upstream pool import refectoring is
merged into ZoL at which point they will be re-enabled.
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Alex Reece <alex@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Richard Laager <rlaager@wiktel.com>
Reviewed by: Tim Chase <tim@chase2k.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Garrett D'Amore <garrett@damore.org>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://www.illumos.org/issues/7614
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/f539f1eb
Closes #6900
2016-09-22 19:30:13 +03:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* There are the same number of allocated segments
|
|
|
|
|
* as free segments, so we will have at least one
|
OpenZFS 9486 - reduce memory used by device removal on fragmented pools
Device removal allocates a new location for each allocated segment on
the disk that's being removed. Each allocation results in one entry in
the mapping table, which maps from old location + length to new
location. When a fragmented disk is removed, this can result in a large
number of mapping entries, and thus a large amount of memory consumed by
the mapping table. In the worst real-world cases, we've seen around 1GB
of RAM per 1TB of storage removed.
We can improve on this situation by allocating larger segments, which
span across both allocated and free regions of the device being removed.
By including free regions in the allocation (and thus mapping), we
reduce the number of mapping entries. For example, if we have a 4K
allocation followed by 1K free and then 4K allocated, we would allocate
4+1+4 = 9KB, and then move the entire region (including allocated and
free parts). In this case we used one mapping where previously we would
have used two, but often the ratio is much higher (up to 20:1 in
real-world use). We then need to mark the regions that were free on the
removing device as free in the new locations, and also obsolete in the
mapping entry.
This method preserves the fragmentation of the removing device, rather
than consolidating its allocated space into a small number of chunks
where possible. But it results in drastic reduction of memory used by
the mapping table - around 20x in the most-fragmented cases.
In the most fragmented real-world cases, this reduces memory used by the
mapping from ~1GB to ~50MB of RAM per 1TB of storage removed. Less
fragmented cases will typically also see around 50-100MB of RAM per 1TB
of storage.
Porting notes:
* Add the following as module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* Document the following module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* zfs_condense_min_mapping_bytes
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://illumos.org/issues/9486
OpenZFS-commit: https://github.com/ahrens/illumos/commit/07152e142e44c
External-issue: DLPX-57962
Closes #7536
2018-02-27 02:33:55 +03:00
|
|
|
* entry per free segment. However, small free
|
|
|
|
|
* segments (smaller than vdev_removal_max_span)
|
|
|
|
|
* will be combined with adjacent allocated segments
|
|
|
|
|
* as a single mapping.
|
OpenZFS 7614, 9064 - zfs device evacuation/removal
OpenZFS 7614 - zfs device evacuation/removal
OpenZFS 9064 - remove_mirror should wait for device removal to complete
This project allows top-level vdevs to be removed from the storage pool
with "zpool remove", reducing the total amount of storage in the pool.
This operation copies all allocated regions of the device to be removed
onto other devices, recording the mapping from old to new location.
After the removal is complete, read and free operations to the removed
(now "indirect") vdev must be remapped and performed at the new location
on disk. The indirect mapping table is kept in memory whenever the pool
is loaded, so there is minimal performance overhead when doing operations
on the indirect vdev.
The size of the in-memory mapping table will be reduced when its entries
become "obsolete" because they are no longer used by any block pointers
in the pool. An entry becomes obsolete when all the blocks that use
it are freed. An entry can also become obsolete when all the snapshots
that reference it are deleted, and the block pointers that reference it
have been "remapped" in all filesystems/zvols (and clones). Whenever an
indirect block is written, all the block pointers in it will be "remapped"
to their new (concrete) locations if possible. This process can be
accelerated by using the "zfs remap" command to proactively rewrite all
indirect blocks that reference indirect (removed) vdevs.
Note that when a device is removed, we do not verify the checksum of
the data that is copied. This makes the process much faster, but if it
were used on redundant vdevs (i.e. mirror or raidz vdevs), it would be
possible to copy the wrong data, when we have the correct data on e.g.
the other side of the mirror.
At the moment, only mirrors and simple top-level vdevs can be removed
and no removal is allowed if any of the top-level vdevs are raidz.
Porting Notes:
* Avoid zero-sized kmem_alloc() in vdev_compact_children().
The device evacuation code adds a dependency that
vdev_compact_children() be able to properly empty the vdev_child
array by setting it to NULL and zeroing vdev_children. Under Linux,
kmem_alloc() and related functions return a sentinel pointer rather
than NULL for zero-sized allocations.
* Remove comment regarding "mpt" driver where zfs_remove_max_segment
is initialized to SPA_MAXBLOCKSIZE.
Change zfs_condense_indirect_commit_entry_delay_ticks to
zfs_condense_indirect_commit_entry_delay_ms for consistency with
most other tunables in which delays are specified in ms.
* ZTS changes:
Use set_tunable rather than mdb
Use zpool sync as appropriate
Use sync_pool instead of sync
Kill jobs during test_removal_with_operation to allow unmount/export
Don't add non-disk names such as "mirror" or "raidz" to $DISKS
Use $TEST_BASE_DIR instead of /tmp
Increase HZ from 100 to 1000 which is more common on Linux
removal_multiple_indirection.ksh
Reduce iterations in order to not time out on the code
coverage builders.
removal_resume_export:
Functionally, the test case is correct but there exists a race
where the kernel thread hasn't been fully started yet and is
not visible. Wait for up to 1 second for the removal thread
to be started before giving up on it. Also, increase the
amount of data copied in order that the removal not finish
before the export has a chance to fail.
* MMP compatibility, the concept of concrete versus non-concrete devices
has slightly changed the semantics of vdev_writeable(). Update
mmp_random_leaf_impl() accordingly.
* Updated dbuf_remap() to handle the org.zfsonlinux:large_dnode pool
feature which is not supported by OpenZFS.
* Added support for new vdev removal tracepoints.
* Test cases removal_with_zdb and removal_condense_export have been
intentionally disabled. When run manually they pass as intended,
but when running in the automated test environment they produce
unreliable results on the latest Fedora release.
They may work better once the upstream pool import refectoring is
merged into ZoL at which point they will be re-enabled.
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Alex Reece <alex@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Richard Laager <rlaager@wiktel.com>
Reviewed by: Tim Chase <tim@chase2k.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Garrett D'Amore <garrett@damore.org>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://www.illumos.org/issues/7614
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/f539f1eb
Closes #6900
2016-09-22 19:30:13 +03:00
|
|
|
*/
|
|
|
|
|
for (int i = 0; i < RANGE_TREE_HISTOGRAM_SIZE; i++) {
|
2020-10-09 02:37:27 +03:00
|
|
|
if (i + 1 < highbit64(vdev_removal_max_span)
|
|
|
|
|
- 1) {
|
OpenZFS 9486 - reduce memory used by device removal on fragmented pools
Device removal allocates a new location for each allocated segment on
the disk that's being removed. Each allocation results in one entry in
the mapping table, which maps from old location + length to new
location. When a fragmented disk is removed, this can result in a large
number of mapping entries, and thus a large amount of memory consumed by
the mapping table. In the worst real-world cases, we've seen around 1GB
of RAM per 1TB of storage removed.
We can improve on this situation by allocating larger segments, which
span across both allocated and free regions of the device being removed.
By including free regions in the allocation (and thus mapping), we
reduce the number of mapping entries. For example, if we have a 4K
allocation followed by 1K free and then 4K allocated, we would allocate
4+1+4 = 9KB, and then move the entire region (including allocated and
free parts). In this case we used one mapping where previously we would
have used two, but often the ratio is much higher (up to 20:1 in
real-world use). We then need to mark the regions that were free on the
removing device as free in the new locations, and also obsolete in the
mapping entry.
This method preserves the fragmentation of the removing device, rather
than consolidating its allocated space into a small number of chunks
where possible. But it results in drastic reduction of memory used by
the mapping table - around 20x in the most-fragmented cases.
In the most fragmented real-world cases, this reduces memory used by the
mapping from ~1GB to ~50MB of RAM per 1TB of storage removed. Less
fragmented cases will typically also see around 50-100MB of RAM per 1TB
of storage.
Porting notes:
* Add the following as module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* Document the following module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* zfs_condense_min_mapping_bytes
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://illumos.org/issues/9486
OpenZFS-commit: https://github.com/ahrens/illumos/commit/07152e142e44c
External-issue: DLPX-57962
Closes #7536
2018-02-27 02:33:55 +03:00
|
|
|
to_alloc +=
|
|
|
|
|
vd->vdev_mg->mg_histogram[i] <<
|
|
|
|
|
(i + 1);
|
|
|
|
|
} else {
|
|
|
|
|
seg_count +=
|
|
|
|
|
vd->vdev_mg->mg_histogram[i];
|
|
|
|
|
}
|
OpenZFS 7614, 9064 - zfs device evacuation/removal
OpenZFS 7614 - zfs device evacuation/removal
OpenZFS 9064 - remove_mirror should wait for device removal to complete
This project allows top-level vdevs to be removed from the storage pool
with "zpool remove", reducing the total amount of storage in the pool.
This operation copies all allocated regions of the device to be removed
onto other devices, recording the mapping from old to new location.
After the removal is complete, read and free operations to the removed
(now "indirect") vdev must be remapped and performed at the new location
on disk. The indirect mapping table is kept in memory whenever the pool
is loaded, so there is minimal performance overhead when doing operations
on the indirect vdev.
The size of the in-memory mapping table will be reduced when its entries
become "obsolete" because they are no longer used by any block pointers
in the pool. An entry becomes obsolete when all the blocks that use
it are freed. An entry can also become obsolete when all the snapshots
that reference it are deleted, and the block pointers that reference it
have been "remapped" in all filesystems/zvols (and clones). Whenever an
indirect block is written, all the block pointers in it will be "remapped"
to their new (concrete) locations if possible. This process can be
accelerated by using the "zfs remap" command to proactively rewrite all
indirect blocks that reference indirect (removed) vdevs.
Note that when a device is removed, we do not verify the checksum of
the data that is copied. This makes the process much faster, but if it
were used on redundant vdevs (i.e. mirror or raidz vdevs), it would be
possible to copy the wrong data, when we have the correct data on e.g.
the other side of the mirror.
At the moment, only mirrors and simple top-level vdevs can be removed
and no removal is allowed if any of the top-level vdevs are raidz.
Porting Notes:
* Avoid zero-sized kmem_alloc() in vdev_compact_children().
The device evacuation code adds a dependency that
vdev_compact_children() be able to properly empty the vdev_child
array by setting it to NULL and zeroing vdev_children. Under Linux,
kmem_alloc() and related functions return a sentinel pointer rather
than NULL for zero-sized allocations.
* Remove comment regarding "mpt" driver where zfs_remove_max_segment
is initialized to SPA_MAXBLOCKSIZE.
Change zfs_condense_indirect_commit_entry_delay_ticks to
zfs_condense_indirect_commit_entry_delay_ms for consistency with
most other tunables in which delays are specified in ms.
* ZTS changes:
Use set_tunable rather than mdb
Use zpool sync as appropriate
Use sync_pool instead of sync
Kill jobs during test_removal_with_operation to allow unmount/export
Don't add non-disk names such as "mirror" or "raidz" to $DISKS
Use $TEST_BASE_DIR instead of /tmp
Increase HZ from 100 to 1000 which is more common on Linux
removal_multiple_indirection.ksh
Reduce iterations in order to not time out on the code
coverage builders.
removal_resume_export:
Functionally, the test case is correct but there exists a race
where the kernel thread hasn't been fully started yet and is
not visible. Wait for up to 1 second for the removal thread
to be started before giving up on it. Also, increase the
amount of data copied in order that the removal not finish
before the export has a chance to fail.
* MMP compatibility, the concept of concrete versus non-concrete devices
has slightly changed the semantics of vdev_writeable(). Update
mmp_random_leaf_impl() accordingly.
* Updated dbuf_remap() to handle the org.zfsonlinux:large_dnode pool
feature which is not supported by OpenZFS.
* Added support for new vdev removal tracepoints.
* Test cases removal_with_zdb and removal_condense_export have been
intentionally disabled. When run manually they pass as intended,
but when running in the automated test environment they produce
unreliable results on the latest Fedora release.
They may work better once the upstream pool import refectoring is
merged into ZoL at which point they will be re-enabled.
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Alex Reece <alex@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Richard Laager <rlaager@wiktel.com>
Reviewed by: Tim Chase <tim@chase2k.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Garrett D'Amore <garrett@damore.org>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://www.illumos.org/issues/7614
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/f539f1eb
Closes #6900
2016-09-22 19:30:13 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
OpenZFS 9486 - reduce memory used by device removal on fragmented pools
Device removal allocates a new location for each allocated segment on
the disk that's being removed. Each allocation results in one entry in
the mapping table, which maps from old location + length to new
location. When a fragmented disk is removed, this can result in a large
number of mapping entries, and thus a large amount of memory consumed by
the mapping table. In the worst real-world cases, we've seen around 1GB
of RAM per 1TB of storage removed.
We can improve on this situation by allocating larger segments, which
span across both allocated and free regions of the device being removed.
By including free regions in the allocation (and thus mapping), we
reduce the number of mapping entries. For example, if we have a 4K
allocation followed by 1K free and then 4K allocated, we would allocate
4+1+4 = 9KB, and then move the entire region (including allocated and
free parts). In this case we used one mapping where previously we would
have used two, but often the ratio is much higher (up to 20:1 in
real-world use). We then need to mark the regions that were free on the
removing device as free in the new locations, and also obsolete in the
mapping entry.
This method preserves the fragmentation of the removing device, rather
than consolidating its allocated space into a small number of chunks
where possible. But it results in drastic reduction of memory used by
the mapping table - around 20x in the most-fragmented cases.
In the most fragmented real-world cases, this reduces memory used by the
mapping from ~1GB to ~50MB of RAM per 1TB of storage removed. Less
fragmented cases will typically also see around 50-100MB of RAM per 1TB
of storage.
Porting notes:
* Add the following as module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* Document the following module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* zfs_condense_min_mapping_bytes
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://illumos.org/issues/9486
OpenZFS-commit: https://github.com/ahrens/illumos/commit/07152e142e44c
External-issue: DLPX-57962
Closes #7536
2018-02-27 02:33:55 +03:00
|
|
|
* The maximum length of a mapping is
|
|
|
|
|
* zfs_remove_max_segment, so we need at least one entry
|
|
|
|
|
* per zfs_remove_max_segment of allocated data.
|
OpenZFS 7614, 9064 - zfs device evacuation/removal
OpenZFS 7614 - zfs device evacuation/removal
OpenZFS 9064 - remove_mirror should wait for device removal to complete
This project allows top-level vdevs to be removed from the storage pool
with "zpool remove", reducing the total amount of storage in the pool.
This operation copies all allocated regions of the device to be removed
onto other devices, recording the mapping from old to new location.
After the removal is complete, read and free operations to the removed
(now "indirect") vdev must be remapped and performed at the new location
on disk. The indirect mapping table is kept in memory whenever the pool
is loaded, so there is minimal performance overhead when doing operations
on the indirect vdev.
The size of the in-memory mapping table will be reduced when its entries
become "obsolete" because they are no longer used by any block pointers
in the pool. An entry becomes obsolete when all the blocks that use
it are freed. An entry can also become obsolete when all the snapshots
that reference it are deleted, and the block pointers that reference it
have been "remapped" in all filesystems/zvols (and clones). Whenever an
indirect block is written, all the block pointers in it will be "remapped"
to their new (concrete) locations if possible. This process can be
accelerated by using the "zfs remap" command to proactively rewrite all
indirect blocks that reference indirect (removed) vdevs.
Note that when a device is removed, we do not verify the checksum of
the data that is copied. This makes the process much faster, but if it
were used on redundant vdevs (i.e. mirror or raidz vdevs), it would be
possible to copy the wrong data, when we have the correct data on e.g.
the other side of the mirror.
At the moment, only mirrors and simple top-level vdevs can be removed
and no removal is allowed if any of the top-level vdevs are raidz.
Porting Notes:
* Avoid zero-sized kmem_alloc() in vdev_compact_children().
The device evacuation code adds a dependency that
vdev_compact_children() be able to properly empty the vdev_child
array by setting it to NULL and zeroing vdev_children. Under Linux,
kmem_alloc() and related functions return a sentinel pointer rather
than NULL for zero-sized allocations.
* Remove comment regarding "mpt" driver where zfs_remove_max_segment
is initialized to SPA_MAXBLOCKSIZE.
Change zfs_condense_indirect_commit_entry_delay_ticks to
zfs_condense_indirect_commit_entry_delay_ms for consistency with
most other tunables in which delays are specified in ms.
* ZTS changes:
Use set_tunable rather than mdb
Use zpool sync as appropriate
Use sync_pool instead of sync
Kill jobs during test_removal_with_operation to allow unmount/export
Don't add non-disk names such as "mirror" or "raidz" to $DISKS
Use $TEST_BASE_DIR instead of /tmp
Increase HZ from 100 to 1000 which is more common on Linux
removal_multiple_indirection.ksh
Reduce iterations in order to not time out on the code
coverage builders.
removal_resume_export:
Functionally, the test case is correct but there exists a race
where the kernel thread hasn't been fully started yet and is
not visible. Wait for up to 1 second for the removal thread
to be started before giving up on it. Also, increase the
amount of data copied in order that the removal not finish
before the export has a chance to fail.
* MMP compatibility, the concept of concrete versus non-concrete devices
has slightly changed the semantics of vdev_writeable(). Update
mmp_random_leaf_impl() accordingly.
* Updated dbuf_remap() to handle the org.zfsonlinux:large_dnode pool
feature which is not supported by OpenZFS.
* Added support for new vdev removal tracepoints.
* Test cases removal_with_zdb and removal_condense_export have been
intentionally disabled. When run manually they pass as intended,
but when running in the automated test environment they produce
unreliable results on the latest Fedora release.
They may work better once the upstream pool import refectoring is
merged into ZoL at which point they will be re-enabled.
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Alex Reece <alex@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Richard Laager <rlaager@wiktel.com>
Reviewed by: Tim Chase <tim@chase2k.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Garrett D'Amore <garrett@damore.org>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://www.illumos.org/issues/7614
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/f539f1eb
Closes #6900
2016-09-22 19:30:13 +03:00
|
|
|
*/
|
2019-06-13 23:12:39 +03:00
|
|
|
seg_count += to_alloc / spa_remove_max_segment(spa);
|
OpenZFS 7614, 9064 - zfs device evacuation/removal
OpenZFS 7614 - zfs device evacuation/removal
OpenZFS 9064 - remove_mirror should wait for device removal to complete
This project allows top-level vdevs to be removed from the storage pool
with "zpool remove", reducing the total amount of storage in the pool.
This operation copies all allocated regions of the device to be removed
onto other devices, recording the mapping from old to new location.
After the removal is complete, read and free operations to the removed
(now "indirect") vdev must be remapped and performed at the new location
on disk. The indirect mapping table is kept in memory whenever the pool
is loaded, so there is minimal performance overhead when doing operations
on the indirect vdev.
The size of the in-memory mapping table will be reduced when its entries
become "obsolete" because they are no longer used by any block pointers
in the pool. An entry becomes obsolete when all the blocks that use
it are freed. An entry can also become obsolete when all the snapshots
that reference it are deleted, and the block pointers that reference it
have been "remapped" in all filesystems/zvols (and clones). Whenever an
indirect block is written, all the block pointers in it will be "remapped"
to their new (concrete) locations if possible. This process can be
accelerated by using the "zfs remap" command to proactively rewrite all
indirect blocks that reference indirect (removed) vdevs.
Note that when a device is removed, we do not verify the checksum of
the data that is copied. This makes the process much faster, but if it
were used on redundant vdevs (i.e. mirror or raidz vdevs), it would be
possible to copy the wrong data, when we have the correct data on e.g.
the other side of the mirror.
At the moment, only mirrors and simple top-level vdevs can be removed
and no removal is allowed if any of the top-level vdevs are raidz.
Porting Notes:
* Avoid zero-sized kmem_alloc() in vdev_compact_children().
The device evacuation code adds a dependency that
vdev_compact_children() be able to properly empty the vdev_child
array by setting it to NULL and zeroing vdev_children. Under Linux,
kmem_alloc() and related functions return a sentinel pointer rather
than NULL for zero-sized allocations.
* Remove comment regarding "mpt" driver where zfs_remove_max_segment
is initialized to SPA_MAXBLOCKSIZE.
Change zfs_condense_indirect_commit_entry_delay_ticks to
zfs_condense_indirect_commit_entry_delay_ms for consistency with
most other tunables in which delays are specified in ms.
* ZTS changes:
Use set_tunable rather than mdb
Use zpool sync as appropriate
Use sync_pool instead of sync
Kill jobs during test_removal_with_operation to allow unmount/export
Don't add non-disk names such as "mirror" or "raidz" to $DISKS
Use $TEST_BASE_DIR instead of /tmp
Increase HZ from 100 to 1000 which is more common on Linux
removal_multiple_indirection.ksh
Reduce iterations in order to not time out on the code
coverage builders.
removal_resume_export:
Functionally, the test case is correct but there exists a race
where the kernel thread hasn't been fully started yet and is
not visible. Wait for up to 1 second for the removal thread
to be started before giving up on it. Also, increase the
amount of data copied in order that the removal not finish
before the export has a chance to fail.
* MMP compatibility, the concept of concrete versus non-concrete devices
has slightly changed the semantics of vdev_writeable(). Update
mmp_random_leaf_impl() accordingly.
* Updated dbuf_remap() to handle the org.zfsonlinux:large_dnode pool
feature which is not supported by OpenZFS.
* Added support for new vdev removal tracepoints.
* Test cases removal_with_zdb and removal_condense_export have been
intentionally disabled. When run manually they pass as intended,
but when running in the automated test environment they produce
unreliable results on the latest Fedora release.
They may work better once the upstream pool import refectoring is
merged into ZoL at which point they will be re-enabled.
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Alex Reece <alex@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Richard Laager <rlaager@wiktel.com>
Reviewed by: Tim Chase <tim@chase2k.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Garrett D'Amore <garrett@damore.org>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://www.illumos.org/issues/7614
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/f539f1eb
Closes #6900
2016-09-22 19:30:13 +03:00
|
|
|
|
|
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_INDIRECT_SIZE,
|
|
|
|
|
seg_count *
|
|
|
|
|
sizeof (vdev_indirect_mapping_entry_phys_t));
|
|
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (!vd->vdev_ops->vdev_op_leaf) {
|
|
|
|
|
nvlist_t **child;
|
panic loop when removing slog device
There is a window in the slog removal code where a panic loop could
ensue if the system crashes during that operation. The original design
of slog removal did not persisted any state because the removal happened
synchronously. This was changed by a later commit which persisted the
vdev_removing flag and exposed this bug. If a slog removal is in
progress and happens to crash after persisting the vdev_removing flag to
the label but before the vdev is removed from the spa config, then the
pool will continue to panic on import. Here's a sample of the panic:
[ 134.387411] VERIFY0(0 == dmu_buf_hold_array(os, object, offset, size,
FALSE, FTAG, &numbufs, &dbp)) failed (0 == 22)
[ 134.393865] PANIC at dmu.c:1135:dmu_write()
[ 134.396035] Kernel panic - not syncing: VERIFY0(0 ==
dmu_buf_hold_array(os, object, offset, size, FALSE, FTAG, &numbufs,
&dbp)) failed (0 == 22)
[ 134.397857] CPU: 2 PID: 5914 Comm: txg_sync Kdump: loaded Tainted:
P OE 5.4.0-1100-dx2023020205-b3751f8c2-azure #106
[ 134.407938] Hardware name: Microsoft Corporation Virtual
Machine/Virtual Machine, BIOS 090008 12/07/2018
[ 134.407938] Call Trace:
[ 134.407938] dump_stack+0x57/0x6d
[ 134.407938] panic+0xfb/0x2d7
[ 134.407938] spl_panic+0xcf/0x102 [spl]
[ 134.407938] ? traverse_impl+0x1ca/0x420 [zfs]
[ 134.407938] ? dmu_object_alloc_impl+0x3b4/0x3c0 [zfs]
[ 134.407938] ? dnode_hold+0x1b/0x20 [zfs]
[ 134.407938] dmu_write+0xc3/0xd0 [zfs]
[ 134.407938] ? space_map_alloc+0x55/0x80 [zfs]
[ 134.407938] metaslab_sync+0x61a/0x830 [zfs]
[ 134.407938] ? queued_spin_unlock+0x9/0x10 [zfs]
[ 134.407938] vdev_sync+0x72/0x190 [zfs]
[ 134.407938] spa_sync_iterate_to_convergence+0x160/0x250 [zfs]
[ 134.407938] spa_sync+0x2f7/0x670 [zfs]
[ 134.407938] txg_sync_thread+0x22d/0x2d0 [zfs]
[ 134.407938] ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
[ 134.407938] thread_generic_wrapper+0x83/0xa0 [spl]
[ 134.407938] kthread+0x104/0x140
[ 134.407938] ? kasan_check_write.constprop.0+0x10/0x10 [spl]
[ 134.407938] ? kthread_park+0x90/0x90
[ 134.457802] ret_from_fork+0x1f/0x40
This change no longer persists the vdev_removing flag when removing slog
devices and also cleans up some code that was added which is not used.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Mark Maybee <mark.maybee@delphix.com>
Signed-off-by: George Wilson <gwilson@delphix.com>
Closes #14652
2023-03-24 20:27:07 +03:00
|
|
|
uint64_t c;
|
2010-05-29 00:45:14 +04:00
|
|
|
|
|
|
|
|
ASSERT(!vd->vdev_ishole);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
child = kmem_alloc(vd->vdev_children * sizeof (nvlist_t *),
|
2014-11-21 03:09:39 +03:00
|
|
|
KM_SLEEP);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
panic loop when removing slog device
There is a window in the slog removal code where a panic loop could
ensue if the system crashes during that operation. The original design
of slog removal did not persisted any state because the removal happened
synchronously. This was changed by a later commit which persisted the
vdev_removing flag and exposed this bug. If a slog removal is in
progress and happens to crash after persisting the vdev_removing flag to
the label but before the vdev is removed from the spa config, then the
pool will continue to panic on import. Here's a sample of the panic:
[ 134.387411] VERIFY0(0 == dmu_buf_hold_array(os, object, offset, size,
FALSE, FTAG, &numbufs, &dbp)) failed (0 == 22)
[ 134.393865] PANIC at dmu.c:1135:dmu_write()
[ 134.396035] Kernel panic - not syncing: VERIFY0(0 ==
dmu_buf_hold_array(os, object, offset, size, FALSE, FTAG, &numbufs,
&dbp)) failed (0 == 22)
[ 134.397857] CPU: 2 PID: 5914 Comm: txg_sync Kdump: loaded Tainted:
P OE 5.4.0-1100-dx2023020205-b3751f8c2-azure #106
[ 134.407938] Hardware name: Microsoft Corporation Virtual
Machine/Virtual Machine, BIOS 090008 12/07/2018
[ 134.407938] Call Trace:
[ 134.407938] dump_stack+0x57/0x6d
[ 134.407938] panic+0xfb/0x2d7
[ 134.407938] spl_panic+0xcf/0x102 [spl]
[ 134.407938] ? traverse_impl+0x1ca/0x420 [zfs]
[ 134.407938] ? dmu_object_alloc_impl+0x3b4/0x3c0 [zfs]
[ 134.407938] ? dnode_hold+0x1b/0x20 [zfs]
[ 134.407938] dmu_write+0xc3/0xd0 [zfs]
[ 134.407938] ? space_map_alloc+0x55/0x80 [zfs]
[ 134.407938] metaslab_sync+0x61a/0x830 [zfs]
[ 134.407938] ? queued_spin_unlock+0x9/0x10 [zfs]
[ 134.407938] vdev_sync+0x72/0x190 [zfs]
[ 134.407938] spa_sync_iterate_to_convergence+0x160/0x250 [zfs]
[ 134.407938] spa_sync+0x2f7/0x670 [zfs]
[ 134.407938] txg_sync_thread+0x22d/0x2d0 [zfs]
[ 134.407938] ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
[ 134.407938] thread_generic_wrapper+0x83/0xa0 [spl]
[ 134.407938] kthread+0x104/0x140
[ 134.407938] ? kasan_check_write.constprop.0+0x10/0x10 [spl]
[ 134.407938] ? kthread_park+0x90/0x90
[ 134.457802] ret_from_fork+0x1f/0x40
This change no longer persists the vdev_removing flag when removing slog
devices and also cleans up some code that was added which is not used.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Mark Maybee <mark.maybee@delphix.com>
Signed-off-by: George Wilson <gwilson@delphix.com>
Closes #14652
2023-03-24 20:27:07 +03:00
|
|
|
for (c = 0; c < vd->vdev_children; c++) {
|
|
|
|
|
child[c] = vdev_config_generate(spa, vd->vdev_child[c],
|
2010-05-29 00:45:14 +04:00
|
|
|
getstats, flags);
|
|
|
|
|
}
|
|
|
|
|
|
panic loop when removing slog device
There is a window in the slog removal code where a panic loop could
ensue if the system crashes during that operation. The original design
of slog removal did not persisted any state because the removal happened
synchronously. This was changed by a later commit which persisted the
vdev_removing flag and exposed this bug. If a slog removal is in
progress and happens to crash after persisting the vdev_removing flag to
the label but before the vdev is removed from the spa config, then the
pool will continue to panic on import. Here's a sample of the panic:
[ 134.387411] VERIFY0(0 == dmu_buf_hold_array(os, object, offset, size,
FALSE, FTAG, &numbufs, &dbp)) failed (0 == 22)
[ 134.393865] PANIC at dmu.c:1135:dmu_write()
[ 134.396035] Kernel panic - not syncing: VERIFY0(0 ==
dmu_buf_hold_array(os, object, offset, size, FALSE, FTAG, &numbufs,
&dbp)) failed (0 == 22)
[ 134.397857] CPU: 2 PID: 5914 Comm: txg_sync Kdump: loaded Tainted:
P OE 5.4.0-1100-dx2023020205-b3751f8c2-azure #106
[ 134.407938] Hardware name: Microsoft Corporation Virtual
Machine/Virtual Machine, BIOS 090008 12/07/2018
[ 134.407938] Call Trace:
[ 134.407938] dump_stack+0x57/0x6d
[ 134.407938] panic+0xfb/0x2d7
[ 134.407938] spl_panic+0xcf/0x102 [spl]
[ 134.407938] ? traverse_impl+0x1ca/0x420 [zfs]
[ 134.407938] ? dmu_object_alloc_impl+0x3b4/0x3c0 [zfs]
[ 134.407938] ? dnode_hold+0x1b/0x20 [zfs]
[ 134.407938] dmu_write+0xc3/0xd0 [zfs]
[ 134.407938] ? space_map_alloc+0x55/0x80 [zfs]
[ 134.407938] metaslab_sync+0x61a/0x830 [zfs]
[ 134.407938] ? queued_spin_unlock+0x9/0x10 [zfs]
[ 134.407938] vdev_sync+0x72/0x190 [zfs]
[ 134.407938] spa_sync_iterate_to_convergence+0x160/0x250 [zfs]
[ 134.407938] spa_sync+0x2f7/0x670 [zfs]
[ 134.407938] txg_sync_thread+0x22d/0x2d0 [zfs]
[ 134.407938] ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
[ 134.407938] thread_generic_wrapper+0x83/0xa0 [spl]
[ 134.407938] kthread+0x104/0x140
[ 134.407938] ? kasan_check_write.constprop.0+0x10/0x10 [spl]
[ 134.407938] ? kthread_park+0x90/0x90
[ 134.457802] ret_from_fork+0x1f/0x40
This change no longer persists the vdev_removing flag when removing slog
devices and also cleans up some code that was added which is not used.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Mark Maybee <mark.maybee@delphix.com>
Signed-off-by: George Wilson <gwilson@delphix.com>
Closes #14652
2023-03-24 20:27:07 +03:00
|
|
|
fnvlist_add_nvlist_array(nv, ZPOOL_CONFIG_CHILDREN,
|
|
|
|
|
(const nvlist_t * const *)child, vd->vdev_children);
|
2010-05-29 00:45:14 +04:00
|
|
|
|
panic loop when removing slog device
There is a window in the slog removal code where a panic loop could
ensue if the system crashes during that operation. The original design
of slog removal did not persisted any state because the removal happened
synchronously. This was changed by a later commit which persisted the
vdev_removing flag and exposed this bug. If a slog removal is in
progress and happens to crash after persisting the vdev_removing flag to
the label but before the vdev is removed from the spa config, then the
pool will continue to panic on import. Here's a sample of the panic:
[ 134.387411] VERIFY0(0 == dmu_buf_hold_array(os, object, offset, size,
FALSE, FTAG, &numbufs, &dbp)) failed (0 == 22)
[ 134.393865] PANIC at dmu.c:1135:dmu_write()
[ 134.396035] Kernel panic - not syncing: VERIFY0(0 ==
dmu_buf_hold_array(os, object, offset, size, FALSE, FTAG, &numbufs,
&dbp)) failed (0 == 22)
[ 134.397857] CPU: 2 PID: 5914 Comm: txg_sync Kdump: loaded Tainted:
P OE 5.4.0-1100-dx2023020205-b3751f8c2-azure #106
[ 134.407938] Hardware name: Microsoft Corporation Virtual
Machine/Virtual Machine, BIOS 090008 12/07/2018
[ 134.407938] Call Trace:
[ 134.407938] dump_stack+0x57/0x6d
[ 134.407938] panic+0xfb/0x2d7
[ 134.407938] spl_panic+0xcf/0x102 [spl]
[ 134.407938] ? traverse_impl+0x1ca/0x420 [zfs]
[ 134.407938] ? dmu_object_alloc_impl+0x3b4/0x3c0 [zfs]
[ 134.407938] ? dnode_hold+0x1b/0x20 [zfs]
[ 134.407938] dmu_write+0xc3/0xd0 [zfs]
[ 134.407938] ? space_map_alloc+0x55/0x80 [zfs]
[ 134.407938] metaslab_sync+0x61a/0x830 [zfs]
[ 134.407938] ? queued_spin_unlock+0x9/0x10 [zfs]
[ 134.407938] vdev_sync+0x72/0x190 [zfs]
[ 134.407938] spa_sync_iterate_to_convergence+0x160/0x250 [zfs]
[ 134.407938] spa_sync+0x2f7/0x670 [zfs]
[ 134.407938] txg_sync_thread+0x22d/0x2d0 [zfs]
[ 134.407938] ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
[ 134.407938] thread_generic_wrapper+0x83/0xa0 [spl]
[ 134.407938] kthread+0x104/0x140
[ 134.407938] ? kasan_check_write.constprop.0+0x10/0x10 [spl]
[ 134.407938] ? kthread_park+0x90/0x90
[ 134.457802] ret_from_fork+0x1f/0x40
This change no longer persists the vdev_removing flag when removing slog
devices and also cleans up some code that was added which is not used.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Mark Maybee <mark.maybee@delphix.com>
Signed-off-by: George Wilson <gwilson@delphix.com>
Closes #14652
2023-03-24 20:27:07 +03:00
|
|
|
for (c = 0; c < vd->vdev_children; c++)
|
2008-11-20 23:01:55 +03:00
|
|
|
nvlist_free(child[c]);
|
|
|
|
|
|
|
|
|
|
kmem_free(child, vd->vdev_children * sizeof (nvlist_t *));
|
|
|
|
|
|
|
|
|
|
} else {
|
2010-05-29 00:45:14 +04:00
|
|
|
const char *aux = NULL;
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
if (vd->vdev_offline && !vd->vdev_tmpoffline)
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_OFFLINE, B_TRUE);
|
|
|
|
|
if (vd->vdev_resilver_txg != 0)
|
|
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_RESILVER_TXG,
|
|
|
|
|
vd->vdev_resilver_txg);
|
2020-07-03 21:05:50 +03:00
|
|
|
if (vd->vdev_rebuild_txg != 0)
|
|
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_REBUILD_TXG,
|
|
|
|
|
vd->vdev_rebuild_txg);
|
2008-11-20 23:01:55 +03:00
|
|
|
if (vd->vdev_faulted)
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_FAULTED, B_TRUE);
|
2008-11-20 23:01:55 +03:00
|
|
|
if (vd->vdev_degraded)
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_DEGRADED, B_TRUE);
|
2008-11-20 23:01:55 +03:00
|
|
|
if (vd->vdev_removed)
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_REMOVED, B_TRUE);
|
2008-11-20 23:01:55 +03:00
|
|
|
if (vd->vdev_unspare)
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_UNSPARE, B_TRUE);
|
2010-05-29 00:45:14 +04:00
|
|
|
if (vd->vdev_ishole)
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_IS_HOLE, B_TRUE);
|
2010-05-29 00:45:14 +04:00
|
|
|
|
2017-05-19 22:30:16 +03:00
|
|
|
/* Set the reason why we're FAULTED/DEGRADED. */
|
2010-05-29 00:45:14 +04:00
|
|
|
switch (vd->vdev_stat.vs_aux) {
|
|
|
|
|
case VDEV_AUX_ERR_EXCEEDED:
|
|
|
|
|
aux = "err_exceeded";
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
case VDEV_AUX_EXTERNAL:
|
|
|
|
|
aux = "external";
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
|
2017-05-19 22:30:16 +03:00
|
|
|
if (aux != NULL && !vd->vdev_tmpoffline) {
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_string(nv, ZPOOL_CONFIG_AUX_STATE, aux);
|
2017-05-19 22:30:16 +03:00
|
|
|
} else {
|
|
|
|
|
/*
|
|
|
|
|
* We're healthy - clear any previous AUX_STATE values.
|
|
|
|
|
*/
|
|
|
|
|
if (nvlist_exists(nv, ZPOOL_CONFIG_AUX_STATE))
|
|
|
|
|
nvlist_remove_all(nv, ZPOOL_CONFIG_AUX_STATE);
|
|
|
|
|
}
|
2010-05-29 00:45:14 +04:00
|
|
|
|
|
|
|
|
if (vd->vdev_splitting && vd->vdev_orig_guid != 0LL) {
|
2013-08-08 00:16:22 +04:00
|
|
|
fnvlist_add_uint64(nv, ZPOOL_CONFIG_ORIG_GUID,
|
|
|
|
|
vd->vdev_orig_guid);
|
2010-05-29 00:45:14 +04:00
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return (nv);
|
|
|
|
|
}
|
|
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
/*
|
|
|
|
|
* Generate a view of the top-level vdevs. If we currently have holes
|
|
|
|
|
* in the namespace, then generate an array which contains a list of holey
|
|
|
|
|
* vdevs. Additionally, add the number of top-level children that currently
|
|
|
|
|
* exist.
|
|
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
vdev_top_config_generate(spa_t *spa, nvlist_t *config)
|
|
|
|
|
{
|
|
|
|
|
vdev_t *rvd = spa->spa_root_vdev;
|
|
|
|
|
uint64_t *array;
|
|
|
|
|
uint_t c, idx;
|
|
|
|
|
|
2014-11-21 03:09:39 +03:00
|
|
|
array = kmem_alloc(rvd->vdev_children * sizeof (uint64_t), KM_SLEEP);
|
2010-05-29 00:45:14 +04:00
|
|
|
|
|
|
|
|
for (c = 0, idx = 0; c < rvd->vdev_children; c++) {
|
|
|
|
|
vdev_t *tvd = rvd->vdev_child[c];
|
|
|
|
|
|
OpenZFS 7614, 9064 - zfs device evacuation/removal
OpenZFS 7614 - zfs device evacuation/removal
OpenZFS 9064 - remove_mirror should wait for device removal to complete
This project allows top-level vdevs to be removed from the storage pool
with "zpool remove", reducing the total amount of storage in the pool.
This operation copies all allocated regions of the device to be removed
onto other devices, recording the mapping from old to new location.
After the removal is complete, read and free operations to the removed
(now "indirect") vdev must be remapped and performed at the new location
on disk. The indirect mapping table is kept in memory whenever the pool
is loaded, so there is minimal performance overhead when doing operations
on the indirect vdev.
The size of the in-memory mapping table will be reduced when its entries
become "obsolete" because they are no longer used by any block pointers
in the pool. An entry becomes obsolete when all the blocks that use
it are freed. An entry can also become obsolete when all the snapshots
that reference it are deleted, and the block pointers that reference it
have been "remapped" in all filesystems/zvols (and clones). Whenever an
indirect block is written, all the block pointers in it will be "remapped"
to their new (concrete) locations if possible. This process can be
accelerated by using the "zfs remap" command to proactively rewrite all
indirect blocks that reference indirect (removed) vdevs.
Note that when a device is removed, we do not verify the checksum of
the data that is copied. This makes the process much faster, but if it
were used on redundant vdevs (i.e. mirror or raidz vdevs), it would be
possible to copy the wrong data, when we have the correct data on e.g.
the other side of the mirror.
At the moment, only mirrors and simple top-level vdevs can be removed
and no removal is allowed if any of the top-level vdevs are raidz.
Porting Notes:
* Avoid zero-sized kmem_alloc() in vdev_compact_children().
The device evacuation code adds a dependency that
vdev_compact_children() be able to properly empty the vdev_child
array by setting it to NULL and zeroing vdev_children. Under Linux,
kmem_alloc() and related functions return a sentinel pointer rather
than NULL for zero-sized allocations.
* Remove comment regarding "mpt" driver where zfs_remove_max_segment
is initialized to SPA_MAXBLOCKSIZE.
Change zfs_condense_indirect_commit_entry_delay_ticks to
zfs_condense_indirect_commit_entry_delay_ms for consistency with
most other tunables in which delays are specified in ms.
* ZTS changes:
Use set_tunable rather than mdb
Use zpool sync as appropriate
Use sync_pool instead of sync
Kill jobs during test_removal_with_operation to allow unmount/export
Don't add non-disk names such as "mirror" or "raidz" to $DISKS
Use $TEST_BASE_DIR instead of /tmp
Increase HZ from 100 to 1000 which is more common on Linux
removal_multiple_indirection.ksh
Reduce iterations in order to not time out on the code
coverage builders.
removal_resume_export:
Functionally, the test case is correct but there exists a race
where the kernel thread hasn't been fully started yet and is
not visible. Wait for up to 1 second for the removal thread
to be started before giving up on it. Also, increase the
amount of data copied in order that the removal not finish
before the export has a chance to fail.
* MMP compatibility, the concept of concrete versus non-concrete devices
has slightly changed the semantics of vdev_writeable(). Update
mmp_random_leaf_impl() accordingly.
* Updated dbuf_remap() to handle the org.zfsonlinux:large_dnode pool
feature which is not supported by OpenZFS.
* Added support for new vdev removal tracepoints.
* Test cases removal_with_zdb and removal_condense_export have been
intentionally disabled. When run manually they pass as intended,
but when running in the automated test environment they produce
unreliable results on the latest Fedora release.
They may work better once the upstream pool import refectoring is
merged into ZoL at which point they will be re-enabled.
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Alex Reece <alex@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Richard Laager <rlaager@wiktel.com>
Reviewed by: Tim Chase <tim@chase2k.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Garrett D'Amore <garrett@damore.org>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://www.illumos.org/issues/7614
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/f539f1eb
Closes #6900
2016-09-22 19:30:13 +03:00
|
|
|
if (tvd->vdev_ishole) {
|
2010-05-29 00:45:14 +04:00
|
|
|
array[idx++] = c;
|
OpenZFS 7614, 9064 - zfs device evacuation/removal
OpenZFS 7614 - zfs device evacuation/removal
OpenZFS 9064 - remove_mirror should wait for device removal to complete
This project allows top-level vdevs to be removed from the storage pool
with "zpool remove", reducing the total amount of storage in the pool.
This operation copies all allocated regions of the device to be removed
onto other devices, recording the mapping from old to new location.
After the removal is complete, read and free operations to the removed
(now "indirect") vdev must be remapped and performed at the new location
on disk. The indirect mapping table is kept in memory whenever the pool
is loaded, so there is minimal performance overhead when doing operations
on the indirect vdev.
The size of the in-memory mapping table will be reduced when its entries
become "obsolete" because they are no longer used by any block pointers
in the pool. An entry becomes obsolete when all the blocks that use
it are freed. An entry can also become obsolete when all the snapshots
that reference it are deleted, and the block pointers that reference it
have been "remapped" in all filesystems/zvols (and clones). Whenever an
indirect block is written, all the block pointers in it will be "remapped"
to their new (concrete) locations if possible. This process can be
accelerated by using the "zfs remap" command to proactively rewrite all
indirect blocks that reference indirect (removed) vdevs.
Note that when a device is removed, we do not verify the checksum of
the data that is copied. This makes the process much faster, but if it
were used on redundant vdevs (i.e. mirror or raidz vdevs), it would be
possible to copy the wrong data, when we have the correct data on e.g.
the other side of the mirror.
At the moment, only mirrors and simple top-level vdevs can be removed
and no removal is allowed if any of the top-level vdevs are raidz.
Porting Notes:
* Avoid zero-sized kmem_alloc() in vdev_compact_children().
The device evacuation code adds a dependency that
vdev_compact_children() be able to properly empty the vdev_child
array by setting it to NULL and zeroing vdev_children. Under Linux,
kmem_alloc() and related functions return a sentinel pointer rather
than NULL for zero-sized allocations.
* Remove comment regarding "mpt" driver where zfs_remove_max_segment
is initialized to SPA_MAXBLOCKSIZE.
Change zfs_condense_indirect_commit_entry_delay_ticks to
zfs_condense_indirect_commit_entry_delay_ms for consistency with
most other tunables in which delays are specified in ms.
* ZTS changes:
Use set_tunable rather than mdb
Use zpool sync as appropriate
Use sync_pool instead of sync
Kill jobs during test_removal_with_operation to allow unmount/export
Don't add non-disk names such as "mirror" or "raidz" to $DISKS
Use $TEST_BASE_DIR instead of /tmp
Increase HZ from 100 to 1000 which is more common on Linux
removal_multiple_indirection.ksh
Reduce iterations in order to not time out on the code
coverage builders.
removal_resume_export:
Functionally, the test case is correct but there exists a race
where the kernel thread hasn't been fully started yet and is
not visible. Wait for up to 1 second for the removal thread
to be started before giving up on it. Also, increase the
amount of data copied in order that the removal not finish
before the export has a chance to fail.
* MMP compatibility, the concept of concrete versus non-concrete devices
has slightly changed the semantics of vdev_writeable(). Update
mmp_random_leaf_impl() accordingly.
* Updated dbuf_remap() to handle the org.zfsonlinux:large_dnode pool
feature which is not supported by OpenZFS.
* Added support for new vdev removal tracepoints.
* Test cases removal_with_zdb and removal_condense_export have been
intentionally disabled. When run manually they pass as intended,
but when running in the automated test environment they produce
unreliable results on the latest Fedora release.
They may work better once the upstream pool import refectoring is
merged into ZoL at which point they will be re-enabled.
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Alex Reece <alex@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Richard Laager <rlaager@wiktel.com>
Reviewed by: Tim Chase <tim@chase2k.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Garrett D'Amore <garrett@damore.org>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://www.illumos.org/issues/7614
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/f539f1eb
Closes #6900
2016-09-22 19:30:13 +03:00
|
|
|
}
|
2010-05-29 00:45:14 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (idx) {
|
|
|
|
|
VERIFY(nvlist_add_uint64_array(config, ZPOOL_CONFIG_HOLE_ARRAY,
|
|
|
|
|
array, idx) == 0);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
VERIFY(nvlist_add_uint64(config, ZPOOL_CONFIG_VDEV_CHILDREN,
|
|
|
|
|
rvd->vdev_children) == 0);
|
|
|
|
|
|
|
|
|
|
kmem_free(array, rvd->vdev_children * sizeof (uint64_t));
|
|
|
|
|
}
|
|
|
|
|
|
2012-12-14 03:24:15 +04:00
|
|
|
/*
|
2012-12-15 00:38:04 +04:00
|
|
|
* Returns the configuration from the label of the given vdev. For vdevs
|
|
|
|
|
* which don't have a txg value stored on their label (i.e. spares/cache)
|
|
|
|
|
* or have not been completely initialized (txg = 0) just return
|
|
|
|
|
* the configuration from the first valid label we find. Otherwise,
|
|
|
|
|
* find the most up-to-date label that does not exceed the specified
|
|
|
|
|
* 'txg' value.
|
2012-12-14 03:24:15 +04:00
|
|
|
*/
|
2008-11-20 23:01:55 +03:00
|
|
|
nvlist_t *
|
2012-12-15 00:38:04 +04:00
|
|
|
vdev_label_read_config(vdev_t *vd, uint64_t txg)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
|
|
|
|
spa_t *spa = vd->vdev_spa;
|
|
|
|
|
nvlist_t *config = NULL;
|
2021-01-13 00:59:56 +03:00
|
|
|
vdev_phys_t *vp[VDEV_LABELS];
|
|
|
|
|
abd_t *vp_abd[VDEV_LABELS];
|
|
|
|
|
zio_t *zio[VDEV_LABELS];
|
2012-12-15 00:38:04 +04:00
|
|
|
uint64_t best_txg = 0;
|
2016-09-14 18:01:40 +03:00
|
|
|
uint64_t label_txg = 0;
|
2012-12-15 00:38:04 +04:00
|
|
|
int error = 0;
|
2009-07-03 02:44:48 +04:00
|
|
|
int flags = ZIO_FLAG_CONFIG_WRITER | ZIO_FLAG_CANFAIL |
|
|
|
|
|
ZIO_FLAG_SPECULATIVE;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2021-01-13 01:25:52 +03:00
|
|
|
ASSERT(vd->vdev_validate_thread == curthread ||
|
|
|
|
|
spa_config_held(spa, SCL_STATE_ALL, RW_WRITER) == SCL_STATE_ALL);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
if (!vdev_readable(vd))
|
|
|
|
|
return (NULL);
|
|
|
|
|
|
Distributed Spare (dRAID) Feature
This patch adds a new top-level vdev type called dRAID, which stands
for Distributed parity RAID. This pool configuration allows all dRAID
vdevs to participate when rebuilding to a distributed hot spare device.
This can substantially reduce the total time required to restore full
parity to pool with a failed device.
A dRAID pool can be created using the new top-level `draid` type.
Like `raidz`, the desired redundancy is specified after the type:
`draid[1,2,3]`. No additional information is required to create the
pool and reasonable default values will be chosen based on the number
of child vdevs in the dRAID vdev.
zpool create <pool> draid[1,2,3] <vdevs...>
Unlike raidz, additional optional dRAID configuration values can be
provided as part of the draid type as colon separated values. This
allows administrators to fully specify a layout for either performance
or capacity reasons. The supported options include:
zpool create <pool> \
draid[<parity>][:<data>d][:<children>c][:<spares>s] \
<vdevs...>
- draid[parity] - Parity level (default 1)
- draid[:<data>d] - Data devices per group (default 8)
- draid[:<children>c] - Expected number of child vdevs
- draid[:<spares>s] - Distributed hot spares (default 0)
Abbreviated example `zpool status` output for a 68 disk dRAID pool
with two distributed spares using special allocation classes.
```
pool: tank
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
slag7 ONLINE 0 0 0
draid2:8d:68c:2s-0 ONLINE 0 0 0
L0 ONLINE 0 0 0
L1 ONLINE 0 0 0
...
U25 ONLINE 0 0 0
U26 ONLINE 0 0 0
spare-53 ONLINE 0 0 0
U27 ONLINE 0 0 0
draid2-0-0 ONLINE 0 0 0
U28 ONLINE 0 0 0
U29 ONLINE 0 0 0
...
U42 ONLINE 0 0 0
U43 ONLINE 0 0 0
special
mirror-1 ONLINE 0 0 0
L5 ONLINE 0 0 0
U5 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
L6 ONLINE 0 0 0
U6 ONLINE 0 0 0
spares
draid2-0-0 INUSE currently in use
draid2-0-1 AVAIL
```
When adding test coverage for the new dRAID vdev type the following
options were added to the ztest command. These options are leverages
by zloop.sh to test a wide range of dRAID configurations.
-K draid|raidz|random - kind of RAID to test
-D <value> - dRAID data drives per group
-S <value> - dRAID distributed hot spares
-R <value> - RAID parity (raidz or dRAID)
The zpool_create, zpool_import, redundancy, replacement and fault
test groups have all been updated provide test coverage for the
dRAID feature.
Co-authored-by: Isaac Huang <he.huang@intel.com>
Co-authored-by: Mark Maybee <mmaybee@cray.com>
Co-authored-by: Don Brady <don.brady@delphix.com>
Co-authored-by: Matthew Ahrens <mahrens@delphix.com>
Co-authored-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Mark Maybee <mmaybee@cray.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #10102
2020-11-14 00:51:51 +03:00
|
|
|
/*
|
|
|
|
|
* The label for a dRAID distributed spare is not stored on disk.
|
|
|
|
|
* Instead it is generated when needed which allows us to bypass
|
|
|
|
|
* the pipeline when reading the config from the label.
|
|
|
|
|
*/
|
|
|
|
|
if (vd->vdev_ops == &vdev_draid_spare_ops)
|
|
|
|
|
return (vdev_draid_read_config_spare(vd));
|
|
|
|
|
|
2021-01-13 00:59:56 +03:00
|
|
|
for (int l = 0; l < VDEV_LABELS; l++) {
|
|
|
|
|
vp_abd[l] = abd_alloc_linear(sizeof (vdev_phys_t), B_TRUE);
|
|
|
|
|
vp[l] = abd_to_buf(vp_abd[l]);
|
|
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2009-07-03 02:44:48 +04:00
|
|
|
retry:
|
2017-11-04 23:25:13 +03:00
|
|
|
for (int l = 0; l < VDEV_LABELS; l++) {
|
2021-01-13 00:59:56 +03:00
|
|
|
zio[l] = zio_root(spa, NULL, NULL, flags);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2021-01-13 00:59:56 +03:00
|
|
|
vdev_label_read(zio[l], vd, l, vp_abd[l],
|
|
|
|
|
offsetof(vdev_label_t, vl_vdev_phys), sizeof (vdev_phys_t),
|
|
|
|
|
NULL, NULL, flags);
|
|
|
|
|
}
|
|
|
|
|
for (int l = 0; l < VDEV_LABELS; l++) {
|
|
|
|
|
nvlist_t *label = NULL;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2021-01-13 00:59:56 +03:00
|
|
|
if (zio_wait(zio[l]) == 0 &&
|
|
|
|
|
nvlist_unpack(vp[l]->vp_nvlist, sizeof (vp[l]->vp_nvlist),
|
2012-12-15 00:38:04 +04:00
|
|
|
&label, 0) == 0) {
|
|
|
|
|
/*
|
|
|
|
|
* Auxiliary vdevs won't have txg values in their
|
|
|
|
|
* labels and newly added vdevs may not have been
|
|
|
|
|
* completely initialized so just return the
|
|
|
|
|
* configuration from the first valid label we
|
|
|
|
|
* encounter.
|
|
|
|
|
*/
|
|
|
|
|
error = nvlist_lookup_uint64(label,
|
|
|
|
|
ZPOOL_CONFIG_POOL_TXG, &label_txg);
|
|
|
|
|
if ((error || label_txg == 0) && !config) {
|
|
|
|
|
config = label;
|
2021-01-13 00:59:56 +03:00
|
|
|
for (l++; l < VDEV_LABELS; l++)
|
|
|
|
|
zio_wait(zio[l]);
|
2012-12-15 00:38:04 +04:00
|
|
|
break;
|
|
|
|
|
} else if (label_txg <= txg && label_txg > best_txg) {
|
|
|
|
|
best_txg = label_txg;
|
|
|
|
|
nvlist_free(config);
|
|
|
|
|
config = fnvlist_dup(label);
|
|
|
|
|
}
|
|
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2012-12-15 00:38:04 +04:00
|
|
|
if (label != NULL) {
|
|
|
|
|
nvlist_free(label);
|
|
|
|
|
label = NULL;
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2009-07-03 02:44:48 +04:00
|
|
|
if (config == NULL && !(flags & ZIO_FLAG_TRYHARD)) {
|
|
|
|
|
flags |= ZIO_FLAG_TRYHARD;
|
|
|
|
|
goto retry;
|
|
|
|
|
}
|
|
|
|
|
|
2016-09-14 18:01:40 +03:00
|
|
|
/*
|
|
|
|
|
* We found a valid label but it didn't pass txg restrictions.
|
|
|
|
|
*/
|
|
|
|
|
if (config == NULL && label_txg != 0) {
|
|
|
|
|
vdev_dbgmsg(vd, "label discarded as txg is too large "
|
|
|
|
|
"(%llu > %llu)", (u_longlong_t)label_txg,
|
|
|
|
|
(u_longlong_t)txg);
|
|
|
|
|
}
|
|
|
|
|
|
2021-01-13 00:59:56 +03:00
|
|
|
for (int l = 0; l < VDEV_LABELS; l++) {
|
|
|
|
|
abd_free(vp_abd[l]);
|
|
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
return (config);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Determine if a device is in use. The 'spare_guid' parameter will be filled
|
|
|
|
|
* in with the device guid if this spare is active elsewhere on the system.
|
|
|
|
|
*/
|
|
|
|
|
static boolean_t
|
|
|
|
|
vdev_inuse(vdev_t *vd, uint64_t crtxg, vdev_labeltype_t reason,
|
|
|
|
|
uint64_t *spare_guid, uint64_t *l2cache_guid)
|
|
|
|
|
{
|
|
|
|
|
spa_t *spa = vd->vdev_spa;
|
|
|
|
|
uint64_t state, pool_guid, device_guid, txg, spare_pool;
|
|
|
|
|
uint64_t vdtxg = 0;
|
|
|
|
|
nvlist_t *label;
|
|
|
|
|
|
|
|
|
|
if (spare_guid)
|
|
|
|
|
*spare_guid = 0ULL;
|
|
|
|
|
if (l2cache_guid)
|
|
|
|
|
*l2cache_guid = 0ULL;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Read the label, if any, and perform some basic sanity checks.
|
|
|
|
|
*/
|
2012-12-15 00:38:04 +04:00
|
|
|
if ((label = vdev_label_read_config(vd, -1ULL)) == NULL)
|
2008-11-20 23:01:55 +03:00
|
|
|
return (B_FALSE);
|
|
|
|
|
|
|
|
|
|
(void) nvlist_lookup_uint64(label, ZPOOL_CONFIG_CREATE_TXG,
|
|
|
|
|
&vdtxg);
|
|
|
|
|
|
|
|
|
|
if (nvlist_lookup_uint64(label, ZPOOL_CONFIG_POOL_STATE,
|
|
|
|
|
&state) != 0 ||
|
|
|
|
|
nvlist_lookup_uint64(label, ZPOOL_CONFIG_GUID,
|
|
|
|
|
&device_guid) != 0) {
|
|
|
|
|
nvlist_free(label);
|
|
|
|
|
return (B_FALSE);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (state != POOL_STATE_SPARE && state != POOL_STATE_L2CACHE &&
|
|
|
|
|
(nvlist_lookup_uint64(label, ZPOOL_CONFIG_POOL_GUID,
|
|
|
|
|
&pool_guid) != 0 ||
|
|
|
|
|
nvlist_lookup_uint64(label, ZPOOL_CONFIG_POOL_TXG,
|
|
|
|
|
&txg) != 0)) {
|
|
|
|
|
nvlist_free(label);
|
|
|
|
|
return (B_FALSE);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
nvlist_free(label);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Check to see if this device indeed belongs to the pool it claims to
|
|
|
|
|
* be a part of. The only way this is allowed is if the device is a hot
|
|
|
|
|
* spare (which we check for later on).
|
|
|
|
|
*/
|
|
|
|
|
if (state != POOL_STATE_SPARE && state != POOL_STATE_L2CACHE &&
|
|
|
|
|
!spa_guid_exists(pool_guid, device_guid) &&
|
2008-12-03 23:09:06 +03:00
|
|
|
!spa_spare_exists(device_guid, NULL, NULL) &&
|
2008-11-20 23:01:55 +03:00
|
|
|
!spa_l2cache_exists(device_guid, NULL))
|
|
|
|
|
return (B_FALSE);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If the transaction group is zero, then this an initialized (but
|
|
|
|
|
* unused) label. This is only an error if the create transaction
|
|
|
|
|
* on-disk is the same as the one we're using now, in which case the
|
|
|
|
|
* user has attempted to add the same vdev multiple times in the same
|
|
|
|
|
* transaction.
|
|
|
|
|
*/
|
|
|
|
|
if (state != POOL_STATE_SPARE && state != POOL_STATE_L2CACHE &&
|
|
|
|
|
txg == 0 && vdtxg == crtxg)
|
|
|
|
|
return (B_TRUE);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Check to see if this is a spare device. We do an explicit check for
|
|
|
|
|
* spa_has_spare() here because it may be on our pending list of spares
|
2021-11-11 22:54:15 +03:00
|
|
|
* to add.
|
2008-11-20 23:01:55 +03:00
|
|
|
*/
|
2008-12-03 23:09:06 +03:00
|
|
|
if (spa_spare_exists(device_guid, &spare_pool, NULL) ||
|
2008-11-20 23:01:55 +03:00
|
|
|
spa_has_spare(spa, device_guid)) {
|
|
|
|
|
if (spare_guid)
|
|
|
|
|
*spare_guid = device_guid;
|
|
|
|
|
|
|
|
|
|
switch (reason) {
|
|
|
|
|
case VDEV_LABEL_CREATE:
|
|
|
|
|
return (B_TRUE);
|
|
|
|
|
|
|
|
|
|
case VDEV_LABEL_REPLACE:
|
|
|
|
|
return (!spa_has_spare(spa, device_guid) ||
|
|
|
|
|
spare_pool != 0ULL);
|
|
|
|
|
|
|
|
|
|
case VDEV_LABEL_SPARE:
|
|
|
|
|
return (spa_has_spare(spa, device_guid));
|
2010-08-26 20:52:41 +04:00
|
|
|
default:
|
|
|
|
|
break;
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Check to see if this is an l2cache device.
|
|
|
|
|
*/
|
2021-11-11 22:54:15 +03:00
|
|
|
if (spa_l2cache_exists(device_guid, NULL) ||
|
|
|
|
|
spa_has_l2cache(spa, device_guid)) {
|
|
|
|
|
if (l2cache_guid)
|
|
|
|
|
*l2cache_guid = device_guid;
|
|
|
|
|
|
|
|
|
|
switch (reason) {
|
|
|
|
|
case VDEV_LABEL_CREATE:
|
|
|
|
|
return (B_TRUE);
|
|
|
|
|
|
|
|
|
|
case VDEV_LABEL_REPLACE:
|
|
|
|
|
return (!spa_has_l2cache(spa, device_guid));
|
|
|
|
|
|
|
|
|
|
case VDEV_LABEL_L2CACHE:
|
|
|
|
|
return (spa_has_l2cache(spa, device_guid));
|
|
|
|
|
default:
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2010-08-27 01:24:34 +04:00
|
|
|
/*
|
|
|
|
|
* We can't rely on a pool's state if it's been imported
|
|
|
|
|
* read-only. Instead we look to see if the pools is marked
|
|
|
|
|
* read-only in the namespace and set the state to active.
|
|
|
|
|
*/
|
2014-04-23 07:18:17 +04:00
|
|
|
if (state != POOL_STATE_SPARE && state != POOL_STATE_L2CACHE &&
|
|
|
|
|
(spa = spa_by_guid(pool_guid, device_guid)) != NULL &&
|
2019-11-21 20:32:57 +03:00
|
|
|
spa_mode(spa) == SPA_MODE_READ)
|
2010-08-27 01:24:34 +04:00
|
|
|
state = POOL_STATE_ACTIVE;
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
/*
|
|
|
|
|
* If the device is marked ACTIVE, then this device is in use by another
|
|
|
|
|
* pool on the system.
|
|
|
|
|
*/
|
|
|
|
|
return (state == POOL_STATE_ACTIVE);
|
|
|
|
|
}
|
|
|
|
|
|
2024-08-09 01:16:46 +03:00
|
|
|
static nvlist_t *
|
|
|
|
|
vdev_aux_label_generate(vdev_t *vd, boolean_t reason_spare)
|
|
|
|
|
{
|
|
|
|
|
/*
|
|
|
|
|
* For inactive hot spares and level 2 ARC devices, we generate
|
|
|
|
|
* a special label that identifies as a mutually shared hot
|
|
|
|
|
* spare or l2cache device. We write the label in case of
|
|
|
|
|
* addition or removal of hot spare or l2cache vdev (in which
|
|
|
|
|
* case we want to revert the labels).
|
|
|
|
|
*/
|
|
|
|
|
nvlist_t *label = fnvlist_alloc();
|
|
|
|
|
fnvlist_add_uint64(label, ZPOOL_CONFIG_VERSION,
|
|
|
|
|
spa_version(vd->vdev_spa));
|
|
|
|
|
fnvlist_add_uint64(label, ZPOOL_CONFIG_POOL_STATE, reason_spare ?
|
|
|
|
|
POOL_STATE_SPARE : POOL_STATE_L2CACHE);
|
|
|
|
|
fnvlist_add_uint64(label, ZPOOL_CONFIG_GUID, vd->vdev_guid);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* This is merely to facilitate reporting the ashift of the
|
|
|
|
|
* cache device through zdb. The actual retrieval of the
|
|
|
|
|
* ashift (in vdev_alloc()) uses the nvlist
|
|
|
|
|
* spa->spa_l2cache->sav_config (populated in
|
|
|
|
|
* spa_ld_open_aux_vdevs()).
|
|
|
|
|
*/
|
|
|
|
|
if (!reason_spare)
|
|
|
|
|
fnvlist_add_uint64(label, ZPOOL_CONFIG_ASHIFT, vd->vdev_ashift);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Add path information to help find it during pool import
|
|
|
|
|
*/
|
|
|
|
|
if (vd->vdev_path != NULL)
|
|
|
|
|
fnvlist_add_string(label, ZPOOL_CONFIG_PATH, vd->vdev_path);
|
|
|
|
|
if (vd->vdev_devid != NULL)
|
|
|
|
|
fnvlist_add_string(label, ZPOOL_CONFIG_DEVID, vd->vdev_devid);
|
|
|
|
|
if (vd->vdev_physpath != NULL) {
|
|
|
|
|
fnvlist_add_string(label, ZPOOL_CONFIG_PHYS_PATH,
|
|
|
|
|
vd->vdev_physpath);
|
|
|
|
|
}
|
|
|
|
|
return (label);
|
|
|
|
|
}
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
/*
|
|
|
|
|
* Initialize a vdev label. We check to make sure each leaf device is not in
|
|
|
|
|
* use, and writable. We put down an initial label which we will later
|
|
|
|
|
* overwrite with a complete label. Note that it's important to do this
|
|
|
|
|
* sequentially, not in parallel, so that we catch cases of multiple use of the
|
|
|
|
|
* same leaf vdev in the vdev we're creating -- e.g. mirroring a disk with
|
|
|
|
|
* itself.
|
|
|
|
|
*/
|
|
|
|
|
int
|
|
|
|
|
vdev_label_init(vdev_t *vd, uint64_t crtxg, vdev_labeltype_t reason)
|
|
|
|
|
{
|
|
|
|
|
spa_t *spa = vd->vdev_spa;
|
|
|
|
|
nvlist_t *label;
|
|
|
|
|
vdev_phys_t *vp;
|
2016-07-22 18:52:49 +03:00
|
|
|
abd_t *vp_abd;
|
2020-05-07 19:36:33 +03:00
|
|
|
abd_t *bootenv;
|
2008-11-20 23:01:55 +03:00
|
|
|
uberblock_t *ub;
|
2016-07-22 18:52:49 +03:00
|
|
|
abd_t *ub_abd;
|
2008-11-20 23:01:55 +03:00
|
|
|
zio_t *zio;
|
|
|
|
|
char *buf;
|
|
|
|
|
size_t buflen;
|
|
|
|
|
int error;
|
2010-08-26 20:58:04 +04:00
|
|
|
uint64_t spare_guid = 0, l2cache_guid = 0;
|
2008-12-03 23:09:06 +03:00
|
|
|
int flags = ZIO_FLAG_CONFIG_WRITER | ZIO_FLAG_CANFAIL;
|
2024-01-04 17:32:53 +03:00
|
|
|
boolean_t reason_spare = (reason == VDEV_LABEL_SPARE || (reason ==
|
|
|
|
|
VDEV_LABEL_REMOVE && vd->vdev_isspare));
|
|
|
|
|
boolean_t reason_l2cache = (reason == VDEV_LABEL_L2CACHE || (reason ==
|
|
|
|
|
VDEV_LABEL_REMOVE && vd->vdev_isl2cache));
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2008-12-03 23:09:06 +03:00
|
|
|
ASSERT(spa_config_held(spa, SCL_ALL, RW_WRITER) == SCL_ALL);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2017-11-04 23:25:13 +03:00
|
|
|
for (int c = 0; c < vd->vdev_children; c++)
|
2008-11-20 23:01:55 +03:00
|
|
|
if ((error = vdev_label_init(vd->vdev_child[c],
|
|
|
|
|
crtxg, reason)) != 0)
|
|
|
|
|
return (error);
|
|
|
|
|
|
2010-05-29 00:45:14 +04:00
|
|
|
/* Track the creation time for this vdev */
|
|
|
|
|
vd->vdev_crtxg = crtxg;
|
|
|
|
|
|
2013-12-12 22:19:54 +04:00
|
|
|
if (!vd->vdev_ops->vdev_op_leaf || !spa_writeable(spa))
|
2008-11-20 23:01:55 +03:00
|
|
|
return (0);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Dead vdevs cannot be initialized.
|
|
|
|
|
*/
|
|
|
|
|
if (vdev_is_dead(vd))
|
2013-03-08 22:41:28 +04:00
|
|
|
return (SET_ERROR(EIO));
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Determine if the vdev is in use.
|
|
|
|
|
*/
|
2010-05-29 00:45:14 +04:00
|
|
|
if (reason != VDEV_LABEL_REMOVE && reason != VDEV_LABEL_SPLIT &&
|
2008-11-20 23:01:55 +03:00
|
|
|
vdev_inuse(vd, crtxg, reason, &spare_guid, &l2cache_guid))
|
2013-03-08 22:41:28 +04:00
|
|
|
return (SET_ERROR(EBUSY));
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If this is a request to add or replace a spare or l2cache device
|
|
|
|
|
* that is in use elsewhere on the system, then we must update the
|
|
|
|
|
* guid (which was initialized to a random value) to reflect the
|
|
|
|
|
* actual GUID (which is shared between multiple pools).
|
|
|
|
|
*/
|
|
|
|
|
if (reason != VDEV_LABEL_REMOVE && reason != VDEV_LABEL_L2CACHE &&
|
|
|
|
|
spare_guid != 0ULL) {
|
2008-12-03 23:09:06 +03:00
|
|
|
uint64_t guid_delta = spare_guid - vd->vdev_guid;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2008-12-03 23:09:06 +03:00
|
|
|
vd->vdev_guid += guid_delta;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2017-11-04 23:25:13 +03:00
|
|
|
for (vdev_t *pvd = vd; pvd != NULL; pvd = pvd->vdev_parent)
|
2008-12-03 23:09:06 +03:00
|
|
|
pvd->vdev_guid_sum += guid_delta;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If this is a replacement, then we want to fallthrough to the
|
|
|
|
|
* rest of the code. If we're adding a spare, then it's already
|
|
|
|
|
* labeled appropriately and we can just return.
|
|
|
|
|
*/
|
|
|
|
|
if (reason == VDEV_LABEL_SPARE)
|
|
|
|
|
return (0);
|
2010-05-29 00:45:14 +04:00
|
|
|
ASSERT(reason == VDEV_LABEL_REPLACE ||
|
|
|
|
|
reason == VDEV_LABEL_SPLIT);
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (reason != VDEV_LABEL_REMOVE && reason != VDEV_LABEL_SPARE &&
|
|
|
|
|
l2cache_guid != 0ULL) {
|
2008-12-03 23:09:06 +03:00
|
|
|
uint64_t guid_delta = l2cache_guid - vd->vdev_guid;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2008-12-03 23:09:06 +03:00
|
|
|
vd->vdev_guid += guid_delta;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2017-11-04 23:25:13 +03:00
|
|
|
for (vdev_t *pvd = vd; pvd != NULL; pvd = pvd->vdev_parent)
|
2008-12-03 23:09:06 +03:00
|
|
|
pvd->vdev_guid_sum += guid_delta;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If this is a replacement, then we want to fallthrough to the
|
|
|
|
|
* rest of the code. If we're adding an l2cache, then it's
|
|
|
|
|
* already labeled appropriately and we can just return.
|
|
|
|
|
*/
|
|
|
|
|
if (reason == VDEV_LABEL_L2CACHE)
|
|
|
|
|
return (0);
|
|
|
|
|
ASSERT(reason == VDEV_LABEL_REPLACE);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Initialize its label.
|
|
|
|
|
*/
|
2016-07-22 18:52:49 +03:00
|
|
|
vp_abd = abd_alloc_linear(sizeof (vdev_phys_t), B_TRUE);
|
|
|
|
|
abd_zero(vp_abd, sizeof (vdev_phys_t));
|
|
|
|
|
vp = abd_to_buf(vp_abd);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Generate a label describing the pool and our top-level vdev.
|
|
|
|
|
* We mark it as being from txg 0 to indicate that it's not
|
|
|
|
|
* really part of an active pool just yet. The labels will
|
|
|
|
|
* be written again with a meaningful txg by spa_sync().
|
|
|
|
|
*/
|
2024-01-04 17:32:53 +03:00
|
|
|
if (reason_spare || reason_l2cache) {
|
2024-08-09 01:16:46 +03:00
|
|
|
label = vdev_aux_label_generate(vd, reason_spare);
|
2024-01-04 17:02:50 +03:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* When spare or l2cache (aux) vdev is added during pool
|
|
|
|
|
* creation, spa->spa_uberblock is not written until this
|
|
|
|
|
* point. Write it on next config sync.
|
|
|
|
|
*/
|
|
|
|
|
if (uberblock_verify(&spa->spa_uberblock))
|
|
|
|
|
spa->spa_aux_sync_uber = B_TRUE;
|
2008-11-20 23:01:55 +03:00
|
|
|
} else {
|
2010-05-29 00:45:14 +04:00
|
|
|
uint64_t txg = 0ULL;
|
|
|
|
|
|
|
|
|
|
if (reason == VDEV_LABEL_SPLIT)
|
|
|
|
|
txg = spa->spa_uberblock.ub_txg;
|
|
|
|
|
label = spa_config_generate(spa, vd, txg, B_FALSE);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Add our creation time. This allows us to detect multiple
|
|
|
|
|
* vdev uses as described above, and automatically expires if we
|
|
|
|
|
* fail.
|
|
|
|
|
*/
|
|
|
|
|
VERIFY(nvlist_add_uint64(label, ZPOOL_CONFIG_CREATE_TXG,
|
|
|
|
|
crtxg) == 0);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
buf = vp->vp_nvlist;
|
|
|
|
|
buflen = sizeof (vp->vp_nvlist);
|
|
|
|
|
|
2014-11-21 03:09:39 +03:00
|
|
|
error = nvlist_pack(label, &buf, &buflen, NV_ENCODE_XDR, KM_SLEEP);
|
2008-11-20 23:01:55 +03:00
|
|
|
if (error != 0) {
|
|
|
|
|
nvlist_free(label);
|
2016-07-22 18:52:49 +03:00
|
|
|
abd_free(vp_abd);
|
2008-11-20 23:01:55 +03:00
|
|
|
/* EFAULT means nvlist_pack ran out of room */
|
2017-08-03 07:16:12 +03:00
|
|
|
return (SET_ERROR(error == EFAULT ? ENAMETOOLONG : EINVAL));
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Initialize uberblock template.
|
|
|
|
|
*/
|
2016-07-22 18:52:49 +03:00
|
|
|
ub_abd = abd_alloc_linear(VDEV_UBERBLOCK_RING, B_TRUE);
|
|
|
|
|
abd_copy_from_buf(ub_abd, &spa->spa_uberblock, sizeof (uberblock_t));
|
2023-11-18 00:28:32 +03:00
|
|
|
abd_zero_off(ub_abd, sizeof (uberblock_t),
|
|
|
|
|
VDEV_UBERBLOCK_RING - sizeof (uberblock_t));
|
2016-07-22 18:52:49 +03:00
|
|
|
ub = abd_to_buf(ub_abd);
|
2008-11-20 23:01:55 +03:00
|
|
|
ub->ub_txg = 0;
|
|
|
|
|
|
2009-07-03 02:44:48 +04:00
|
|
|
/* Initialize the 2nd padding area. */
|
2020-05-07 19:36:33 +03:00
|
|
|
bootenv = abd_alloc_for_io(VDEV_PAD_SIZE, B_TRUE);
|
|
|
|
|
abd_zero(bootenv, VDEV_PAD_SIZE);
|
2009-07-03 02:44:48 +04:00
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
/*
|
|
|
|
|
* Write everything in parallel.
|
|
|
|
|
*/
|
2009-07-03 02:44:48 +04:00
|
|
|
retry:
|
2008-11-20 23:01:55 +03:00
|
|
|
zio = zio_root(spa, NULL, NULL, flags);
|
|
|
|
|
|
2017-11-04 23:25:13 +03:00
|
|
|
for (int l = 0; l < VDEV_LABELS; l++) {
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2016-07-22 18:52:49 +03:00
|
|
|
vdev_label_write(zio, vd, l, vp_abd,
|
2008-11-20 23:01:55 +03:00
|
|
|
offsetof(vdev_label_t, vl_vdev_phys),
|
|
|
|
|
sizeof (vdev_phys_t), NULL, NULL, flags);
|
|
|
|
|
|
2009-07-03 02:44:48 +04:00
|
|
|
/*
|
|
|
|
|
* Skip the 1st padding area.
|
|
|
|
|
* Zero out the 2nd padding area where it might have
|
|
|
|
|
* left over data from previous filesystem format.
|
|
|
|
|
*/
|
2020-05-07 19:36:33 +03:00
|
|
|
vdev_label_write(zio, vd, l, bootenv,
|
|
|
|
|
offsetof(vdev_label_t, vl_be),
|
2009-07-03 02:44:48 +04:00
|
|
|
VDEV_PAD_SIZE, NULL, NULL, flags);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2016-07-22 18:52:49 +03:00
|
|
|
vdev_label_write(zio, vd, l, ub_abd,
|
2009-08-18 22:43:27 +04:00
|
|
|
offsetof(vdev_label_t, vl_uberblock),
|
|
|
|
|
VDEV_UBERBLOCK_RING, NULL, NULL, flags);
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
error = zio_wait(zio);
|
|
|
|
|
|
2009-07-03 02:44:48 +04:00
|
|
|
if (error != 0 && !(flags & ZIO_FLAG_TRYHARD)) {
|
|
|
|
|
flags |= ZIO_FLAG_TRYHARD;
|
|
|
|
|
goto retry;
|
|
|
|
|
}
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
nvlist_free(label);
|
2020-05-07 19:36:33 +03:00
|
|
|
abd_free(bootenv);
|
2016-07-22 18:52:49 +03:00
|
|
|
abd_free(ub_abd);
|
|
|
|
|
abd_free(vp_abd);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If this vdev hasn't been previously identified as a spare, then we
|
|
|
|
|
* mark it as such only if a) we are labeling it as a spare, or b) it
|
|
|
|
|
* exists as a spare elsewhere in the system. Do the same for
|
|
|
|
|
* level 2 ARC devices.
|
|
|
|
|
*/
|
|
|
|
|
if (error == 0 && !vd->vdev_isspare &&
|
|
|
|
|
(reason == VDEV_LABEL_SPARE ||
|
2008-12-03 23:09:06 +03:00
|
|
|
spa_spare_exists(vd->vdev_guid, NULL, NULL)))
|
2008-11-20 23:01:55 +03:00
|
|
|
spa_spare_add(vd);
|
|
|
|
|
|
|
|
|
|
if (error == 0 && !vd->vdev_isl2cache &&
|
|
|
|
|
(reason == VDEV_LABEL_L2CACHE ||
|
|
|
|
|
spa_l2cache_exists(vd->vdev_guid, NULL)))
|
|
|
|
|
spa_l2cache_add(vd);
|
|
|
|
|
|
|
|
|
|
return (error);
|
|
|
|
|
}
|
|
|
|
|
|
2020-05-07 19:36:33 +03:00
|
|
|
/*
|
|
|
|
|
* Done callback for vdev_label_read_bootenv_impl. If this is the first
|
|
|
|
|
* callback to finish, store our abd in the callback pointer. Otherwise, we
|
|
|
|
|
* just free our abd and return.
|
|
|
|
|
*/
|
|
|
|
|
static void
|
|
|
|
|
vdev_label_read_bootenv_done(zio_t *zio)
|
|
|
|
|
{
|
|
|
|
|
zio_t *rio = zio->io_private;
|
|
|
|
|
abd_t **cbp = rio->io_private;
|
|
|
|
|
|
|
|
|
|
ASSERT3U(zio->io_size, ==, VDEV_PAD_SIZE);
|
|
|
|
|
|
|
|
|
|
if (zio->io_error == 0) {
|
|
|
|
|
mutex_enter(&rio->io_lock);
|
|
|
|
|
if (*cbp == NULL) {
|
|
|
|
|
/* Will free this buffer in vdev_label_read_bootenv. */
|
|
|
|
|
*cbp = zio->io_abd;
|
|
|
|
|
} else {
|
|
|
|
|
abd_free(zio->io_abd);
|
|
|
|
|
}
|
|
|
|
|
mutex_exit(&rio->io_lock);
|
|
|
|
|
} else {
|
|
|
|
|
abd_free(zio->io_abd);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
vdev_label_read_bootenv_impl(zio_t *zio, vdev_t *vd, int flags)
|
|
|
|
|
{
|
|
|
|
|
for (int c = 0; c < vd->vdev_children; c++)
|
|
|
|
|
vdev_label_read_bootenv_impl(zio, vd->vdev_child[c], flags);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* We just use the first label that has a correct checksum; the
|
|
|
|
|
* bootloader should have rewritten them all to be the same on boot,
|
|
|
|
|
* and any changes we made since boot have been the same across all
|
|
|
|
|
* labels.
|
|
|
|
|
*/
|
|
|
|
|
if (vd->vdev_ops->vdev_op_leaf && vdev_readable(vd)) {
|
2020-09-16 01:42:27 +03:00
|
|
|
for (int l = 0; l < VDEV_LABELS; l++) {
|
2020-05-07 19:36:33 +03:00
|
|
|
vdev_label_read(zio, vd, l,
|
|
|
|
|
abd_alloc_linear(VDEV_PAD_SIZE, B_FALSE),
|
|
|
|
|
offsetof(vdev_label_t, vl_be), VDEV_PAD_SIZE,
|
|
|
|
|
vdev_label_read_bootenv_done, zio, flags);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
int
|
2020-09-16 01:42:27 +03:00
|
|
|
vdev_label_read_bootenv(vdev_t *rvd, nvlist_t *bootenv)
|
2020-05-07 19:36:33 +03:00
|
|
|
{
|
2020-09-16 01:42:27 +03:00
|
|
|
nvlist_t *config;
|
2020-05-07 19:36:33 +03:00
|
|
|
spa_t *spa = rvd->vdev_spa;
|
|
|
|
|
abd_t *abd = NULL;
|
|
|
|
|
int flags = ZIO_FLAG_CONFIG_WRITER | ZIO_FLAG_CANFAIL |
|
|
|
|
|
ZIO_FLAG_SPECULATIVE | ZIO_FLAG_TRYHARD;
|
|
|
|
|
|
2020-09-16 01:42:27 +03:00
|
|
|
ASSERT(bootenv);
|
2020-05-07 19:36:33 +03:00
|
|
|
ASSERT(spa_config_held(spa, SCL_ALL, RW_WRITER) == SCL_ALL);
|
|
|
|
|
|
|
|
|
|
zio_t *zio = zio_root(spa, NULL, &abd, flags);
|
|
|
|
|
vdev_label_read_bootenv_impl(zio, rvd, flags);
|
|
|
|
|
int err = zio_wait(zio);
|
|
|
|
|
|
|
|
|
|
if (abd != NULL) {
|
2020-09-16 01:42:27 +03:00
|
|
|
char *buf;
|
2020-05-07 19:36:33 +03:00
|
|
|
vdev_boot_envblock_t *vbe = abd_to_buf(abd);
|
2020-09-16 01:42:27 +03:00
|
|
|
|
|
|
|
|
vbe->vbe_version = ntohll(vbe->vbe_version);
|
|
|
|
|
switch (vbe->vbe_version) {
|
|
|
|
|
case VB_RAW:
|
|
|
|
|
/*
|
|
|
|
|
* if we have textual data in vbe_bootenv, create nvlist
|
|
|
|
|
* with key "envmap".
|
|
|
|
|
*/
|
|
|
|
|
fnvlist_add_uint64(bootenv, BOOTENV_VERSION, VB_RAW);
|
|
|
|
|
vbe->vbe_bootenv[sizeof (vbe->vbe_bootenv) - 1] = '\0';
|
|
|
|
|
fnvlist_add_string(bootenv, GRUB_ENVMAP,
|
|
|
|
|
vbe->vbe_bootenv);
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
case VB_NVLIST:
|
|
|
|
|
err = nvlist_unpack(vbe->vbe_bootenv,
|
|
|
|
|
sizeof (vbe->vbe_bootenv), &config, 0);
|
|
|
|
|
if (err == 0) {
|
|
|
|
|
fnvlist_merge(bootenv, config);
|
|
|
|
|
nvlist_free(config);
|
|
|
|
|
break;
|
|
|
|
|
}
|
2022-02-15 19:58:59 +03:00
|
|
|
zfs_fallthrough;
|
2020-09-16 01:42:27 +03:00
|
|
|
default:
|
|
|
|
|
/* Check for FreeBSD zfs bootonce command string */
|
|
|
|
|
buf = abd_to_buf(abd);
|
|
|
|
|
if (*buf == '\0') {
|
|
|
|
|
fnvlist_add_uint64(bootenv, BOOTENV_VERSION,
|
|
|
|
|
VB_NVLIST);
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
fnvlist_add_string(bootenv, FREEBSD_BOOTONCE, buf);
|
2020-05-07 19:36:33 +03:00
|
|
|
}
|
2020-09-16 01:42:27 +03:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* abd was allocated in vdev_label_read_bootenv_impl()
|
|
|
|
|
*/
|
2020-05-07 19:36:33 +03:00
|
|
|
abd_free(abd);
|
2020-09-16 01:42:27 +03:00
|
|
|
/*
|
|
|
|
|
* If we managed to read any successfully,
|
|
|
|
|
* return success.
|
|
|
|
|
*/
|
2020-05-07 19:36:33 +03:00
|
|
|
return (0);
|
|
|
|
|
}
|
|
|
|
|
return (err);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
int
|
2020-09-16 01:42:27 +03:00
|
|
|
vdev_label_write_bootenv(vdev_t *vd, nvlist_t *env)
|
2020-05-07 19:36:33 +03:00
|
|
|
{
|
|
|
|
|
zio_t *zio;
|
|
|
|
|
spa_t *spa = vd->vdev_spa;
|
|
|
|
|
vdev_boot_envblock_t *bootenv;
|
|
|
|
|
int flags = ZIO_FLAG_CONFIG_WRITER | ZIO_FLAG_CANFAIL;
|
2020-09-16 01:42:27 +03:00
|
|
|
int error;
|
|
|
|
|
size_t nvsize;
|
|
|
|
|
char *nvbuf;
|
2023-03-11 21:39:24 +03:00
|
|
|
const char *tmp;
|
2020-05-07 19:36:33 +03:00
|
|
|
|
2020-09-16 01:42:27 +03:00
|
|
|
error = nvlist_size(env, &nvsize, NV_ENCODE_XDR);
|
|
|
|
|
if (error != 0)
|
|
|
|
|
return (SET_ERROR(error));
|
|
|
|
|
|
|
|
|
|
if (nvsize >= sizeof (bootenv->vbe_bootenv)) {
|
2020-05-07 19:36:33 +03:00
|
|
|
return (SET_ERROR(E2BIG));
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
ASSERT(spa_config_held(spa, SCL_ALL, RW_WRITER) == SCL_ALL);
|
|
|
|
|
|
2020-09-16 01:42:27 +03:00
|
|
|
error = ENXIO;
|
2020-05-07 19:36:33 +03:00
|
|
|
for (int c = 0; c < vd->vdev_children; c++) {
|
2020-09-16 01:42:27 +03:00
|
|
|
int child_err;
|
|
|
|
|
|
|
|
|
|
child_err = vdev_label_write_bootenv(vd->vdev_child[c], env);
|
2020-05-07 19:36:33 +03:00
|
|
|
/*
|
|
|
|
|
* As long as any of the disks managed to write all of their
|
|
|
|
|
* labels successfully, return success.
|
|
|
|
|
*/
|
|
|
|
|
if (child_err == 0)
|
|
|
|
|
error = child_err;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (!vd->vdev_ops->vdev_op_leaf || vdev_is_dead(vd) ||
|
|
|
|
|
!vdev_writeable(vd)) {
|
|
|
|
|
return (error);
|
|
|
|
|
}
|
|
|
|
|
ASSERT3U(sizeof (*bootenv), ==, VDEV_PAD_SIZE);
|
|
|
|
|
abd_t *abd = abd_alloc_for_io(VDEV_PAD_SIZE, B_TRUE);
|
|
|
|
|
abd_zero(abd, VDEV_PAD_SIZE);
|
2020-09-16 01:42:27 +03:00
|
|
|
|
2020-05-07 19:36:33 +03:00
|
|
|
bootenv = abd_borrow_buf_copy(abd, VDEV_PAD_SIZE);
|
2020-09-16 01:42:27 +03:00
|
|
|
nvbuf = bootenv->vbe_bootenv;
|
|
|
|
|
nvsize = sizeof (bootenv->vbe_bootenv);
|
|
|
|
|
|
|
|
|
|
bootenv->vbe_version = fnvlist_lookup_uint64(env, BOOTENV_VERSION);
|
|
|
|
|
switch (bootenv->vbe_version) {
|
|
|
|
|
case VB_RAW:
|
2023-03-11 21:39:24 +03:00
|
|
|
if (nvlist_lookup_string(env, GRUB_ENVMAP, &tmp) == 0) {
|
|
|
|
|
(void) strlcpy(bootenv->vbe_bootenv, tmp, nvsize);
|
2020-09-16 01:42:27 +03:00
|
|
|
}
|
|
|
|
|
error = 0;
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
case VB_NVLIST:
|
|
|
|
|
error = nvlist_pack(env, &nvbuf, &nvsize, NV_ENCODE_XDR,
|
|
|
|
|
KM_SLEEP);
|
|
|
|
|
break;
|
2020-05-07 19:36:33 +03:00
|
|
|
|
2020-09-16 01:42:27 +03:00
|
|
|
default:
|
|
|
|
|
error = EINVAL;
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (error == 0) {
|
|
|
|
|
bootenv->vbe_version = htonll(bootenv->vbe_version);
|
|
|
|
|
abd_return_buf_copy(abd, bootenv, VDEV_PAD_SIZE);
|
|
|
|
|
} else {
|
|
|
|
|
abd_free(abd);
|
|
|
|
|
return (SET_ERROR(error));
|
|
|
|
|
}
|
2020-05-07 19:36:33 +03:00
|
|
|
|
|
|
|
|
retry:
|
|
|
|
|
zio = zio_root(spa, NULL, NULL, flags);
|
2020-09-16 01:42:27 +03:00
|
|
|
for (int l = 0; l < VDEV_LABELS; l++) {
|
2020-05-07 19:36:33 +03:00
|
|
|
vdev_label_write(zio, vd, l, abd,
|
|
|
|
|
offsetof(vdev_label_t, vl_be),
|
|
|
|
|
VDEV_PAD_SIZE, NULL, NULL, flags);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
error = zio_wait(zio);
|
|
|
|
|
if (error != 0 && !(flags & ZIO_FLAG_TRYHARD)) {
|
|
|
|
|
flags |= ZIO_FLAG_TRYHARD;
|
|
|
|
|
goto retry;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
abd_free(abd);
|
|
|
|
|
return (error);
|
|
|
|
|
}
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
/*
|
|
|
|
|
* ==========================================================================
|
|
|
|
|
* uberblock load/sync
|
|
|
|
|
* ==========================================================================
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Consider the following situation: txg is safely synced to disk. We've
|
|
|
|
|
* written the first uberblock for txg + 1, and then we lose power. When we
|
|
|
|
|
* come back up, we fail to see the uberblock for txg + 1 because, say,
|
|
|
|
|
* it was on a mirrored device and the replica to which we wrote txg + 1
|
|
|
|
|
* is now offline. If we then make some changes and sync txg + 1, and then
|
2012-12-14 03:24:15 +04:00
|
|
|
* the missing replica comes back, then for a few seconds we'll have two
|
2008-11-20 23:01:55 +03:00
|
|
|
* conflicting uberblocks on disk with the same txg. The solution is simple:
|
|
|
|
|
* among uberblocks with equal txg, choose the one with the latest timestamp.
|
|
|
|
|
*/
|
|
|
|
|
static int
|
2016-08-27 21:12:53 +03:00
|
|
|
vdev_uberblock_compare(const uberblock_t *ub1, const uberblock_t *ub2)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
Reduce loaded range tree memory usage
This patch implements a new tree structure for ZFS, and uses it to
store range trees more efficiently.
The new structure is approximately a B-tree, though there are some
small differences from the usual characterizations. The tree has core
nodes and leaf nodes; each contain data elements, which the elements
in the core nodes acting as separators between its children. The
difference between core and leaf nodes is that the core nodes have an
array of children, while leaf nodes don't. Every node in the tree may
be only partially full; in most cases, they are all at least 50% full
(in terms of element count) except for the root node, which can be
less full. Underfull nodes will steal from their neighbors or merge to
remain full enough, while overfull nodes will split in two. The data
elements are contained in tree-controlled buffers; they are copied
into these on insertion, and overwritten on deletion. This means that
the elements are not independently allocated, which reduces overhead,
but also means they can't be shared between trees (and also that
pointers to them are only valid until a side-effectful tree operation
occurs). The overhead varies based on how dense the tree is, but is
usually on the order of about 50% of the element size; the per-node
overheads are very small, and so don't make a significant difference.
The trees can accept arbitrary records; they accept a size and a
comparator to allow them to be used for a variety of purposes.
The new trees replace the AVL trees used in the range trees today.
Currently, the range_seg_t structure contains three 8 byte integers
of payload and two 24 byte avl_tree_node_ts to handle its storage in
both an offset-sorted tree and a size-sorted tree (total size: 64
bytes). In the new model, the range seg structures are usually two 4
byte integers, but a separate one needs to exist for the size-sorted
and offset-sorted tree. Between the raw size, the 50% overhead, and
the double storage, the new btrees are expected to use 8*1.5*2 = 24
bytes per record, or 33.3% as much memory as the AVL trees (this is
for the purposes of storing metaslab range trees; for other purposes,
like scrubs, they use ~50% as much memory).
We reduced the size of the payload in the range segments by teaching
range trees about starting offsets and shifts; since metaslabs have a
fixed starting offset, and they all operate in terms of disk sectors,
we can store the ranges using 4-byte integers as long as the size of
the metaslab divided by the sector size is less than 2^32. For 512-byte
sectors, this is a 2^41 (or 2TB) metaslab, which with the default
settings corresponds to a 256PB disk. 4k sector disks can handle
metaslabs up to 2^46 bytes, or 2^63 byte disks. Since we do not
anticipate disks of this size in the near future, there should be
almost no cases where metaslabs need 64-byte integers to store their
ranges. We do still have the capability to store 64-byte integer ranges
to account for cases where we are storing per-vdev (or per-dnode) trees,
which could reasonably go above the limits discussed. We also do not
store fill information in the compact version of the node, since it
is only used for sorted scrub.
We also optimized the metaslab loading process in various other ways
to offset some inefficiencies in the btree model. While individual
operations (find, insert, remove_from) are faster for the btree than
they are for the avl tree, remove usually requires a find operation,
while in the AVL tree model the element itself suffices. Some clever
changes actually caused an overall speedup in metaslab loading; we use
approximately 40% less cpu to load metaslabs in our tests on Illumos.
Another memory and performance optimization was achieved by changing
what is stored in the size-sorted trees. When a disk is heavily
fragmented, the df algorithm used by default in ZFS will almost always
find a number of small regions in its initial cursor-based search; it
will usually only fall back to the size-sorted tree to find larger
regions. If we increase the size of the cursor-based search slightly,
and don't store segments that are smaller than a tunable size floor
in the size-sorted tree, we can further cut memory usage down to
below 20% of what the AVL trees store. This also results in further
reductions in CPU time spent loading metaslabs.
The 16KiB size floor was chosen because it results in substantial memory
usage reduction while not usually resulting in situations where we can't
find an appropriate chunk with the cursor and are forced to use an
oversized chunk from the size-sorted tree. In addition, even if we do
have to use an oversized chunk from the size-sorted tree, the chunk
would be too small to use for ZIL allocations, so it isn't as big of a
loss as it might otherwise be. And often, more small allocations will
follow the initial one, and the cursor search will now find the
remainder of the chunk we didn't use all of and use it for subsequent
allocations. Practical testing has shown little or no change in
fragmentation as a result of this change.
If the size-sorted tree becomes empty while the offset sorted one still
has entries, it will load all the entries from the offset sorted tree
and disregard the size floor until it is unloaded again. This operation
occurs rarely with the default setting, only on incredibly thoroughly
fragmented pools.
There are some other small changes to zdb to teach it to handle btrees,
but nothing major.
Reviewed-by: George Wilson <gwilson@delphix.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed by: Sebastien Roy seb@delphix.com
Reviewed-by: Igor Kozhukhov <igor@dilos.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #9181
2019-10-09 20:36:03 +03:00
|
|
|
int cmp = TREE_CMP(ub1->ub_txg, ub2->ub_txg);
|
MMP interval and fail_intervals in uberblock
When Multihost is enabled, and a pool is imported, uberblock writes
include ub_mmp_delay to allow an importing node to calculate the
duration of an activity test. This value, is not enough information.
If zfs_multihost_fail_intervals > 0 on the node with the pool imported,
the safe minimum duration of the activity test is well defined, but does
not depend on ub_mmp_delay:
zfs_multihost_fail_intervals * zfs_multihost_interval
and if zfs_multihost_fail_intervals == 0 on that node, there is no such
well defined safe duration, but the importing host cannot tell whether
mmp_delay is high due to I/O delays, or due to a very large
zfs_multihost_interval setting on the host which last imported the pool.
As a result, it may use a far longer period for the activity test than
is necessary.
This patch renames ub_mmp_sequence to ub_mmp_config and uses it to
record the zfs_multihost_interval and zfs_multihost_fail_intervals
values, as well as the mmp sequence. This allows a shorter activity
test duration to be calculated by the importing host in most situations.
These values are also added to the multihost_history kstat records.
It calculates the activity test duration differently depending on
whether the new fields are present or not; for importing pools with
only ub_mmp_delay, it uses
(zfs_multihost_interval + ub_mmp_delay) * zfs_multihost_import_intervals
Which results in an activity test duration less sensitive to the leaf
count.
In addition, it makes a few other improvements:
* It updates the "sequence" part of ub_mmp_config when MMP writes
in between syncs occur. This allows an importing host to detect MMP
on the remote host sooner, when the pool is idle, as it is not limited
to the granularity of ub_timestamp (1 second).
* It issues writes immediately when zfs_multihost_interval is changed
so remote hosts see the updated value as soon as possible.
* It fixes a bug where setting zfs_multihost_fail_intervals = 1 results
in immediate pool suspension.
* Update tests to verify activity check duration is based on recorded
tunable values, not tunable values on importing host.
* Update tests to verify the expected number of uberblocks have valid
MMP fields - fail_intervals, mmp_interval, mmp_seq (sequence number),
that sequence number is incrementing, and that uberblock values match
tunable settings.
Reviewed-by: Andreas Dilger <andreas.dilger@whamcloud.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #7842
2019-03-21 22:47:57 +03:00
|
|
|
|
2016-08-27 21:12:53 +03:00
|
|
|
if (likely(cmp))
|
|
|
|
|
return (cmp);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
Reduce loaded range tree memory usage
This patch implements a new tree structure for ZFS, and uses it to
store range trees more efficiently.
The new structure is approximately a B-tree, though there are some
small differences from the usual characterizations. The tree has core
nodes and leaf nodes; each contain data elements, which the elements
in the core nodes acting as separators between its children. The
difference between core and leaf nodes is that the core nodes have an
array of children, while leaf nodes don't. Every node in the tree may
be only partially full; in most cases, they are all at least 50% full
(in terms of element count) except for the root node, which can be
less full. Underfull nodes will steal from their neighbors or merge to
remain full enough, while overfull nodes will split in two. The data
elements are contained in tree-controlled buffers; they are copied
into these on insertion, and overwritten on deletion. This means that
the elements are not independently allocated, which reduces overhead,
but also means they can't be shared between trees (and also that
pointers to them are only valid until a side-effectful tree operation
occurs). The overhead varies based on how dense the tree is, but is
usually on the order of about 50% of the element size; the per-node
overheads are very small, and so don't make a significant difference.
The trees can accept arbitrary records; they accept a size and a
comparator to allow them to be used for a variety of purposes.
The new trees replace the AVL trees used in the range trees today.
Currently, the range_seg_t structure contains three 8 byte integers
of payload and two 24 byte avl_tree_node_ts to handle its storage in
both an offset-sorted tree and a size-sorted tree (total size: 64
bytes). In the new model, the range seg structures are usually two 4
byte integers, but a separate one needs to exist for the size-sorted
and offset-sorted tree. Between the raw size, the 50% overhead, and
the double storage, the new btrees are expected to use 8*1.5*2 = 24
bytes per record, or 33.3% as much memory as the AVL trees (this is
for the purposes of storing metaslab range trees; for other purposes,
like scrubs, they use ~50% as much memory).
We reduced the size of the payload in the range segments by teaching
range trees about starting offsets and shifts; since metaslabs have a
fixed starting offset, and they all operate in terms of disk sectors,
we can store the ranges using 4-byte integers as long as the size of
the metaslab divided by the sector size is less than 2^32. For 512-byte
sectors, this is a 2^41 (or 2TB) metaslab, which with the default
settings corresponds to a 256PB disk. 4k sector disks can handle
metaslabs up to 2^46 bytes, or 2^63 byte disks. Since we do not
anticipate disks of this size in the near future, there should be
almost no cases where metaslabs need 64-byte integers to store their
ranges. We do still have the capability to store 64-byte integer ranges
to account for cases where we are storing per-vdev (or per-dnode) trees,
which could reasonably go above the limits discussed. We also do not
store fill information in the compact version of the node, since it
is only used for sorted scrub.
We also optimized the metaslab loading process in various other ways
to offset some inefficiencies in the btree model. While individual
operations (find, insert, remove_from) are faster for the btree than
they are for the avl tree, remove usually requires a find operation,
while in the AVL tree model the element itself suffices. Some clever
changes actually caused an overall speedup in metaslab loading; we use
approximately 40% less cpu to load metaslabs in our tests on Illumos.
Another memory and performance optimization was achieved by changing
what is stored in the size-sorted trees. When a disk is heavily
fragmented, the df algorithm used by default in ZFS will almost always
find a number of small regions in its initial cursor-based search; it
will usually only fall back to the size-sorted tree to find larger
regions. If we increase the size of the cursor-based search slightly,
and don't store segments that are smaller than a tunable size floor
in the size-sorted tree, we can further cut memory usage down to
below 20% of what the AVL trees store. This also results in further
reductions in CPU time spent loading metaslabs.
The 16KiB size floor was chosen because it results in substantial memory
usage reduction while not usually resulting in situations where we can't
find an appropriate chunk with the cursor and are forced to use an
oversized chunk from the size-sorted tree. In addition, even if we do
have to use an oversized chunk from the size-sorted tree, the chunk
would be too small to use for ZIL allocations, so it isn't as big of a
loss as it might otherwise be. And often, more small allocations will
follow the initial one, and the cursor search will now find the
remainder of the chunk we didn't use all of and use it for subsequent
allocations. Practical testing has shown little or no change in
fragmentation as a result of this change.
If the size-sorted tree becomes empty while the offset sorted one still
has entries, it will load all the entries from the offset sorted tree
and disregard the size floor until it is unloaded again. This operation
occurs rarely with the default setting, only on incredibly thoroughly
fragmented pools.
There are some other small changes to zdb to teach it to handle btrees,
but nothing major.
Reviewed-by: George Wilson <gwilson@delphix.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed by: Sebastien Roy seb@delphix.com
Reviewed-by: Igor Kozhukhov <igor@dilos.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #9181
2019-10-09 20:36:03 +03:00
|
|
|
cmp = TREE_CMP(ub1->ub_timestamp, ub2->ub_timestamp);
|
MMP interval and fail_intervals in uberblock
When Multihost is enabled, and a pool is imported, uberblock writes
include ub_mmp_delay to allow an importing node to calculate the
duration of an activity test. This value, is not enough information.
If zfs_multihost_fail_intervals > 0 on the node with the pool imported,
the safe minimum duration of the activity test is well defined, but does
not depend on ub_mmp_delay:
zfs_multihost_fail_intervals * zfs_multihost_interval
and if zfs_multihost_fail_intervals == 0 on that node, there is no such
well defined safe duration, but the importing host cannot tell whether
mmp_delay is high due to I/O delays, or due to a very large
zfs_multihost_interval setting on the host which last imported the pool.
As a result, it may use a far longer period for the activity test than
is necessary.
This patch renames ub_mmp_sequence to ub_mmp_config and uses it to
record the zfs_multihost_interval and zfs_multihost_fail_intervals
values, as well as the mmp sequence. This allows a shorter activity
test duration to be calculated by the importing host in most situations.
These values are also added to the multihost_history kstat records.
It calculates the activity test duration differently depending on
whether the new fields are present or not; for importing pools with
only ub_mmp_delay, it uses
(zfs_multihost_interval + ub_mmp_delay) * zfs_multihost_import_intervals
Which results in an activity test duration less sensitive to the leaf
count.
In addition, it makes a few other improvements:
* It updates the "sequence" part of ub_mmp_config when MMP writes
in between syncs occur. This allows an importing host to detect MMP
on the remote host sooner, when the pool is idle, as it is not limited
to the granularity of ub_timestamp (1 second).
* It issues writes immediately when zfs_multihost_interval is changed
so remote hosts see the updated value as soon as possible.
* It fixes a bug where setting zfs_multihost_fail_intervals = 1 results
in immediate pool suspension.
* Update tests to verify activity check duration is based on recorded
tunable values, not tunable values on importing host.
* Update tests to verify the expected number of uberblocks have valid
MMP fields - fail_intervals, mmp_interval, mmp_seq (sequence number),
that sequence number is incrementing, and that uberblock values match
tunable settings.
Reviewed-by: Andreas Dilger <andreas.dilger@whamcloud.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #7842
2019-03-21 22:47:57 +03:00
|
|
|
if (likely(cmp))
|
|
|
|
|
return (cmp);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If MMP_VALID(ub) && MMP_SEQ_VALID(ub) then the host has an MMP-aware
|
2020-10-09 06:10:13 +03:00
|
|
|
* ZFS, e.g. OpenZFS >= 0.7.
|
MMP interval and fail_intervals in uberblock
When Multihost is enabled, and a pool is imported, uberblock writes
include ub_mmp_delay to allow an importing node to calculate the
duration of an activity test. This value, is not enough information.
If zfs_multihost_fail_intervals > 0 on the node with the pool imported,
the safe minimum duration of the activity test is well defined, but does
not depend on ub_mmp_delay:
zfs_multihost_fail_intervals * zfs_multihost_interval
and if zfs_multihost_fail_intervals == 0 on that node, there is no such
well defined safe duration, but the importing host cannot tell whether
mmp_delay is high due to I/O delays, or due to a very large
zfs_multihost_interval setting on the host which last imported the pool.
As a result, it may use a far longer period for the activity test than
is necessary.
This patch renames ub_mmp_sequence to ub_mmp_config and uses it to
record the zfs_multihost_interval and zfs_multihost_fail_intervals
values, as well as the mmp sequence. This allows a shorter activity
test duration to be calculated by the importing host in most situations.
These values are also added to the multihost_history kstat records.
It calculates the activity test duration differently depending on
whether the new fields are present or not; for importing pools with
only ub_mmp_delay, it uses
(zfs_multihost_interval + ub_mmp_delay) * zfs_multihost_import_intervals
Which results in an activity test duration less sensitive to the leaf
count.
In addition, it makes a few other improvements:
* It updates the "sequence" part of ub_mmp_config when MMP writes
in between syncs occur. This allows an importing host to detect MMP
on the remote host sooner, when the pool is idle, as it is not limited
to the granularity of ub_timestamp (1 second).
* It issues writes immediately when zfs_multihost_interval is changed
so remote hosts see the updated value as soon as possible.
* It fixes a bug where setting zfs_multihost_fail_intervals = 1 results
in immediate pool suspension.
* Update tests to verify activity check duration is based on recorded
tunable values, not tunable values on importing host.
* Update tests to verify the expected number of uberblocks have valid
MMP fields - fail_intervals, mmp_interval, mmp_seq (sequence number),
that sequence number is incrementing, and that uberblock values match
tunable settings.
Reviewed-by: Andreas Dilger <andreas.dilger@whamcloud.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #7842
2019-03-21 22:47:57 +03:00
|
|
|
*
|
|
|
|
|
* If one ub has MMP and the other does not, they were written by
|
|
|
|
|
* different hosts, which matters for MMP. So we treat no MMP/no SEQ as
|
|
|
|
|
* a 0 value.
|
|
|
|
|
*
|
|
|
|
|
* Since timestamp and txg are the same if we get this far, either is
|
|
|
|
|
* acceptable for importing the pool.
|
|
|
|
|
*/
|
|
|
|
|
unsigned int seq1 = 0;
|
|
|
|
|
unsigned int seq2 = 0;
|
|
|
|
|
|
|
|
|
|
if (MMP_VALID(ub1) && MMP_SEQ_VALID(ub1))
|
|
|
|
|
seq1 = MMP_SEQ(ub1);
|
|
|
|
|
|
|
|
|
|
if (MMP_VALID(ub2) && MMP_SEQ_VALID(ub2))
|
|
|
|
|
seq2 = MMP_SEQ(ub2);
|
|
|
|
|
|
Reduce loaded range tree memory usage
This patch implements a new tree structure for ZFS, and uses it to
store range trees more efficiently.
The new structure is approximately a B-tree, though there are some
small differences from the usual characterizations. The tree has core
nodes and leaf nodes; each contain data elements, which the elements
in the core nodes acting as separators between its children. The
difference between core and leaf nodes is that the core nodes have an
array of children, while leaf nodes don't. Every node in the tree may
be only partially full; in most cases, they are all at least 50% full
(in terms of element count) except for the root node, which can be
less full. Underfull nodes will steal from their neighbors or merge to
remain full enough, while overfull nodes will split in two. The data
elements are contained in tree-controlled buffers; they are copied
into these on insertion, and overwritten on deletion. This means that
the elements are not independently allocated, which reduces overhead,
but also means they can't be shared between trees (and also that
pointers to them are only valid until a side-effectful tree operation
occurs). The overhead varies based on how dense the tree is, but is
usually on the order of about 50% of the element size; the per-node
overheads are very small, and so don't make a significant difference.
The trees can accept arbitrary records; they accept a size and a
comparator to allow them to be used for a variety of purposes.
The new trees replace the AVL trees used in the range trees today.
Currently, the range_seg_t structure contains three 8 byte integers
of payload and two 24 byte avl_tree_node_ts to handle its storage in
both an offset-sorted tree and a size-sorted tree (total size: 64
bytes). In the new model, the range seg structures are usually two 4
byte integers, but a separate one needs to exist for the size-sorted
and offset-sorted tree. Between the raw size, the 50% overhead, and
the double storage, the new btrees are expected to use 8*1.5*2 = 24
bytes per record, or 33.3% as much memory as the AVL trees (this is
for the purposes of storing metaslab range trees; for other purposes,
like scrubs, they use ~50% as much memory).
We reduced the size of the payload in the range segments by teaching
range trees about starting offsets and shifts; since metaslabs have a
fixed starting offset, and they all operate in terms of disk sectors,
we can store the ranges using 4-byte integers as long as the size of
the metaslab divided by the sector size is less than 2^32. For 512-byte
sectors, this is a 2^41 (or 2TB) metaslab, which with the default
settings corresponds to a 256PB disk. 4k sector disks can handle
metaslabs up to 2^46 bytes, or 2^63 byte disks. Since we do not
anticipate disks of this size in the near future, there should be
almost no cases where metaslabs need 64-byte integers to store their
ranges. We do still have the capability to store 64-byte integer ranges
to account for cases where we are storing per-vdev (or per-dnode) trees,
which could reasonably go above the limits discussed. We also do not
store fill information in the compact version of the node, since it
is only used for sorted scrub.
We also optimized the metaslab loading process in various other ways
to offset some inefficiencies in the btree model. While individual
operations (find, insert, remove_from) are faster for the btree than
they are for the avl tree, remove usually requires a find operation,
while in the AVL tree model the element itself suffices. Some clever
changes actually caused an overall speedup in metaslab loading; we use
approximately 40% less cpu to load metaslabs in our tests on Illumos.
Another memory and performance optimization was achieved by changing
what is stored in the size-sorted trees. When a disk is heavily
fragmented, the df algorithm used by default in ZFS will almost always
find a number of small regions in its initial cursor-based search; it
will usually only fall back to the size-sorted tree to find larger
regions. If we increase the size of the cursor-based search slightly,
and don't store segments that are smaller than a tunable size floor
in the size-sorted tree, we can further cut memory usage down to
below 20% of what the AVL trees store. This also results in further
reductions in CPU time spent loading metaslabs.
The 16KiB size floor was chosen because it results in substantial memory
usage reduction while not usually resulting in situations where we can't
find an appropriate chunk with the cursor and are forced to use an
oversized chunk from the size-sorted tree. In addition, even if we do
have to use an oversized chunk from the size-sorted tree, the chunk
would be too small to use for ZIL allocations, so it isn't as big of a
loss as it might otherwise be. And often, more small allocations will
follow the initial one, and the cursor search will now find the
remainder of the chunk we didn't use all of and use it for subsequent
allocations. Practical testing has shown little or no change in
fragmentation as a result of this change.
If the size-sorted tree becomes empty while the offset sorted one still
has entries, it will load all the entries from the offset sorted tree
and disregard the size floor until it is unloaded again. This operation
occurs rarely with the default setting, only on incredibly thoroughly
fragmented pools.
There are some other small changes to zdb to teach it to handle btrees,
but nothing major.
Reviewed-by: George Wilson <gwilson@delphix.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed by: Sebastien Roy seb@delphix.com
Reviewed-by: Igor Kozhukhov <igor@dilos.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #9181
2019-10-09 20:36:03 +03:00
|
|
|
return (TREE_CMP(seq1, seq2));
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
2012-12-14 03:24:15 +04:00
|
|
|
struct ubl_cbdata {
|
RAID-Z expansion feature
This feature allows disks to be added one at a time to a RAID-Z group,
expanding its capacity incrementally. This feature is especially useful
for small pools (typically with only one RAID-Z group), where there
isn't sufficient hardware to add capacity by adding a whole new RAID-Z
group (typically doubling the number of disks).
== Initiating expansion ==
A new device (disk) can be attached to an existing RAIDZ vdev, by
running `zpool attach POOL raidzP-N NEW_DEVICE`, e.g. `zpool attach tank
raidz2-0 sda`. The new device will become part of the RAIDZ group. A
"raidz expansion" will be initiated, and the new device will contribute
additional space to the RAIDZ group once the expansion completes.
The `feature@raidz_expansion` on-disk feature flag must be `enabled` to
initiate an expansion, and it remains `active` for the life of the pool.
In other words, pools with expanded RAIDZ vdevs can not be imported by
older releases of the ZFS software.
== During expansion ==
The expansion entails reading all allocated space from existing disks in
the RAIDZ group, and rewriting it to the new disks in the RAIDZ group
(including the newly added device).
The expansion progress can be monitored with `zpool status`.
Data redundancy is maintained during (and after) the expansion. If a
disk fails while the expansion is in progress, the expansion pauses
until the health of the RAIDZ vdev is restored (e.g. by replacing the
failed disk and waiting for reconstruction to complete).
The pool remains accessible during expansion. Following a reboot or
export/import, the expansion resumes where it left off.
== After expansion ==
When the expansion completes, the additional space is available for use,
and is reflected in the `available` zfs property (as seen in `zfs list`,
`df`, etc).
Expansion does not change the number of failures that can be tolerated
without data loss (e.g. a RAIDZ2 is still a RAIDZ2 even after
expansion).
A RAIDZ vdev can be expanded multiple times.
After the expansion completes, old blocks remain with their old
data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but
distributed among the larger set of disks. New blocks will be written
with the new data-to-parity ratio (e.g. a 5-wide RAIDZ2 which has been
expanded once to 6-wide, has 4 data to 2 parity). However, the RAIDZ
vdev's "assumed parity ratio" does not change, so slightly less space
than is expected may be reported for newly-written blocks, according to
`zfs list`, `df`, `ls -s`, and similar tools.
Sponsored-by: The FreeBSD Foundation
Sponsored-by: iXsystems, Inc.
Sponsored-by: vStack
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Mark Maybee <mark.maybee@delphix.com>
Authored-by: Matthew Ahrens <mahrens@delphix.com>
Contributions-by: Fedor Uporov <fuporov.vstack@gmail.com>
Contributions-by: Stuart Maybee <stuart.maybee@comcast.net>
Contributions-by: Thorsten Behrens <tbehrens@outlook.com>
Contributions-by: Fmstrat <nospam@nowsci.com>
Contributions-by: Don Brady <dev.fs.zfs@gmail.com>
Signed-off-by: Don Brady <dev.fs.zfs@gmail.com>
Closes #15022
2023-11-08 21:19:41 +03:00
|
|
|
uberblock_t ubl_latest; /* Most recent uberblock */
|
|
|
|
|
uberblock_t *ubl_ubbest; /* Best uberblock (w/r/t max_txg) */
|
2012-12-14 03:24:15 +04:00
|
|
|
vdev_t *ubl_vd; /* vdev associated with the above */
|
|
|
|
|
};
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
static void
|
|
|
|
|
vdev_uberblock_load_done(zio_t *zio)
|
|
|
|
|
{
|
2012-12-14 03:24:15 +04:00
|
|
|
vdev_t *vd = zio->io_vd;
|
2010-05-29 00:45:14 +04:00
|
|
|
spa_t *spa = zio->io_spa;
|
2008-12-03 23:09:06 +03:00
|
|
|
zio_t *rio = zio->io_private;
|
2016-07-22 18:52:49 +03:00
|
|
|
uberblock_t *ub = abd_to_buf(zio->io_abd);
|
2012-12-14 03:24:15 +04:00
|
|
|
struct ubl_cbdata *cbp = rio->io_private;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2012-12-14 03:24:15 +04:00
|
|
|
ASSERT3U(zio->io_size, ==, VDEV_UBERBLOCK_SIZE(vd));
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
if (zio->io_error == 0 && uberblock_verify(ub) == 0) {
|
2008-12-03 23:09:06 +03:00
|
|
|
mutex_enter(&rio->io_lock);
|
RAID-Z expansion feature
This feature allows disks to be added one at a time to a RAID-Z group,
expanding its capacity incrementally. This feature is especially useful
for small pools (typically with only one RAID-Z group), where there
isn't sufficient hardware to add capacity by adding a whole new RAID-Z
group (typically doubling the number of disks).
== Initiating expansion ==
A new device (disk) can be attached to an existing RAIDZ vdev, by
running `zpool attach POOL raidzP-N NEW_DEVICE`, e.g. `zpool attach tank
raidz2-0 sda`. The new device will become part of the RAIDZ group. A
"raidz expansion" will be initiated, and the new device will contribute
additional space to the RAIDZ group once the expansion completes.
The `feature@raidz_expansion` on-disk feature flag must be `enabled` to
initiate an expansion, and it remains `active` for the life of the pool.
In other words, pools with expanded RAIDZ vdevs can not be imported by
older releases of the ZFS software.
== During expansion ==
The expansion entails reading all allocated space from existing disks in
the RAIDZ group, and rewriting it to the new disks in the RAIDZ group
(including the newly added device).
The expansion progress can be monitored with `zpool status`.
Data redundancy is maintained during (and after) the expansion. If a
disk fails while the expansion is in progress, the expansion pauses
until the health of the RAIDZ vdev is restored (e.g. by replacing the
failed disk and waiting for reconstruction to complete).
The pool remains accessible during expansion. Following a reboot or
export/import, the expansion resumes where it left off.
== After expansion ==
When the expansion completes, the additional space is available for use,
and is reflected in the `available` zfs property (as seen in `zfs list`,
`df`, etc).
Expansion does not change the number of failures that can be tolerated
without data loss (e.g. a RAIDZ2 is still a RAIDZ2 even after
expansion).
A RAIDZ vdev can be expanded multiple times.
After the expansion completes, old blocks remain with their old
data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but
distributed among the larger set of disks. New blocks will be written
with the new data-to-parity ratio (e.g. a 5-wide RAIDZ2 which has been
expanded once to 6-wide, has 4 data to 2 parity). However, the RAIDZ
vdev's "assumed parity ratio" does not change, so slightly less space
than is expected may be reported for newly-written blocks, according to
`zfs list`, `df`, `ls -s`, and similar tools.
Sponsored-by: The FreeBSD Foundation
Sponsored-by: iXsystems, Inc.
Sponsored-by: vStack
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Mark Maybee <mark.maybee@delphix.com>
Authored-by: Matthew Ahrens <mahrens@delphix.com>
Contributions-by: Fedor Uporov <fuporov.vstack@gmail.com>
Contributions-by: Stuart Maybee <stuart.maybee@comcast.net>
Contributions-by: Thorsten Behrens <tbehrens@outlook.com>
Contributions-by: Fmstrat <nospam@nowsci.com>
Contributions-by: Don Brady <dev.fs.zfs@gmail.com>
Signed-off-by: Don Brady <dev.fs.zfs@gmail.com>
Closes #15022
2023-11-08 21:19:41 +03:00
|
|
|
if (vdev_uberblock_compare(ub, &cbp->ubl_latest) > 0) {
|
|
|
|
|
cbp->ubl_latest = *ub;
|
|
|
|
|
}
|
2010-05-29 00:45:14 +04:00
|
|
|
if (ub->ub_txg <= spa->spa_load_max_txg &&
|
2012-12-14 03:24:15 +04:00
|
|
|
vdev_uberblock_compare(ub, cbp->ubl_ubbest) > 0) {
|
|
|
|
|
/*
|
2012-12-15 00:38:04 +04:00
|
|
|
* Keep track of the vdev in which this uberblock
|
|
|
|
|
* was found. We will use this information later
|
|
|
|
|
* to obtain the config nvlist associated with
|
2012-12-14 03:24:15 +04:00
|
|
|
* this uberblock.
|
|
|
|
|
*/
|
|
|
|
|
*cbp->ubl_ubbest = *ub;
|
|
|
|
|
cbp->ubl_vd = vd;
|
|
|
|
|
}
|
2008-12-03 23:09:06 +03:00
|
|
|
mutex_exit(&rio->io_lock);
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
2016-07-22 18:52:49 +03:00
|
|
|
abd_free(zio->io_abd);
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
2012-12-14 03:24:15 +04:00
|
|
|
static void
|
|
|
|
|
vdev_uberblock_load_impl(zio_t *zio, vdev_t *vd, int flags,
|
|
|
|
|
struct ubl_cbdata *cbp)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
Multi-modifier protection (MMP)
Add multihost=on|off pool property to control MMP. When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp. Property defaults to off.
During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock. Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".
Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval. The period is specified in milliseconds. The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.
Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path. Abbreviated
output below.
$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg timestamp mmp_delay vdev_guid vdev_label vdev_path
20468 261337 250274925 68396651780 3 /dev/sda
20468 261339 252023374 6267402363293 1 /dev/sdc
20468 261340 252000858 6698080955233 1 /dev/sdx
20468 261341 251980635 783892869810 2 /dev/sdy
20468 261342 253385953 8923255792467 3 /dev/sdd
20468 261344 253336622 042125143176 0 /dev/sdab
20468 261345 253310522 1200778101278 2 /dev/sde
20468 261346 253286429 0950576198362 2 /dev/sdt
20468 261347 253261545 96209817917 3 /dev/sds
20468 261349 253238188 8555725937673 3 /dev/sdb
Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.
When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test. For example, the
pool is exported to run zdb and then imported again. Add a new ztest
function, "-M", to alter ztest behavior to prevent this.
Add new tests to verify the new functionality. Tests provided by
Giuseppe Di Natale.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #745
Closes #6279
2017-07-08 06:20:35 +03:00
|
|
|
for (int c = 0; c < vd->vdev_children; c++)
|
2012-12-14 03:24:15 +04:00
|
|
|
vdev_uberblock_load_impl(zio, vd->vdev_child[c], flags, cbp);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
Distributed Spare (dRAID) Feature
This patch adds a new top-level vdev type called dRAID, which stands
for Distributed parity RAID. This pool configuration allows all dRAID
vdevs to participate when rebuilding to a distributed hot spare device.
This can substantially reduce the total time required to restore full
parity to pool with a failed device.
A dRAID pool can be created using the new top-level `draid` type.
Like `raidz`, the desired redundancy is specified after the type:
`draid[1,2,3]`. No additional information is required to create the
pool and reasonable default values will be chosen based on the number
of child vdevs in the dRAID vdev.
zpool create <pool> draid[1,2,3] <vdevs...>
Unlike raidz, additional optional dRAID configuration values can be
provided as part of the draid type as colon separated values. This
allows administrators to fully specify a layout for either performance
or capacity reasons. The supported options include:
zpool create <pool> \
draid[<parity>][:<data>d][:<children>c][:<spares>s] \
<vdevs...>
- draid[parity] - Parity level (default 1)
- draid[:<data>d] - Data devices per group (default 8)
- draid[:<children>c] - Expected number of child vdevs
- draid[:<spares>s] - Distributed hot spares (default 0)
Abbreviated example `zpool status` output for a 68 disk dRAID pool
with two distributed spares using special allocation classes.
```
pool: tank
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
slag7 ONLINE 0 0 0
draid2:8d:68c:2s-0 ONLINE 0 0 0
L0 ONLINE 0 0 0
L1 ONLINE 0 0 0
...
U25 ONLINE 0 0 0
U26 ONLINE 0 0 0
spare-53 ONLINE 0 0 0
U27 ONLINE 0 0 0
draid2-0-0 ONLINE 0 0 0
U28 ONLINE 0 0 0
U29 ONLINE 0 0 0
...
U42 ONLINE 0 0 0
U43 ONLINE 0 0 0
special
mirror-1 ONLINE 0 0 0
L5 ONLINE 0 0 0
U5 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
L6 ONLINE 0 0 0
U6 ONLINE 0 0 0
spares
draid2-0-0 INUSE currently in use
draid2-0-1 AVAIL
```
When adding test coverage for the new dRAID vdev type the following
options were added to the ztest command. These options are leverages
by zloop.sh to test a wide range of dRAID configurations.
-K draid|raidz|random - kind of RAID to test
-D <value> - dRAID data drives per group
-S <value> - dRAID distributed hot spares
-R <value> - RAID parity (raidz or dRAID)
The zpool_create, zpool_import, redundancy, replacement and fault
test groups have all been updated provide test coverage for the
dRAID feature.
Co-authored-by: Isaac Huang <he.huang@intel.com>
Co-authored-by: Mark Maybee <mmaybee@cray.com>
Co-authored-by: Don Brady <don.brady@delphix.com>
Co-authored-by: Matthew Ahrens <mahrens@delphix.com>
Co-authored-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Mark Maybee <mmaybee@cray.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #10102
2020-11-14 00:51:51 +03:00
|
|
|
if (vd->vdev_ops->vdev_op_leaf && vdev_readable(vd) &&
|
|
|
|
|
vd->vdev_ops != &vdev_draid_spare_ops) {
|
Multi-modifier protection (MMP)
Add multihost=on|off pool property to control MMP. When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp. Property defaults to off.
During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock. Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".
Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval. The period is specified in milliseconds. The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.
Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path. Abbreviated
output below.
$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg timestamp mmp_delay vdev_guid vdev_label vdev_path
20468 261337 250274925 68396651780 3 /dev/sda
20468 261339 252023374 6267402363293 1 /dev/sdc
20468 261340 252000858 6698080955233 1 /dev/sdx
20468 261341 251980635 783892869810 2 /dev/sdy
20468 261342 253385953 8923255792467 3 /dev/sdd
20468 261344 253336622 042125143176 0 /dev/sdab
20468 261345 253310522 1200778101278 2 /dev/sde
20468 261346 253286429 0950576198362 2 /dev/sdt
20468 261347 253261545 96209817917 3 /dev/sds
20468 261349 253238188 8555725937673 3 /dev/sdb
Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.
When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test. For example, the
pool is exported to run zdb and then imported again. Add a new ztest
function, "-M", to alter ztest behavior to prevent this.
Add new tests to verify the new functionality. Tests provided by
Giuseppe Di Natale.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #745
Closes #6279
2017-07-08 06:20:35 +03:00
|
|
|
for (int l = 0; l < VDEV_LABELS; l++) {
|
|
|
|
|
for (int n = 0; n < VDEV_UBERBLOCK_COUNT(vd); n++) {
|
2008-12-03 23:09:06 +03:00
|
|
|
vdev_label_read(zio, vd, l,
|
2016-07-22 18:52:49 +03:00
|
|
|
abd_alloc_linear(VDEV_UBERBLOCK_SIZE(vd),
|
|
|
|
|
B_TRUE), VDEV_UBERBLOCK_OFFSET(vd, n),
|
2008-12-03 23:09:06 +03:00
|
|
|
VDEV_UBERBLOCK_SIZE(vd),
|
|
|
|
|
vdev_uberblock_load_done, zio, flags);
|
|
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
}
|
2012-12-14 03:24:15 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Reads the 'best' uberblock from disk along with its associated
|
|
|
|
|
* configuration. First, we read the uberblock array of each label of each
|
|
|
|
|
* vdev, keeping track of the uberblock with the highest txg in each array.
|
2012-12-15 00:38:04 +04:00
|
|
|
* Then, we read the configuration from the same vdev as the best uberblock.
|
2012-12-14 03:24:15 +04:00
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
vdev_uberblock_load(vdev_t *rvd, uberblock_t *ub, nvlist_t **config)
|
|
|
|
|
{
|
|
|
|
|
zio_t *zio;
|
|
|
|
|
spa_t *spa = rvd->vdev_spa;
|
|
|
|
|
struct ubl_cbdata cb;
|
|
|
|
|
int flags = ZIO_FLAG_CONFIG_WRITER | ZIO_FLAG_CANFAIL |
|
|
|
|
|
ZIO_FLAG_SPECULATIVE | ZIO_FLAG_TRYHARD;
|
|
|
|
|
|
|
|
|
|
ASSERT(ub);
|
|
|
|
|
ASSERT(config);
|
2008-12-03 23:09:06 +03:00
|
|
|
|
2022-02-25 16:26:54 +03:00
|
|
|
memset(ub, 0, sizeof (uberblock_t));
|
RAID-Z expansion feature
This feature allows disks to be added one at a time to a RAID-Z group,
expanding its capacity incrementally. This feature is especially useful
for small pools (typically with only one RAID-Z group), where there
isn't sufficient hardware to add capacity by adding a whole new RAID-Z
group (typically doubling the number of disks).
== Initiating expansion ==
A new device (disk) can be attached to an existing RAIDZ vdev, by
running `zpool attach POOL raidzP-N NEW_DEVICE`, e.g. `zpool attach tank
raidz2-0 sda`. The new device will become part of the RAIDZ group. A
"raidz expansion" will be initiated, and the new device will contribute
additional space to the RAIDZ group once the expansion completes.
The `feature@raidz_expansion` on-disk feature flag must be `enabled` to
initiate an expansion, and it remains `active` for the life of the pool.
In other words, pools with expanded RAIDZ vdevs can not be imported by
older releases of the ZFS software.
== During expansion ==
The expansion entails reading all allocated space from existing disks in
the RAIDZ group, and rewriting it to the new disks in the RAIDZ group
(including the newly added device).
The expansion progress can be monitored with `zpool status`.
Data redundancy is maintained during (and after) the expansion. If a
disk fails while the expansion is in progress, the expansion pauses
until the health of the RAIDZ vdev is restored (e.g. by replacing the
failed disk and waiting for reconstruction to complete).
The pool remains accessible during expansion. Following a reboot or
export/import, the expansion resumes where it left off.
== After expansion ==
When the expansion completes, the additional space is available for use,
and is reflected in the `available` zfs property (as seen in `zfs list`,
`df`, etc).
Expansion does not change the number of failures that can be tolerated
without data loss (e.g. a RAIDZ2 is still a RAIDZ2 even after
expansion).
A RAIDZ vdev can be expanded multiple times.
After the expansion completes, old blocks remain with their old
data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but
distributed among the larger set of disks. New blocks will be written
with the new data-to-parity ratio (e.g. a 5-wide RAIDZ2 which has been
expanded once to 6-wide, has 4 data to 2 parity). However, the RAIDZ
vdev's "assumed parity ratio" does not change, so slightly less space
than is expected may be reported for newly-written blocks, according to
`zfs list`, `df`, `ls -s`, and similar tools.
Sponsored-by: The FreeBSD Foundation
Sponsored-by: iXsystems, Inc.
Sponsored-by: vStack
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Mark Maybee <mark.maybee@delphix.com>
Authored-by: Matthew Ahrens <mahrens@delphix.com>
Contributions-by: Fedor Uporov <fuporov.vstack@gmail.com>
Contributions-by: Stuart Maybee <stuart.maybee@comcast.net>
Contributions-by: Thorsten Behrens <tbehrens@outlook.com>
Contributions-by: Fmstrat <nospam@nowsci.com>
Contributions-by: Don Brady <dev.fs.zfs@gmail.com>
Signed-off-by: Don Brady <dev.fs.zfs@gmail.com>
Closes #15022
2023-11-08 21:19:41 +03:00
|
|
|
memset(&cb, 0, sizeof (cb));
|
2012-12-14 03:24:15 +04:00
|
|
|
*config = NULL;
|
|
|
|
|
|
|
|
|
|
cb.ubl_ubbest = ub;
|
|
|
|
|
|
|
|
|
|
spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);
|
|
|
|
|
zio = zio_root(spa, NULL, &cb, flags);
|
|
|
|
|
vdev_uberblock_load_impl(zio, rvd, flags, &cb);
|
|
|
|
|
(void) zio_wait(zio);
|
2012-12-15 00:38:04 +04:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* It's possible that the best uberblock was discovered on a label
|
|
|
|
|
* that has a configuration which was written in a future txg.
|
|
|
|
|
* Search all labels on this vdev to find the configuration that
|
|
|
|
|
* matches the txg for our uberblock.
|
|
|
|
|
*/
|
2016-03-10 18:16:02 +03:00
|
|
|
if (cb.ubl_vd != NULL) {
|
|
|
|
|
vdev_dbgmsg(cb.ubl_vd, "best uberblock found for spa %s. "
|
|
|
|
|
"txg %llu", spa->spa_name, (u_longlong_t)ub->ub_txg);
|
|
|
|
|
|
RAID-Z expansion feature
This feature allows disks to be added one at a time to a RAID-Z group,
expanding its capacity incrementally. This feature is especially useful
for small pools (typically with only one RAID-Z group), where there
isn't sufficient hardware to add capacity by adding a whole new RAID-Z
group (typically doubling the number of disks).
== Initiating expansion ==
A new device (disk) can be attached to an existing RAIDZ vdev, by
running `zpool attach POOL raidzP-N NEW_DEVICE`, e.g. `zpool attach tank
raidz2-0 sda`. The new device will become part of the RAIDZ group. A
"raidz expansion" will be initiated, and the new device will contribute
additional space to the RAIDZ group once the expansion completes.
The `feature@raidz_expansion` on-disk feature flag must be `enabled` to
initiate an expansion, and it remains `active` for the life of the pool.
In other words, pools with expanded RAIDZ vdevs can not be imported by
older releases of the ZFS software.
== During expansion ==
The expansion entails reading all allocated space from existing disks in
the RAIDZ group, and rewriting it to the new disks in the RAIDZ group
(including the newly added device).
The expansion progress can be monitored with `zpool status`.
Data redundancy is maintained during (and after) the expansion. If a
disk fails while the expansion is in progress, the expansion pauses
until the health of the RAIDZ vdev is restored (e.g. by replacing the
failed disk and waiting for reconstruction to complete).
The pool remains accessible during expansion. Following a reboot or
export/import, the expansion resumes where it left off.
== After expansion ==
When the expansion completes, the additional space is available for use,
and is reflected in the `available` zfs property (as seen in `zfs list`,
`df`, etc).
Expansion does not change the number of failures that can be tolerated
without data loss (e.g. a RAIDZ2 is still a RAIDZ2 even after
expansion).
A RAIDZ vdev can be expanded multiple times.
After the expansion completes, old blocks remain with their old
data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but
distributed among the larger set of disks. New blocks will be written
with the new data-to-parity ratio (e.g. a 5-wide RAIDZ2 which has been
expanded once to 6-wide, has 4 data to 2 parity). However, the RAIDZ
vdev's "assumed parity ratio" does not change, so slightly less space
than is expected may be reported for newly-written blocks, according to
`zfs list`, `df`, `ls -s`, and similar tools.
Sponsored-by: The FreeBSD Foundation
Sponsored-by: iXsystems, Inc.
Sponsored-by: vStack
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Mark Maybee <mark.maybee@delphix.com>
Authored-by: Matthew Ahrens <mahrens@delphix.com>
Contributions-by: Fedor Uporov <fuporov.vstack@gmail.com>
Contributions-by: Stuart Maybee <stuart.maybee@comcast.net>
Contributions-by: Thorsten Behrens <tbehrens@outlook.com>
Contributions-by: Fmstrat <nospam@nowsci.com>
Contributions-by: Don Brady <dev.fs.zfs@gmail.com>
Signed-off-by: Don Brady <dev.fs.zfs@gmail.com>
Closes #15022
2023-11-08 21:19:41 +03:00
|
|
|
if (ub->ub_raidz_reflow_info !=
|
|
|
|
|
cb.ubl_latest.ub_raidz_reflow_info) {
|
|
|
|
|
vdev_dbgmsg(cb.ubl_vd,
|
|
|
|
|
"spa=%s best uberblock (txg=%llu info=0x%llx) "
|
|
|
|
|
"has different raidz_reflow_info than latest "
|
|
|
|
|
"uberblock (txg=%llu info=0x%llx)",
|
|
|
|
|
spa->spa_name,
|
|
|
|
|
(u_longlong_t)ub->ub_txg,
|
|
|
|
|
(u_longlong_t)ub->ub_raidz_reflow_info,
|
|
|
|
|
(u_longlong_t)cb.ubl_latest.ub_txg,
|
|
|
|
|
(u_longlong_t)cb.ubl_latest.ub_raidz_reflow_info);
|
|
|
|
|
memset(ub, 0, sizeof (uberblock_t));
|
|
|
|
|
spa_config_exit(spa, SCL_ALL, FTAG);
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
|
2012-12-15 00:38:04 +04:00
|
|
|
*config = vdev_label_read_config(cb.ubl_vd, ub->ub_txg);
|
OpenZFS 9075 - Improve ZFS pool import/load process and corrupted pool recovery
Some work has been done lately to improve the debugability of the ZFS pool
load (and import) process. This includes:
7638 Refactor spa_load_impl into several functions
8961 SPA load/import should tell us why it failed
7277 zdb should be able to print zfs_dbgmsg's
To iterate on top of that, there's a few changes that were made to make the
import process more resilient and crash free. One of the first tasks during the
pool load process is to parse a config provided from userland that describes
what devices the pool is composed of. A vdev tree is generated from that config,
and then all the vdevs are opened.
The Meta Object Set (MOS) of the pool is accessed, and several metadata objects
that are necessary to load the pool are read. The exact configuration of the
pool is also stored inside the MOS. Since the configuration provided from
userland is external and might not accurately describe the vdev tree
of the pool at the txg that is being loaded, it cannot be relied upon to safely
operate the pool. For that reason, the configuration in the MOS is read early
on. In the past, the two configurations were compared together and if there was
a mismatch then the load process was aborted and an error was returned.
The latter was a good way to ensure a pool does not get corrupted, however it
made the pool load process needlessly fragile in cases where the vdev
configuration changed or the userland configuration was outdated. Since the MOS
is stored in 3 copies, the configuration provided by userland doesn't have to be
perfect in order to read its contents. Hence, a new approach has been adopted:
The pool is first opened with the untrusted userland configuration just so that
the real configuration can be read from the MOS. The trusted MOS configuration
is then used to generate a new vdev tree and the pool is re-opened.
When the pool is opened with an untrusted configuration, writes are disabled
to avoid accidentally damaging it. During reads, some sanity checks are
performed on block pointers to see if each DVA points to a known vdev;
when the configuration is untrusted, instead of panicking the system if those
checks fail we simply avoid issuing reads to the invalid DVAs.
This new two-step pool load process now allows rewinding pools accross
vdev tree changes such as device replacement, addition, etc. Loading a pool
from an external config file in a clustering environment also becomes much
safer now since the pool will import even if the config is outdated and didn't,
for instance, register a recent device addition.
With this code in place, it became relatively easy to implement a
long-sought-after feature: the ability to import a pool with missing top level
(i.e. non-redundant) devices. Note that since this almost guarantees some loss
of data, this feature is for now restricted to a read-only import.
Porting notes (ZTS):
* Fix 'make dist' target in zpool_import
* The maximum path length allowed by tar is 99 characters. Several
of the new test cases exceeded this limit resulting in them not
being included in the tarball. Shorten the names slightly.
* Set/get tunables using accessor functions.
* Get last synced txg via the "zfs_txg_history" mechanism.
* Clear zinject handlers in cleanup for import_cache_device_replaced
and import_rewind_device_replaced in order that the zpool can be
exported if there is an error.
* Increase FILESIZE to 8G in zfs-test.sh to allow for a larger
ext4 file system to be created on ZFS_DISK2. Also, there's
no need to partition ZFS_DISK2 at all. The partitioning had
already been disabled for multipath devices. Among other things,
the partitioning steals some space from the ext4 file system,
makes it difficult to accurately calculate the paramters to
parted and can make some of the tests fail.
* Increase FS_SIZE and FILE_SIZE in the zpool_import test
configuration now that FILESIZE is larger.
* Write more data in order that device evacuation take lonnger in
a couple tests.
* Use mkdir -p to avoid errors when the directory already exists.
* Remove use of sudo in import_rewind_config_changed.
Authored by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Andrew Stormont <andyjstormont@gmail.com>
Approved by: Hans Rosenfeld <rosenfeld@grumpf.hope-2000.org>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://illumos.org/issues/9075
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/619c0123
Closes #7459
2016-07-22 17:39:36 +03:00
|
|
|
if (*config == NULL && spa->spa_extreme_rewind) {
|
|
|
|
|
vdev_dbgmsg(cb.ubl_vd, "failed to read label config. "
|
|
|
|
|
"Trying again without txg restrictions.");
|
|
|
|
|
*config = vdev_label_read_config(cb.ubl_vd, UINT64_MAX);
|
|
|
|
|
}
|
2016-03-10 18:16:02 +03:00
|
|
|
if (*config == NULL) {
|
|
|
|
|
vdev_dbgmsg(cb.ubl_vd, "failed to read label config");
|
|
|
|
|
}
|
|
|
|
|
}
|
2012-12-14 03:24:15 +04:00
|
|
|
spa_config_exit(spa, SCL_ALL, FTAG);
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
2017-05-02 23:55:24 +03:00
|
|
|
/*
|
|
|
|
|
* For use when a leaf vdev is expanded.
|
|
|
|
|
* The location of labels 2 and 3 changed, and at the new location the
|
|
|
|
|
* uberblock rings are either empty or contain garbage. The sync will write
|
|
|
|
|
* new configs there because the vdev is dirty, but expansion also needs the
|
|
|
|
|
* uberblock rings copied. Read them from label 0 which did not move.
|
|
|
|
|
*
|
|
|
|
|
* Since the point is to populate labels {2,3} with valid uberblocks,
|
|
|
|
|
* we zero uberblocks we fail to read or which are not valid.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
vdev_copy_uberblocks(vdev_t *vd)
|
|
|
|
|
{
|
|
|
|
|
abd_t *ub_abd;
|
|
|
|
|
zio_t *write_zio;
|
|
|
|
|
int locks = (SCL_L2ARC | SCL_ZIO);
|
|
|
|
|
int flags = ZIO_FLAG_CONFIG_WRITER | ZIO_FLAG_CANFAIL |
|
|
|
|
|
ZIO_FLAG_SPECULATIVE;
|
|
|
|
|
|
|
|
|
|
ASSERT(spa_config_held(vd->vdev_spa, SCL_STATE, RW_READER) ==
|
|
|
|
|
SCL_STATE);
|
|
|
|
|
ASSERT(vd->vdev_ops->vdev_op_leaf);
|
|
|
|
|
|
Distributed Spare (dRAID) Feature
This patch adds a new top-level vdev type called dRAID, which stands
for Distributed parity RAID. This pool configuration allows all dRAID
vdevs to participate when rebuilding to a distributed hot spare device.
This can substantially reduce the total time required to restore full
parity to pool with a failed device.
A dRAID pool can be created using the new top-level `draid` type.
Like `raidz`, the desired redundancy is specified after the type:
`draid[1,2,3]`. No additional information is required to create the
pool and reasonable default values will be chosen based on the number
of child vdevs in the dRAID vdev.
zpool create <pool> draid[1,2,3] <vdevs...>
Unlike raidz, additional optional dRAID configuration values can be
provided as part of the draid type as colon separated values. This
allows administrators to fully specify a layout for either performance
or capacity reasons. The supported options include:
zpool create <pool> \
draid[<parity>][:<data>d][:<children>c][:<spares>s] \
<vdevs...>
- draid[parity] - Parity level (default 1)
- draid[:<data>d] - Data devices per group (default 8)
- draid[:<children>c] - Expected number of child vdevs
- draid[:<spares>s] - Distributed hot spares (default 0)
Abbreviated example `zpool status` output for a 68 disk dRAID pool
with two distributed spares using special allocation classes.
```
pool: tank
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
slag7 ONLINE 0 0 0
draid2:8d:68c:2s-0 ONLINE 0 0 0
L0 ONLINE 0 0 0
L1 ONLINE 0 0 0
...
U25 ONLINE 0 0 0
U26 ONLINE 0 0 0
spare-53 ONLINE 0 0 0
U27 ONLINE 0 0 0
draid2-0-0 ONLINE 0 0 0
U28 ONLINE 0 0 0
U29 ONLINE 0 0 0
...
U42 ONLINE 0 0 0
U43 ONLINE 0 0 0
special
mirror-1 ONLINE 0 0 0
L5 ONLINE 0 0 0
U5 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
L6 ONLINE 0 0 0
U6 ONLINE 0 0 0
spares
draid2-0-0 INUSE currently in use
draid2-0-1 AVAIL
```
When adding test coverage for the new dRAID vdev type the following
options were added to the ztest command. These options are leverages
by zloop.sh to test a wide range of dRAID configurations.
-K draid|raidz|random - kind of RAID to test
-D <value> - dRAID data drives per group
-S <value> - dRAID distributed hot spares
-R <value> - RAID parity (raidz or dRAID)
The zpool_create, zpool_import, redundancy, replacement and fault
test groups have all been updated provide test coverage for the
dRAID feature.
Co-authored-by: Isaac Huang <he.huang@intel.com>
Co-authored-by: Mark Maybee <mmaybee@cray.com>
Co-authored-by: Don Brady <don.brady@delphix.com>
Co-authored-by: Matthew Ahrens <mahrens@delphix.com>
Co-authored-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Mark Maybee <mmaybee@cray.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #10102
2020-11-14 00:51:51 +03:00
|
|
|
/*
|
|
|
|
|
* No uberblocks are stored on distributed spares, they may be
|
|
|
|
|
* safely skipped when expanding a leaf vdev.
|
|
|
|
|
*/
|
|
|
|
|
if (vd->vdev_ops == &vdev_draid_spare_ops)
|
|
|
|
|
return;
|
|
|
|
|
|
2017-05-02 23:55:24 +03:00
|
|
|
spa_config_enter(vd->vdev_spa, locks, FTAG, RW_READER);
|
|
|
|
|
|
2017-10-06 05:30:02 +03:00
|
|
|
ub_abd = abd_alloc_linear(VDEV_UBERBLOCK_SIZE(vd), B_TRUE);
|
2017-05-02 23:55:24 +03:00
|
|
|
|
|
|
|
|
write_zio = zio_root(vd->vdev_spa, NULL, NULL, flags);
|
|
|
|
|
for (int n = 0; n < VDEV_UBERBLOCK_COUNT(vd); n++) {
|
|
|
|
|
const int src_label = 0;
|
|
|
|
|
zio_t *zio;
|
|
|
|
|
|
|
|
|
|
zio = zio_root(vd->vdev_spa, NULL, NULL, flags);
|
|
|
|
|
vdev_label_read(zio, vd, src_label, ub_abd,
|
|
|
|
|
VDEV_UBERBLOCK_OFFSET(vd, n), VDEV_UBERBLOCK_SIZE(vd),
|
|
|
|
|
NULL, NULL, flags);
|
|
|
|
|
|
|
|
|
|
if (zio_wait(zio) || uberblock_verify(abd_to_buf(ub_abd)))
|
|
|
|
|
abd_zero(ub_abd, VDEV_UBERBLOCK_SIZE(vd));
|
|
|
|
|
|
|
|
|
|
for (int l = 2; l < VDEV_LABELS; l++)
|
|
|
|
|
vdev_label_write(write_zio, vd, l, ub_abd,
|
|
|
|
|
VDEV_UBERBLOCK_OFFSET(vd, n),
|
|
|
|
|
VDEV_UBERBLOCK_SIZE(vd), NULL, NULL,
|
|
|
|
|
flags | ZIO_FLAG_DONT_PROPAGATE);
|
|
|
|
|
}
|
|
|
|
|
(void) zio_wait(write_zio);
|
|
|
|
|
|
|
|
|
|
spa_config_exit(vd->vdev_spa, locks, FTAG);
|
|
|
|
|
|
|
|
|
|
abd_free(ub_abd);
|
|
|
|
|
}
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
/*
|
|
|
|
|
* On success, increment root zio's count of good writes.
|
|
|
|
|
* We only get credit for writes to known-visible vdevs; see spa_vdev_add().
|
|
|
|
|
*/
|
|
|
|
|
static void
|
|
|
|
|
vdev_uberblock_sync_done(zio_t *zio)
|
|
|
|
|
{
|
|
|
|
|
uint64_t *good_writes = zio->io_private;
|
|
|
|
|
|
|
|
|
|
if (zio->io_error == 0 && zio->io_vd->vdev_top->vdev_ms_array != 0)
|
2016-01-14 03:37:41 +03:00
|
|
|
atomic_inc_64(good_writes);
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Write the uberblock to all labels of all leaves of the specified vdev.
|
|
|
|
|
*/
|
|
|
|
|
static void
|
2016-12-16 21:31:01 +03:00
|
|
|
vdev_uberblock_sync(zio_t *zio, uint64_t *good_writes,
|
|
|
|
|
uberblock_t *ub, vdev_t *vd, int flags)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
2016-12-16 21:31:01 +03:00
|
|
|
for (uint64_t c = 0; c < vd->vdev_children; c++) {
|
|
|
|
|
vdev_uberblock_sync(zio, good_writes,
|
|
|
|
|
ub, vd->vdev_child[c], flags);
|
|
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
if (!vd->vdev_ops->vdev_op_leaf)
|
|
|
|
|
return;
|
|
|
|
|
|
2008-12-03 23:09:06 +03:00
|
|
|
if (!vdev_writeable(vd))
|
2008-11-20 23:01:55 +03:00
|
|
|
return;
|
|
|
|
|
|
Distributed Spare (dRAID) Feature
This patch adds a new top-level vdev type called dRAID, which stands
for Distributed parity RAID. This pool configuration allows all dRAID
vdevs to participate when rebuilding to a distributed hot spare device.
This can substantially reduce the total time required to restore full
parity to pool with a failed device.
A dRAID pool can be created using the new top-level `draid` type.
Like `raidz`, the desired redundancy is specified after the type:
`draid[1,2,3]`. No additional information is required to create the
pool and reasonable default values will be chosen based on the number
of child vdevs in the dRAID vdev.
zpool create <pool> draid[1,2,3] <vdevs...>
Unlike raidz, additional optional dRAID configuration values can be
provided as part of the draid type as colon separated values. This
allows administrators to fully specify a layout for either performance
or capacity reasons. The supported options include:
zpool create <pool> \
draid[<parity>][:<data>d][:<children>c][:<spares>s] \
<vdevs...>
- draid[parity] - Parity level (default 1)
- draid[:<data>d] - Data devices per group (default 8)
- draid[:<children>c] - Expected number of child vdevs
- draid[:<spares>s] - Distributed hot spares (default 0)
Abbreviated example `zpool status` output for a 68 disk dRAID pool
with two distributed spares using special allocation classes.
```
pool: tank
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
slag7 ONLINE 0 0 0
draid2:8d:68c:2s-0 ONLINE 0 0 0
L0 ONLINE 0 0 0
L1 ONLINE 0 0 0
...
U25 ONLINE 0 0 0
U26 ONLINE 0 0 0
spare-53 ONLINE 0 0 0
U27 ONLINE 0 0 0
draid2-0-0 ONLINE 0 0 0
U28 ONLINE 0 0 0
U29 ONLINE 0 0 0
...
U42 ONLINE 0 0 0
U43 ONLINE 0 0 0
special
mirror-1 ONLINE 0 0 0
L5 ONLINE 0 0 0
U5 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
L6 ONLINE 0 0 0
U6 ONLINE 0 0 0
spares
draid2-0-0 INUSE currently in use
draid2-0-1 AVAIL
```
When adding test coverage for the new dRAID vdev type the following
options were added to the ztest command. These options are leverages
by zloop.sh to test a wide range of dRAID configurations.
-K draid|raidz|random - kind of RAID to test
-D <value> - dRAID data drives per group
-S <value> - dRAID distributed hot spares
-R <value> - RAID parity (raidz or dRAID)
The zpool_create, zpool_import, redundancy, replacement and fault
test groups have all been updated provide test coverage for the
dRAID feature.
Co-authored-by: Isaac Huang <he.huang@intel.com>
Co-authored-by: Mark Maybee <mmaybee@cray.com>
Co-authored-by: Don Brady <don.brady@delphix.com>
Co-authored-by: Matthew Ahrens <mahrens@delphix.com>
Co-authored-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Mark Maybee <mmaybee@cray.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #10102
2020-11-14 00:51:51 +03:00
|
|
|
/*
|
|
|
|
|
* There's no need to write uberblocks to a distributed spare, they
|
|
|
|
|
* are already stored on all the leaves of the parent dRAID. For
|
|
|
|
|
* this same reason vdev_uberblock_load_impl() skips distributed
|
|
|
|
|
* spares when reading uberblocks.
|
|
|
|
|
*/
|
|
|
|
|
if (vd->vdev_ops == &vdev_draid_spare_ops)
|
|
|
|
|
return;
|
|
|
|
|
|
2017-05-02 23:55:24 +03:00
|
|
|
/* If the vdev was expanded, need to copy uberblock rings. */
|
|
|
|
|
if (vd->vdev_state == VDEV_STATE_HEALTHY &&
|
|
|
|
|
vd->vdev_copy_uberblocks == B_TRUE) {
|
|
|
|
|
vdev_copy_uberblocks(vd);
|
|
|
|
|
vd->vdev_copy_uberblocks = B_FALSE;
|
|
|
|
|
}
|
|
|
|
|
|
RAID-Z expansion feature
This feature allows disks to be added one at a time to a RAID-Z group,
expanding its capacity incrementally. This feature is especially useful
for small pools (typically with only one RAID-Z group), where there
isn't sufficient hardware to add capacity by adding a whole new RAID-Z
group (typically doubling the number of disks).
== Initiating expansion ==
A new device (disk) can be attached to an existing RAIDZ vdev, by
running `zpool attach POOL raidzP-N NEW_DEVICE`, e.g. `zpool attach tank
raidz2-0 sda`. The new device will become part of the RAIDZ group. A
"raidz expansion" will be initiated, and the new device will contribute
additional space to the RAIDZ group once the expansion completes.
The `feature@raidz_expansion` on-disk feature flag must be `enabled` to
initiate an expansion, and it remains `active` for the life of the pool.
In other words, pools with expanded RAIDZ vdevs can not be imported by
older releases of the ZFS software.
== During expansion ==
The expansion entails reading all allocated space from existing disks in
the RAIDZ group, and rewriting it to the new disks in the RAIDZ group
(including the newly added device).
The expansion progress can be monitored with `zpool status`.
Data redundancy is maintained during (and after) the expansion. If a
disk fails while the expansion is in progress, the expansion pauses
until the health of the RAIDZ vdev is restored (e.g. by replacing the
failed disk and waiting for reconstruction to complete).
The pool remains accessible during expansion. Following a reboot or
export/import, the expansion resumes where it left off.
== After expansion ==
When the expansion completes, the additional space is available for use,
and is reflected in the `available` zfs property (as seen in `zfs list`,
`df`, etc).
Expansion does not change the number of failures that can be tolerated
without data loss (e.g. a RAIDZ2 is still a RAIDZ2 even after
expansion).
A RAIDZ vdev can be expanded multiple times.
After the expansion completes, old blocks remain with their old
data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but
distributed among the larger set of disks. New blocks will be written
with the new data-to-parity ratio (e.g. a 5-wide RAIDZ2 which has been
expanded once to 6-wide, has 4 data to 2 parity). However, the RAIDZ
vdev's "assumed parity ratio" does not change, so slightly less space
than is expected may be reported for newly-written blocks, according to
`zfs list`, `df`, `ls -s`, and similar tools.
Sponsored-by: The FreeBSD Foundation
Sponsored-by: iXsystems, Inc.
Sponsored-by: vStack
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Mark Maybee <mark.maybee@delphix.com>
Authored-by: Matthew Ahrens <mahrens@delphix.com>
Contributions-by: Fedor Uporov <fuporov.vstack@gmail.com>
Contributions-by: Stuart Maybee <stuart.maybee@comcast.net>
Contributions-by: Thorsten Behrens <tbehrens@outlook.com>
Contributions-by: Fmstrat <nospam@nowsci.com>
Contributions-by: Don Brady <dev.fs.zfs@gmail.com>
Signed-off-by: Don Brady <dev.fs.zfs@gmail.com>
Closes #15022
2023-11-08 21:19:41 +03:00
|
|
|
/*
|
|
|
|
|
* We chose a slot based on the txg. If this uberblock has a special
|
|
|
|
|
* RAIDZ expansion state, then it is essentially an update of the
|
|
|
|
|
* current uberblock (it has the same txg). However, the current
|
|
|
|
|
* state is committed, so we want to write it to a different slot. If
|
|
|
|
|
* we overwrote the same slot, and we lose power during the uberblock
|
|
|
|
|
* write, and the disk does not do single-sector overwrites
|
|
|
|
|
* atomically (even though it is required to - i.e. we should see
|
|
|
|
|
* either the old or the new uberblock), then we could lose this
|
|
|
|
|
* txg's uberblock. Rewinding to the previous txg's uberblock may not
|
|
|
|
|
* be possible because RAIDZ expansion may have already overwritten
|
|
|
|
|
* some of the data, so we need the progress indicator in the
|
|
|
|
|
* uberblock.
|
|
|
|
|
*/
|
Multi-modifier protection (MMP)
Add multihost=on|off pool property to control MMP. When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp. Property defaults to off.
During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock. Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".
Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval. The period is specified in milliseconds. The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.
Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path. Abbreviated
output below.
$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg timestamp mmp_delay vdev_guid vdev_label vdev_path
20468 261337 250274925 68396651780 3 /dev/sda
20468 261339 252023374 6267402363293 1 /dev/sdc
20468 261340 252000858 6698080955233 1 /dev/sdx
20468 261341 251980635 783892869810 2 /dev/sdy
20468 261342 253385953 8923255792467 3 /dev/sdd
20468 261344 253336622 042125143176 0 /dev/sdab
20468 261345 253310522 1200778101278 2 /dev/sde
20468 261346 253286429 0950576198362 2 /dev/sdt
20468 261347 253261545 96209817917 3 /dev/sds
20468 261349 253238188 8555725937673 3 /dev/sdb
Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.
When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test. For example, the
pool is exported to run zdb and then imported again. Add a new ztest
function, "-M", to alter ztest behavior to prevent this.
Add new tests to verify the new functionality. Tests provided by
Giuseppe Di Natale.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #745
Closes #6279
2017-07-08 06:20:35 +03:00
|
|
|
int m = spa_multihost(vd->vdev_spa) ? MMP_BLOCKS_PER_LABEL : 0;
|
RAID-Z expansion feature
This feature allows disks to be added one at a time to a RAID-Z group,
expanding its capacity incrementally. This feature is especially useful
for small pools (typically with only one RAID-Z group), where there
isn't sufficient hardware to add capacity by adding a whole new RAID-Z
group (typically doubling the number of disks).
== Initiating expansion ==
A new device (disk) can be attached to an existing RAIDZ vdev, by
running `zpool attach POOL raidzP-N NEW_DEVICE`, e.g. `zpool attach tank
raidz2-0 sda`. The new device will become part of the RAIDZ group. A
"raidz expansion" will be initiated, and the new device will contribute
additional space to the RAIDZ group once the expansion completes.
The `feature@raidz_expansion` on-disk feature flag must be `enabled` to
initiate an expansion, and it remains `active` for the life of the pool.
In other words, pools with expanded RAIDZ vdevs can not be imported by
older releases of the ZFS software.
== During expansion ==
The expansion entails reading all allocated space from existing disks in
the RAIDZ group, and rewriting it to the new disks in the RAIDZ group
(including the newly added device).
The expansion progress can be monitored with `zpool status`.
Data redundancy is maintained during (and after) the expansion. If a
disk fails while the expansion is in progress, the expansion pauses
until the health of the RAIDZ vdev is restored (e.g. by replacing the
failed disk and waiting for reconstruction to complete).
The pool remains accessible during expansion. Following a reboot or
export/import, the expansion resumes where it left off.
== After expansion ==
When the expansion completes, the additional space is available for use,
and is reflected in the `available` zfs property (as seen in `zfs list`,
`df`, etc).
Expansion does not change the number of failures that can be tolerated
without data loss (e.g. a RAIDZ2 is still a RAIDZ2 even after
expansion).
A RAIDZ vdev can be expanded multiple times.
After the expansion completes, old blocks remain with their old
data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but
distributed among the larger set of disks. New blocks will be written
with the new data-to-parity ratio (e.g. a 5-wide RAIDZ2 which has been
expanded once to 6-wide, has 4 data to 2 parity). However, the RAIDZ
vdev's "assumed parity ratio" does not change, so slightly less space
than is expected may be reported for newly-written blocks, according to
`zfs list`, `df`, `ls -s`, and similar tools.
Sponsored-by: The FreeBSD Foundation
Sponsored-by: iXsystems, Inc.
Sponsored-by: vStack
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Mark Maybee <mark.maybee@delphix.com>
Authored-by: Matthew Ahrens <mahrens@delphix.com>
Contributions-by: Fedor Uporov <fuporov.vstack@gmail.com>
Contributions-by: Stuart Maybee <stuart.maybee@comcast.net>
Contributions-by: Thorsten Behrens <tbehrens@outlook.com>
Contributions-by: Fmstrat <nospam@nowsci.com>
Contributions-by: Don Brady <dev.fs.zfs@gmail.com>
Signed-off-by: Don Brady <dev.fs.zfs@gmail.com>
Closes #15022
2023-11-08 21:19:41 +03:00
|
|
|
int n = (ub->ub_txg - (RRSS_GET_STATE(ub) == RRSS_SCRATCH_VALID)) %
|
|
|
|
|
(VDEV_UBERBLOCK_COUNT(vd) - m);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2016-07-22 18:52:49 +03:00
|
|
|
/* Copy the uberblock_t into the ABD */
|
Multi-modifier protection (MMP)
Add multihost=on|off pool property to control MMP. When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp. Property defaults to off.
During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock. Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".
Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval. The period is specified in milliseconds. The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.
Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path. Abbreviated
output below.
$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg timestamp mmp_delay vdev_guid vdev_label vdev_path
20468 261337 250274925 68396651780 3 /dev/sda
20468 261339 252023374 6267402363293 1 /dev/sdc
20468 261340 252000858 6698080955233 1 /dev/sdx
20468 261341 251980635 783892869810 2 /dev/sdy
20468 261342 253385953 8923255792467 3 /dev/sdd
20468 261344 253336622 042125143176 0 /dev/sdab
20468 261345 253310522 1200778101278 2 /dev/sde
20468 261346 253286429 0950576198362 2 /dev/sdt
20468 261347 253261545 96209817917 3 /dev/sds
20468 261349 253238188 8555725937673 3 /dev/sdb
Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.
When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test. For example, the
pool is exported to run zdb and then imported again. Add a new ztest
function, "-M", to alter ztest behavior to prevent this.
Add new tests to verify the new functionality. Tests provided by
Giuseppe Di Natale.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #745
Closes #6279
2017-07-08 06:20:35 +03:00
|
|
|
abd_t *ub_abd = abd_alloc_for_io(VDEV_UBERBLOCK_SIZE(vd), B_TRUE);
|
2016-07-22 18:52:49 +03:00
|
|
|
abd_copy_from_buf(ub_abd, ub, sizeof (uberblock_t));
|
2023-11-18 00:28:32 +03:00
|
|
|
abd_zero_off(ub_abd, sizeof (uberblock_t),
|
|
|
|
|
VDEV_UBERBLOCK_SIZE(vd) - sizeof (uberblock_t));
|
2008-11-20 23:01:55 +03:00
|
|
|
|
Multi-modifier protection (MMP)
Add multihost=on|off pool property to control MMP. When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp. Property defaults to off.
During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock. Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".
Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval. The period is specified in milliseconds. The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.
Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path. Abbreviated
output below.
$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg timestamp mmp_delay vdev_guid vdev_label vdev_path
20468 261337 250274925 68396651780 3 /dev/sda
20468 261339 252023374 6267402363293 1 /dev/sdc
20468 261340 252000858 6698080955233 1 /dev/sdx
20468 261341 251980635 783892869810 2 /dev/sdy
20468 261342 253385953 8923255792467 3 /dev/sdd
20468 261344 253336622 042125143176 0 /dev/sdab
20468 261345 253310522 1200778101278 2 /dev/sde
20468 261346 253286429 0950576198362 2 /dev/sdt
20468 261347 253261545 96209817917 3 /dev/sds
20468 261349 253238188 8555725937673 3 /dev/sdb
Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.
When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test. For example, the
pool is exported to run zdb and then imported again. Add a new ztest
function, "-M", to alter ztest behavior to prevent this.
Add new tests to verify the new functionality. Tests provided by
Giuseppe Di Natale.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #745
Closes #6279
2017-07-08 06:20:35 +03:00
|
|
|
for (int l = 0; l < VDEV_LABELS; l++)
|
2016-07-22 18:52:49 +03:00
|
|
|
vdev_label_write(zio, vd, l, ub_abd,
|
2008-12-03 23:09:06 +03:00
|
|
|
VDEV_UBERBLOCK_OFFSET(vd, n), VDEV_UBERBLOCK_SIZE(vd),
|
2016-12-16 21:31:01 +03:00
|
|
|
vdev_uberblock_sync_done, good_writes,
|
2008-12-03 23:09:06 +03:00
|
|
|
flags | ZIO_FLAG_DONT_PROPAGATE);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2016-07-22 18:52:49 +03:00
|
|
|
abd_free(ub_abd);
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
2013-06-11 21:12:34 +04:00
|
|
|
/* Sync the uberblocks to all vdevs in svd[] */
|
RAID-Z expansion feature
This feature allows disks to be added one at a time to a RAID-Z group,
expanding its capacity incrementally. This feature is especially useful
for small pools (typically with only one RAID-Z group), where there
isn't sufficient hardware to add capacity by adding a whole new RAID-Z
group (typically doubling the number of disks).
== Initiating expansion ==
A new device (disk) can be attached to an existing RAIDZ vdev, by
running `zpool attach POOL raidzP-N NEW_DEVICE`, e.g. `zpool attach tank
raidz2-0 sda`. The new device will become part of the RAIDZ group. A
"raidz expansion" will be initiated, and the new device will contribute
additional space to the RAIDZ group once the expansion completes.
The `feature@raidz_expansion` on-disk feature flag must be `enabled` to
initiate an expansion, and it remains `active` for the life of the pool.
In other words, pools with expanded RAIDZ vdevs can not be imported by
older releases of the ZFS software.
== During expansion ==
The expansion entails reading all allocated space from existing disks in
the RAIDZ group, and rewriting it to the new disks in the RAIDZ group
(including the newly added device).
The expansion progress can be monitored with `zpool status`.
Data redundancy is maintained during (and after) the expansion. If a
disk fails while the expansion is in progress, the expansion pauses
until the health of the RAIDZ vdev is restored (e.g. by replacing the
failed disk and waiting for reconstruction to complete).
The pool remains accessible during expansion. Following a reboot or
export/import, the expansion resumes where it left off.
== After expansion ==
When the expansion completes, the additional space is available for use,
and is reflected in the `available` zfs property (as seen in `zfs list`,
`df`, etc).
Expansion does not change the number of failures that can be tolerated
without data loss (e.g. a RAIDZ2 is still a RAIDZ2 even after
expansion).
A RAIDZ vdev can be expanded multiple times.
After the expansion completes, old blocks remain with their old
data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but
distributed among the larger set of disks. New blocks will be written
with the new data-to-parity ratio (e.g. a 5-wide RAIDZ2 which has been
expanded once to 6-wide, has 4 data to 2 parity). However, the RAIDZ
vdev's "assumed parity ratio" does not change, so slightly less space
than is expected may be reported for newly-written blocks, according to
`zfs list`, `df`, `ls -s`, and similar tools.
Sponsored-by: The FreeBSD Foundation
Sponsored-by: iXsystems, Inc.
Sponsored-by: vStack
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Mark Maybee <mark.maybee@delphix.com>
Authored-by: Matthew Ahrens <mahrens@delphix.com>
Contributions-by: Fedor Uporov <fuporov.vstack@gmail.com>
Contributions-by: Stuart Maybee <stuart.maybee@comcast.net>
Contributions-by: Thorsten Behrens <tbehrens@outlook.com>
Contributions-by: Fmstrat <nospam@nowsci.com>
Contributions-by: Don Brady <dev.fs.zfs@gmail.com>
Signed-off-by: Don Brady <dev.fs.zfs@gmail.com>
Closes #15022
2023-11-08 21:19:41 +03:00
|
|
|
int
|
2008-11-20 23:01:55 +03:00
|
|
|
vdev_uberblock_sync_list(vdev_t **svd, int svdcount, uberblock_t *ub, int flags)
|
|
|
|
|
{
|
|
|
|
|
spa_t *spa = svd[0]->vdev_spa;
|
|
|
|
|
zio_t *zio;
|
|
|
|
|
uint64_t good_writes = 0;
|
|
|
|
|
|
2016-12-16 21:31:01 +03:00
|
|
|
zio = zio_root(spa, NULL, NULL, flags);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2017-11-04 23:25:13 +03:00
|
|
|
for (int v = 0; v < svdcount; v++)
|
2016-12-16 21:31:01 +03:00
|
|
|
vdev_uberblock_sync(zio, &good_writes, ub, svd[v], flags);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2024-01-04 17:02:50 +03:00
|
|
|
if (spa->spa_aux_sync_uber) {
|
|
|
|
|
for (int v = 0; v < spa->spa_spares.sav_count; v++) {
|
|
|
|
|
vdev_uberblock_sync(zio, &good_writes, ub,
|
|
|
|
|
spa->spa_spares.sav_vdevs[v], flags);
|
|
|
|
|
}
|
|
|
|
|
for (int v = 0; v < spa->spa_l2cache.sav_count; v++) {
|
|
|
|
|
vdev_uberblock_sync(zio, &good_writes, ub,
|
|
|
|
|
spa->spa_l2cache.sav_vdevs[v], flags);
|
|
|
|
|
}
|
|
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
(void) zio_wait(zio);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Flush the uberblocks to disk. This ensures that the odd labels
|
|
|
|
|
* are no longer needed (because the new uberblocks and the even
|
|
|
|
|
* labels are safely on disk), so it is safe to overwrite them.
|
|
|
|
|
*/
|
|
|
|
|
zio = zio_root(spa, NULL, NULL, flags);
|
|
|
|
|
|
OpenZFS 7614, 9064 - zfs device evacuation/removal
OpenZFS 7614 - zfs device evacuation/removal
OpenZFS 9064 - remove_mirror should wait for device removal to complete
This project allows top-level vdevs to be removed from the storage pool
with "zpool remove", reducing the total amount of storage in the pool.
This operation copies all allocated regions of the device to be removed
onto other devices, recording the mapping from old to new location.
After the removal is complete, read and free operations to the removed
(now "indirect") vdev must be remapped and performed at the new location
on disk. The indirect mapping table is kept in memory whenever the pool
is loaded, so there is minimal performance overhead when doing operations
on the indirect vdev.
The size of the in-memory mapping table will be reduced when its entries
become "obsolete" because they are no longer used by any block pointers
in the pool. An entry becomes obsolete when all the blocks that use
it are freed. An entry can also become obsolete when all the snapshots
that reference it are deleted, and the block pointers that reference it
have been "remapped" in all filesystems/zvols (and clones). Whenever an
indirect block is written, all the block pointers in it will be "remapped"
to their new (concrete) locations if possible. This process can be
accelerated by using the "zfs remap" command to proactively rewrite all
indirect blocks that reference indirect (removed) vdevs.
Note that when a device is removed, we do not verify the checksum of
the data that is copied. This makes the process much faster, but if it
were used on redundant vdevs (i.e. mirror or raidz vdevs), it would be
possible to copy the wrong data, when we have the correct data on e.g.
the other side of the mirror.
At the moment, only mirrors and simple top-level vdevs can be removed
and no removal is allowed if any of the top-level vdevs are raidz.
Porting Notes:
* Avoid zero-sized kmem_alloc() in vdev_compact_children().
The device evacuation code adds a dependency that
vdev_compact_children() be able to properly empty the vdev_child
array by setting it to NULL and zeroing vdev_children. Under Linux,
kmem_alloc() and related functions return a sentinel pointer rather
than NULL for zero-sized allocations.
* Remove comment regarding "mpt" driver where zfs_remove_max_segment
is initialized to SPA_MAXBLOCKSIZE.
Change zfs_condense_indirect_commit_entry_delay_ticks to
zfs_condense_indirect_commit_entry_delay_ms for consistency with
most other tunables in which delays are specified in ms.
* ZTS changes:
Use set_tunable rather than mdb
Use zpool sync as appropriate
Use sync_pool instead of sync
Kill jobs during test_removal_with_operation to allow unmount/export
Don't add non-disk names such as "mirror" or "raidz" to $DISKS
Use $TEST_BASE_DIR instead of /tmp
Increase HZ from 100 to 1000 which is more common on Linux
removal_multiple_indirection.ksh
Reduce iterations in order to not time out on the code
coverage builders.
removal_resume_export:
Functionally, the test case is correct but there exists a race
where the kernel thread hasn't been fully started yet and is
not visible. Wait for up to 1 second for the removal thread
to be started before giving up on it. Also, increase the
amount of data copied in order that the removal not finish
before the export has a chance to fail.
* MMP compatibility, the concept of concrete versus non-concrete devices
has slightly changed the semantics of vdev_writeable(). Update
mmp_random_leaf_impl() accordingly.
* Updated dbuf_remap() to handle the org.zfsonlinux:large_dnode pool
feature which is not supported by OpenZFS.
* Added support for new vdev removal tracepoints.
* Test cases removal_with_zdb and removal_condense_export have been
intentionally disabled. When run manually they pass as intended,
but when running in the automated test environment they produce
unreliable results on the latest Fedora release.
They may work better once the upstream pool import refectoring is
merged into ZoL at which point they will be re-enabled.
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Alex Reece <alex@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Richard Laager <rlaager@wiktel.com>
Reviewed by: Tim Chase <tim@chase2k.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Garrett D'Amore <garrett@damore.org>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://www.illumos.org/issues/7614
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/f539f1eb
Closes #6900
2016-09-22 19:30:13 +03:00
|
|
|
for (int v = 0; v < svdcount; v++) {
|
|
|
|
|
if (vdev_writeable(svd[v])) {
|
|
|
|
|
zio_flush(zio, svd[v]);
|
|
|
|
|
}
|
|
|
|
|
}
|
2024-01-04 17:02:50 +03:00
|
|
|
if (spa->spa_aux_sync_uber) {
|
|
|
|
|
spa->spa_aux_sync_uber = B_FALSE;
|
|
|
|
|
for (int v = 0; v < spa->spa_spares.sav_count; v++) {
|
|
|
|
|
if (vdev_writeable(spa->spa_spares.sav_vdevs[v])) {
|
|
|
|
|
zio_flush(zio, spa->spa_spares.sav_vdevs[v]);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
for (int v = 0; v < spa->spa_l2cache.sav_count; v++) {
|
|
|
|
|
if (vdev_writeable(spa->spa_l2cache.sav_vdevs[v])) {
|
|
|
|
|
zio_flush(zio, spa->spa_l2cache.sav_vdevs[v]);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
(void) zio_wait(zio);
|
|
|
|
|
|
|
|
|
|
return (good_writes >= 1 ? 0 : EIO);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* On success, increment the count of good writes for our top-level vdev.
|
|
|
|
|
*/
|
|
|
|
|
static void
|
|
|
|
|
vdev_label_sync_done(zio_t *zio)
|
|
|
|
|
{
|
|
|
|
|
uint64_t *good_writes = zio->io_private;
|
|
|
|
|
|
|
|
|
|
if (zio->io_error == 0)
|
2016-01-14 03:37:41 +03:00
|
|
|
atomic_inc_64(good_writes);
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If there weren't enough good writes, indicate failure to the parent.
|
|
|
|
|
*/
|
|
|
|
|
static void
|
|
|
|
|
vdev_label_sync_top_done(zio_t *zio)
|
|
|
|
|
{
|
|
|
|
|
uint64_t *good_writes = zio->io_private;
|
|
|
|
|
|
|
|
|
|
if (*good_writes == 0)
|
2013-03-08 22:41:28 +04:00
|
|
|
zio->io_error = SET_ERROR(EIO);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
kmem_free(good_writes, sizeof (uint64_t));
|
|
|
|
|
}
|
|
|
|
|
|
2008-12-03 23:09:06 +03:00
|
|
|
/*
|
|
|
|
|
* We ignore errors for log and cache devices, simply free the private data.
|
|
|
|
|
*/
|
|
|
|
|
static void
|
|
|
|
|
vdev_label_sync_ignore_done(zio_t *zio)
|
|
|
|
|
{
|
|
|
|
|
kmem_free(zio->io_private, sizeof (uint64_t));
|
|
|
|
|
}
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
/*
|
|
|
|
|
* Write all even or odd labels to all leaves of the specified vdev.
|
|
|
|
|
*/
|
|
|
|
|
static void
|
2016-12-16 21:31:01 +03:00
|
|
|
vdev_label_sync(zio_t *zio, uint64_t *good_writes,
|
|
|
|
|
vdev_t *vd, int l, uint64_t txg, int flags)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
|
|
|
|
nvlist_t *label;
|
|
|
|
|
vdev_phys_t *vp;
|
2016-07-22 18:52:49 +03:00
|
|
|
abd_t *vp_abd;
|
2008-11-20 23:01:55 +03:00
|
|
|
char *buf;
|
|
|
|
|
size_t buflen;
|
2024-08-09 01:16:46 +03:00
|
|
|
vdev_t *pvd = vd->vdev_parent;
|
|
|
|
|
boolean_t spare_in_use = B_FALSE;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2016-12-16 21:31:01 +03:00
|
|
|
for (int c = 0; c < vd->vdev_children; c++) {
|
|
|
|
|
vdev_label_sync(zio, good_writes,
|
|
|
|
|
vd->vdev_child[c], l, txg, flags);
|
|
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
if (!vd->vdev_ops->vdev_op_leaf)
|
|
|
|
|
return;
|
|
|
|
|
|
2008-12-03 23:09:06 +03:00
|
|
|
if (!vdev_writeable(vd))
|
2008-11-20 23:01:55 +03:00
|
|
|
return;
|
|
|
|
|
|
Distributed Spare (dRAID) Feature
This patch adds a new top-level vdev type called dRAID, which stands
for Distributed parity RAID. This pool configuration allows all dRAID
vdevs to participate when rebuilding to a distributed hot spare device.
This can substantially reduce the total time required to restore full
parity to pool with a failed device.
A dRAID pool can be created using the new top-level `draid` type.
Like `raidz`, the desired redundancy is specified after the type:
`draid[1,2,3]`. No additional information is required to create the
pool and reasonable default values will be chosen based on the number
of child vdevs in the dRAID vdev.
zpool create <pool> draid[1,2,3] <vdevs...>
Unlike raidz, additional optional dRAID configuration values can be
provided as part of the draid type as colon separated values. This
allows administrators to fully specify a layout for either performance
or capacity reasons. The supported options include:
zpool create <pool> \
draid[<parity>][:<data>d][:<children>c][:<spares>s] \
<vdevs...>
- draid[parity] - Parity level (default 1)
- draid[:<data>d] - Data devices per group (default 8)
- draid[:<children>c] - Expected number of child vdevs
- draid[:<spares>s] - Distributed hot spares (default 0)
Abbreviated example `zpool status` output for a 68 disk dRAID pool
with two distributed spares using special allocation classes.
```
pool: tank
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
slag7 ONLINE 0 0 0
draid2:8d:68c:2s-0 ONLINE 0 0 0
L0 ONLINE 0 0 0
L1 ONLINE 0 0 0
...
U25 ONLINE 0 0 0
U26 ONLINE 0 0 0
spare-53 ONLINE 0 0 0
U27 ONLINE 0 0 0
draid2-0-0 ONLINE 0 0 0
U28 ONLINE 0 0 0
U29 ONLINE 0 0 0
...
U42 ONLINE 0 0 0
U43 ONLINE 0 0 0
special
mirror-1 ONLINE 0 0 0
L5 ONLINE 0 0 0
U5 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
L6 ONLINE 0 0 0
U6 ONLINE 0 0 0
spares
draid2-0-0 INUSE currently in use
draid2-0-1 AVAIL
```
When adding test coverage for the new dRAID vdev type the following
options were added to the ztest command. These options are leverages
by zloop.sh to test a wide range of dRAID configurations.
-K draid|raidz|random - kind of RAID to test
-D <value> - dRAID data drives per group
-S <value> - dRAID distributed hot spares
-R <value> - RAID parity (raidz or dRAID)
The zpool_create, zpool_import, redundancy, replacement and fault
test groups have all been updated provide test coverage for the
dRAID feature.
Co-authored-by: Isaac Huang <he.huang@intel.com>
Co-authored-by: Mark Maybee <mmaybee@cray.com>
Co-authored-by: Don Brady <don.brady@delphix.com>
Co-authored-by: Matthew Ahrens <mahrens@delphix.com>
Co-authored-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Mark Maybee <mmaybee@cray.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #10102
2020-11-14 00:51:51 +03:00
|
|
|
/*
|
|
|
|
|
* The top-level config never needs to be written to a distributed
|
|
|
|
|
* spare. When read vdev_dspare_label_read_config() will generate
|
|
|
|
|
* the config for the vdev_label_read_config().
|
|
|
|
|
*/
|
|
|
|
|
if (vd->vdev_ops == &vdev_draid_spare_ops)
|
|
|
|
|
return;
|
|
|
|
|
|
2024-08-09 01:16:46 +03:00
|
|
|
if (pvd && pvd->vdev_ops == &vdev_spare_ops)
|
|
|
|
|
spare_in_use = B_TRUE;
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
/*
|
|
|
|
|
* Generate a label describing the top-level config to which we belong.
|
|
|
|
|
*/
|
2024-08-09 01:16:46 +03:00
|
|
|
if ((vd->vdev_isspare && !spare_in_use) || vd->vdev_isl2cache) {
|
|
|
|
|
label = vdev_aux_label_generate(vd, vd->vdev_isspare);
|
|
|
|
|
} else {
|
|
|
|
|
label = spa_config_generate(vd->vdev_spa, vd, txg, B_FALSE);
|
|
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2016-07-22 18:52:49 +03:00
|
|
|
vp_abd = abd_alloc_linear(sizeof (vdev_phys_t), B_TRUE);
|
|
|
|
|
abd_zero(vp_abd, sizeof (vdev_phys_t));
|
|
|
|
|
vp = abd_to_buf(vp_abd);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
buf = vp->vp_nvlist;
|
|
|
|
|
buflen = sizeof (vp->vp_nvlist);
|
|
|
|
|
|
2014-11-21 03:09:39 +03:00
|
|
|
if (!nvlist_pack(label, &buf, &buflen, NV_ENCODE_XDR, KM_SLEEP)) {
|
2008-11-20 23:01:55 +03:00
|
|
|
for (; l < VDEV_LABELS; l += 2) {
|
2016-07-22 18:52:49 +03:00
|
|
|
vdev_label_write(zio, vd, l, vp_abd,
|
2008-11-20 23:01:55 +03:00
|
|
|
offsetof(vdev_label_t, vl_vdev_phys),
|
|
|
|
|
sizeof (vdev_phys_t),
|
2016-12-16 21:31:01 +03:00
|
|
|
vdev_label_sync_done, good_writes,
|
2008-12-03 23:09:06 +03:00
|
|
|
flags | ZIO_FLAG_DONT_PROPAGATE);
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2016-07-22 18:52:49 +03:00
|
|
|
abd_free(vp_abd);
|
2008-11-20 23:01:55 +03:00
|
|
|
nvlist_free(label);
|
|
|
|
|
}
|
|
|
|
|
|
2020-06-15 21:30:37 +03:00
|
|
|
static int
|
2008-12-03 23:09:06 +03:00
|
|
|
vdev_label_sync_list(spa_t *spa, int l, uint64_t txg, int flags)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
2008-12-03 23:09:06 +03:00
|
|
|
list_t *dl = &spa->spa_config_dirty_list;
|
2008-11-20 23:01:55 +03:00
|
|
|
vdev_t *vd;
|
|
|
|
|
zio_t *zio;
|
|
|
|
|
int error;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Write the new labels to disk.
|
|
|
|
|
*/
|
|
|
|
|
zio = zio_root(spa, NULL, NULL, flags);
|
|
|
|
|
|
|
|
|
|
for (vd = list_head(dl); vd != NULL; vd = list_next(dl, vd)) {
|
2010-08-26 20:52:39 +04:00
|
|
|
uint64_t *good_writes;
|
2010-05-29 00:45:14 +04:00
|
|
|
|
|
|
|
|
ASSERT(!vd->vdev_ishole);
|
|
|
|
|
|
2014-11-21 03:09:39 +03:00
|
|
|
good_writes = kmem_zalloc(sizeof (uint64_t), KM_SLEEP);
|
2017-11-04 23:25:13 +03:00
|
|
|
zio_t *vio = zio_null(zio, spa, NULL,
|
2008-12-03 23:09:06 +03:00
|
|
|
(vd->vdev_islog || vd->vdev_aux != NULL) ?
|
|
|
|
|
vdev_label_sync_ignore_done : vdev_label_sync_top_done,
|
2008-11-20 23:01:55 +03:00
|
|
|
good_writes, flags);
|
2016-12-16 21:31:01 +03:00
|
|
|
vdev_label_sync(vio, good_writes, vd, l, txg, flags);
|
2008-11-20 23:01:55 +03:00
|
|
|
zio_nowait(vio);
|
|
|
|
|
}
|
|
|
|
|
|
2024-08-09 01:16:46 +03:00
|
|
|
/*
|
|
|
|
|
* AUX path may have changed during import
|
|
|
|
|
*/
|
|
|
|
|
spa_aux_vdev_t *sav[2] = {&spa->spa_spares, &spa->spa_l2cache};
|
|
|
|
|
for (int i = 0; i < 2; i++) {
|
|
|
|
|
for (int v = 0; v < sav[i]->sav_count; v++) {
|
|
|
|
|
uint64_t *good_writes;
|
|
|
|
|
if (!sav[i]->sav_label_sync)
|
|
|
|
|
continue;
|
|
|
|
|
good_writes = kmem_zalloc(sizeof (uint64_t), KM_SLEEP);
|
|
|
|
|
zio_t *vio = zio_null(zio, spa, NULL,
|
|
|
|
|
vdev_label_sync_ignore_done, good_writes, flags);
|
|
|
|
|
vdev_label_sync(vio, good_writes, sav[i]->sav_vdevs[v],
|
|
|
|
|
l, txg, flags);
|
|
|
|
|
zio_nowait(vio);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
error = zio_wait(zio);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Flush the new labels to disk.
|
|
|
|
|
*/
|
|
|
|
|
zio = zio_root(spa, NULL, NULL, flags);
|
|
|
|
|
|
|
|
|
|
for (vd = list_head(dl); vd != NULL; vd = list_next(dl, vd))
|
|
|
|
|
zio_flush(zio, vd);
|
|
|
|
|
|
2024-08-09 01:16:46 +03:00
|
|
|
for (int i = 0; i < 2; i++) {
|
|
|
|
|
if (!sav[i]->sav_label_sync)
|
|
|
|
|
continue;
|
|
|
|
|
for (int v = 0; v < sav[i]->sav_count; v++)
|
|
|
|
|
zio_flush(zio, sav[i]->sav_vdevs[v]);
|
|
|
|
|
if (l == 1)
|
|
|
|
|
sav[i]->sav_label_sync = B_FALSE;
|
|
|
|
|
}
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
(void) zio_wait(zio);
|
|
|
|
|
|
|
|
|
|
return (error);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Sync the uberblock and any changes to the vdev configuration.
|
|
|
|
|
*
|
|
|
|
|
* The order of operations is carefully crafted to ensure that
|
|
|
|
|
* if the system panics or loses power at any time, the state on disk
|
|
|
|
|
* is still transactionally consistent. The in-line comments below
|
|
|
|
|
* describe the failure semantics at each stage.
|
|
|
|
|
*
|
|
|
|
|
* Moreover, vdev_config_sync() is designed to be idempotent: if it fails
|
|
|
|
|
* at any time, you can just call it again, and it will resume its work.
|
|
|
|
|
*/
|
|
|
|
|
int
|
2016-01-27 04:27:46 +03:00
|
|
|
vdev_config_sync(vdev_t **svd, int svdcount, uint64_t txg)
|
2008-11-20 23:01:55 +03:00
|
|
|
{
|
|
|
|
|
spa_t *spa = svd[0]->vdev_spa;
|
|
|
|
|
uberblock_t *ub = &spa->spa_uberblock;
|
2016-01-27 04:27:46 +03:00
|
|
|
int error = 0;
|
2008-12-03 23:09:06 +03:00
|
|
|
int flags = ZIO_FLAG_CONFIG_WRITER | ZIO_FLAG_CANFAIL;
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2016-12-17 01:11:29 +03:00
|
|
|
ASSERT(svdcount != 0);
|
2016-01-27 04:27:46 +03:00
|
|
|
retry:
|
2009-07-03 02:44:48 +04:00
|
|
|
/*
|
|
|
|
|
* Normally, we don't want to try too hard to write every label and
|
|
|
|
|
* uberblock. If there is a flaky disk, we don't want the rest of the
|
|
|
|
|
* sync process to block while we retry. But if we can't write a
|
|
|
|
|
* single label out, we should retry with ZIO_FLAG_TRYHARD before
|
|
|
|
|
* bailing out and declaring the pool faulted.
|
|
|
|
|
*/
|
2016-01-27 04:27:46 +03:00
|
|
|
if (error != 0) {
|
|
|
|
|
if ((flags & ZIO_FLAG_TRYHARD) != 0)
|
|
|
|
|
return (error);
|
2009-07-03 02:44:48 +04:00
|
|
|
flags |= ZIO_FLAG_TRYHARD;
|
2016-01-27 04:27:46 +03:00
|
|
|
}
|
2009-07-03 02:44:48 +04:00
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
ASSERT(ub->ub_txg <= txg);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If this isn't a resync due to I/O errors,
|
|
|
|
|
* and nothing changed in this transaction group,
|
2024-04-30 00:35:53 +03:00
|
|
|
* and multihost protection isn't enabled,
|
2008-11-20 23:01:55 +03:00
|
|
|
* and the vdev configuration hasn't changed,
|
|
|
|
|
* then there's nothing to do.
|
|
|
|
|
*/
|
Multi-modifier protection (MMP)
Add multihost=on|off pool property to control MMP. When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp. Property defaults to off.
During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock. Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".
Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval. The period is specified in milliseconds. The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.
Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path. Abbreviated
output below.
$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg timestamp mmp_delay vdev_guid vdev_label vdev_path
20468 261337 250274925 68396651780 3 /dev/sda
20468 261339 252023374 6267402363293 1 /dev/sdc
20468 261340 252000858 6698080955233 1 /dev/sdx
20468 261341 251980635 783892869810 2 /dev/sdy
20468 261342 253385953 8923255792467 3 /dev/sdd
20468 261344 253336622 042125143176 0 /dev/sdab
20468 261345 253310522 1200778101278 2 /dev/sde
20468 261346 253286429 0950576198362 2 /dev/sdt
20468 261347 253261545 96209817917 3 /dev/sds
20468 261349 253238188 8555725937673 3 /dev/sdb
Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.
When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test. For example, the
pool is exported to run zdb and then imported again. Add a new ztest
function, "-M", to alter ztest behavior to prevent this.
Add new tests to verify the new functionality. Tests provided by
Giuseppe Di Natale.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #745
Closes #6279
2017-07-08 06:20:35 +03:00
|
|
|
if (ub->ub_txg < txg) {
|
|
|
|
|
boolean_t changed = uberblock_update(ub, spa->spa_root_vdev,
|
|
|
|
|
txg, spa->spa_mmp.mmp_delay);
|
|
|
|
|
|
2024-04-30 00:35:53 +03:00
|
|
|
if (!changed && list_is_empty(&spa->spa_config_dirty_list) &&
|
|
|
|
|
!spa_multihost(spa))
|
Multi-modifier protection (MMP)
Add multihost=on|off pool property to control MMP. When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp. Property defaults to off.
During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock. Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".
Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval. The period is specified in milliseconds. The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.
Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path. Abbreviated
output below.
$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg timestamp mmp_delay vdev_guid vdev_label vdev_path
20468 261337 250274925 68396651780 3 /dev/sda
20468 261339 252023374 6267402363293 1 /dev/sdc
20468 261340 252000858 6698080955233 1 /dev/sdx
20468 261341 251980635 783892869810 2 /dev/sdy
20468 261342 253385953 8923255792467 3 /dev/sdd
20468 261344 253336622 042125143176 0 /dev/sdab
20468 261345 253310522 1200778101278 2 /dev/sde
20468 261346 253286429 0950576198362 2 /dev/sdt
20468 261347 253261545 96209817917 3 /dev/sds
20468 261349 253238188 8555725937673 3 /dev/sdb
Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.
When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test. For example, the
pool is exported to run zdb and then imported again. Add a new ztest
function, "-M", to alter ztest behavior to prevent this.
Add new tests to verify the new functionality. Tests provided by
Giuseppe Di Natale.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #745
Closes #6279
2017-07-08 06:20:35 +03:00
|
|
|
return (0);
|
|
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
if (txg > spa_freeze_txg(spa))
|
|
|
|
|
return (0);
|
|
|
|
|
|
|
|
|
|
ASSERT(txg <= spa->spa_final_txg);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Flush the write cache of every disk that's been written to
|
|
|
|
|
* in this transaction group. This ensures that all blocks
|
|
|
|
|
* written in this txg will be committed to stable storage
|
|
|
|
|
* before any uberblock that references them.
|
|
|
|
|
*/
|
2016-12-17 01:11:29 +03:00
|
|
|
zio_t *zio = zio_root(spa, NULL, NULL, flags);
|
2008-11-20 23:01:55 +03:00
|
|
|
|
2016-12-17 01:11:29 +03:00
|
|
|
for (vdev_t *vd =
|
|
|
|
|
txg_list_head(&spa->spa_vdev_txg_list, TXG_CLEAN(txg)); vd != NULL;
|
2008-11-20 23:01:55 +03:00
|
|
|
vd = txg_list_next(&spa->spa_vdev_txg_list, vd, TXG_CLEAN(txg)))
|
|
|
|
|
zio_flush(zio, vd);
|
|
|
|
|
|
|
|
|
|
(void) zio_wait(zio);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Sync out the even labels (L0, L2) for every dirty vdev. If the
|
|
|
|
|
* system dies in the middle of this process, that's OK: all of the
|
|
|
|
|
* even labels that made it to disk will be newer than any uberblock,
|
|
|
|
|
* and will therefore be considered invalid. The odd labels (L1, L3),
|
|
|
|
|
* which have not yet been touched, will still be valid. We flush
|
|
|
|
|
* the new labels to disk to ensure that all even-label updates
|
|
|
|
|
* are committed to stable storage before the uberblock update.
|
|
|
|
|
*/
|
2016-12-17 01:11:29 +03:00
|
|
|
if ((error = vdev_label_sync_list(spa, 0, txg, flags)) != 0) {
|
|
|
|
|
if ((flags & ZIO_FLAG_TRYHARD) != 0) {
|
|
|
|
|
zfs_dbgmsg("vdev_label_sync_list() returned error %d "
|
|
|
|
|
"for pool '%s' when syncing out the even labels "
|
|
|
|
|
"of dirty vdevs", error, spa_name(spa));
|
|
|
|
|
}
|
2016-01-27 04:27:46 +03:00
|
|
|
goto retry;
|
2016-12-17 01:11:29 +03:00
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Sync the uberblocks to all vdevs in svd[].
|
|
|
|
|
* If the system dies in the middle of this step, there are two cases
|
|
|
|
|
* to consider, and the on-disk state is consistent either way:
|
|
|
|
|
*
|
|
|
|
|
* (1) If none of the new uberblocks made it to disk, then the
|
|
|
|
|
* previous uberblock will be the newest, and the odd labels
|
|
|
|
|
* (which had not yet been touched) will be valid with respect
|
|
|
|
|
* to that uberblock.
|
|
|
|
|
*
|
|
|
|
|
* (2) If one or more new uberblocks made it to disk, then they
|
|
|
|
|
* will be the newest, and the even labels (which had all
|
|
|
|
|
* been successfully committed) will be valid with respect
|
|
|
|
|
* to the new uberblocks.
|
|
|
|
|
*/
|
2016-12-17 01:11:29 +03:00
|
|
|
if ((error = vdev_uberblock_sync_list(svd, svdcount, ub, flags)) != 0) {
|
|
|
|
|
if ((flags & ZIO_FLAG_TRYHARD) != 0) {
|
|
|
|
|
zfs_dbgmsg("vdev_uberblock_sync_list() returned error "
|
|
|
|
|
"%d for pool '%s'", error, spa_name(spa));
|
|
|
|
|
}
|
2016-01-27 04:27:46 +03:00
|
|
|
goto retry;
|
2016-12-17 01:11:29 +03:00
|
|
|
}
|
2008-11-20 23:01:55 +03:00
|
|
|
|
Multi-modifier protection (MMP)
Add multihost=on|off pool property to control MMP. When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp. Property defaults to off.
During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock. Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".
Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval. The period is specified in milliseconds. The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.
Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path. Abbreviated
output below.
$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg timestamp mmp_delay vdev_guid vdev_label vdev_path
20468 261337 250274925 68396651780 3 /dev/sda
20468 261339 252023374 6267402363293 1 /dev/sdc
20468 261340 252000858 6698080955233 1 /dev/sdx
20468 261341 251980635 783892869810 2 /dev/sdy
20468 261342 253385953 8923255792467 3 /dev/sdd
20468 261344 253336622 042125143176 0 /dev/sdab
20468 261345 253310522 1200778101278 2 /dev/sde
20468 261346 253286429 0950576198362 2 /dev/sdt
20468 261347 253261545 96209817917 3 /dev/sds
20468 261349 253238188 8555725937673 3 /dev/sdb
Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.
When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test. For example, the
pool is exported to run zdb and then imported again. Add a new ztest
function, "-M", to alter ztest behavior to prevent this.
Add new tests to verify the new functionality. Tests provided by
Giuseppe Di Natale.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #745
Closes #6279
2017-07-08 06:20:35 +03:00
|
|
|
if (spa_multihost(spa))
|
|
|
|
|
mmp_update_uberblock(spa, ub);
|
|
|
|
|
|
2008-11-20 23:01:55 +03:00
|
|
|
/*
|
|
|
|
|
* Sync out odd labels for every dirty vdev. If the system dies
|
|
|
|
|
* in the middle of this process, the even labels and the new
|
|
|
|
|
* uberblocks will suffice to open the pool. The next time
|
|
|
|
|
* the pool is opened, the first thing we'll do -- before any
|
|
|
|
|
* user data is modified -- is mark every vdev dirty so that
|
|
|
|
|
* all labels will be brought up to date. We flush the new labels
|
|
|
|
|
* to disk to ensure that all odd-label updates are committed to
|
|
|
|
|
* stable storage before the next transaction group begins.
|
|
|
|
|
*/
|
2016-12-17 01:11:29 +03:00
|
|
|
if ((error = vdev_label_sync_list(spa, 1, txg, flags)) != 0) {
|
|
|
|
|
if ((flags & ZIO_FLAG_TRYHARD) != 0) {
|
|
|
|
|
zfs_dbgmsg("vdev_label_sync_list() returned error %d "
|
|
|
|
|
"for pool '%s' when syncing out the odd labels of "
|
|
|
|
|
"dirty vdevs", error, spa_name(spa));
|
|
|
|
|
}
|
2016-01-27 04:27:46 +03:00
|
|
|
goto retry;
|
2016-12-17 01:11:29 +03:00
|
|
|
}
|
2016-01-27 04:27:46 +03:00
|
|
|
|
|
|
|
|
return (0);
|
2008-11-20 23:01:55 +03:00
|
|
|
}
|