Go to file
Matthew Ahrens 2e5dc449c1 OpenZFS 9337 - zfs get all is slow due to uncached metadata
This project's goal is to make read-heavy channel programs and zfs(1m)
administrative commands faster by caching all the metadata that they will
need in the dbuf layer. This will prevent the data from being evicted, so
that any future call to i.e. zfs get all won't have to go to disk (very
much). There are two parts:

The dbuf_metadata_cache. We identify what to put into the cache based on
the object type of each dbuf.  Caching objset properties os
{version,normalization,utf8only,casesensitivity} in the objset_t. The reason
these needed to be cached is that although they are queried frequently,
they aren't stored in a dbuf type which we can easily recognize and cache in
the dbuf layer; instead, we have to explicitly store them. There's already
existing infrastructure for maintaining cached properties in the objset
setup code, so I simply used that.

Performance Testing:

 - Disabled kmem_flags
 - Tuned dbuf_cache_max_bytes very low (128K)
 - Tuned zfs_arc_max very low (64M)

Created test pool with 400 filesystems, and 100 snapshots per filesystem.
Later on in testing, added 600 more filesystems (with no snapshots) to make
sure scaling didn't look different between snapshots and filesystems.

Results:

    | Test                   | Time (trunk / diff) | I/Os (trunk / diff) |
    +------------------------+---------------------+---------------------+
    | zpool import           |     0:05 / 0:06     |    12.9k / 12.9k    |
    | zfs get all (uncached) |     1:36 / 0:53     |    16.7k / 5.7k     |
    | zfs get all (cached)   |     1:36 / 0:51     |    16.0k / 6.0k     |

Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Thomas Caputi <tcaputi@datto.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Richard Lowe <richlowe@richlowe.net>
Ported-by: Alek Pinchuk <apinchuk@datto.com>
Signed-off-by: Alek Pinchuk <apinchuk@datto.com>

OpenZFS-issue: https://illumos.org/issues/9337
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/7dec52f
Closes #7668
2018-07-12 10:49:27 -07:00
.github Small cleanup of PR and issue templates 2018-05-15 09:02:57 -07:00
cmd Reduce zdb output when pool contains checkpoint 2018-07-10 21:23:17 -07:00
config Linux 4.14 compat: blk_queue_stackable() 2018-06-19 21:52:45 -07:00
contrib Add pyzfs build directories to gitignore 2018-06-11 18:42:12 -07:00
etc Minor documentation, logging, and testing typos 2018-06-07 09:38:39 -07:00
include OpenZFS 9337 - zfs get all is slow due to uncached metadata 2018-07-12 10:49:27 -07:00
lib Fix kernel unaligned access on sparc64 2018-07-11 13:10:40 -07:00
man OpenZFS 9337 - zfs get all is slow due to uncached metadata 2018-07-12 10:49:27 -07:00
module OpenZFS 9337 - zfs get all is slow due to uncached metadata 2018-07-12 10:49:27 -07:00
rpm Update build system and packaging 2018-05-29 16:00:33 -07:00
scripts OpenZFS 9330 - stack overflow when creating a deeply nested dataset 2018-07-09 13:02:50 -07:00
tests OpenZFS 9337 - zfs get all is slow due to uncached metadata 2018-07-12 10:49:27 -07:00
udev Add kernel module auto-loading 2018-03-13 10:45:55 -07:00
.gitignore Ignore *.o.ur-safe build artifacts 2018-05-13 18:59:02 -07:00
.gitmodules Add zimport.sh compatibility test script 2014-02-21 12:10:31 -08:00
.travis.yml Add .travis.yml 2017-11-13 09:18:18 -08:00
AUTHORS Update build system and packaging 2018-05-29 16:00:33 -07:00
autogen.sh Cause autogen.sh to fail if autoreconf fails 2018-07-06 09:27:37 -07:00
configure.ac OpenZFS 9166 - zfs storage pool checkpoint 2018-06-26 10:07:42 -07:00
copy-builtin copy-builtin: SPL must be in Kbuild first 2018-06-15 15:16:29 -07:00
COPYRIGHT Update build system and packaging 2018-05-29 16:00:33 -07:00
LICENSE Update build system and packaging 2018-05-29 16:00:33 -07:00
Makefile.am make install only works once 2018-05-31 09:19:59 -07:00
META Linux compat 4.18: check_disk_size_change() 2018-06-15 15:05:21 -07:00
NOTICE Update build system and packaging 2018-05-29 16:00:33 -07:00
README.md Explicitly state supported Linux versions 2018-05-30 20:11:19 -07:00
TEST Update build system and packaging 2018-05-29 16:00:33 -07:00
zfs.release.in Move zfs.release generation to configure step 2012-07-12 12:22:51 -07:00

img

ZFS on Linux is an advanced file system and volume manager which was originally developed for Solaris and is now maintained by the OpenZFS community.

codecov coverity

Official Resources

Installation

Full documentation for installing ZoL on your favorite Linux distribution can be found at our site.

Contribute & Develop

We have a separate document with contribution guidelines.

Release

ZFS on Linux is released under a CDDL license.
For more details see the NOTICE, LICENSE and COPYRIGHT files; UCRL-CODE-235197

Supported Kernels

  • The META file contains the officially recognized supported kernel versions.