mirror_zfs/include/sys
Matthew Ahrens df4474f92d Illumos #3805 arc shouldn't cache freed blocks
3805 arc shouldn't cache freed blocks
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Richard Elling <richard.elling@dey-sys.com>
Reviewed by: Will Andrews <will@firepipe.net>
Approved by: Dan McDonald <danmcd@nexenta.com>

References:
  illumos/illumos-gate@6e6d5868f5
  https://www.illumos.org/issues/3805

ZFS should proactively evict freed blocks from the cache.

On dcenter, we saw that we were caching ~256GB of metadata, while the
pool only had <4GB of metadata on disk.  We were wasting about half the
system's RAM (252GB) on blocks that have been freed.

Even though these freed blocks will never be used again, and thus will
eventually be evicted, this causes us to use memory inefficiently for 2
reasons:

1. A block that is freed has no chance of being accessed again, but will
be kept in memory preferentially to a block that was accessed before it
(and is thus older) but has not been freed and thus has at least some
chance of being accessed again.

2. We partition the ARC into several buckets:
user data that has been accessed only once (MRU)
metadata that has been accessed only once (MRU)
user data that has been accessed more than once (MFU)
metadata that has been accessed more than once (MFU)

The user data vs metadata split is somewhat arbitrary, and the primary
control on how much memory is used to cache data vs metadata is to
simply try to keep the proportion the same as it has been in the past
(each bucket "evicts against" itself).  The secondary control is to
evict data before evicting metadata.

Because of this bucketing, we may end up with one bucket mostly
containing freed blocks that are very old, while another bucket has more
recently accessed, still-allocated blocks.  Data in the useful bucket
(with still-allocated blocks) may be evicted in preference to data in
the useless bucket (with old, freed blocks).

On dcenter, we saw that the MFU metadata bucket was 230MB, while the MFU
data bucket was 27GB and the MRU metadata bucket was 256GB.  However,
the vast majority of data in the MRU metadata bucket (256GB) was freed
blocks, and thus useless.  Meanwhile, the MFU metadata bucket (230MB)
was constantly evicting useful blocks that will be soon needed.

The problem of cache segmentation is a larger problem that needs more
investigation.  However, if we stop caching freed blocks, it should
reduce the impact of this more fundamental issue.

Ported-by: Richard Yao <ryao@cs.stonybrook.edu>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1503
2013-06-20 09:55:52 -07:00
..
fm 3246 ZFS I/O deadman thread 2013-05-01 17:05:52 -07:00
fs Change zfs-kmod-devel install path 2013-03-13 13:42:16 -07:00
arc.h Illumos #3805 arc shouldn't cache freed blocks 2013-06-20 09:55:52 -07:00
avl_impl.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
avl.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
bplist.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
bpobj.h Illumos #3104: eliminate empty bpobjs 2013-01-08 10:35:43 -08:00
bptree.h Illumos #2619 and #2747 2013-01-08 10:35:35 -08:00
dbuf.h Switch KM_SLEEP to KM_PUSHPAGE 2012-08-27 12:01:37 -07:00
ddt.h Add ddt_object_count() error handling 2012-10-29 08:57:45 -07:00
dmu_impl.h Illumos #2703: add mechanism to report ZFS send progress 2012-09-19 13:39:06 -07:00
dmu_objset.h Illumos #3100: zvol rename fails with EBUSY when dirty. 2012-10-03 13:59:02 -07:00
dmu_traverse.h Illumos #2619 and #2747 2013-01-08 10:35:35 -08:00
dmu_tx.h Add --enable-debug-dmu-tx configure option 2012-03-23 12:25:17 -07:00
dmu_zfetch.h Add missing ZFS tunables 2011-05-04 10:02:37 -07:00
dmu.h build: resolve orthographic and other grammatical errors 2013-04-02 10:44:52 -07:00
dnode.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
dsl_dataset.h Use dsl_dataset_snap_lookup() 2013-01-25 15:07:40 -08:00
dsl_deadlist.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
dsl_deleg.h Illumos #1644, #1645, #1646, #1647, #1708 2012-07-31 09:25:30 -07:00
dsl_dir.h Switch KM_SLEEP to KM_PUSHPAGE 2012-08-27 12:01:37 -07:00
dsl_pool.h 3246 ZFS I/O deadman thread 2013-05-01 17:05:52 -07:00
dsl_prop.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
dsl_scan.h Illumos #2619 and #2747 2013-01-08 10:35:35 -08:00
dsl_synctask.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
efi_partition.h Move partition scanning from userspace to module. 2012-07-17 09:17:31 -07:00
Makefile.am Change zfs-kmod-devel install path 2013-03-13 13:42:16 -07:00
metaslab_impl.h Illumos #3552, #3564 2013-06-19 16:22:39 -07:00
metaslab.h Add FASTWRITE algorithm for synchronous writes. 2012-10-17 08:56:41 -07:00
nvpair_impl.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
nvpair.h Illumos #2619 and #2747 2013-01-08 10:35:35 -08:00
refcount.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
rrwlock.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
sa_impl.h Illumos #3208 cross-endian incorrect user/group accounting 2013-01-14 09:32:22 -08:00
sa.h Add sa_spill_rele() interface 2012-03-07 16:28:00 -08:00
spa_boot.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
spa_impl.h Use taskq for dump_bytes() 2013-05-06 14:05:42 -07:00
spa.h Illumos #3329, #3330, #3331, #3335 2013-05-06 12:39:34 -07:00
space_map.h Illumos #3552, #3564 2013-06-19 16:22:39 -07:00
txg_impl.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
txg.h Illumos #3086: unnecessarily setting DS_FLAG_INCONSISTENT on async 2013-01-08 10:35:43 -08:00
u8_textprep_data.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
u8_textprep.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
uberblock_impl.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
uberblock.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
uio_impl.h Add basic uio support 2011-02-10 09:21:43 -08:00
unique.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
uuid.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
vdev_disk.h Linux 2.6.x compat, blkdev_compat.h 2011-02-23 12:29:38 -08:00
vdev_file.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
vdev_impl.h 3246 ZFS I/O deadman thread 2013-05-01 17:05:52 -07:00
vdev.h 3246 ZFS I/O deadman thread 2013-05-01 17:05:52 -07:00
xvattr.h Add xvattr support 2011-03-02 11:43:50 -08:00
zap_impl.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
zap_leaf.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
zap.h Illumos #3104: eliminate empty bpobjs 2013-01-08 10:35:43 -08:00
zfeature.h Illumos #2762: zpool command should have better support for feature flags 2013-01-08 10:35:43 -08:00
zfs_acl.h Fix build failures on PaX/GRSecurity patched kernels 2012-07-17 09:22:43 -07:00
zfs_context.h Illumos #3006 2013-06-19 15:14:10 -07:00
zfs_ctldir.h Linux 3.7 compat, schedule_delayed_work() 2012-12-12 10:47:05 -08:00
zfs_debug.h Cleanly support debug packages 2012-02-27 14:08:17 -08:00
zfs_dir.h Prototype/structure update for Linux 2011-02-10 09:27:21 -08:00
zfs_fuid.h Prototype/structure update for Linux 2011-02-10 09:27:21 -08:00
zfs_ioctl.h 3246 ZFS I/O deadman thread 2013-05-01 17:05:52 -07:00
zfs_onexit.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
zfs_rlock.h Range lock performance improvements 2011-03-08 12:44:06 -08:00
zfs_sa.h Implement SA based xattrs 2011-11-28 15:45:51 -08:00
zfs_stat.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
zfs_vfsops.h Fix 'zfs rollback' on mounted file systems 2013-01-17 09:51:20 -08:00
zfs_vnops.h Update SAs when an inode is dirtied 2012-12-14 12:18:54 -08:00
zfs_znode.h Constify structures containing function pointers 2013-03-04 08:49:32 -08:00
zil_impl.h Illumos #3086: unnecessarily setting DS_FLAG_INCONSISTENT on async 2013-01-08 10:35:43 -08:00
zil.h Constify structures containing function pointers 2013-03-04 08:49:32 -08:00
zio_checksum.h Constify structures containing function pointers 2013-03-04 08:49:32 -08:00
zio_compress.h Constify structures containing function pointers 2013-03-04 08:49:32 -08:00
zio_impl.h Illumos #3329, #3330, #3331, #3335 2013-05-06 12:39:34 -07:00
zio.h 3246 ZFS I/O deadman thread 2013-05-01 17:05:52 -07:00
zpl.h Fix 'zfs rollback' on mounted file systems 2013-01-17 09:51:20 -08:00
zrlock.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
zvol.h Add snapdev=[hidden|visible] dataset property 2013-03-05 12:37:54 -08:00