2014-12-08 21:35:51 +03:00
|
|
|
/*
|
2014-12-08 21:04:42 +03:00
|
|
|
* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
|
|
|
|
|
* Copyright (C) 2007 The Regents of the University of California.
|
|
|
|
|
* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
|
|
|
|
|
* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
|
|
|
|
|
* UCRL-CODE-235197
|
|
|
|
|
*
|
|
|
|
|
* This file is part of the SPL, Solaris Porting Layer.
|
|
|
|
|
* For details, see <http://zfsonlinux.org/>.
|
|
|
|
|
*
|
|
|
|
|
* The SPL is free software; you can redistribute it and/or modify it
|
|
|
|
|
* under the terms of the GNU General Public License as published by the
|
|
|
|
|
* Free Software Foundation; either version 2 of the License, or (at your
|
|
|
|
|
* option) any later version.
|
|
|
|
|
*
|
|
|
|
|
* The SPL is distributed in the hope that it will be useful, but WITHOUT
|
|
|
|
|
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
|
|
|
|
|
* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
|
|
|
|
|
* for more details.
|
|
|
|
|
*
|
|
|
|
|
* You should have received a copy of the GNU General Public License along
|
|
|
|
|
* with the SPL. If not, see <http://www.gnu.org/licenses/>.
|
2014-12-08 21:35:51 +03:00
|
|
|
*/
|
2014-12-08 21:04:42 +03:00
|
|
|
|
|
|
|
|
#ifndef _SPL_VMEM_H
|
|
|
|
|
#define _SPL_VMEM_H
|
|
|
|
|
|
|
|
|
|
#include <sys/kmem.h>
|
|
|
|
|
#include <linux/sched.h>
|
|
|
|
|
#include <linux/vmalloc.h>
|
|
|
|
|
|
|
|
|
|
typedef struct vmem { } vmem_t;
|
|
|
|
|
|
|
|
|
|
extern vmem_t *heap_arena;
|
|
|
|
|
extern vmem_t *zio_alloc_arena;
|
|
|
|
|
extern vmem_t *zio_arena;
|
|
|
|
|
|
|
|
|
|
extern size_t vmem_size(vmem_t *vmp, int typemask);
|
2014-07-13 22:45:20 +04:00
|
|
|
extern void *spl_vmalloc(unsigned long size, gfp_t lflags, pgprot_t prot);
|
2014-12-08 21:04:42 +03:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Memory allocation interfaces
|
|
|
|
|
*/
|
2014-12-08 21:35:51 +03:00
|
|
|
#define VMEM_ALLOC 0x01
|
|
|
|
|
#define VMEM_FREE 0x02
|
2014-12-08 21:04:42 +03:00
|
|
|
|
|
|
|
|
#ifndef VMALLOC_TOTAL
|
2014-12-08 21:35:51 +03:00
|
|
|
#define VMALLOC_TOTAL (VMALLOC_END - VMALLOC_START)
|
2014-12-08 21:04:42 +03:00
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
/*
|
Refactor generic memory allocation interfaces
This patch achieves the following goals:
1. It replaces the preprocessor kmem flag to gfp flag mapping with
proper translation logic. This eliminates the potential for
surprises that were previously possible where kmem flags were
mapped to gfp flags.
2. It maps vmem_alloc() allocations to kmem_alloc() for allocations
sized less than or equal to the newly-added spl_kmem_alloc_max
parameter. This ensures that small allocations will not contend
on a single global lock, large allocations can still be handled,
and potentially limited virtual address space will not be squandered.
This behavior is entirely different than under Illumos due to
different memory management strategies employed by the respective
kernels. However, this functionally provides the semantics required.
3. The --disable-debug-kmem, --enable-debug-kmem (default), and
--enable-debug-kmem-tracking allocators have been unified in to
a single spl_kmem_alloc_impl() allocation function. This was
done to simplify the code and make it more maintainable.
4. Improve portability by exposing an implementation of the memory
allocations functions that can be safely used in the same way
they are used on Illumos. Specifically, callers may safely
use KM_SLEEP in contexts which perform filesystem IO. This
allows us to eliminate an entire class of Linux specific changes
which were previously required to avoid deadlocking the system.
This change will be largely transparent to existing callers but there
are a few caveats:
1. Because the headers were refactored and extraneous includes removed
callers may find they need to explicitly add additional #includes.
In particular, kmem_cache.h must now be explicitly includes to
access the SPL's kmem cache implementation. This behavior is
different from Illumos but it was done to avoid always masking
the Linux slab functions when kmem.h is included.
2. Callers, like Lustre, which made assumptions about the definitions
of KM_SLEEP, KM_NOSLEEP, and KM_PUSHPAGE will need to be updated.
Other callers such as ZFS which did not will not require changes.
3. KM_PUSHPAGE is no longer overloaded to imply GFP_NOIO. It retains
its original meaning of allowing allocations to access reserved
memory. KM_PUSHPAGE callers can be converted back to KM_SLEEP.
4. The KM_NODEBUG flags has been retired and the default warning
threshold increased to 32k.
5. The kmem_virt() functions has been removed. For callers which
need to distinguish between a physical and virtual address use
is_vmalloc_addr().
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
2014-12-08 23:37:14 +03:00
|
|
|
* vmem_* is an interface to a low level arena-based memory allocator on
|
|
|
|
|
* Illumos that is used to allocate virtual address space. The kmem SLAB
|
|
|
|
|
* allocator allocates slabs from it. Then the generic allocation functions
|
|
|
|
|
* kmem_{alloc,zalloc,free}() are layered on top of SLAB allocators.
|
2014-12-08 21:04:42 +03:00
|
|
|
*
|
Refactor generic memory allocation interfaces
This patch achieves the following goals:
1. It replaces the preprocessor kmem flag to gfp flag mapping with
proper translation logic. This eliminates the potential for
surprises that were previously possible where kmem flags were
mapped to gfp flags.
2. It maps vmem_alloc() allocations to kmem_alloc() for allocations
sized less than or equal to the newly-added spl_kmem_alloc_max
parameter. This ensures that small allocations will not contend
on a single global lock, large allocations can still be handled,
and potentially limited virtual address space will not be squandered.
This behavior is entirely different than under Illumos due to
different memory management strategies employed by the respective
kernels. However, this functionally provides the semantics required.
3. The --disable-debug-kmem, --enable-debug-kmem (default), and
--enable-debug-kmem-tracking allocators have been unified in to
a single spl_kmem_alloc_impl() allocation function. This was
done to simplify the code and make it more maintainable.
4. Improve portability by exposing an implementation of the memory
allocations functions that can be safely used in the same way
they are used on Illumos. Specifically, callers may safely
use KM_SLEEP in contexts which perform filesystem IO. This
allows us to eliminate an entire class of Linux specific changes
which were previously required to avoid deadlocking the system.
This change will be largely transparent to existing callers but there
are a few caveats:
1. Because the headers were refactored and extraneous includes removed
callers may find they need to explicitly add additional #includes.
In particular, kmem_cache.h must now be explicitly includes to
access the SPL's kmem cache implementation. This behavior is
different from Illumos but it was done to avoid always masking
the Linux slab functions when kmem.h is included.
2. Callers, like Lustre, which made assumptions about the definitions
of KM_SLEEP, KM_NOSLEEP, and KM_PUSHPAGE will need to be updated.
Other callers such as ZFS which did not will not require changes.
3. KM_PUSHPAGE is no longer overloaded to imply GFP_NOIO. It retains
its original meaning of allowing allocations to access reserved
memory. KM_PUSHPAGE callers can be converted back to KM_SLEEP.
4. The KM_NODEBUG flags has been retired and the default warning
threshold increased to 32k.
5. The kmem_virt() functions has been removed. For callers which
need to distinguish between a physical and virtual address use
is_vmalloc_addr().
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
2014-12-08 23:37:14 +03:00
|
|
|
* On Linux, the primary means of doing allocations is via kmalloc(), which
|
|
|
|
|
* is similarly layered on top of something called the buddy allocator. The
|
|
|
|
|
* buddy allocator is not available to kernel modules, it uses physical
|
|
|
|
|
* memory addresses rather than virtual memory addresses and is prone to
|
|
|
|
|
* fragmentation.
|
2014-12-08 21:04:42 +03:00
|
|
|
*
|
Refactor generic memory allocation interfaces
This patch achieves the following goals:
1. It replaces the preprocessor kmem flag to gfp flag mapping with
proper translation logic. This eliminates the potential for
surprises that were previously possible where kmem flags were
mapped to gfp flags.
2. It maps vmem_alloc() allocations to kmem_alloc() for allocations
sized less than or equal to the newly-added spl_kmem_alloc_max
parameter. This ensures that small allocations will not contend
on a single global lock, large allocations can still be handled,
and potentially limited virtual address space will not be squandered.
This behavior is entirely different than under Illumos due to
different memory management strategies employed by the respective
kernels. However, this functionally provides the semantics required.
3. The --disable-debug-kmem, --enable-debug-kmem (default), and
--enable-debug-kmem-tracking allocators have been unified in to
a single spl_kmem_alloc_impl() allocation function. This was
done to simplify the code and make it more maintainable.
4. Improve portability by exposing an implementation of the memory
allocations functions that can be safely used in the same way
they are used on Illumos. Specifically, callers may safely
use KM_SLEEP in contexts which perform filesystem IO. This
allows us to eliminate an entire class of Linux specific changes
which were previously required to avoid deadlocking the system.
This change will be largely transparent to existing callers but there
are a few caveats:
1. Because the headers were refactored and extraneous includes removed
callers may find they need to explicitly add additional #includes.
In particular, kmem_cache.h must now be explicitly includes to
access the SPL's kmem cache implementation. This behavior is
different from Illumos but it was done to avoid always masking
the Linux slab functions when kmem.h is included.
2. Callers, like Lustre, which made assumptions about the definitions
of KM_SLEEP, KM_NOSLEEP, and KM_PUSHPAGE will need to be updated.
Other callers such as ZFS which did not will not require changes.
3. KM_PUSHPAGE is no longer overloaded to imply GFP_NOIO. It retains
its original meaning of allowing allocations to access reserved
memory. KM_PUSHPAGE callers can be converted back to KM_SLEEP.
4. The KM_NODEBUG flags has been retired and the default warning
threshold increased to 32k.
5. The kmem_virt() functions has been removed. For callers which
need to distinguish between a physical and virtual address use
is_vmalloc_addr().
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
2014-12-08 23:37:14 +03:00
|
|
|
* Linux sets aside a relatively small address space for in-kernel virtual
|
|
|
|
|
* memory from which allocations can be done using vmalloc(). It might seem
|
|
|
|
|
* like a good idea to use vmalloc() to implement something similar to
|
|
|
|
|
* Illumos' allocator. However, this has the following problems:
|
|
|
|
|
*
|
|
|
|
|
* 1. Page directory table allocations are hard coded to use GFP_KERNEL.
|
|
|
|
|
* Consequently, any KM_PUSHPAGE or KM_NOSLEEP allocations done using
|
|
|
|
|
* vmalloc() will not have proper semantics.
|
|
|
|
|
*
|
|
|
|
|
* 2. Address space exhaustion is a real issue on 32-bit platforms where
|
|
|
|
|
* only a few 100MB are available. The kernel will handle it by spinning
|
|
|
|
|
* when it runs out of address space.
|
|
|
|
|
*
|
|
|
|
|
* 3. All vmalloc() allocations and frees are protected by a single global
|
|
|
|
|
* lock which serializes all allocations.
|
2014-12-08 21:04:42 +03:00
|
|
|
*
|
Refactor generic memory allocation interfaces
This patch achieves the following goals:
1. It replaces the preprocessor kmem flag to gfp flag mapping with
proper translation logic. This eliminates the potential for
surprises that were previously possible where kmem flags were
mapped to gfp flags.
2. It maps vmem_alloc() allocations to kmem_alloc() for allocations
sized less than or equal to the newly-added spl_kmem_alloc_max
parameter. This ensures that small allocations will not contend
on a single global lock, large allocations can still be handled,
and potentially limited virtual address space will not be squandered.
This behavior is entirely different than under Illumos due to
different memory management strategies employed by the respective
kernels. However, this functionally provides the semantics required.
3. The --disable-debug-kmem, --enable-debug-kmem (default), and
--enable-debug-kmem-tracking allocators have been unified in to
a single spl_kmem_alloc_impl() allocation function. This was
done to simplify the code and make it more maintainable.
4. Improve portability by exposing an implementation of the memory
allocations functions that can be safely used in the same way
they are used on Illumos. Specifically, callers may safely
use KM_SLEEP in contexts which perform filesystem IO. This
allows us to eliminate an entire class of Linux specific changes
which were previously required to avoid deadlocking the system.
This change will be largely transparent to existing callers but there
are a few caveats:
1. Because the headers were refactored and extraneous includes removed
callers may find they need to explicitly add additional #includes.
In particular, kmem_cache.h must now be explicitly includes to
access the SPL's kmem cache implementation. This behavior is
different from Illumos but it was done to avoid always masking
the Linux slab functions when kmem.h is included.
2. Callers, like Lustre, which made assumptions about the definitions
of KM_SLEEP, KM_NOSLEEP, and KM_PUSHPAGE will need to be updated.
Other callers such as ZFS which did not will not require changes.
3. KM_PUSHPAGE is no longer overloaded to imply GFP_NOIO. It retains
its original meaning of allowing allocations to access reserved
memory. KM_PUSHPAGE callers can be converted back to KM_SLEEP.
4. The KM_NODEBUG flags has been retired and the default warning
threshold increased to 32k.
5. The kmem_virt() functions has been removed. For callers which
need to distinguish between a physical and virtual address use
is_vmalloc_addr().
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
2014-12-08 23:37:14 +03:00
|
|
|
* 4. Accessing /proc/meminfo and /proc/vmallocinfo will iterate the entire
|
|
|
|
|
* list. The former will sum the allocations while the latter will print
|
|
|
|
|
* them to user space in a way that user space can keep the lock held
|
|
|
|
|
* indefinitely. When the total number of mapped allocations is large
|
|
|
|
|
* (several 100,000) a large amount of time will be spent waiting on locks.
|
|
|
|
|
*
|
|
|
|
|
* 5. Linux has a wait_on_bit() locking primitive that assumes physical
|
|
|
|
|
* memory is used, it simply does not work on virtual memory. Certain
|
|
|
|
|
* Linux structures (e.g. the superblock) use them and might be embedded
|
|
|
|
|
* into a structure from Illumos. This makes using Linux virtual memory
|
|
|
|
|
* unsafe in certain situations.
|
|
|
|
|
*
|
|
|
|
|
* It follows that we cannot obtain identical semantics to those on Illumos.
|
|
|
|
|
* Consequently, we implement the kmem_{alloc,zalloc,free}() functions in
|
|
|
|
|
* such a way that they can be used as drop-in replacements for small vmem_*
|
|
|
|
|
* allocations (8MB in size or smaller) and map vmem_{alloc,zalloc,free}()
|
|
|
|
|
* to them.
|
2014-12-08 21:04:42 +03:00
|
|
|
*/
|
|
|
|
|
|
Refactor generic memory allocation interfaces
This patch achieves the following goals:
1. It replaces the preprocessor kmem flag to gfp flag mapping with
proper translation logic. This eliminates the potential for
surprises that were previously possible where kmem flags were
mapped to gfp flags.
2. It maps vmem_alloc() allocations to kmem_alloc() for allocations
sized less than or equal to the newly-added spl_kmem_alloc_max
parameter. This ensures that small allocations will not contend
on a single global lock, large allocations can still be handled,
and potentially limited virtual address space will not be squandered.
This behavior is entirely different than under Illumos due to
different memory management strategies employed by the respective
kernels. However, this functionally provides the semantics required.
3. The --disable-debug-kmem, --enable-debug-kmem (default), and
--enable-debug-kmem-tracking allocators have been unified in to
a single spl_kmem_alloc_impl() allocation function. This was
done to simplify the code and make it more maintainable.
4. Improve portability by exposing an implementation of the memory
allocations functions that can be safely used in the same way
they are used on Illumos. Specifically, callers may safely
use KM_SLEEP in contexts which perform filesystem IO. This
allows us to eliminate an entire class of Linux specific changes
which were previously required to avoid deadlocking the system.
This change will be largely transparent to existing callers but there
are a few caveats:
1. Because the headers were refactored and extraneous includes removed
callers may find they need to explicitly add additional #includes.
In particular, kmem_cache.h must now be explicitly includes to
access the SPL's kmem cache implementation. This behavior is
different from Illumos but it was done to avoid always masking
the Linux slab functions when kmem.h is included.
2. Callers, like Lustre, which made assumptions about the definitions
of KM_SLEEP, KM_NOSLEEP, and KM_PUSHPAGE will need to be updated.
Other callers such as ZFS which did not will not require changes.
3. KM_PUSHPAGE is no longer overloaded to imply GFP_NOIO. It retains
its original meaning of allowing allocations to access reserved
memory. KM_PUSHPAGE callers can be converted back to KM_SLEEP.
4. The KM_NODEBUG flags has been retired and the default warning
threshold increased to 32k.
5. The kmem_virt() functions has been removed. For callers which
need to distinguish between a physical and virtual address use
is_vmalloc_addr().
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
2014-12-08 23:37:14 +03:00
|
|
|
#define vmem_alloc(sz, fl) spl_vmem_alloc((sz), (fl), __func__, __LINE__)
|
|
|
|
|
#define vmem_zalloc(sz, fl) spl_vmem_zalloc((sz), (fl), __func__, __LINE__)
|
|
|
|
|
#define vmem_free(ptr, sz) spl_vmem_free((ptr), (sz))
|
|
|
|
|
|
|
|
|
|
extern void *spl_vmem_alloc(size_t sz, int fl, const char *func, int line);
|
|
|
|
|
extern void *spl_vmem_zalloc(size_t sz, int fl, const char *func, int line);
|
|
|
|
|
extern void spl_vmem_free(const void *ptr, size_t sz);
|
2014-12-08 21:04:42 +03:00
|
|
|
|
|
|
|
|
int spl_vmem_init(void);
|
|
|
|
|
void spl_vmem_fini(void);
|
|
|
|
|
|
|
|
|
|
#endif /* _SPL_VMEM_H */
|