ZFS even under Solaris does not strictly require libshare to be
available. The current implementation attempts to dlopen() the
library to access the needed symbols. If this fails libshare
support is simply disabled.
This means that on Linux we only need the most minimal libshare
implementation. In fact just enough to prevent the build from
failing. Longer term we can decide if we want to implement a
libshare library like Solaris. At best this would be an abstraction
layer between ZFS and NFS/SMB. Alternately, we can drop libshare
entirely and directly integrate ZFS with Linux's NFS/SMB.
Finally the bare bones user-libshare.m4 test was dropped. If we
do decide to implement libshare at some point it will surely be
as part of this package so the check is not needed.
By design the zfs utility is supposed to handle mounting and unmounting
a zfs filesystem. We could allow zfs to do this directly. There are
system calls available to mount/umount a filesystem. And there are
library calls available to manipulate /etc/mtab. But there are a
couple very good reasons not to take this appraoch... for now.
Instead of directly calling the system and library calls to (u)mount
the filesystem we fork and exec a (u)mount process. The principle
reason for this is to delegate the responsibility for locking and
updating /etc/mtab to (u)mount(8). This ensures maximum portability
and ensures the right locking scheme for your version of (u)mount
will be used. If we didn't do this we would have to resort to an
autoconf test to determine what locking mechanism is used.
The downside to using mount(8) instead of mount(2) is that we lose
the exact errno which was returned by the kernel. The return code
from mount(8) provides some insight in to what went wrong but it
not quite as good. For the moment this is translated as a best
guess in to a errno for the higher layers of zfs.
In the long term a shared library called libmount is under development
which provides a common API to address the locking and errno issues.
Once the standard mount utility has been updated to use this library
we can then leverage it. Until then this is the only safe solution.
http://www.kernel.org/pub/linux/utils/util-linux/libmount-docs/index.html
Recently helper functions were added to libzfs_util to load a kernel
module or execute a process. Initially this functionality was limited
to libzfs but it has become clear there will be other consumers. This
change opens up the interface so it may be used where appropriate.
If libselinux is detected on your system at configure time link
against it. This allows us to use a library call to detect if
selinux is enabled and if it is to pass the mount option:
"context=\"system_u:object_r:file_t:s0"
For now this is required because none of the existing selinux
policies are aware of the zfs filesystem type. Because of this
they do not properly enable xattr based labeling even though
zfs supports all of the required hooks.
Until distro's add zfs as a known xattr friendly fs type we
must use mntpoint labeling. Alternately, end users could modify
their existing selinux policy with a little guidance.
These compiler warnings were introduced when code which was
previously #ifdef'ed out by HAVE_ZPL was re-added for use
by the posix layer. All of the following changes should be
obviously correct and will cause no semantic changes.
For while now mkdirp has been built as part of libspl however
the protoype was never added to libgen.h. This went unnoticed
until enabling the mount support which uses mkdirp().
Specifically, this fixes the two following errors in zdb when a pool
is composed of block devices:
1) 'Value too large for defined data type' when running 'zdb <dataset>'.
2) 'character device required' when running 'zdb -l <block-device>'.
Signed-off-by: Ricardo M. Correia <ricardo.correia@oracle.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
If libblkid does not contain ZFS support, then 'zpool import' will scan
all block devices in /dev/ to determine which ones are components of a
ZFS filesystem. It does this by opening all the devices and stat'ing
them to determine which ones are block devices. If the device turns
out not to be a block device it is skipped.
Usually, this whole process is pretty harmless (although slow). But
there are certain devices in /dev/ which must be handled in a very
specific way or your system may crash. For example, if /dev/watchdog
is simply opened the watchdog timer will be started and your system
will panic when the timer expires.
It turns out the /dev/hpet causes similiar problems although only when
accessed under a virtual machine. For some reason accessing /dev/hpet
causes qemu to crash. To address this issue this commit adds /dev/hpet
to the device blacklist, it will be skipped solely based on its name.
This commit fixes a sign extension bug affecting l2arc devices. Extremely
large offsets may be passed down to the low level block device driver on
reads, generating errors similar to
attempt to access beyond end of device
sdbi1: rw=14, want=36028797014862705, limit=125026959
The unwanted sign extension occurrs because the function arc_read_nolock()
stores the offset as a daddr_t, a 32-bit signed int type in the Linux kernel.
This offset is then passed to zio_read_phys() as a uint64_t argument, causing
sign extension for values of 0x80000000 or greater. To avoid this, we store
the offset in a uint64_t.
This change also changes a few daddr_t struct members to uint64_t in the libspl
headers to avoid similar bugs cropping up in the future. We also add an ASSERT
to __vdev_disk_physio() to check for invalid offsets.
Closes#66
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Some sudo configurations may not include /sbin in the PATH.
libzfs_load_module() currently does not call modprobe with an absolute path, so
it may fail under such configurations if called under sudo. This change adds
the absolute path to modprobe so we no longer rely on how PATH is set.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
zpool status displays abbreviated vdev names without leading path components
and, in the case of whole disks, without partition information. Also, the
zpool subcommands 'create' and 'add' support using shorthand devices names
without qualified paths. Prior to this change, however, removing a device
generally required specifying its name as it is stored in the vdev label. So
while zpool status might list a cache disk with a name like A16, removing it
would require a full path such as /dev/disk/zpool/A16-part1, which is
non-intuitive.
This change adds support for shorthand device names with the remove subcommand
so one can simply type, for example,
zpool remove tank A16
A consequence of this change is that including the partition information when
removing a whole-disk vdev now results in an error. While this is arguably the
correct behavior, it is a departure from how zpool previously worked in this
project.
This change removes the only reference to ctd_check_path(), so that function is
also removed to avoid compiler warnings.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
This change adds two helper functions for working with vdev names and paths.
zfs_resolve_shortname() resolves a shorthand vdev name to an absolute path
of a file in /dev, /dev/disk/by-id, /dev/disk/by-label, /dev/disk/by-path,
/dev/disk/by-uuid, /dev/disk/zpool. This was previously done only in the
function is_shorthand_path(), but we need a general helper function to
implement shorthand names for additional zpool subcommands like remove.
is_shorthand_path() is accordingly updated to call the helper function.
There is a minor change in the way zfs_resolve_shortname() tests if a file
exists. is_shorthand_path() effectively used open() and stat64() to test for
file existence, since its scope includes testing if a device is a whole disk
and collecting file status information. zfs_resolve_shortname(), on the other
hand, only uses access() to test for existence and leaves it to the caller to
perform any additional file operations. This seemed like the most general and
lightweight approach, and still preserves the semantics of is_shorthand_path().
zfs_append_partition() appends a partition suffix to a device path. This
should be used to generate the name of a whole disk as it is stored in the vdev
label. The user-visible names of whole disks do not contain the partition
information, while the name in the vdev label does. The code was lifted from
the function make_disks(), which now just calls the helper function. Again,
having a helper function to do this supports general handling of shorthand
names in the user interface.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
ZFS works best when it is notified as soon as possible when a device
failure occurs. This allows it to immediately start any recovery
actions which may be needed. In theory Linux supports a flag which
can be set on bio's called FAILFAST which provides this quick
notification by disabling the retry logic in the lower scsi layers.
That's the theory at least. In practice is turns out that while the
flag exists you oddly have to set it with the BIO_RW_AHEAD flag.
And even when it's set it you may get retries in the low level
drivers decides that's the right behavior, or if you don't get the
right error codes reported to the scsi midlayer.
Unfortunately, without additional kernels patchs there's not much
which can be done to improve this. Basically, this just means that
it may take 2-3 minutes before a ZFS is notified properly that a
device has failed. This can be improved and I suspect I'll be
submitting patches upstream to handle this.
This commit modifies libzfs_init() to attempt to load the zfs kernel module if
it is not already loaded. This is done to simplify initialization by letting
users simply import their zpools without having to first load the module.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Under Solaris, the slice number is chopped off when displaying the device name
if the vdev is a whole disk. Under Linux we should similarly discard the
partition number. This commit adds the logic to perform the name truncation
for devices ending in -partX, XpX, or X, where X is a string of digits. The
second case handles devices like md0p0. The third case is limited to scsi and
ide disks, i.e. those beginning with "sd" or "hd", in order to avoid stripping
the number from names like "loop0".
This commit removes the Solaris-specific code for removing slices, since we no
longer reasonably expect our changes to be merged in upstream. The partition
stripping code was moved off to a helper function to improve readability.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Top-level vdev names in zpool status output should follow a <type-id> naming
convention. In the case of raidz devices, the type portion of the name was
missing.
This commit fixes a bug in zpool_vdev_name() where in this snprintf call
(void) snprintf(buf, sizeof (buf), "%s-%llu", path,
(u_longlong_t)id);
buf and path may point to the same location. The result is that buf ends up
containing only the "-id" part. This only occurred for raidz devices because
the code for appending the parity level to the type string stored its result in
buf then set path to point there. To fix this we allocate a new temporary
buffer on the stack instead of reusing buf.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#57
The zfs package supports the option --with-config=srpm which
is used to bootstrap configure to allow the 'make srpm' target
to work. This has the advantage of allowing creation of source
rpms without having all your -devel packages installed. This
source package can then be feed back in to an automated build
farm which only installs the required packages listed by the
srpm. This ensures that all proper dependencies are expressed
by the source package, because if they are not you will get
configure/build failures.
The trouble here is that --with-config=srpm prevents the
architecture check from running resulting in TARGET_ASM_DIR
being set to the default asm-generic. The 'make dist' rule
then fails because there is no asm-generic/atomic.S file
because it is generated at build time. To handle this I
have added an empty file asm-generic/atomic.S simply as a
place holder for 'make dist'.
Under Ubuntu 10.04 the default compiler flags include -Wformat
and -Wformat-security which cause the above warning. In particular,
cases where "%s" was forgotten as part of the format specifier.
https://wiki.ubuntu.com/CompilerFlags
One of the neat tricks an autoconf style project is capable of
is allow configurion/building in a directory other than the
source directory. The major advantage to this is that you can
build the project various different ways while making changes
in a single source tree.
For example, this project is designed to work on various different
Linux distributions each of which work slightly differently. This
means that changes need to verified on each of those supported
distributions perferably before the change is committed to the
public git repo.
Using nfs and custom build directories makes this much easier.
I now have a single source tree in nfs mounted on several different
systems each running a supported distribution. When I make a
change to the source base I suspect may break things I can
concurrently build from the same source on all the systems each
in their own subdirectory.
wget -c http://github.com/downloads/behlendorf/zfs/zfs-x.y.z.tar.gz
tar -xzf zfs-x.y.z.tar.gz
cd zfs-x-y-z
------------------------- run concurrently ----------------------
<ubuntu system> <fedora system> <debian system> <rhel6 system>
mkdir ubuntu mkdir fedora mkdir debian mkdir rhel6
cd ubuntu cd fedora cd debian cd rhel6
../configure ../configure ../configure ../configure
make make make make
make check make check make check make check
This change also moves many of the include headers from individual
incude/sys directories under the modules directory in to a single
top level include directory. This has the advantage of making
the build rules cleaner and logically it makes a bit more sense.
The GIT file was removed from the tree because I have stopped
using TopGit. Because of this is must also be removed from
the top level Makefile.am as will as the zfs.spec.in file
which referenced it.
Fix type in lib/libzpool/Makefile.am which was preventing
the needed zrlock.h header from being included by 'make dist'.
I simply had the name wrong in the Makefile.am.
Regenerated autogen.sh build products.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Add the initial products from autogen.sh. These products will
be updated incrementally after this point as development occurs.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
This topic branch contains required changes to the user space
utilities to allow them to integrate cleanly with Linux.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
This topic branch contains all the changes needed to integrate the user
side zfs tools with Linux style devices. Primarily this includes fixing
up the Solaris libefi library to be Linux friendly, and integrating with
the libblkid library which is provided by e2fsprogs.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Track various large hunks which have been dropped simply
because they are not relevant to this port.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Solaris recently introduced the idea of drive topology because
where a drive is located does matter. I have already handled
this with udev/blkid integration under Linux so I'm hopeful
this case can simply be removed but for now I've just stubbed
out what is needed in libspl and commented out the rest here.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
The upstream ZFS code has correctly moved to a faster native sha2
implementation. Unfortunately, under Linux that's going to be a little
problematic so we revert the code to the more portable version contained
in earlier ZFS releases. Using the native sha2 implementation in Linux
is possible but the API is slightly different in kernel version user
space depending on which libraries are used. Ideally, we need a fast
implementation of SHA256 which builds as part of ZFS this shouldn't be
that hard to do but it will take some effort.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
All changes needed for the libspl layer. This includes modifications
to files directly copied from OpenSolaris and the addition of new
files needed to fill in the gaps.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
This branch contains the majority of the changes required to cleanly
intergrate with Linux style special devices (/dev/zfs). Mainly this
means dropping all the Solaris style callbacks and replacing them
with the Linux equivilants.
This patch also adds the onexit infrastructure needed to track
some minimal state between ioctls. Under Linux it would be easy
to do this simply using the file->private_data. But under Solaris
they apparent need to pass the file descriptor as part of the ioctl
data and then perform a lookup in the kernel. Once again to keep
code change to a minimum I've implemented the Solaris solution.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
The ZFS update to onnv_141 brought with it support for a
security label attribute called mlslabel. This feature
depends on zones to work correctly and thus I am disabling
it under Linux. Equivilant functionality could be added
at some point in the future.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
This topic branch leverages the Solaris style FMA call points
in ZFS to create a user space visible event notification system
under Linux. This new system is called zevent and it unifies
all previous Solaris style ereports and sysevent notifications.
Under this Linux specific scheme when a sysevent or ereport event
occurs an nvlist describing the event is created which looks almost
exactly like a Solaris ereport. These events are queued up in the
kernel when they occur and conditionally logged to the console.
It is then up to a user space application to consume the events
and do whatever it likes with them.
To make this possible the existing /dev/zfs ABI has been extended
with two new ioctls which behave as follows.
* ZFS_IOC_EVENTS_NEXT
Get the next pending event. The kernel will keep track of the last
event consumed by the file descriptor and provide the next one if
available. If no new events are available the ioctl() will block
waiting for the next event. This ioctl may also be called in a
non-blocking mode by setting zc.zc_guid = ZEVENT_NONBLOCK. In the
non-blocking case if no events are available ENOENT will be returned.
It is possible that ESHUTDOWN will be returned if the ioctl() is
called while module unloading is in progress. And finally ENOMEM
may occur if the provided nvlist buffer is not large enough to
contain the entire event.
* ZFS_IOC_EVENTS_CLEAR
Clear are events queued by the kernel. The kernel will keep a fairly
large number of recent events queued, use this ioctl to clear the
in kernel list. This will effect all user space processes consuming
events.
The zpool command has been extended to use this events ABI with the
'events' subcommand. You may run 'zpool events -v' to output a
verbose log of all recent events. This is very similar to the
Solaris 'fmdump -ev' command with the key difference being it also
includes what would be considered sysevents under Solaris. You
may also run in follow mode with the '-f' option. To clear the
in kernel event queue use the '-c' option.
$ sudo cmd/zpool/zpool events -fv
TIME CLASS
May 13 2010 16:31:15.777711000 ereport.fs.zfs.config.sync
class = "ereport.fs.zfs.config.sync"
ena = 0x40982b7897700001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0xed976600de75dfa6
(end detector)
time = 0x4bec8bc3 0x2e5aed98
pool = "zpios"
pool_guid = 0xed976600de75dfa6
pool_context = 0x0
While the 'zpool events' command is handy for interactive debugging
it is not expected to be the primary consumer of zevents. This ABI
was primarily added to facilitate the addition of a user space
monitoring daemon. This daemon would consume all events posted by
the kernel and based on the type of event perform an action. For
most events simply forwarding them on to syslog is likely enough.
But this interface also cleanly allows for more sophisticated
actions to be taken such as generating an email for a failed drive.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Add autoconf style build infrastructure to the ZFS tree. This
includes autogen.sh, configure.ac, m4 macros, some scripts/*,
and makefiles for all the core ZFS components.
We should not put a 4k maxpathlen buffer on the stack, instead
locate it to the heap. Even in user space we run ztest with 8K
stacks to verify correctness
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Certain function must never be automatically inlined by gcc because
they are stack heavy or called recursively. This patch flags all
such functions I've found as 'noinline' to prevent gcc from making
the optimization.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
This is a portability change which removes the dependence of the Solaris
thread library. All locations where Solaris thread API was used before
have been replaced with equivilant Solaris kernel style thread calls.
In user space the kernel style threading API is implemented in term of
the portable pthreads library. This includes all threads, mutexs,
condition variables, reader/writer locks, and taskqs.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
This look like a typo. The intention was to use strlcat() however
strncat() was used instead accidentally this may lead to a buffer
overflow. This was caught by gcc -D_FORTIFY_SOURCE=2.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Do not use zmod.h in userspace.
This has also been filed with the ZFS team. It makes the userspace
libzpool code use the zlib API, instead of the Solaris-only and
non-standard zmod.h. The zlib API is almost identical and is a de
facto standard, so this is a no-brainer.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
1) In vn_open(), if fstat64() returned an error, the real errno
was being obscured by calling close().
2) Add error handling for both pwrite64() calls in vn_rdwr().
Signed-off-by: Ricardo M. Correia <Ricardo.M.Correia@Sun.COM>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Remove deadcode. It's possible the code should be in use
somewhere, but as the source code is laid out it currently
is not.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Use constructor attribute on non-Solaris platforms.
The #pragma init/fini ->__attribute__((constructor/destructor))
conversions, these should go upstream.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Resolve issues uncovered by -D_FORTIFY_SOURCE=2, the default redhat
macro's file adds this option to the cflags. This causes warnings
of the following type designed to keep the developer honest:
warning: ignoring return value of 'foo', declared
with attribute warn_unused_result
The short term fix is to wrap these calls in VERIFY() to check the
return code. The code was already assusing these would never fail.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Fix non-c90 compliant code, for the most part these changes
simply deal with where a particular variable is declared.
Under c90 it must alway be done at the very start of a block.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
