Implement zeventd daemon

[behlendorf]

As of zfs-0.4.9 the infrastructure is in place to monitor events generated by zfs from user space. Currently the only consumer of this ABI is the 'zpool events' command which can be used to print the events which are just nvlists. What we need now is a zeventd daemon which uses this interface to monitor zevents. Initially this daemon can simply forward the events cleanly off to syslog but longer term we want to be able to have specific events trigger actions. For example, in the case of a disk failure event you may want to send an email or perhaps kick off a suite of diagnostics against the newly failed drive.

[behlendorf]

Additionally one case we need to make sure to handle here is to detect when a disk is going bad but has not yet failed. Often an educated guess about this can be made based on non-success sense codes returned by the drive during IO. The Linux port promotes non-success sense code to zevents for consumption by the zeventd. These events can be used to make a determination that the drive should be proactively failed and a hot space kicked in.

[behlendorf]

The zeventd daemon also looks like the correct way to automatically handle autoexpand. See my first comment in issue #120 for how this should be done.

[colbygk]

I'd like to work on this.

[behlendorf]

Wonderful, we'd love the help. This is a daemon we know we're going to need but just haven't found the time to write yet. However, we have given it some thought so let me sketch out for you what we're thinking and we can start a discussion plus implementation from there.

At a high level here's how the zeventd needs to work. The daemon will most likely be started by the zfs init.d script at system boot time. It will need some minimal configuration from a /etc/zfs/zeventd.conf file. It would be reasonable to simply have the init script source this file and pass any needed configuration in as command line options. Alternately, a parser could be added to zeventd to read in the config file.

Once started the zeventd needs to consume events posted from the kernel via the /dev/zfs ioctl interface. These events are nvlists which are packed in the kernel then passed via ioctl() to user space when they are unpacked. The zpool events command show exactly how this can be done. Depending on the configuration options the event should then be sent to syslog.

Next, we want to add the ability for zeventd to exec an arbitrary helper application. Typically, this will just be a script which allows us to easily customize the behavior. The zeventd should pass the full event to stdin of the helper in an easily parsed form, for example the verbose zpool events -v form. We can then provide a default helper script which parses this and handles the common cases.

• Disk failure - The kernel will automatically fail a disk but the helper may need to kick in a hot spare, start diagnostics on the failed drive, email someone the drive has failed, etc.
• Autoreplace - When a drive is replaced the autoreplace property should be checked and zfs replace run accordingly.
• Autoexpand - When a vdev is expanded online the autoexpand property should be checked and zpool online -e run accordingly.

For all the above we should try to be consistent with the Solaris FMA behavior as much as possible. If people want different behavior they can of course tweak the script to their liking. That is exactly why we're making the policy bit a script and not placing it in the heart of zeventd. We can also provide some helper functions in the script to make customization easy and list all the possible events and what they mean.

The easiest way to implement the core of the zeventd will be as a user space daemon written in C. This way the daemon can easily leverage certain existing libraries in the zfs package to simplify the kernel ioctl, the nvlist manipulation, and event formatting. There are already helper functions to do all these things so it should be relatively straight forward.

[colbygk]

Sounds good to me. I will set up a stubbed out server to be in zfs/cmds/zeventd and will work towards having a parser for a config file that won't do much at first. Following on through reading events from /dev/zfs etc...

I'm only cursorily familiar with the behavior of Solaris FMA.

I have a dev environment set up under openSUSE 11.3 now, built both zfs/spl and have created a zpool that mounted. Will follow up with more later. Any suggested resources to look at to learn more about Solaris FMA?

[behlendorf]

Sounds like a plan. To be honest I'm not all that familiar with Solaris FMA either, but I don't think we need/want to replicate it entirely. At least initially we just want to make sure we have enough infrastructure in place to handle the common events like disk failure, autoreplace, and autoreplace. We can determine if we need a more complicated analysis engine latter.

[colbygk]

Quick questions:

Do zfs events have some sort of computationally unique ID's? Can individual events be cleared vs clearing all events? Consider the following:

zeventd is running, recognizes a disk failure event, runs a script that is configured to launch in the event of such a failure, presumably this script attempts to move a hot spare into place.

Some time later, an admin restarts zeventd using the init.d script. zeventd rereads events and sees the same series that it acted upon earlier. That would be bad! This is moot if doing a zpool_next_event(...) using blocking I/O will only read new events and not old ones (which it looks like is what happens in my initial testing).

Initially, I was thinking that zeventd could keep a log of the events it has acted upon, using some unique id associated with each event (an ID produced by the underlying ZFS kernel module) and not act on any event that it has already logged.

and/or

zeventd could remove events after it has acted on them (I don't like this idea, I'd rather leave the event log intact).

Perhaps zeventd could inject log events as it acts on other events, so that the chain of event->action/resolution is maintained in the zfs events log itself. This would also be a way for it to keep track of events that it has already acted upon.

[behlendorf]

There is supposed to be an ENA (Error Numeric Association) nvpair in the nvlist which is close to this. However, I just noticed there is not an ENA for all event types. This field is not added for events generated by zfs_post_common(), we may need to add this. If you look at the header in module/zfs/zfs_fm.c you can read a pretty good description of how ENAs are support to be used under Solaris FMA.

The way events work at the moment is the kernel maintains a list of the last 64 events. These events are then accessed via an ioctl() on /dev/zfs. Each process when it opens /dev/zfs is allowed to read events as a private stream. This allows multiple consumers of /dev/zfs events to operate concurrently without destroying each others events.

However, when the device is closed and opened the process will start getting against from the oldest to the newest. Your absolutely right this could be very bad if the zeventd was restart and reprocessed the same events. Well once we get the ENA added to all event you could start tracking those in a log file. Or you could try something even simpler to start with. All the events are time stamped by the kernel when they were generated. If you simply store the time stamp of the last event processed by zeventd, then you would know which events to skip if the daemon was restarted. Events older than the logged time stamp must already have been processed and can be ignored. There may be a small race here with something like ntp adjusting your clock but that seems like an exceptional rare case.

[colbygk]

Ahh yes, I had noticed the ENA and thought it might be this. I like your suggestion of storing a high resolution copy of the time as an initial start and look forward to having ENA on all events.

[Rudd-O]

Seems to me like udev or udisks already provide what you are looking for, and you should route events through them (making zeventd, if needed, a consumer of those interfaces).

[colbygk]

I have been busy with a job transition but am coming back to this.

Re: udev and udisks.

udev is for devices that the kernel notices are new or have been removed and is not appropriate in the case of passing messages for zeventd
udisks I had not heard of before, but it seems to be a sub-component of free desktop and is a glue layer between disk related events and d-bus listeners.

zeventd would most likely be a path for messages to end up in udisks.

[pyavdr]

Hi Brian,
i really like this ZFS port to linux for the last 6 months. In these 6 months there was a lot of bugs beeing fixed and zol becomes more stable then ever. BUT the autoreplace feature for failing disks does not work and this issue is numbered as #2. I would say, this is a very basic feature.
As you guys are really good in solving issues, is there any chance to made this work for the next rc9 ?

[behlendorf]

No, I can't see this code being finished by -rc9. But your right it absolutely needs to happen, and sooner would be better than later. We just haven't had enough time with the other outstanding issues to get to this. You could probably cobble together some scripts to handle this job if your desperate and need it right away.

[pyavdr]

@behlendorf

Hi Brian,

while composing some C-code for an zeventd, the timeshifts driving me crazy. There are always timeshifts within a system. On startup there are a timeshift if the timezone gets involved, which is a most common point for vdev failures. If we think about traveling around with a zpool, then there are lots. Also NTP timeshifts occur really periodically.
So zpool events shows timestamps in future or in the past, which is not predictable, so not reliable for an eventd.
For me, the whole event correlation leads to parse a log file, filled with events, searching for enas ( asuming they are unique ..), which are not in there, thus making the assumption, that this is a new event, and writing again timpestamps in there, probably leading to the same problems with timeshifts. But doing so, this is not really efficient. Any ideas? (Maybe creating a database with keys for ENA, or trees … ?)
Next questions for the target of triggering a “zpool_vdev_attach(zhp, old_disk, new_disk, nvroot, replacing)” call are: Which event (class, ena, whatever ...) should really trigger the replacement ?
There are the unsolved issues #250 and #544. That makes things complicated too. Is there any solution in sight for them?

[pyavdr]

@behlendorf

Hi Brian,

while composing some C-code for an zeventd, the timeshifts driving me crazy. There are always timeshifts within a system. On startup there are a timeshift if the timezone gets involved, which is a most common point for vdev failures. If we think about traveling around with a zpool, then there are lots. Also NTP timeshifts occur really periodically.
So zpool events shows timestamps in future or in the past, which is not predictable, so not reliable for an eventd.
For me, the whole event correlation leads to parse a log file, filled with events, searching for enas ( asuming they are unique ..), which are not in there, thus making the assumption, that this is a new event, and writing again timpestamps in there, probably leading to the same problems with timeshifts. But doing so, this is not really efficient. Any ideas? (Maybe creating a database with keys for ENA, or trees … ?)
Next questions for the target of triggering a “zpool_vdev_attach(zhp, old_disk, new_disk, nvroot, replacing)” call are: Which event (class, ena, whatever ...) should really trigger the replacement ?
There are the unsolved issues #250 and #544. That makes things complicated too. Is there any solution in sight for them?

[behlendorf]

@pyavdr Concerning the timeshift for ENAs we could perhaps include the current value of the kernels monotonically increasing clock along with the timestamp. Or event simpler a monotonically increasing number of ENAs which have been generated since the zfs module was loaded (although that might break down if you were to unload/reload the module). Really, the contents of the event are entirely up to us so don't feel to locked in to what's there now.

Offhand I'm not exactly sure which events should be used to trigger a replacement. I'd need to look at it carefully, it wouldn't be the worst idea to look and see what Solaris does in their FMA.

I haven't had a chance to seriously look in to either #250 or #544 yet. However, your right getting them fixed is a bit of a prerequisite for this work.

[pyavdr]

Brian,

while searching i found this : http://gdamore.blogspot.com/2010/07/zfs-disk-monitoring.html

Is there any chance you can contact him to ask for the code of his monitor ?

[behlendorf]

Garrett's been busy. It sounds like he's willing to share the code with anyone, it's certainly worth taking a look at. Although since it's a FMA module I'm not sure how useful it will be.

[pyavdr]

@behlendorf

Brian,

while testing around failed disks the zpool events for some cases are missing.

In my vmware suse guest i created a quite large pool with a zfs with 16 vdevs. Next i remove one vdev with vmware device manager. Copied a file to the pool. After that the pool shows state (zpool status) "degraded" with the missing vdev marked first as "faulted" and then "removed". But there is only a simply "state change" event in the zpool events.
If the events are not giving more information, it would be better to analyse the state of the pool and the vdevs itself, but this seems to be double effort. As you mentioned, the events interface isn´t complete right now. For an zeventd, the events are essential. I would like to wish some more events which clearly reflects the vdev/zpool state. Alternatively just parsing the zpool status output maybe gives a faster approach to the target of replacing a "unavailable or removed" disk automatically . Pls comment. Thank you!