execlib.listeners.path module

class execlib.listeners.path.PathListener(router)

Bases: Listener[FileEvent]

__init__(router)

Parameters:

router – associated Router instance that events should be passed to

Note

Due to the nature of INotify, you cannot watch the same path with two separate flag settings (without creating a new INotify instance). Under the same instance, calling add_watch for an already watched path location will simply return the watch descriptor already associated with that location (and may update the flags to whatever is passed). However, this location will only ever be “watched once” by a given INotify instance, so keep this in mind if multiple downstream callbacks (packaged up by the Router) want to react to the same path but with different flags (i.e., it won’t work as expected).

handle_events(events)

Note

If handle_events is called externally, note that this loop will block in the calling thread until the jobs have been submitted. It will not block until jobs have completed, however, as a list of futures is returned. The calling Listener instance may have already been started, in which case run() will already be executing in a separate thread. Calling this method externally will not interfere with this loop insofar as it adds jobs to the same thread pool.

Because this method only submits jobs associated with the provided events, the calling thread can await the returned list of futures and be confident that top-level callbacks associated with these file events have completed. Do note that, if the Listener has already been started, any propagating file events will be picked up and possibly processed simultaneously (although their associated callbacks will have nothing to do with the returned list of futures).

listen(path, flags=None)

Listen to file events occurring under a provided path, optionally excluding those not matching the provided iNotify flags.

Parameters:

• path – Path (directory) to watch with inotify

• flags – inotify_simple flags matching FS event types allowed to trigger the callback

run()

Start the (blocking) iNotify event loop

Note: On usage

start() is a blocking call. This will hog your main thread if not properly threaded. If handling this manually in your outer context, you will also need to make sure to call .stop()

stop()

Shutdown active listener processes, including the attached router thread pool and the iNotify event loop.

Note

Shutting down the thread pool will wait until pending futures are finished executing before actually returning. A common source of error is having the main process exit before final tasks can be submitted, resulting in RuntimeErrors that cannot “schedule new futures after interpreter shutdown.” So you either need to ensure the final tasks are scheduled before calling stop() (this means more than just a submit() call; it must have actually propagated through to submit_callback and reached thread_pool.submit) to allow them to be handled automatically prior to shutdown, or manually wait on their futures to complete. Otherwise, thread pool shutdown will occur, and they’ll still be making their way out of the queue only to reach the thread_pool.submit after it’s had its final boarding call.

update_moved_from(path, lead)

Update directories on MOVED_FROM events.

Additional details

This method gets the existing WD, removes the old path associated with that WD from the watchmap (preventing events originating from this old path when the new path, which has the same WD, receives an inotify event), and queues the (WD, base path) tuple to be matched later in a MOVED_TO handler.

This method isn’t a part of a MOVED_TO handler because it may be called without ever having a MOVED_TO that follows up. We respond right away in handle_events to MOVED_FROM events, keeping the watchmap in sync, regardless of whether we can expect a MOVED_TO to sweep through after the fact.

Note that the lead is unique for a given WD and base path. WDs are unique for filepaths, but inotify uses the same WD for new directories when they experience a rename (it’s the same inode). However, during such a transition, the watchmap can see two different entries for the same WD and basepath: the old tracked path, and the newly named one (again, still the same inode). So: this method can be called 1) directly from MOVED_FROM events, preemptively wiping the old path from the tracked dicts, or 2) during handling of a MOVED_TO event (in case we don’t allow MOVED_FROM events, for instance), given both the new and old paths can be seen in the watchmap.

update_moved_to(path, lead)

Construct synthetic MOVED events. Events are constructed from the path’s WD. If the provided path is not watched, an empty list of events is returned.

Design details

This method is nuanced. It can only be called once a MOVED_TO occurs, since we can’t act on a MOVED_FROM (we don’t know the new target location to look so we can send file events). When called, we first look for the path’s WD in the pathmap. We then check if this WD points to more than one entry with the same base path (WDs are unique to the path; under the same WD, the same base path implies the same lead). If so, we know one is the outdated path, and we push the outdated lead to update_moved_from. This would be evidence that the MOVED_FROM event for the move operation wasn’t handled in the main event handling loop. We then check for unmatched move-froms, which should provide any renamed directories, regardless of whether MOVED_FROMs were allowed, to be detected. Finally, the appropriate MOVED_FROMs and MOVED_TOs are handled. To ensure only the correct events match upon handling, we do the following:

• First, if a MOVED_FROM path is not available, we assume it wasn’t queued by the event and not a watched flag. Given we by default ensure MOVED events are tracked, regardless of listened paths, this shouldn’t be possible, but if this standard were to change, we won’t recursively respond to MOVED_FROMs. This will mean that we can’t prevent events from being matched to old directory names (we’ve rooted out the ability to tell when they’ve changed), and thus can’t remove them from the watchpath accordingly. (One functional caveat here: this MOVED_TO handling method explicitly calls updated_moved_from, which should clean up lingering renamed path targets. This happens recursively if we’re watching MOVED_TOs, so even if standards do change and you don’t watch MOVED_FROMs, you’ll still get clean up for free due to the robustness of this method.

• If a MOVED_FROM lead is found, either due to an inferred matching base lingering in the watchmap or through previously handled MOVED_FROM response, add this path/lead back to the watchmap, remove the new path/lead, and call handle_events for the synthetic MOVED_FROM events across files and directories. Once finished, again remove the old path/lead and add back the new one.

• Submit MOVED_TO events to handle_events. This will recursively propagate for subdirectories, each submitting their own update_moved_to call, resetting its own outdated leads and changing them back, all the way down to the bottom.

In the odd case where MOVED_FROM is registered but not MOVED_TO, you will simply remove the directory causing a MOVED_FROM event, with no recursive propagation. This should likely be changed.