helm-org-rifle

This is my rifle. There are many like it, but this one is mine. My rifle is my best friend. It is my life. I must master it as I must master my life.

What does my rifle do? It searches rapidly through my Org files, quickly bringing me the information I need to defeat the enemy.

This package is inspired by org-search-goto/org-search-goto-ml. It searches both headings and contents of entries in Org buffers, and it displays entries that match all search terms, whether the terms appear in the heading, the contents, or both. Matching portions of entries’ contents are displayed with surrounding context and grouped by buffer to make it easy to acquire your target.

In contrast with org-occur and similar commands, helm-org-rifle is entry-based (i.e. a heading and all of its contents, not including subheadings), while org-occur is line-based. So org-occur will show you entire lines that contain matching words, without any reference to the heading the line is under, while helm-org-rifle will show the heading of the entry that matches, followed by context around each matching word in the entry. In other words, helm-org-rifle is sort of like Google, while org-occur is sort of like grep.

Entries are fontified by default to match the appearance of an Org buffer, and optionally the entire path can be displayed for each entry, rather than just its own heading.

Screenshots

An animation is worth…a million words?

With helm-org-rifle-show-path set to t, the whole path to each heading is shown:

Note: These screenshots were taken with solarized-theme and spacemacs-dark, and these org-level face styles are part of those themes, not part of this package. If you install this, they will be fontified according to your own theme and faces.

Contents

• Installation
• Usage
• Changelog
• Credits
• Development

Installation

MELPA

If you installed from MELPA, your rifle is ready. Just run one of the commands below.

Manual

Install Helm, dash.el, f.el, and s.el. Then require this package in your init file:

(require 'helm-org-rifle)

Usage

Run one of the rifle commands, type some words, and results will be displayed, grouped by buffer. Hit RET to show the selected entry, or <C-return> to show it in an indirect buffer.

Helm commands: show results in a Helm buffer

• helm-org-rifle: Show results from all open Org buffers
• helm-org-rifle-agenda-files: Show results from Org agenda files
• helm-org-rifle-current-buffer: Show results from current buffer
• helm-org-rifle-directories: Show results from selected directories; with prefix, recursively
• helm-org-rifle-files: Show results from selected files
• helm-org-rifle-org-directory: Show results from Org files in org-directory

Occur commands: show results in an occur-like, persistent buffer

• helm-org-rifle-occur: Show results from all open Org buffers
• helm-org-rifle-occur-agenda-files: Show results from Org agenda files
• helm-org-rifle-occur-current-buffer: Show results from current buffer
• helm-org-rifle-occur-directories: Show results from selected directories; with prefix, recursively
• helm-org-rifle-occur-files: Show results from selected files
• helm-org-rifle-occur-org-directory: Show results from Org files in org-directory

Tips

• Select multiple entries in the Helm buffer to display selected entries in a read-only, occur-style buffer.
• Save all results in a Helm buffer to a helm-org-rifle-occur buffer by pressing C-s (like helm-grep-save-results).
• Show results from certain buffers by typing the name of the buffer (usually the filename).
• Show headings with certain to-do keywords by typing the keyword, e.g. TODO or DONE.
  • Multiple to-do keywords are matched with boolean OR.
• Show headings with certain priorities by typing, e.g. #A or [#A].
• Show headings with certain tags by searching for, e.g. :tag1:tag2:.
• Negate matches with a !, e.g. pepperoni !anchovies.
• Sort results by timestamp or buffer-order (the default) by calling commands with a universal prefix (C-u).
• Show entries in an indirect buffer by selecting that action from the Helm actions list, or by pressing <C-return>.
• The keymap for helm-org-rifle-occur results buffers imitates the org-speed keys, making it quicker to navigate. You can also collapse and expand headings and drawers with TAB and S-TAB, just like in regular Org buffers. Results buffers are marked read-only so you cannot modify them by accidental keypresses.
• Delete the result at point in helm-org-rifle-occur buffers by pressing d. This does not alter the source buffers but simply removes uninteresting results from view.
• You can customize the helm-org-rifle group if you like.

Changelog

1.5.0-pre

Additions

• Match to-do keywords specifically.
  • Previously, to-do keywords were matched as normal words, so searching for WAITING something would find any entry with the words waiting or something in it. Now, to-do keywords are recognized as such and are compared to the actual to-do keyword of each entry, using the list of to-do keywords specific to each buffer. So, searching for WAITING something will find entries containing the word something with the to-do status WAITING; searching for waiting something will search for waiting as a normal word, finding entries containing waiting or something without regard for their to-do status.
  • Multiple to-do keywords are matched with a boolean OR. Since each entry can only have one to-do status, searching for multiple to-do keywords would logically exclude all entries and therefore match nothing. To make it more useful (and less error-prone), multiple to-do keywords are matched with OR. For example, searching for TODO DONE something will find entries containing the word something that are marked either TODO or DONE.
• In occur results:
  • Show a header for each source buffer.
  • Show separators between each result.
  • Fold search results individually, instead of folding the “fake” subtrees which are effectively created by inserting Org entries into the results buffer.
  • Fold every entry in a source when the org-cycle key is pressed with point on a source header.

Fixes

• Use (org-agenda-files) function instead of the org-agenda-files variable in helm-org-rifle-agenda-files and helm-org-rifle-occur-agenda-files. This way, directory entries are expanded, and other Org customizations are respected. Thanks to Joonatan O’Rourke.
• Fix tag negation. Negated tags were not actually being negated, and the test was wrong. Oops.
• When helm-org-rifle-show-path is on, match against path elements. This only happens when that option is enabled, so WYSIWYG: if you can’t see the paths, they aren’t matched against. Note, however, that they are always checked for excluded patterns, even if the option is off. Thanks to George Singer.

Internal

• Rewrote input parsing.
• Factored out entry-matching.

1.4.2

• Fix node positions in occur commands. This prevented the user from jumping to the position of results in source buffers.

1.4.1

• In occur commands, get only entries, not entire subtrees. This is the intended behavior and makes it consistent with the non occur commands. (It’s so easy to forget that org-get-entry gets “the entry text, after heading, entire subtree.”)

1.4.0

Additions

• New commands helm-org-rifle-occur, helm-org-rifle-occur-current-buffer, helm-org-rifle-occur-files, helm-org-rifle-occur-agenda-files, helm-org-rifle-occur-org-directories, and helm-org-rifle-occur-org-directory, which display results in an occur-like, persistent buffer. These are handy when you aren’t as certain of what you’re looking for and you want to keep the results visible while looking at each result’s source buffer. When you click on or press RET on a result, the source buffer will be popped to alongside the results buffer, and the node will be revealed, cycled to, and the point moved to the same place. These commands do not actually use Helm at all, so maybe they should be renamed to simply org-rifle…?
• Results can now be sorted by either the order that nodes appear in their buffers (the default) or the latest timestamp in each node. To change the sort order, run a command with a universal prefix (C-u). If helm-org-rifle-sort-order-persist is set, the sort order remains after setting it, and the default sort order may also be customized.
• The Helm commands now support multiple selection. If multiple entries are selected, they will be displayed in-full in a read-only, occur-style buffer, like the helm-org-rifle-occur commands do.
• All of the results in a Helm buffer can be saved to a helm-org-rifle-occur buffer by pressing C-s in the Helm buffer.
• New option helm-org-rifle-always-show-entry-contents-chars to show some entry contents when the query only matches the heading or metadata, defaults to 50 characters.
• New option helm-org-rifle-show-full-contents which displays each result’s entire contents rather than just the context around each matching word. This is off by default, but the helm-org-rifle-occur commands activate it for their results, and you might use it selectively by calling one of the rifle commands inside a let that sets this variable.
• New options helm-org-rifle-heading-contents-separator and helm-org-rifle-multiline. These may be useful to compact the results display when defining custom commands.
• The functions helm-org-rifle-files and helm-org-rifle-directories (and their new -occur counterparts) now accept either a string or a list of strings.
• Tests have been added to prevent future breakage. Whew.

Fixes

• Tag order is now irrelevant. Previously, searching for :tag1:tag2: would not show a heading tagged :tag2:tag1:. It could be worked around by searching for :tag1: :tag2:, but that was non-obvious and counter-intuitive. This could be considered a bug-fix, but the change is significant enough that it belongs in a feature update to get more testing.
• Negations are now matched against each node’s entire outline path and against buffer names. Previously they were only matched against the node’s own heading, not any of its parents’ headings.
• Org links are “unlinkified” when showing match context, preventing Org syntax characters from cluttering the results.
• The display of full outline paths in Helm buffers is tidier.
• Fixed bug that may have prevented the first or last heading in a file from being matched.
• Fixed bug that prevented negated tags (e.g. !:tag1:) from being negated properly.
• Fixed very minor bug in customization settings that caused the setting for helm-org-rifle-show-entry-function to display the wrong function name, even though it worked correctly.

1.3.0

• Add helm-org-rifle-agenda-files command.
• Add helm-org-rifle-org-directory command.

1.2.0

• New commands helm-org-rifle-files and helm-org-rifle-directories to search through files that may or may not already be open.
  • New option helm-org-rifle-directories-filename-regexp to control what files are searched with helm-org-rifle-directories (e.g. including .org_archive files).
  • New option helm-org-rifle-close-unopened-file-buffers to control whether new buffers opened for searching remain open. Leaving them open will make subsequent searches faster, but most users will probably prefer to not have their buffer list cluttered, so this is enabled by default.
  • New option helm-org-rifle-directories-recursive to control whether helm-org-rifle-directories recursively scans subdirectories, enabled by default. When helm-org-rifle-directories is called with a prefix, this option is inverted.
  • Add dependency on f.el.
• When helm-org-rifle-show-path is enabled, replace Org links in headings with their descriptions. This prevents org-format-outline-path from truncating the links, making them useless for reading.
• Show results in the order they appear in the Org file (they were shown in reverse order).

1.1.1

• Fix helm-org-rifle-show-path. A bug caused no results to be displayed for entries below the top level.

1.1

• Restore context display. This was accidentally broken when adding the negation feature, before the tagging of 1.0.0, so it’s like a new feature.
• Turn on the show-tags feature and remove the option to disable it. It fixes a bug, and I don’t think anyone would want to turn it off anyway. It was off by default before, which might mean that users who didn’t turn it on were getting incorrect results by default. Oops.
• Bind <C-return> to open entries in indirect buffers with org-tree-to-indirect-buffer. This is super-duper handy, and seems to be an under-appreciated Org feature. Try indirect buffers, today!
• Add option to customize the ellipses and use comment face by default.
• Use dash.el for some things.
• Set helm-input-idle-delay to prevent flickering as the user types, customizable through helm-org-rifle-input-idle-delay.

1.0.1

• Handle Org in-buffer settings (#5). Thanks to @jonmoore.

Credits

• This package is inspired by org-search-goto-ml by Tom. Its unofficial-official home is on EmacsWiki, but I’ve mirrored it on GitHub with some small fixes. It’s a really great package, and the only thing that could make it better is to make it work with Helm. To avoid confusion, this package has a completely different name.
• Thanks to Thierry Volpiatto for doing such an amazing job with Helm. Without him, this would not be possible.
• Thanks to Jack, aka /u/washy99999 for great feedback and suggestions.
• Thanks to Jorgen Schäfer for Buttercup, which makes testing simple.
• Thanks to Joonatan O’Rourke for suggesting the (org-agenda-files) function.
• Thanks to @zeltak for helping to test many changes.

Development

I can’t recommend Outorg enough. If you edit source code and use Emacs, check it out!

Testing

Running tests

Run make test from the main project directory. Testing requires Cask and Buttercup. It’s helpful to put this in a git pre-push hook.

Adding test data

When adding new data to test/data.org, it should go at the bottom to avoid breaking the test data embedded in test/helm-org-rifle-test.el, which includes buffer positions. Adding data anywhere else in the file will invalidate those. However, if necessary, the helm-org-rifle--test-update-result function can be used to easily update test data after making such changes.

Test interactive functionality

Future testing should include testing interactive functionality, like Helm commands. This will probably be easier with ecukes and espuds. It would be nice to use assess, but it requires Emacs 25 by way of its dependency on m-buffer.

Update: Ryan C. Thompson graciously updated his new with-simulated-input library to support interactive use with simulated idle timers, which seems to work perfectly for this!

Bugs

Multiple tags negated in a single token

Negating multiple tags in a single token (e.g. !:negatedtag1:negatedtag2:) doesn’t work properly. Fairly minor issue, but will need fixing.

Ideas

[#B] Testing

After reading about Emacs testing packages, it looks like the best way to test this package is with some combination of Assess, Buttercup, Ecukes, ERT, and Espuds. Espuds’s steps should help testing interactive things, like Helm (although this will still be difficult), and Buttercup should make unit testing easier, and Assess should help with everything. Buttercup is intended as an alternative to ERT, but ERT might be useful too.

[#B] jump-to-next/prev-match command for occur buffers

It would be handy to have a built-in command to jump to the next match instance in the occur buffers, maybe something like M-g n. Suggested by washy99999.

[#B] Phrase matching

Don’t know how I overlooked this for this long. Shouldn’t be too hard to implement searching for phrases in quotes. Should probably match multiple spaces (but probably not newlines or tabs) between words; wouldn’t want an accidental double-spacebar press in the searched file to prevent a match.

MAYBE [#C] follow-mode

helm-follow-mode can be activated from within Helm already with C-c C-f, and on an individual-item basis with C-j, and anyone can define a custom command to set it themselves, but it might be worth having an argument to enable it too.

Use prefix arg to toggle full-path mode

Along the lines of:

(defun my/helm-org-rifle-with-full-paths ()
    (interactive)
    (let ((helm-org-rifle-show-path (not helm-org-rifle-show-path))
      (helm-org-rifle))))

Make Helm highlight all matches

Helm only seems to highlight the first match in each candidate.

MAYBE Timestamp searching

It would be interesting to be able to search for timestamps, e.g. for nodes timestamped on a certain day, or within a certain date range. Might be a bit slow, because it would require comparing every timestamp in every result, but if it’s what you need, then it would probably be usable and worth it.

MAYBE git grep support

By setting a custom xfuncname for a git repo containing org files (see man 5 gitattributes), git diff will display the org heading as the hunk header in its output. Then running git grep -W shows entire org entries that match. And git grep has boolean operators. And git grep is very fast. Plug these into an async Helm source and boom, lightning-fast searching of org files, even if they aren’t open in an Emacs buffer. Well, as long as the files are in a git repo–but you are storing your org files in a git repo, aren’t you? =)

MAYBE sift support

Sift sounds like it might be a perfect solution here, since it supports multi-line matching, replacements, etc.

MAYBE ripgrep support

ripgrep might also be useful, although I don’t think it supports multi-line yet.

MAYBE async matching

It might be interesting to use emacs-async to do matching in files that aren’t already open in the current Emacs process. I’m not sure if it would be worth it, because even if it were faster in some cases for unopened files, it wouldn’t be faster compared to searching already opened files. And even though loading large files can be slow, once they are opened, the price is paid, and searching is faster; doing it in external Emacs processes would be slow every time, not just the first.

But there might be some cases where it would be helpful. It might be possible to do it without loading the files in Org in the other processes, and it might be helpful to do all the searching in one process instead of one for each file. For the case of opening many small files that don’t need to be frequently accessed, that the user doesn’t want to keep open, doing it in another process might actually be good.

But it might also be complicated to keep the search process open while the user is changing the query; and without doing that, a new search process would be started every time the user changed the query, which would mean loading the files all over again. So I’m not sure this idea would be generally useful.

UNDERWAY Non-substring matching

Currently matches are made against substrings, like most other commands in Helm. However, this might not always lead to the best results. For example, if someone were searching for “Sol”, referring to the sun, he probably wouldn’t want to match “solution” or “solvent” or “soliloquy”. But if someone were trying to dig up a note he made a while back about apple pie, did he write about “an apple pie” or “some apple pies”? Dessert hangs in the balance!

To solve this, matches could be made against word, punctuation, or symbol boundaries. However, this is less “Helm-like,” and it might not be what most users expect. So it would be good to make this a configurable default. A prefix could override the default, and/or it could be toggleable from within a Helm session.

Collapse overlapping context strings

Right now, if more than one term appears in the same range, parts of that range will show up more than once in the context. Not a big deal, but should be fixable.

MAYBE Further profiling

helm-org-rifle-get-candidates-in-buffer might be able to be optimized more with elp. But the “low-hanging fruit” is probably gone, and performance seems good.

MAYBE Regexp matching

It would be nice to have a regexp mode…maybe.

MAYBE Match limit

org-search-goto had a match limit. I removed it to simplify things, but it might still be useful, depending on how big one’s org files are. However, performance seems good now, so this probably isn’t needed.

[#C] always-show-entry-text truncates by 3 too many

s-truncate truncates and adds ..., which means that the chosen length of entry text gets reduced by 3. Could fix this by using a setter for the defcustom that adds 3.

License

GPLv3