nnir notmuch search on specific groups?

[Andreas Goesele]

Hi,

with quite some effort I was able to get nnir to work together with the
notmuch search engine. But then, differently from what I expected from
the documentation in the manual, I discovered that obviously with the
notmuch engine the searches are not restricted to the groups selected.

Looking at the code I found this error message, commented out:

;; (when group
;; (error "The notmuch backend cannot search specific groups")

I guess this indeed means what it says, but as it is commented out and
not mentioned in the manual, I would like to get some confirmation:

Is it possible or not to search specific groups with the notmuch
backend?

Thanks a lot

Andreas

[Eric Abrahamsen]

Post by Andreas Goesele
Hi,
with quite some effort I was able to get nnir to work together with the
notmuch search engine. But then, differently from what I expected from
the documentation in the manual, I discovered that obviously with the
notmuch engine the searches are not restricted to the groups selected.
;; (when group
;; (error "The notmuch backend cannot search specific groups")
I guess this indeed means what it says, but as it is commented out and
Is it possible or not to search specific groups with the notmuch
backend?

People have been struggling with this for a while, here's the approach
that looks most likely to solve your problem:

https://lists.gnu.org/archive/html/info-gnus-english/2016-09/msg00065.html

What mail backend are you using for this, by the way?

Eric

[Adam Sjøgren]

Post by Eric Abrahamsen
People have been struggling with this for a while, here's the approach
https://lists.gnu.org/archive/html/info-gnus-english/2016-09/msg00065.html

That doesn't seem to have any hints on how to search only the email in a
specific group?

The nnir-integration built into Gnus, i.e. setting (nnir-search-engine
notmuch) on my local mail back end of choice (nnml), works for me, but
it doesn't allow searching in a specific group.

But maybe notmuch has some search term that can be used.

Yes, there is a path:<directory-path> term, which could be useful. There
is also a folder:<maildir-folder>, if you use maildir.

Maybe nnir could figure out the paths of selected groups and include
them in the search string for notmuch?


Best regards,

Adam

[Eric Abrahamsen]

Post by Adam Sjøgren

Post by Eric Abrahamsen
People have been struggling with this for a while, here's the approach
https://lists.gnu.org/archive/html/info-gnus-english/2016-09/msg00065.html

That doesn't seem to have any hints on how to search only the email in a
specific group?

You're right, I misremembered the point of that conversation.

Post by Adam Sjøgren
The nnir-integration built into Gnus, i.e. setting (nnir-search-engine
notmuch) on my local mail back end of choice (nnml), works for me, but
it doesn't allow searching in a specific group.
But maybe notmuch has some search term that can be used.
Yes, there is a path:<directory-path> term, which could be useful. There
is also a folder:<maildir-folder>, if you use maildir.
Maybe nnir could figure out the paths of selected groups and include
them in the search string for notmuch?

I went back and looked at my old work on gnus-search (which will land
someday... sniff), and it looks like I bailed on making this work there,
too. I vaguely remember that I couldn't even make it work properly on
the command line, couldn't figure out why "folder" and "path" returned
the results they did.

But `nnir-run-notmuch' filters its results same as all the other search
routines, so I'm not sure why we can't do the filtering on group name,
after the fact. It has this:

;; maybe limit results to matching groups.
(when (or (not groupspec)
(string-match groupspec dirnam))
(nnir-add-result dirnam artno "" prefix server artlist)))))

That should be usable for filtering, right?

I have the feeling that I've gone over all this before, then forgotten
what the conclusion was...

Eric

[Adam Sjøgren]

I vaguely remember that I couldn't even make it work properly on the
command line, couldn't figure out why "folder" and "path" returned the
results they did.

path:{folder} works for me, using notmuch on the command line.

My email is in nnml format in various folders in ~/Mail/{folder}, and my
.notmuch-config has:

[database]
path=/home/asjo/Mail

Searching for "Adam" gives me a lot of results:

***@tullinup:~$ time notmuch search Adam | wc -l
32911

real 0m6.574s
user 0m5.016s
sys 0m1.577s

If I limit the search using path:, I get less, so it looks like it
works:

***@tullinup:~$ time notmuch search "Adam path:cron" | wc -l
507

real 0m0.148s
user 0m0.128s
sys 0m0.024s

My notmuch index has ~119K entries:

$ time notmuch count
119018

But `nnir-run-notmuch' filters its results same as all the other search
routines, so I'm not sure why we can't do the filtering on group name,
;; maybe limit results to matching groups.
(when (or (not groupspec)
(string-match groupspec dirnam))
(nnir-add-result dirnam artno "" prefix server artlist)))))
That should be usable for filtering, right?

Yeah! I guess it could matter when the results are excluded,
performance-wise, but in practise, I would guess it would be "fast
enough"™ to do it post hoc.

I have the feeling that I've gone over all this before, then forgotten
what the conclusion was...

I know that feeling all too well... :-)


Best regards,

Adam

[Eric Abrahamsen]

Post by Adam Sjøgren

I vaguely remember that I couldn't even make it work properly on the
command line, couldn't figure out why "folder" and "path" returned the
results they did.

path:{folder} works for me, using notmuch on the command line.
My email is in nnml format in various folders in ~/Mail/{folder}, and my
[database]
path=/home/asjo/Mail
32911
real 0m6.574s
user 0m5.016s
sys 0m1.577s
If I limit the search using path:, I get less, so it looks like it
507
real 0m0.148s
user 0m0.128s
sys 0m0.024s

Beats me what my problem was, then! But no matter.

Post by Adam Sjøgren

But `nnir-run-notmuch' filters its results same as all the other search
routines, so I'm not sure why we can't do the filtering on group name,
;; maybe limit results to matching groups.
(when (or (not groupspec)
(string-match groupspec dirnam))
(nnir-add-result dirnam artno "" prefix server artlist)))))
That should be usable for filtering, right?

Yeah! I guess it could matter when the results are excluded,
performance-wise, but in practise, I would guess it would be "fast
enough"™ to do it post hoc.

Sure. It would be preferable to do it up front though!

This whole thing looks a little weird. The "groupspec" filter criteria
above is supposed to come from a "notmuch-group" query parameter (ie
nnir would promppt you for it), but as that parameter doesn't appear in
`nnir-engines', the prompt never happens, and we never get a "groupspec"
at all.

I don't see why we couldn't just use the GROUP argument. I think it
should be optional, because we can't always be sure the user doesn't
have some odd name transformation stuff going on, but for instance in
your case with nnml it seems like it ought to work fine.

Would you try this and see if it does what you'd expect?


(defun nnir-run-notmuch (query server &optional groups)
"Run QUERY against notmuch.
Returns a vector of (group name, file name) pairs (also vectors,
actually). If GROUPS is a list of group names, use them to
construct path: search terms."

(save-excursion
(let* ((qstring (cdr (assq 'query query)))
(prefix (nnir-read-server-parm 'nnir-notmuch-remove-prefix server))
artlist
(article-pattern (if (string-match "\\`nnmaildir:"
(gnus-group-server server))
":[0-9]+"
"^[0-9]+$"))
(groups (mapcar #'gnus-group-short-name groups))
(pathquery (when groups
(mapconcat (lambda (g)
(format " path:%s" g))
groups " or")))
artno dirnam filenam)

(when (equal "" qstring)
(error "notmuch: You didn't enter anything"))

(set-buffer (get-buffer-create nnir-tmp-buffer))
(erase-buffer)

(if groups
(message "Doing notmuch query %s on %s..."
qstring (mapconcat #'identity groups " "))
(message "Doing notmuch query %s..." qstring))

(when groups
(setq qstring (concat qstring pathquery)))

(let* ((cp-list `( ,nnir-notmuch-program
nil ; input from /dev/null
t ; output
nil ; don't redisplay
"search"
"--format=text"
"--output=files"
,@(nnir-read-server-parm 'nnir-notmuch-additional-switches server)
,qstring ; the query, in notmuch format
))
(exitstatus
(progn
(message "%s args: %s" nnir-notmuch-program
(mapconcat #'identity (nthcdr 4 cp-list) " ")) ;; ???
(apply #'call-process cp-list))))
(unless (or (null exitstatus)
(zerop exitstatus))
(nnheader-report 'nnir "Couldn't run notmuch: %s" exitstatus)
;; notmuch failure reason is in this buffer, show it if
;; the user wants it.
(when (> gnus-verbose 6)
(display-buffer nnir-tmp-buffer))))

;; The results are output in the format of:
;; absolute-path-name
(goto-char (point-min))
(while (not (eobp))
(setq filenam (buffer-substring-no-properties (line-beginning-position)
(line-end-position))
artno (file-name-nondirectory filenam)
dirnam (file-name-directory filenam))
(forward-line 1)

;; don't match directories
(when (string-match article-pattern artno)
(when (not (null dirnam))

(nnir-add-result dirnam artno "" prefix server artlist))))

(message "Massaging notmuch output...done")

artlist)))

[Andreas Goesele]

Post by Eric Abrahamsen
Would you try this and see if it does what you'd expect?

That's great: It works like charme with folders directly in the Mail
directory. For subfolders it only would be necessary to replace the "."
by a slash "/". So instead of searching "mail.subfolder" (as subfolders
are shown in the group buffer) it should search "mail/subfolder".

Post by Eric Abrahamsen
(defun nnir-run-notmuch (query server &optional groups)
"Run QUERY against notmuch.
Returns a vector of (group name, file name) pairs (also vectors,
actually). If GROUPS is a list of group names, use them to
construct path: search terms."
(save-excursion
(let* ((qstring (cdr (assq 'query query)))
(prefix (nnir-read-server-parm 'nnir-notmuch-remove-prefix server))
artlist
(article-pattern (if (string-match "\\`nnmaildir:"
(gnus-group-server server))
":[0-9]+"
"^[0-9]+$"))
(groups (mapcar #'gnus-group-short-name groups))
(pathquery (when groups
(mapconcat (lambda (g)
(format " path:%s" g))
groups " or")))
artno dirnam filenam)
(when (equal "" qstring)
(error "notmuch: You didn't enter anything"))
(set-buffer (get-buffer-create nnir-tmp-buffer))
(erase-buffer)
(if groups
(message "Doing notmuch query %s on %s..."
qstring (mapconcat #'identity groups " "))
(message "Doing notmuch query %s..." qstring))
(when groups
(setq qstring (concat qstring pathquery)))
(let* ((cp-list `( ,nnir-notmuch-program
nil ; input from /dev/null
t ; output
nil ; don't redisplay
"search"
"--format=text"
"--output=files"
,qstring ; the query, in notmuch format
))
(exitstatus
(progn
(message "%s args: %s" nnir-notmuch-program
(mapconcat #'identity (nthcdr 4 cp-list) " ")) ;; ???
(apply #'call-process cp-list))))
(unless (or (null exitstatus)
(zerop exitstatus))
(nnheader-report 'nnir "Couldn't run notmuch: %s" exitstatus)
;; notmuch failure reason is in this buffer, show it if
;; the user wants it.
(when (> gnus-verbose 6)
(display-buffer nnir-tmp-buffer))))
;; absolute-path-name
(goto-char (point-min))
(while (not (eobp))
(setq filenam (buffer-substring-no-properties (line-beginning-position)
(line-end-position))
artno (file-name-nondirectory filenam)
dirnam (file-name-directory filenam))
(forward-line 1)
;; don't match directories
(when (string-match article-pattern artno)
(when (not (null dirnam))
(nnir-add-result dirnam artno "" prefix server artlist))))
(message "Massaging notmuch output...done")
artlist)))

[Eric Abrahamsen]

Post by Andreas Goesele

Post by Eric Abrahamsen
Would you try this and see if it does what you'd expect?

That's great: It works like charme with folders directly in the Mail
directory.

Great!

Post by Andreas Goesele
For subfolders it only would be necessary to replace the "." by a
slash "/". So instead of searching "mail.subfolder" (as subfolders are
shown in the group buffer) it should search "mail/subfolder".

Well that's where I'm a little hesitant. I'm not really familiar with
how this path transformation stuff usually works, and don't want to

I suppose we could add an option that's either nil (don't use groups and
paths at all), t (use untransformed paths), or a function that munges
the group names appropriately. Then it would just be a matter of finding
a reasonable default behavior.

Eric

[Andreas Goesele]

Post by Eric Abrahamsen
I went back and looked at my old work on gnus-search (which will land
someday... sniff), and it looks like I bailed on making this work there,
too. I vaguely remember that I couldn't even make it work properly on
the command line, couldn't figure out why "folder" and "path" returned
the results they did.

I find the results using "path" rather consistent. The only - pleasant -
surprise was "that terms with a common prefix will be implicitly combined
with OR".

Post by Eric Abrahamsen
But `nnir-run-notmuch' filters its results same as all the other search
routines, so I'm not sure why we can't do the filtering on group name,
after the fact.

This would be a real improvement.

But as notmuch isn't *very* fast with a huge amount of hits I would
prefer a solution which restricts the search beforehand. (Unfortunately I
can't contribute.)

Andreas

[Andreas Goesele]

Post by Adam Sjøgren

Post by Eric Abrahamsen
People have been struggling with this for a while, here's the approach
https://lists.gnu.org/archive/html/info-gnus-english/2016-09/msg00065.html

That doesn't seem to have any hints on how to search only the email in a
specific group?
The nnir-integration built into Gnus, i.e. setting (nnir-search-engine
notmuch) on my local mail back end of choice (nnml), works for me, but
it doesn't allow searching in a specific group.

I finally got the built in nnir-notmuch integration to work, and I'm
using nnml too.

Post by Adam Sjøgren
But maybe notmuch has some search term that can be used.
Yes, there is a path:<directory-path> term, which could be useful. There
is also a folder:<maildir-folder>, if you use maildir.
Maybe nnir could figure out the paths of selected groups and include
them in the search string for notmuch?

Yes, I'm using this path:<directory-path> search term. But the
possibility to search the group on the current line or the groups marked
would be very handy.

Andreas