summaryrefslogtreecommitdiff
path: root/worker
AgeCommit message (Collapse)Author
2022-07-24seqmap: sync seqNum to uid after expungeTim Culverhouse
This patch updates the seqNums after an Expunge operation. When an expunge operation occurs, the seqNum of the deleted message is reported. The Imap spec [0] states that an immediate decrement of all seqnums greater than the deleted occurs, even before the next reporting of an expunge update. [0]: https://datatracker.ietf.org/doc/html/rfc3501#section-7.4.1 Fixes: https://todo.sr.ht/~rjarry/aerc/61 Signed-off-by: Tim Culverhouse <tim@timculverhouse.com> Signed-off-by: Robin Jarry <robin@jarry.cc>
2022-07-23logging: use level-based logger functionsRobin Jarry
Do not pass logger objects around anymore. Shuffle some messages to make them consistent with the new logging API. Avoid using %v when a more specific verb exists for the argument types. The loggers are completely disabled (i.e. Sprintf is not even called) by default. They are only enabled when redirecting stdout to a file. Signed-off-by: Robin Jarry <robin@jarry.cc> Acked-by: Moritz Poldrack <moritz@poldrack.dev>
2022-07-23imap: disable debug loggerRobin Jarry
This causes all raw email bodies to be dumped along with actual debugging messages. I don't believe we neither need nor want such a thing. Signed-off-by: Robin Jarry <robin@jarry.cc> Acked-by: Moritz Poldrack <moritz@poldrack.dev>
2022-07-23imap: fix error when server returns a message without body sectionRobin Jarry
When opening unread emails from certain people (I won't name any names, sorry), an annoying error message is displayed on the status line: could not get section &imap.BodySectionName{BodyPartName: imap.BodyPartName{Specifier:"", Path:[]int(nil), Fields:[]string(nil), NotFields:false}, Peek:false, Partial:[]int(nil), value} This does not occur for already read messages. This issue is similar to the one that was fixed in commit 8ed95b0d2ad2 ("imap: avoid crash when replying to unread message"). This happens because the flags are updated in the callback that receives the message itself. It causes the flag update to arrive in the same channel/request. Ignore the messages that have an empty body (i.e. only containing flag updates). This is inherently racy but there seems no way to get rid of these extra messages. Signed-off-by: Robin Jarry <robin@jarry.cc> Acked-by: Koni Marti <koni.marti@gmail.com>
2022-07-14account: import mbox file to a folderKoni Marti
Append all messages from an mbox file to the selected folder with the import-mbox command. User confirmation is required when the folder already contains messages. A failed append will be retried a few times. If a backend timeout occurs, the entire import is stopped to prevent a hang. Signed-off-by: Koni Marti <koni.marti@gmail.com> Acked-by: Robin Jarry <robin@jarry.cc>
2022-07-14account: export folder to mbox fileKoni Marti
Export all message in the current folder to an mbox file. If an error occurs during the export, aerc retries a few times before giving up to prevent a hang. Signed-off-by: Koni Marti <koni.marti@gmail.com> Acked-by: Robin Jarry <robin@jarry.cc>
2022-07-14mbox: implement an mbox backend workerKoni Marti
Implement an mbox backend worker. Worker can be used for testing and development by mocking a backend for the message store. Worker does not modify the actual mbox file on disk; all operations are performed in memory. To use the mbox backend, create an mbox account in the accounts.conf where the source uses the "mbox://" scheme, such as source = mbox://~/mbox/ or source = mbox://~/mbox/file.mbox If the mbox source points to a directory, all files in this directory with the .mbox suffix will be opened as folders. If an outgoing smtp server is defined for the mbox account, replies can be sent to emails that are stored in the mbox file. Signed-off-by: Koni Marti <koni.marti@gmail.com> Acked-by: Robin Jarry <robin@jarry.cc>
2022-07-10threading: enable filtering of server-side threadsTim Culverhouse
This patch enables the filtering of a threaded view which uses server-built threads. Filtering is done server-side, in order to preserve the use of server-built threads. In adding this feature, the filtering of notmuch folders was brought up to feature parity with the other workers. The filters function the same (ie: they can be stacked). The notmuch filters, however, still use notmuch syntax for the filtering. Signed-off-by: Tim Culverhouse <tim@timculverhouse.com> Acked-by: Robin Jarry <robin@jarry.cc>
2022-07-10threading: add backend capabilities to workersTim Culverhouse
This patch provides a method to report backend capabilities to the UI. The intial capabilities included in the report are Sort and Thread. Having these available to the UI enables the client to better handle server side threading. Signed-off-by: Koni Marti <koni.marti@gmail.com> Signed-off-by: Tim Culverhouse <tim@timculverhouse.com> Acked-by: Robin Jarry <robin@jarry.cc>
2022-07-10worker/maildir: implement Maildir++ supportAdnan Maolood
See https://www.courier-mta.org/maildir.html#maildircontents Signed-off-by: Adnan Maolood <me@adnano.co> Acked-by: Koni Marti <koni.marti@gmail.com>
2022-07-02notmuch: fix server-side threadsTim Culverhouse
Notmuch server-side threading added messages within a thread that didn't match the query into the uidstore. By doing so, several UI issues presented: * All "hidden" messages displayed at the bottom of the msglist * Selected messages wouldn't open properly This patch stops these messages from being put into the message store, thereby resolving the UI issues Signed-off-by: Tim Culverhouse <tim@timculverhouse.com> Acked-by: Koni Marti <koni.marti@gmail.com>
2022-06-26lint: fix composite literal using unkeyed fieldsMoritz Poldrack
Fix the following go vet error: # git.sr.ht/~rjarry/aerc/worker/notmuch worker/notmuch/worker.go:86:19: git.sr.ht/~rjarry/aerc/worker/types.Done composite literal uses unkeyed fields Signed-off-by: Moritz Poldrack <git@moritz.sh> Acked-by: Robin Jarry <robin@jarry.cc> Acked-by: Tim Culverhouse <tim@timculverhouse.com>
2022-06-24imap: fix data race on seqMap arrayRobin Jarry
There are concurrent threads that are accessing and modifying IMAPWorker.seqMap (the mapping of sequence numbers to message UIDs). This can lead to crashes when trying to add and remove a message ID. panic: runtime error: index out of range [391] with length 390 goroutine 1834 [running]: git.sr.ht/~rjarry/aerc/logging.PanicHandler() logging/panic-logger.go:47 +0x6de panic({0xa41760, 0xc0019b3290}) /usr/lib/golang/src/runtime/panic.go:838 +0x207 git.sr.ht/~rjarry/aerc/worker/imap.(*IMAPWorker).handleFetchMessages.func1() worker/imap/fetch.go:214 +0x185 created by git.sr.ht/~rjarry/aerc/worker/imap.(*IMAPWorker).handleFetchMessages worker/imap/fetch.go:209 +0x12b Use a map which makes more sense than a simple array for random access operations. Also, it allows better typing for the key values. Protect the map with a mutex. Add internal API to access the map. Add basic unit tests to ensure that concurrent access works. Fixes: https://todo.sr.ht/~rjarry/aerc/49 Signed-off-by: Robin Jarry <robin@jarry.cc> Acked-by: Moritz Poldrack <moritz@poldrack.dev>
2022-06-22imap: add option to cache headersTim Culverhouse
Add option to cache headers for imap accounts. Cache db is located at $XDG_CACHE_DIR/aerc/{account name}. The cache is cleaned of stale entries when aerc is first opened. Two new account level configuration options are introduced: * cache-headers (Default: false) * cache-max-age (Default: 30 days (720 hours)) The change in worker/imap/open.go is to set the selected directory. This is required to access the UIDVALIDITY field, which is used in combination with the message ID to form the key for use in the cache db. The key structure is: "header.{UIDVALIDITY}.{UID}" Where reasonable, cache does not stop aerc from running. In general, if there is an error in the cache, aerc should continue working as usual. Errors are either displayed to the user or logged. All messages are stored without flags, and when retrieved have the flags set to SEEN. This is to prevent UI flashes. A new method to FetchMessageFlags is introduced to update flags of cached headers. This is done asynchronously, and the user will see their messages appear and then any flags updated. The message will initially show as SEEN, but will update to unread. I considered updating the cache with the last-known flag state, however it seems prudent to spare the R/W cycle and assume that - eventually - all messages will end up read, and if it isn't the update will occur rather quickly. Note that leveldb puts a lock on the database, preventing multiple instances of aerc from accessing the cache at the same time. Much of this work is based on previous efforts by Vladimír Magyar. Implements: https://todo.sr.ht/~rjarry/aerc/2 Thanks: Vladimír Magyar <vladimir@mgyar.me> Signed-off-by: Tim Culverhouse <tim@timculverhouse.com> Tested-by: inwit <inwit@sindominio.net> Reviewed-by: Koni Marti <koni.marti@gmail.com> Acked-by: Robin Jarry <robin@jarry.cc>
2022-06-14maildir: fix dirinfo.Unseen and Exists countingTim Culverhouse
The maildir worker was adding Recent messages to the counts of Unseen and Exists, however these messages were already included in those counts. Signed-off-by: Tim Culverhouse <tim@timculverhouse.com> Acked-by: Robin Jarry <robin@jarry.cc>
2022-06-07parse: fix content-type parsing errorKoni Marti
If an error occurs when parsing the content-type, check if the content-type is quoted; if so, remove quotes. If this is not the case, then return a text/plain content-type as a sane fallback option, so that the message can be at least viewed in plaintext. Fixes: https://todo.sr.ht/~rjarry/aerc/44 Signed-off-by: Koni Marti <koni.marti@gmail.com> Acked-by: Robin Jarry <robin@jarry.cc>
2022-06-07dirlist: update RUE counts for imap/maildir on move|copy|delete|archiveTim Culverhouse
When moving/copying/deleting/archiving a message in imap, the RUE counts displayed in the dirlist would not update properly. Maildir has (had) an implementation that recounts the entire directory and updates the DirectoryInfo after one of these actions. This patch implements a more efficient method of updating, and also enables it to apply to IMAP without any additional requests. Upon completion of the action, the counts are manually updated with the count of messages in the action and recent and/or unseen states of those messages. This is more efficient for maildir, because we aren't counting everything in the store. For IMAP, we get the updates for free because we are only performing the update after confirmation from the server that the action has happened. Signed-off-by: Tim Culverhouse <tim@timculverhouse.com> Acked-by: Robin Jarry <robin@jarry.cc>
2022-05-31feat: add background mail polling option for all workersTim Culverhouse
Check for new mail (recent, unseen, exists counts) with an external command, or for imap with the STATUS command, at start or on reconnection and every X time duration IMAP: The selected folder is skipped, per specification. Additional config options are included for including/excluding folders explicitly. Maildir/Notmuch: An external command will be run in the background to check for new mail. An optional timeout can be used with maildir/notmuch. Default is 10s New account options: check-mail check-mail-cmd (maildir/notmuch only) check-mail-timeout (maildir/notmuch only), default 10s check-mail-include (IMAP only) check-mail-exclude (IMAP only) If unset, or set less than or equal to 0, check-mail will be ignored Signed-off-by: Tim Culverhouse <tim@timculverhouse.com> Tested-by: Moritz Poldrack <moritz@poldrack.dev> Acked-by: Robin Jarry <robin@jarry.cc>
2022-05-04imap: add timeout to tcp connect functionsKoni Marti
Extract the tcp connection details and timeout the tcp connect functions (net.ResolveTCPAddr and net.DialTCP). If timed out, ensure that the connection is properly closed. Signed-off-by: Koni Marti <koni.marti@gmail.com> Acked-by: Robin Jarry <robin@jarry.cc>
2022-05-04imap: add debouncer to the idlerKoni Marti
Add a debouncer to the idle mode. Avoid unnecessary idling when another job arrives within a certain time frame. For example, the ui sends three messages to the worker at the same time when we open a message (FlagMessage, FetchMessageBodyPart, and the FetchMessageHeaders). The debouncer prevents the unnecessary entering and leaving of the idle mode between those messages. Signed-off-by: Koni Marti <koni.marti@gmail.com> Acked-by: Robin Jarry <robin@jarry.cc>
2022-05-04imap: monitor the logout channel with an observerKoni Marti
Untangle the observer functionality from the message handling routine. Observe the imap client's logout channel and trigger a connection error when necessary to start the reconnect cycle. Signed-off-by: Koni Marti <koni.marti@gmail.com> Acked-by: Robin Jarry <robin@jarry.cc>
2022-05-04imap: manage idle mode with an idlerKoni Marti
Untangle the idle functionality from the message handling routine. Wait for the idle mode to properly exit every time to ensure a consistent imap state. Timeout when hanging in idle mode and inform the ui. Signed-off-by: Koni Marti <koni.marti@gmail.com> Acked-by: Robin Jarry <robin@jarry.cc>
2022-05-04imap: extract imap config and configure handlingKoni Marti
Extract the imap config and move the configure part out of the message handler. Signed-off-by: Koni Marti <koni.marti@gmail.com> Acked-by: Robin Jarry <robin@jarry.cc>
2022-04-29imap: fix out-of-range panic for imap updatesKoni Marti
Check slice bounds before using it for the message and expunge updates. Log the error but ignore the affected updates. Link: https://lists.sr.ht/~rjarry/aerc-devel/%3CCJEHBFFUI11T.1AYGOMVGZ87ZS%40rek2system%3E Reported-by: ReK2 <rek2@hispagatos.org> Signed-off-by: Koni Marti <koni.marti@gmail.com> Acked-by: Robin Jarry <robin@jarry.cc>
2022-04-28imap: avoid crash when replying to unread messageRobin Jarry
When running `:reply -q` on an unread message, aerc crashes after opening the editor: panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x5d1019] goroutine 63 [running]: bufio.(*Reader).fill(0xc000086ef8) /usr/lib/golang/src/bufio/bufio.go:106 +0xd9 bufio.(*Reader).Peek(0xc00020bef8, 0x1) /usr/lib/golang/src/bufio/bufio.go:144 +0x5d github.com/emersion/go-message/textproto.ReadHeader(0xc00004a700?) emersion/go-message@v0.15.0/textproto/header.go:525 +0x5f git.sr.ht/~rjarry/aerc/worker/imap.(*IMAPWorker).handleFetchMessageBodyPart.func1(0xc00056e280) worker/imap/fetch.go:99 +0x1ab git.sr.ht/~rjarry/aerc/worker/imap.(*IMAPWorker).handleFetchMessages.func1() worker/imap/fetch.go:178 +0xd7 created by git.sr.ht/~rjarry/aerc/worker/imap.(*IMAPWorker).handleFetchMessages worker/imap/fetch.go:172 +0x12b This happens because the flags are updated in the callback that receives the message itself. It causes the flag update to arrive in the same channel/request. Ignore the messages that have an empty body (i.e. only containing flag updates). This is inherently racy but there seems no way to get rid of these extra messages. Signed-off-by: Robin Jarry <robin@jarry.cc> Tested-by: Connor Kuehl <cipkuehl@gmail.com>
2022-04-25pgp: ensure CRLF line endings in pgpmail readerKoni Marti
Ensure CRLF line endings in the pgpmail reader. Fix the pgp signature verification for maildir and notmuch. These backends do not return the full message body with CRLF line endings. But the accepted OpenPGP convention is for signed data to end with a <CR><LF> sequence (see RFC3156). If this is not the case the signed and transmitted data are considered not the same and thus signature verification fails. Link: https://datatracker.ietf.org/doc/html/rfc3156 Reported-by: Tim Culverhouse <tim@timculverhouse.com> Signed-off-by: Koni Marti <koni.marti@gmail.com> Tested-by: Tim Culverhouse <tim@timculverhouse.com>
2022-03-23logging: added a log on panicMoritz Poldrack
Since panics still regularly "destroy" the terminal, it is hard to get a stack trace for panics you do not anticipate. This commit adds a panic handler that automatically creates a logfile inside the current working directory. It has to be added to every goroutine that is started and will repair the terminal on a panic. Signed-off-by: Moritz Poldrack <git@moritz.sh> Acked-by: Robin Jarry <robin@jarry.cc>
2022-03-18go vet: composite literal uses unkeyed fieldsMoritz Poldrack
This commit fixes all occurrences of the abovementioned lint-error in the codebase. Signed-off-by: Moritz Poldrack <git@moritz.sh> Acked-by: Robin Jarry <robin@jarry.cc>
2022-03-09threading: honor user-defined sort criteriaKoni Marti
Apply the user-defined sort criteria to the message with the highest uid in a threaded discussion. Restore the default sort order when leaving threading mode. Commit 7811620eb809 ("threading: implement on-the-fly message threading") introduced message threading with the threaded messages being only sorted by their message uids irrespective of the defined sorting criteria. It did not restore the default sort order either. Reported-by: Sebastien Binet <s@sbinet.org> Signed-off-by: Koni Marti <koni.marti@gmail.com> Acked-by: Robin Jarry <robin@jarry.cc>
2022-03-07maildir: fix data race in maildir workerwagner riffel
Fix a data race due dirInfo pointer being read in the main goroutine by NewMessageStore and written in the anonymous goroutine launched in Worker.getDirectoryInfo. To address the issue raised in https://todo.sr.ht/~rjarry/aerc/16, we use readdir(3) once, parse and cache its results, this replaces go-maildir library Dir.Flags based stat(3) and filepath.Glob causing the issue when N (emails) is large. Signed-off-by: wagner riffel <w@104d.net>
2022-02-24threading: implement on-the-fly message threadingKoni Marti
implement message threading on the message store level using the jwz algorithm. Build threads on-the-fly when new message headers arrive. Use the references header to create the threads and the in-reply-to header as a fall-back option in case no references header is present. Does not run when the worker provides its own threading (e.g. imap server threads). Include only those message headers that have been fetched and are stored in the message store. References: https://www.jwz.org/doc/threading.html Signed-off-by: Koni Marti <koni.marti@gmail.com> Tested-by: Inwit <inwit@sindominio.net> Tested-by: akspecs <akspecs@gmail.com> Acked-by: Robin Jarry <robin@jarry.cc>
2022-02-23thread: add method to append new nodeKoni Marti
implement a method function for a *types.Thread receiver to append a new node to its linked list. Signed-off-by: Koni Marti <koni.marti@gmail.com>
2022-02-20imap: start reconnect when initial connect failsKoni Marti
Start the reconnect cycle when the initial connect fails. Make the connection observer send a connection error when the imap client is nil. Signed-off-by: Koni Marti <koni.marti@gmail.com>
2022-02-19imap: reconnect with exponential backoffKoni Marti
waits an increasing amount of time before attempting a reconnect. Wait is capped at 16s. Prevents many reconnect attemps in a short time period. Fixes commit 05ad96a30cb8 ("imap: improve reconnect stability") that improved the reliability of the reconnect mechanism but did not implement controls to prevent the triggering of too many reconnects within a short period of time. Fixes: 05ad96a30cb8 ("imap: improve reconnect stability") Signed-off-by: Koni Marti <koni.marti@gmail.com>
2022-02-06imap: improve reconnect stabilityKoni Marti
improves the robustness of the imap reconnect feature which was introduced in commit beae17a6da37 ("imap: auto-reconnects on connection error"). If a connection error is emitted, the message list is cleared and a corresponding error message is shown in the ui. Status bar is updated as well. Upon reconnect, the directories and the message list will be re-fetched (same behavior as the connect command). Reconnect can be enabled and disabled with the connect and the disconnect commands. Signed-off-by: Koni Marti <koni.marti@gmail.com>
2022-02-06imap: do not abort on malformed messagesRobin Jarry
Some messages contain invalid headers, textproto.ReadHeader fails with an error: malformed MIME header key: From nobody Fri Jan 14 19 And all other messages of the selected folder are ignored. Return an explicit error message to the UI and continue processing other messages. This is consistent with what the maildir worker does. Signed-off-by: Robin Jarry <robin@jarry.cc>
2022-01-30maildir: defer the count of recent messagesRobin Jarry
Since commit 01c96e78dfe8 ("Update DirectoryInfo handling for maildir"), flags are checked for every message of a folder when entering it. Iterating over all messages of a folder takes a long time for large collections of emails. Only count the number of messages and state that the directory info counts are not accurate. Defer the parsing of message flags in a goroutine to have a more responsive UI. Fixes: https://todo.sr.ht/~rjarry/aerc/16 Suggested-by: Koni Marti <koni.marti@gmail.com> Signed-off-by: Robin Jarry <robin@jarry.cc>
2022-01-20maildir,notmuch: pass in-memory message to callbackNguyễn Gia Phong
This fixes piped full message (:pipe -m) being empty. Fixes: 904ffacb0e52 ("maildir,notmuch: avoid leaking open files") Signed-off-by: Nguyễn Gia Phong <mcsinyx@disroot.org>
2022-01-19maildir,notmuch: avoid leaking open filesNguyễn Gia Phong
Previously, Message.NewReader returned the wrapped buffered reader without a reference to the opened file, so the files descriptors were left unclosed after reading. Now, the file reader is returned directly and closed on the call site. Buffering is not needed here because it is an implementation detail of go-message. Fixes: https://todo.sr.ht/~rjarry/aerc/9
2022-01-19imap: auto-reconnects on connection errorKoni Marti
if the worker emits a connection error, the ui will automatically send back a reconnect command. The worker then establishes a new connection. Auto-reconnect is disabled when the user sends the disconnect command. Fixes: https://todo.sr.ht/~rjarry/aerc/1 Signed-off-by: Koni Marti <koni.marti@gmail.com>
2022-01-19imap: emits connection error on logoutKoni Marti
implements a new connection error message. This allows the worker to emit a connection-related error message to the ui when the imap client closes the loggedOut channel. Signed-off-by: Koni Marti <koni.marti@gmail.com>
2022-01-19imap: fix panic when sending multiple connect cmdsKoni Marti
fixes the panic when the user sends multiple connect commands and is already connected. The panic is caused by closing an already closed channel. This happens when the idle re-init code is not executed, e.g. when there's a return statement in the switch block. A defer func() before the switch block will prevent this. The existing behavior of only creating a new idleStop channel when properly connected is preseverd. Signed-off-by: Koni Marti <koni.marti@gmail.com>
2022-01-15Revert "imap: attempt automatic reconnection on error"Robin Jarry
This reverts commit c605ada3ddc7569ebfc153b07db12a21b30d0569. This breaks reading message bodies. I am not sure why, I'll take some time to fix it later. Signed-off-by: Robin Jarry <robin@jarry.cc>
2022-01-14imap: attempt automatic reconnection on errorRobin Jarry
Attempt to reconnect to the server when there is an unexpected disconnection or network error. Use the Client.LoggedOut() channel which is closed when the connection is closed. This patch is rather flaky and is certainly bugged. However, it is a start. Signed-off-by: Robin Jarry <robin@jarry.cc>
2021-12-11format: reformat code with go 1.17Robin Jarry
There was a change in how build tags are formatted. Use this as new reference. Link: https://go.dev/doc/go1.17#gofmt Signed-off-by: Robin Jarry <robin@jarry.cc>
2021-12-11imap: fix build on macosRobin Jarry
Fix the following build error on mac os: worker/imap/worker.go:368:29: undefined: syscall.TCP_KEEPCNT worker/imap/worker.go:376:29: undefined: syscall.TCP_KEEPINTVL These symbols are not defined on darwin. Fixes: 5dfeff75f368 ("imap: add tcp connection options") Signed-off-by: Robin Jarry <robin@jarry.cc>
2021-12-07imap: add tcp connection optionsRobin Jarry
Allow fine tuning tcp connection options. Signed-off-by: Robin Jarry <robin@jarry.cc>
2021-12-07imap: use low level tcp connectionRobin Jarry
In preparation for tcp keepalive options, we need access to the net.TCPConn object associated with an IMAP connection. The only way to do this is to create the connection ourselves. No functional change. Signed-off-by: Robin Jarry <robin@jarry.cc>
2021-12-07imap: move connect procedure into a dedicated functionRobin Jarry
This will prepare for extra tcp connection options support and for automatic reconnect. No functional change. Signed-off-by: Robin Jarry <robin@jarry.cc>
2021-12-06maildir: watch for message renamesRobin Jarry
Messages flags can also be changed without changing directories. Changing flags in maildirs means renaming the message files. Also take renames into account. Link: https://cr.yp.to/proto/maildir.html Fixes: f4d3c8fc77f9 ("maildir: watch for external changes") Signed-off-by: Robin Jarry <robin@jarry.cc>