sync-mail

created by Jason Cox

I’m a bit of an email nerd, and my setup is rather complicated. I wrote a sync-mail script to sync email between the mail server and my laptop, keeping it organized in the process.

(I have lots of scripts to do various things on my computer, most of which I don’t publish. But someone asked to see this one, so here it is.)

Below, I examine the script bit by bit. If you’d prefer to see the full script, click to expand the following box. (Note that I’ve redacted a few things, using all caps placeholders like ACCT1 and MY_EMAIL.)

Full sync-mail script

#!/bin/sh

set -e

acct="$1"
folder="$2"
profiles="${acct:-ACCT1 ACCT2}"

maildir_mv() {
    src_path="$1"
    dst_maildir="$2"
    [ -f "$src_path" ] || return 0

    # rename the file to remove the U=<uid> and prevent mbsync errors
    subdir="$(basename "$(dirname "$src_path")")"
    file="$(basename "$src_path")"
    id="${file%%,*}"; id="${id%%:*}"
    flags="${file##*,}"

    if [ "$subdir" = cur ] || [ -n "$flags" ]; then
        new_file="$id:2,$flags"
    else
        new_file="$id"
    fi

    mv "$src_path" ~/pim/mail/"$dst_maildir/$subdir/$new_file"
}

maildir_mvs() {
    while read -r file; do
        maildir_mv "$file" "$1"
    done
}

set_vars() {
    profile="$1"
    trash_dir=Trash
    spam_dir=Junk
    sent_dir=Sent
    to_me=to:MY_EMAIL

    if [ "$profile" = ACCT2 ]; then
        trash_dir="Deleted Items"
        spam_dir="Junk Email"
        sent_dir="Sent Items"
        to_me=to:MY_OTHER_EMAIL
    fi

    export NOTMUCH_PROFILE="$profile"
}

# PRE-SYNC
#
# Move emails between folders based on tag changes. It's important not to move
# any emails that are still tagged as new; they've been indexed by notmuch but
# haven't yet received initial tags based on what folder they arrived in using
# the post-sync logic below.
#
# Searches are carefully constructed to ensure that each email is moved at most
# one time.
for profile in $profiles; do
    set_vars "$profile"

    # if tagged inbox, spam, or trash but not in corresponding folder, move it
    notmuch search --output=files not tag:new \
        and tag:inbox and not folder:INBOX \
        | maildir_mvs "$profile/INBOX"
    notmuch search --output=files not tag:new \
        and tag:trash and not tag:inbox and not folder:"\"$trash_dir\"" \
        | maildir_mvs "$profile/$trash_dir"
    notmuch search --output=files not tag:new \
        and tag:spam and not tag:inbox and not tag:trash and not folder:"\"$spam_dir\"" \
        | maildir_mvs "$profile/$spam_dir"

    # if in inbox, spam, or trash but missing corresponding tag, move to archive
    notmuch search --output=files not tag:new and \
        "(folder:INBOX and not tag:inbox)" \
        or "(folder:\"$spam_dir\" and not tag:spam)" \
        or "(folder:\"$trash_dir\" and not tag:trash)" \
        | maildir_mvs "$profile/Archive"
done

# SYNC
set +e
mbsync "${acct:--a}${folder:+:$folder}" || status=$?

# POST-SYNC
#
# Update notmuch, tagging new emails based on folders. Once initial tags are
# applied, the new tag must be removed so that the above pre-sync logic to move
# emails to the correct folders based on tags will work on the next run.
for profile in $profiles; do
    set_vars "$profile"
    notmuch new && notmuch tag --batch <<EOF
# tag based on folder (inbox, drafts, spam, trash)
+inbox -- tag:new and folder:INBOX
+draft -- tag:new and folder:Drafts
+spam -- tag:new and folder:"$spam_dir"
+trash -- tag:new and folder:"$trash_dir"

# remove unread tag if in sent or drafts
-unread -- tag:new and (folder:"$sent_dir" or folder:Drafts)

# tag as trash if in a trashed, muted thread and not addressed to me
+trash +mute -inbox -- tag:new and not $to_me and thread:"{tag:mute and tag:trash}"

# remove new tag
-new -- tag:new
EOF
    status="${status:-$?}"
done

exit "${status:-0}"

Background

I use mbsync (a.k.a. isync) to sync messages from my mail server to my laptop in maildir format. On top of that, I use notmuch to organize my email with tags and get powerful search capabilities. My email client, aerc, interfaces directly with notmuch.

I try to stick to notmuch’s philosophy as much as possible, primarily using tags for organization rather than folders. This approach works for me because I almost always access email on my laptop, where I have notmuch set up. I do keep email properly sorted in the “standard” folders – INBOX, Sent, Archive, Drafts, Trash, Junk – for the rare occasions when I need to check email on my phone or via the web client, though. I have notmuch tags that correspond to these folders, with the exception of Sent and Archive.

Within my email client, I only add and remove tags, never move messages directly between folders. Instead, my sync-mail script moves messages for me based on their tags. I settled on this method because I found that aerc didn’t handle moving messages well via notmuch.

For this tag-only approach to work, the script must tag new emails based on the folder that they start in. For example, a new email in the INBOX folder will have the inbox tag applied to it. The script also moves existing emails to the correct folder when their tags change. For example, when I delete a message by removing the inbox tag and adding the trash tag, it will be moved to the Trash folder on the next run of the script.

I make use of notmuch profiles to keep my email accounts separate. As a result, the sync script is a bit more complex.

My script was inspired by some snippets I found on Reddit.

Setup

First, I do some basic setup. The script expects 0-2 arguments: sync-mail [account [folder]]. If no arguments are provided, all folders in all accounts are synced. If only an account is provided, all folders in that account are synced. If both account and folder are provided, only that specific folder is synced.

#!/bin/sh

set -e

acct="$1"
folder="$2"
profiles="${acct:-ACCT1 ACCT2}"

Helper functions

Next, I define some helper functions. The first moves a single file between maildirs, being careful to remove mbsync’s UID from the filename in the process. I keep the same subdirectory (either cur or new); I’m not sure if that’s “correct” behavior or if I should always move the file into new, but it’s been working fine so far.

The function takes the file’s full path as the first argument. The second argument is the destination folder (without the final cur or new) relative to my base maildir (~/pim/mail/), which is hardcoded in the function.

maildir_mv() {
    src_path="$1"
    dst_maildir="$2"
    [ -f "$src_path" ] || return 0

    # rename the file to remove the U=<uid> and prevent mbsync errors
    subdir="$(basename "$(dirname "$src_path")")"
    file="$(basename "$src_path")"
    id="${file%%,*}"; id="${id%%:*}"
    flags="${file##*,}"

    if [ "$subdir" = cur ] || [ -n "$flags" ]; then
        new_file="$id:2,$flags"
    else
        new_file="$id"
    fi

    mv "$src_path" ~/pim/mail/"$dst_maildir/$subdir/$new_file"
}

The next helper function takes a destination folder as its first argument, reads source files from stdin, and then calls maildir_mv() for each file. This function is designed to work on the receiving end of a notmuch search --output=files pipe.

maildir_mvs() {
    while read -r file; do
        maildir_mv "$file" "$1"
    done
}

I also have a helper function for setting necessary variables for each account. Some of the folder names differ between accounts, and I set the notmuch profile. The to_me variable is used to filter emails that are addressed to my email address; I use this variable for the mute feature described below in the post-sync section.

set_vars() {
    profile="$1"
    trash_dir=Trash
    spam_dir=Junk
    sent_dir=Sent
    to_me=to:MY_EMAIL

    if [ "$profile" = ACCT2 ]; then
        trash_dir="Deleted Items"
        spam_dir="Junk Email"
        sent_dir="Sent Items"
        to_me=to:MY_OTHER_EMAIL
    fi

    export NOTMUCH_PROFILE="$profile"
}

Pre-sync

Before syncing, I move emails between folders based on their tags. I don’t move anything that still has the new tag; those messages have been indexed by notmuch but haven’t yet received initial folder-based tags in the post-sync logic below.

I’ve carefully crafted my search queries to ensure that each message is moved at most one time. The first moves messages that have the inbox tag to the INBOX folder. The second moves messages that have the trash tag but not the inbox tag to the Trash folder. The third moves messages that have the spam tag but not the inbox or trash tags to the Junk folder. The last moves messages that from the INBOX, Trash, and Junk folders that don’t have the corresponding tag to the Archive folder.

My mail server auto-deletes messages from Trash and Junk after a month or so. As a result, my goal here is to err on the side of caution by giving priority to the inbox. If I accidentally tag a message as spam or trash without removing the inbox tag, it will stay in the INBOX folder and not get auto-deleted.

# PRE-SYNC
for profile in $profiles; do
    set_vars "$profile"

    # if tagged inbox, spam, or trash but not in corresponding folder, move it
    notmuch search --output=files not tag:new \
        and tag:inbox and not folder:INBOX \
        | maildir_mvs "$profile/INBOX"
    notmuch search --output=files not tag:new \
        and tag:trash and not tag:inbox and not folder:"\"$trash_dir\"" \
        | maildir_mvs "$profile/$trash_dir"
    notmuch search --output=files not tag:new \
        and tag:spam and not tag:inbox and not tag:trash and not folder:"\"$spam_dir\"" \
        | maildir_mvs "$profile/$spam_dir"

    # if in inbox, spam, or trash but missing corresponding tag, move to archive
    notmuch search --output=files not tag:new and \
        "(folder:INBOX and not tag:inbox)" \
        or "(folder:\"$spam_dir\" and not tag:spam)" \
        or "(folder:\"$trash_dir\" and not tag:trash)" \
        | maildir_mvs "$profile/Archive"
done

Sync

Next, I run mbsync to actually sync messages between the mail server and my computer. If no account is specified, they’re all synced. If no folder is specified, all folders in the specified account are synced.

Starting here, I don’t abort on errors because I want to run the post-sync logic for all synced accounts even if the sync isn’t fully successful. I do keep track of any failed exit status in order to exit the script with a failure code if something went wrong, though.

# SYNC
set +e
mbsync "${acct:--a}${folder:+:$folder}" || status=$?

Post-sync

After syncing, I run notmuch new for each synced account and then apply some folder-based tagging. Once these initial tags have been applied, the new tag is removed so that the pre-sync logic above will move the messages between maildir folders when their tags change.

I don’t apply any tags for mail in the Sent folder. Mail only ends up in Sent when I send it – I never manually move messages there – and I can find sent messages in my email client with a simple from:<my email> notmuch query.

Similarly, I don’t apply any tags for mail in the Archive folder. Instead, I consider a message archived if it doesn’t have the inbox, draft, spam, or trash tag.

The other special tagging feature that I’ve implemented is muted threads. Once I tag a message with the mute and trash tags, all future messages in that thread will automatically get those same tags applied. This feature is nice for mailing lists, where threads often go on for quite a while; once I decide I’m no longer interested in a thread, I can mute it and make its subsequent messages skip the inbox. (There’s an exception for emails that are directly addressed to me – they’ll still land in my inbox.)

# POST-SYNC
for profile in $profiles; do
    set_vars "$profile"
    notmuch new && notmuch tag --batch <<EOF
# tag based on folder (inbox, drafts, spam, trash)
+inbox -- tag:new and folder:INBOX
+draft -- tag:new and folder:Drafts
+spam -- tag:new and folder:"$spam_dir"
+trash -- tag:new and folder:"$trash_dir"

# remove unread tag if in sent or drafts
-unread -- tag:new and (folder:"$sent_dir" or folder:Drafts)

# tag as trash if in a trashed, muted thread and not addressed to me
+trash +mute -inbox -- tag:new and not $to_me and thread:"{tag:mute and tag:trash}"

# remove new tag
-new -- tag:new
EOF
    status="${status:-$?}"
done

Last but not least, I exit with a failure code if mbsync or the post-sync logic had any errors.

exit "${status:-0}"