Archiving Mail

[Ref: One Mailbox]

We have a need to archive mail, but our 'mail chain' doesn't allow us to have it configured on the mailbox server (user access) or other servers. We have to install a new server, but want to minimise anything that gets installed on that system.

Our solution: use Postfix as a relay host to nowhere.

The relayhost to nowhere design(?) requires us to set up a mail server that accepts e-mail for delivery to someone else with the following behaviour:-

  1. copy all mail to a local account
  2. discard (do not relay) all mail
  3. Review Default Limits

The local account configuration uses standard Unix/Postfix behaviour:

  1. Forward mail to [procmail](http://www.procmail.org/)
  2. Use procmail to set a systemic manner of storage

Configuration Settings

Copy to Local Account

[Ref: Postfix Configuration Parameters]

As is our goal, we want to accept all incoming messages and have it copied to an account from where we will manage the e-mail content.

File extract: /etc/postfix/main.cf

always_bcc = mailarchive

For our configuration, we'll use a local account mailarchive

always_bcc
Optional address that receives a "blind carbon copy" of each message that 
is received by the Postfix mail system. 

The BCC address is added as if it was specified with NOTIFY=NONE. The sender 
will not be notified when the BCC address is undeliverable, as long as all 
down-stream software implements RFC 3461. 

Automatic BCC recipients are produced only for new mail. To avoid mailer loops, 
automatic BCC recipients are not generated after Postfix forwards mail internally, 
or after Postfix generates mail itself. 

Discard all outgoing mail

a) Set a relayhost to a non-existent ip in your network

File extract: /etc/postfix/main.cf

relayhost = 127.0.0.1:2025

There isn't any service running port localhost port 2025 (or if there is, obviously you want to set the port to something else.)

b) Change the bounce queue settings so that bounced mail will be discarded instantly

File extract: /etc/postfix/main.cf

bounce_queue_lifetime = 0
maximal_queue_lifetime = 0

Reviewing Default Limits

As the mail archiving process is going to get a lot of email messages into the above mailarchive single e-mail account, our server's mail processing behaviour is significantly different than the standard mail server.

We need to review standard behaviours designed to limit abuse, and manage them per our requirements.

  • Mailbox size limit
  • Message size limit
  • Who to archive for
mailbox_size_limit = 0
message_size_limit = 40960000
my_networks = servers_to_accept_mail_from

Mailbox Size Limit

File extract: /etc/postfix/main.cf

mailbox_size_limit = 0
Symptoms

[Ref: 49MB mailbox size limit (postfix & procmail)]

If you encounter errors in your log file (procmail.log) such as the below, then it is likely that you will need to configure the filesize limits for postfix:

Acquiring kernel-lock
Error while writing to "/path/mail-file"
Truncated file to former size
Assigning "LASTFOLDER=today"

Your installation limits can be determined using postconf

postconf mailbox_size_limit
mailbox_size_limit = 512000000

Which indicates a 512,000,000 byte file size limit. As shown by the above, for our configuration we need the file size to be unlimited.

Message Size Limit

[Ref:message_size_limit]

File extract: /etc/postfix/main.cf

message_size_limit = 40960000
message_size_limit (default: 10240000)
The maximal size in bytes of a message, including envelope information

Our internal corporate policy is to allow 10MB, with certain levels of flexibility up to 20MB (depending on who has the biggest stick.) Since we're unlikely to remember that we have these limits in our archiving system, we're consciously deciding to expand the archiving well above our operational standards. We set the above configuration to be 40MB (40960000 bytes)

Who to archive for

As you can tell from the above, the increased 'limit's leaves the box very vulnerable to Denial of Service attacks that will readily fill up all disc capacity. To mitigate against rogue mailers, we set up at least one barrier by limiting the servers/clients allowed to send email to this box.

File extract: /etc/postfix/main.cf

mynetworks = 127.0.0.1/8, 192.168.3.14/32, 192.168.3.25/32, 192.168.3.42/32

My above example specifies my hosts ip addresses, and then adds two hosts from which the server will accept email. All other hosts should show up in the logs with a NOQUEUE error message.

Local Account

For our system, we use the local user account mailarchive and now have to set up some configuration and adjust standard restrictions on the user account.

Standard Limits

[Ref: login.conf(5)]

We're creating an archiving system, so standard limits do not make sense. To isolate the changes we want to make, we will use a new login class:

File extract: /etc/login.conf

archiver: \
        :datasize-cur=512M:\
        :datasize-max=infinity:\
        :maxproc-max=512:\
        :maxproc-cur=128:\
        :filesize=infinity:\
        :tc=default:

Obviously, we want to increase the maximum filesize as we have no real data on what the filesizes should/could be, and we're an archiving solution so we need to be able to accept a broad range of things.

datasize    size       Maximum data size limit.
maxproc     number     Maximum number of processes.
filesize    size       Maximum file size limit.
...
number  A number.  A leading 0x implies the number is expressed in
        hexadecimal.  A leading 0 implies the number is expressed in
        octal.  Any other number is treated as decimal.

size    A number which expresses a size.  By default, the size is
        specified in bytes.  It may have a trailing b, k, m, g or t to
        indicate that the value is in 512-byte blocks, kilobytes,
        megabytes, gigabytes, or terrabytes, respectively.

[Ref: cap_mkdb(1)]

Once you've made modification such as the above login.conf, make sure you update the database by executing

$ sudo cap_makedb /etc/login.conf

As per our above configuration in login.conf, you'll want to ensure that your local user account is now connected using the new filesize limits, by using something like vipw, or usermod(8) "-L login-class" to set the login class:

  • mailarchive --> login class --> archiver

File extract /etc/passwd:

mailarchive:*:1003:1003:archiver:0:0:mailarchive:/home/mailarchive:/bin/ksh

Forward mail to procmail

With postfix configured to always_bcc email to our mailarchive account, our mailbox will grow ad-infinitum, which isn't too useful a configuration. We need to do some processing of the incoming email and the simplest (?) way to do this is to forward all email to a program that is better suited for manipulating mail procmail

File: /home/mailarchive/.forward

"|IFS=' ' &&p=/usr/local/bin/procmail&&test -f $p&&exec $p -f-||exit 75#mailarchive"

Procmail Recipe

[Ref: Timo Salmi's Tips]

Procmail is flexible, and with this its configuration can be difficult to decipher.

My basic recipe uses macros to specify/outline where I want to store all email messages (except for special messages with the Subject ending with Procmail TEST)

File: /home/mailarchive/.procmailrc

# Store as ~/.procmailrc
# Make configuration files, directories exist
#       $STORAGEDIR/mail
#       RECIPES 

# Debugging
VERBOSE=off

SHELL=/bin/sh
STORAGEDIR=/storage/archive
TMP=/tmp

MAILDIR=$STORAGEDIR/mail
RECIPES=$STORAGEDIR/rcs
LOCKFILE=${STORAGEDIR}/log/procmail.lck
LOGFILE=${STORAGEDIR}/log/procmail.log
MAILFILE=`date "+%Y/%m-%B/%d-%A-week-%V"`

SUBDIR=`dirname $MAILFILE`

# --- make sure the paths exist
:0
* ? test -d ${MAILDIR}/${SUBDIR} || mkdir -p ${MAILDIR}/${SUBDIR}
* ? test -d ${STORAGEDIR}/log/ || mkdir -p ${STORAGEDIR}/log
* ? test -d ${RECIPES} || mkdir -p ${RECIPES}
:0E
{
    # Bail out if any of the above fails
    EXITCODE=127
    HOST
}

# ------------------------------------------------
MAILBOX=${MAILDIR}/${MAILFILE}
DEFAULT=${MAILBOX}

# -------------------
# Processing
# -------------------

# Send all messages with the specific Subject Line
# "Procmail TEST" to a 'testing' mailbox
# -------------------------------------------------------------
:0:             # Note the tight subject [Procmail TEST]
* ^Subject: .*Procmail TEST$
testing     # will go to $MAILDIR/testing


# INCLUDERC=${RECIPES}/rc.all <--- put other recipes into a separate file

# Default/Catch All (or technically could be left out
#                             since we've defined $DEFAULT above)
# send all other messages to the specified $MAILBOX
# -------------------------------------------------------------

:0
* .*
${MAILBOX}

Trawling the Archives

[Ref: Procmail FAQ]

Now we have 6 months worth of email, and they've grown to between 30GB a month. Someone finally decides they want mail from the archives and there is a huge amount of mail to wade through.

How do we do it?

formail -s procmail recipe.rc < mailfile

There's some fancy stuff out there, but I have found procmail and formail already installed because of the archiving solution.

Recipe - Mail To/From mailclient

A user has these requirements for retrieving emails from the archives:

  • time period of 5 days (specified to us)
  • sent to a particular user (specified to us)

We collect messages by the 'day' so this part is simplified. We use the below procmail recipe for each day's email messages.

formail -s procmail /path-to/tofrom.recipe.rc < /path-to/mailfile

File: tofrom.recipe.rc

# Debugging
VERBOSE=off
UMASK=007
SHELL=/bin/sh

# http://www.iki.fi/era/procmail/mini-faq.html#from
CMDFROM="^(From[   ]|(Old-|X-)?(Resent-)?(From|Reply-To|Sender):)(.*\<)?"
CMDTO="^TO_"

## ---
## Customise Me!!
## ---
# - set MAILDIR to where you will do work (note: all relative paths go from here)
MAILDIR=/path-to/store/work
MAILCLIENT="<--emailaddress-here-->"
CMD=${CMDTO}

## ---

LOCKFILE=${MAILDIR}/procmail.lck
LOGFILE=${MAILDIR}/procmail.log
TMPDIR=/tmp/$USER

MAILBOX=${MAILDIR}/${MAILCLIENT}
DEFAULT=/dev/null

# --- Check a few paths first
:0
* ? test -d ${MAILDIR}
* ? test -d ${TMPDIR} || mkdir ${TMPDIR}
{ }

:0E
{
    # Bail out if any of the above fails
    EXITCODE=127
    HOST
}

# Deliver Mail to our file ${MAILBOX}

:0:
* $ ${CMD}${USER}
${MAILBOX}

Customisation areas are 'blocked off' above.

  • MAILDIR - set the path where files are to be written (i.e. results of the process, lock file, log file) e.g. /var/data/mail/recovery
  • MAILCLIENT - recipient (e.g. samt@example.com)
  • CMD - choose either CMDTO or CMDFROM to specify which you want (CMDTO messages sent to $MAILCLIENT or CMDFROM for messages sent from) We use ${CMDTO}
File Permissions

[Ref: Check for Permission Problems]

One aspect of procmail that is important to remember (or at least it wasted too much of my time until I re-discovered this.)

Recipe file permissions:

  • Use a path/file permission of 0640 and make sure the
  • running user 'owns' the recipe file

Mutt

[Ref: tags , Mutt, port]

We now have a 'maildir' file as /var/data/mail/recovery/samt@example.com, but our users are Windows/Outlook users.

I can't readily give them the files (which they would accept as individual EML files) because I don't have free tools for that conversion, so we use another tool mutt.

  • Using
  • T to tag messages by a pattern
  • Use Patterns
  • ~A to tag all messages
  • Using ";" Bounce b, to send all messages to the appropriate person
Stepping through

Launch "mutt" with the "-f" option to open the mail message file

mutt -f /var/data/mail/recovery/samt@example.com
q:Quit  d:Del  u:Undel  s:Save  m:Mail  r:Reply  g:Grouop  ?:Help
1  N   Month Day From-address ( size) Subject line
2  N   Month Day From-address ( size) Subject line
3  N   Month Day From-address ( size) Subject line
...
...
---Mutt: samt@example.com [Msgs:xyz Old:xyz xyzM]---(date/date)---

From the email index list, the command sequence looks this:

  • T
...
---Mutt: samt@example.com [Msgs:xyz Old:xyz xyzM]---(date/date)---
Tag message matching:
  • ~A

The Mail Index should show an "*" asterisk beside all messages

q:Quit  d:Del  u:Undel  s:Save  m:Mail  r:Reply  g:Grouop  ?:Help
1  N * Month Day From-address ( size) Subject line
2  N * Month Day From-address ( size) Subject line
3  N * Month Day From-address ( size) Subject line
...
...
---Mutt: samt@example.com [Msgs:xyz Old:xyz Tag:xyz xyzM]---(date/date)---

and the status bar should include the number of messages tagged:

To bounce a message, we would use 'b', but we want to bounce all tagged messages, and therefore precede 'b' with the semi-colon ';'

  • ;
...
...
---Mutt: samt@example.com [Msgs:xyz Old:xyz Tag:xyz xyzM]---(date/date)---
tag-
  • b
...
...
---Mutt: samt@example.com [Msgs:xyz Old:xyz Tag:xyz xyzM]---(date/date)---
Bounce tagged messages to:

and we get a prompt for whom/where we wish to bounce messages, enter our destination address

  • samt@example.net

and we get a confirmation prompt

Bounce messages to samt@example.net? ([yes]/no):
  • yes
...
...
---Mutt: samt@example.com [Msgs:xyz Old:xyz Tag:xyz xyzM]---(date/date)---
Messages bounced.