Table of Content
[Ref: One Mailbox]
We have a need to archive mail, but our ‘mail chain’ doesn’t allow us to have it configured on the mailbox server (user access) or other servers. We have to install a new server, but want to minimise anything that gets installed on that system.
Our solution: use Postfix as a relay host to nowhere.
The relayhost to nowhere design(?) requires us to set up a mail server that accepts e-mail for delivery to someone else with the following behaviour:-
* copy all mail to a local account
* discard (do not relay) all mail
* Review Default Limits
The local account configuration uses standard Unix/Postfix behaviour:
* Forward mail to [procmail](http://www.procmail.org/)
* Use procmail to set a systemic manner of storage
[Ref: Postfix Configuration Parameters]
As is our goal, we want to accept all incoming messages and have it copied to an account from where we will manage the e-mail content.
File extract: /etc/postfix/main.cf
always_bcc = mailarchiveFor our configuration, we’ll use a local account mailarchive
always_bcc Optional address that receives a "blind carbon copy" of each message that is received by the Postfix mail system. The BCC address is added as if it was specified with NOTIFY=NONE. The sender will not be notified when the BCC address is undeliverable, as long as all down-stream software implements RFC 3461. Automatic BCC recipients are produced only for new mail. To avoid mailer loops, automatic BCC recipients are not generated after Postfix forwards mail internally, or after Postfix generates mail itself.
a) Set a relayhost to a non-existent ip in your network
File extract: /etc/postfix/main.cf
relayhost = 127.0.0.1:2025There isn’t any service running port localhost port 2025 (or if there is, obviously you want to set the port to something else.)
b) Change the bounce queue settings so that bounced mail will be discarded instantly
File extract: /etc/postfix/main.cf
bounce_queue_lifetime = 0
maximal_queue_lifetime = 0As the mail archiving process is going to get a lot of email messages into the above mailarchive single e-mail account, our server’s mail processing behaviour is significantly different than the standard mail server.
We need to review standard behaviours designed to limit abuse, and manage them per our requirements.
mailbox_size_limit = 0
message_size_limit = 40960000
my_networks = servers_to_accept_mail_fromFile extract: /etc/postfix/main.cf
mailbox_size_limit = 0[Ref: 49MB mailbox size limit (postfix & procmail)]
If you encounter errors in your log file (procmail.log) such as the below, then it is likely that you will need to configure the filesize limits for postfix:
Acquiring kernel-lock Error while writing to "/path/mail-file" Truncated file to former size Assigning "LASTFOLDER=today"
Your installation limits can be determined using postconf
postconf mailbox_size_limitmailbox_size_limit = 512000000
Which indicates a 512,000,000 byte file size limit. As shown by the above, for our configuration we need the file size to be unlimited.
[Ref:message_size_limit]
File extract: /etc/postfix/main.cf
message_size_limit = 40960000message_size_limit (default: 10240000) The maximal size in bytes of a message, including envelope information
Our internal corporate policy is to allow 10MB, with certain levels of flexibility up to 20MB (depending on who has the biggest stick.) Since we’re unlikely to remember that we have these limits in our archiving system, we’re consciously deciding to expand the archiving well above our operational standards. We set the above configuration to be 40MB (40960000 bytes)
As you can tell from the above, the increased ’limit’s leaves the box very vulnerable to Denial of Service attacks that will readily fill up all disc capacity. To mitigate against rogue mailers, we set up at least one barrier by limiting the servers/clients allowed to send email to this box.
File extract: /etc/postfix/main.cf
mynetworks = 127.0.0.1/8, 192.168.3.14/32, 192.168.3.25/32, 192.168.3.42/32My above example specifies my hosts ip addresses, and then adds two hosts from which the server will accept email. All other hosts should show up in the logs with a NOQUEUE error message.
For our system, we use the local user account mailarchive and now have to set up some configuration and adjust standard restrictions on the user account.
[Ref: login.conf(5)]
We’re creating an archiving system, so standard limits do not make sense. To isolate the changes we want to make, we will use a new login class:
File extract: /etc/login.conf
archiver: \
:datasize-cur=512M:\
:datasize-max=infinity:\
:maxproc-max=512:\
:maxproc-cur=128:\
:filesize=infinity:\
:tc=default:Obviously, we want to increase the maximum filesize as we have no real data on what the filesizes should/could be, and we’re an archiving solution so we need to be able to accept a broad range of things.
datasize size Maximum data size limit.
maxproc number Maximum number of processes.
filesize size Maximum file size limit.
...
number A number. A leading 0x implies the number is expressed in
hexadecimal. A leading 0 implies the number is expressed in
octal. Any other number is treated as decimal.
size A number which expresses a size. By default, the size is
specified in bytes. It may have a trailing b, k, m, g or t to
indicate that the value is in 512-byte blocks, kilobytes,
megabytes, gigabytes, or terrabytes, respectively.
[Ref: cap_mkdb(1)]
Once you’ve made modification such as the above login.conf, make sure you update the database by executing
$ [ -f /etc/login.conf.db ] && sudo /usr/bin/cap_mkdb /etc/login.confAs per our above configuration in login.conf, you’ll want to ensure that your local user account is now connected using the new filesize limits, by using something like vipw, or usermod(8) "-L login-class" to set the login class:
File extract /etc/passwd:
mailarchive:*:1003:1003:archiver:0:0:mailarchive:/home/mailarchive:/bin/ksh
With postfix configured to always_bcc email to our mailarchive account, our mailbox will grow ad-infinitum, which isn’t too useful a configuration. We need to do some processing of the incoming email and the simplest (?) way to do this is to forward all email to a program that is better suited for manipulating mail procmail
File: /home/mailarchive/.forward
"|IFS=' ' &&p=/usr/local/bin/procmail&&test -f $p&&exec $p -f-||exit 75#mailarchive"[Ref: Timo Salmi’s Tips]
Procmail is flexible, and with this its configuration can be difficult to decipher.
My basic recipe uses macros to specify/outline where I want to store all email messages (except for special messages with the Subject ending with Procmail TEST)
File: /home/mailarchive/.procmailrc
# Store as ~/.procmailrc
# Make configuration files, directories exist
# $STORAGEDIR/mail
# RECIPES
# Debugging
VERBOSE=off
SHELL=/bin/sh
STORAGEDIR=/storage/archive
TMP=/tmp
MAILDIR=$STORAGEDIR/mail
RECIPES=$STORAGEDIR/rcs
LOCKFILE=${STORAGEDIR}/log/procmail.lck
LOGFILE=${STORAGEDIR}/log/procmail.log
MAILFILE=`date "+%Y/%m-%B/%d-%A-week-%V"`
SUBDIR=`dirname $MAILFILE`
# --- make sure the paths exist
:0
* ? test -d ${MAILDIR}/${SUBDIR} || mkdir -p ${MAILDIR}/${SUBDIR}
* ? test -d ${STORAGEDIR}/log/ || mkdir -p ${STORAGEDIR}/log
* ? test -d ${RECIPES} || mkdir -p ${RECIPES}
:0E
{
# Bail out if any of the above fails
EXITCODE=127
HOST
}
# ------------------------------------------------
MAILBOX=${MAILDIR}/${MAILFILE}
DEFAULT=${MAILBOX}
# -------------------
# Processing
# -------------------
# Send all messages with the specific Subject Line
# "Procmail TEST" to a 'testing' mailbox
# -------------------------------------------------------------
:0: # Note the tight subject [Procmail TEST]
* ^Subject: .*Procmail TEST$
testing # will go to $MAILDIR/testing
# INCLUDERC=${RECIPES}/rc.all <--- put other recipes into a separate file
# Default/Catch All (or technically could be left out
# since we've defined $DEFAULT above)
# send all other messages to the specified $MAILBOX
# -------------------------------------------------------------
:0
* .*
${MAILBOX}
[Ref: Procmail FAQ]
Now we have 6 months worth of email, and they’ve grown to between 30GB a month. Someone finally decides they want mail from the archives and there is a huge amount of mail to wade through.
How do we do it?
formail -s procmail recipe.rc < mailfileThere’s some fancy stuff out there, but I have found procmail and formail already installed because of the archiving solution.
A user has these requirements for retrieving emails from the archives:
We collect messages by the ‘day’ so this part is simplified. We use the below procmail recipe for each day’s email messages.
formail -s procmail /path-to/tofrom.recipe.rc < /path-to/mailfileFile: tofrom.recipe.rc
# Debugging
VERBOSE=off
UMASK=007
SHELL=/bin/sh
# http://www.iki.fi/era/procmail/mini-faq.html#from
CMDFROM="^(From[ ]|(Old-|X-)?(Resent-)?(From|Reply-To|Sender):)(.*\<)?"
CMDTO="^TO_"
## ---
## Customise Me!!
## ---
# - set MAILDIR to where you will do work (note: all relative paths go from here)
MAILDIR=/path-to/store/work
MAILCLIENT="<--emailaddress-here-->"
CMD=${CMDTO}
## ---
LOCKFILE=${MAILDIR}/procmail.lck
LOGFILE=${MAILDIR}/procmail.log
TMPDIR=/tmp/$USER
MAILBOX=${MAILDIR}/${MAILCLIENT}
DEFAULT=/dev/null
# --- Check a few paths first
:0
* ? test -d ${MAILDIR}
* ? test -d ${TMPDIR} || mkdir ${TMPDIR}
{ }
:0E
{
# Bail out if any of the above fails
EXITCODE=127
HOST
}
# Deliver Mail to our file ${MAILBOX}
:0:
* $ ${CMD}${USER}
${MAILBOX}Customisation areas are ‘blocked off’ above.
[Ref: Check for Permission Problems]
One aspect of procmail that is important to remember (or at least it wasted too much of my time until I re-discovered this.)
Recipe file permissions:
We now have a ‘maildir’ file as /var/data/mail/recovery/samt@example.com, but our users are Windows/Outlook users.
I can’t readily give them the files (which they would accept as individual EML files) because I don’t have free tools for that conversion, so we use another tool mutt.
Launch “mutt” with the “-f” option to open the mail message file
mutt -f /var/data/mail/recovery/samt@example.comq:Quit d:Del u:Undel s:Save m:Mail r:Reply g:Grouop ?:Help 1 N Month Day From-address ( size) Subject line 2 N Month Day From-address ( size) Subject line 3 N Month Day From-address ( size) Subject line ... ... ---Mutt: samt@example.com [Msgs:xyz Old:xyz xyzM]---(date/date)---
From the email index list, the command sequence looks this:
... ---Mutt: samt@example.com [Msgs:xyz Old:xyz xyzM]---(date/date)--- Tag message matching:
The Mail Index should show an “*” asterisk beside all messages
q:Quit d:Del u:Undel s:Save m:Mail r:Reply g:Grouop ?:Help 1 N * Month Day From-address ( size) Subject line 2 N * Month Day From-address ( size) Subject line 3 N * Month Day From-address ( size) Subject line ... ... ---Mutt: samt@example.com [Msgs:xyz Old:xyz Tag:xyz xyzM]---(date/date)---
and the status bar should include the number of messages tagged:
To bounce a message, we would use ‘b’, but we want to bounce all tagged messages, and therefore precede ‘b’ with the semi-colon ‘;’
... ... ---Mutt: samt@example.com [Msgs:xyz Old:xyz Tag:xyz xyzM]---(date/date)--- tag-
... ... ---Mutt: samt@example.com [Msgs:xyz Old:xyz Tag:xyz xyzM]---(date/date)--- Bounce tagged messages to:
and we get a prompt for whom/where we wish to bounce messages, enter our destination address
and we get a confirmation prompt
Bounce messages to samt@example.net? ([yes]/no):
... ... ---Mutt: samt@example.com [Msgs:xyz Old:xyz Tag:xyz xyzM]---(date/date)--- Messages bounced.