Building a Reputable Mail Server
================================
:author:  Aaron Ball
:email:   nullspoon@oper.io


Several months back, I moved my server to a different hosting provider and
naturally received a different IP address. Since then, I have been struggling
with one of the most prevalant problems one has when running their own mail
server: building a good reputation to avoid spam directories. Since changing
IPs, all of my emails to both Hotmail and Gmail have been going to spam,
regardless of policy compliance.

I did a [metric] truck load of research to remediate the situation and have
finally resolved it. Most of the things I tried are relatively standard, but I
realized during my research that there aren't very many good documents oriented
to personal mail server owners. Most of the resources are aimed at bulk senders
looking to accost users with [strikethrough]#spam# marketing emails. This is
not me (and hopefully not you).

To be clear before I proceed. This post is **not** about how to avoid spam
filters so you can spam people. If you're spamming people, this guide will only
help you for so long (and quite honestly, you can shove your spam where the sun
don't shine). This post also won't go into the specifics of setting up a
particular software like Exim or Postfix.

This post is for the folks who might be interested in running their own servers
for educational reasons. Or, perhaps you like to own your data. Maybe you hold
the principle of freedom of the internet. You might also want to avoid
mega-corporations. Or perhaps you may just have an affinity for tin-foil hats.

Let's get to it!


The Checklist
-------------

* <<_the_easy_stuff, The Easy Stuff>>
** <<_static_ip, Static IP>>
** <<_primary_hostname, Primary Hostname>>
** <<_port_25_outbound, Port 25 Outbound>>
** <<_have_accounts_with_major_providers, Have Accounts with Major Providers>>
* <<_ip_reputation, IP Reputation>>
* <<_spf_records, SPF Records>>
* <<_dkim_signing, DKIM signing>>


The Easy Stuff
--------------

I'll just cover these quickly since they are relatively simple.


Static IP
~~~~~~~~~

If you understand how email works fundamentally (as in, you've read the SMTP
link:https://tools.ietf.org/html/rfc821[RFC 821]), you know this isn't required
by the spec. However, because of spammers (and other reasons), this is now an
unofficial requirement to get emails through to the major providers.

**TL;DR**: Don't run a mail server if you can't get a static IP. If you do, you
will probably have big problems getting your emails delivered.

**Detailed explanation**: Most providers won't deliver emails anywhere other
than spam from IP addresses on which they cannot do
link:https://en.wikipedia.org/wiki/Reverse_DNS_lookup[reverse DNS lookups].
If you do this from a dynamic IP, you likely don't have control the PTR record
and thus all of your emails will go straight to spam.

The other reason you need a static IP is because major mail providers will
often not deliver emails to inboxes unless the domain referenced in the _from_
field has a DNS link:https://en.wikipedia.org/wiki/MX_record[MX record] that
resolves to the IP address (your server) that sent the email. Because of
minimum DNS TTLs, you can't [easily] get your DNS MX record to update fast
enough when your IP changes for the major mail providers to correctly reconcile
the new domain MX record with the sending IP (your server). This means when
you're IP changes, even if you have automation updating DNS on the fly, you'll
still have a window until the TTL expires for all of the various providers to
re-query for the updated IP and approve delivery from it.


Primary Hostname
~~~~~~~~~~~~~~~~

This is the name your mail server identifies as when connected to. If you want
to see what this looks like, you can run `telnet <server> 25` to connect to a
server's smtp service. The response you get back should say something like...

  220 <primary_hostname> ESMTP Exim 4.91 Sat, 09 Feb 2019 03:57:22 +0000

NOTE: You probably cannot do this from your home internet, as most ISPs block
port 25 outbound to prevent botnets from spamming the world. Marketing and
current political issues aside, we haven't had actual net neutrality for years
and this is one example of that fact, though it arguably does the internet some
good.

The primary hostname can actually be set to anything, as long as it is
resolvable in DNS. We'll get into this a bit later with SPF records, but any
mail server can send email on behalf any other domain. Whether that server is
**permitted** to send email on behalf of that domain is another issue. We'll
get more into the permissions aspect in the SPF records section. Regardless of
all the flexibility you have here, just set this to the domain name from which
you are sending email.


Port 25 Outbound
~~~~~~~~~~~~~~~~

As mentioned in the last section, to send email you must be able to connect on
port tcp/25, which most ISPs don't allow. If you can't do this for whatever
reason, your emails will never reach their destination.


Have Accounts with Major Providers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Troubleshooting any issues with a particular provider is very difficult without
being able to see what your recipients are seeing. This is why I recommend you
have an account with all of the major providers, so you can replicate any
delivery issues your users might be having. Major providers you might consider
having an account with are... 

* GMail
* Hotmail/Live/Outlook
* +++Mail.com+++
* GMX
* Proton Mail

I personally have accounts with both Hotmail and Gmail, both of which I don't
use for anything other than validating that I can send email to users of the
respective services. I don't bother having an account with the other providers,
since none of the people I email use those services (except Yahoo, but I've
never had any delivery problems with them).




IP Reputation
-------------

If you have an IP address that was previously used by spammers, it likely has a
bad reputation. In an attempt to fight spammers, engineers created something
called DNSBLs, or **DNS** **B**ased **L**ists. This is defined in the IETF
link:https://tools.ietf.org/html/rfc5782[RFC 5782]. Effectively, an DNSBL can
be used to define a whitelist or a blacklist of IPS and/or domains. If your IP
address has a bad reputation, this is where you go to find out. Unfortunately
however, there are hundreds of these available, and different providers use
different lists, which makes reputation remediation a bit difficult.
Fortunately, folks have built services to help you resolve the situation.
Before I wrote link:/?p=checking-dnsbl-lists[my own script for this], I used
link:https://mxtoolbox.com/blacklists.aspx[https://mxtoolbox.com] all the time.

If your IP shows up as blacklisted on any of the providers, you should
remediate that by their stated process. Each has their own process
unfortunately, though most are relatively simple and quick. If you do not
repair your server's IP reputation, all of your email's will be sent to spam by
the major providers. For a better explanation of DNSBLs, see my post on
link:/?p=checking-dnsbl-lists[checking dnsbl lists] (yes, I know that's
redundant, like ATM machine).


SPF Records
-----------

Here we have yet another spam-fighting mechanism, one with which I have no
issues, unlike with DKIM. SPF stands for **S**ender **P**olicy **F**ramework.
It uses DNS TXT records to public a list of IP address permitted to send emails
on behalf of a given domain. These exist less as a whitelist and more as a sort
of blacklist.

Before SPF, any IP address could send email with from fields claiming to
originate from any given domain. They still can do that, but now mail providers
have a method with which they can verify permission to send emails on behalf of
a given domain. I wrote link:/?p=Aol_Email_Hacked[a detailed post about this]
back when AOL was "hacked" in 2014 if you want to read more details on SPF
records and how not having them (or having them be too permissive) is
dangerous.

To better explain how these work, here is the spf record for my website
(oper.io).
  
----
$ dig +short -t spf oper.io
"v=spf1 mx a -all"
----

My spf [version 1] record permits any IP address that is pointed to by my MX or
A records to send email on behalf of oper.io. In other words, emails with
_...@oper.io_ in the _from_ field will be delivered field when the originating
IP is in my MX or A records. Note though that I have a `-all` at the end of my
SPF record. That is a hard deny for all other IP addresses. Other options are
`+all`, which is a hard approval, and `?all`, which is a soft deny that
effectively leaves the decision up to the mail server receiving the email.

For most people running their own servers, the SPF record will likely only
contain one IP. However, for businesses, these records have much greater
application. Most businesses that send out marketing emails do not use their
own mail servers to do so. They pay a third party and use SPF records to permit
that third party to send emails on their behalf.


DKIM Signing
------------

DKIM is _yet another_ method for validating email as coming from its specified
source. DKIM stands for **D**omain**K**eys **I**dentified **M**ail. It is
asymmetric cryptographic signing used to validate that an email came from the
domain specified. To more easily describe this, I am going to simplify the
architecture about which we are talking. I will assume that we have **one
server**, and the DKIM selector is **default**.

At a high level, your one server has an RSA key somewhere on it. Your email
server is configured to sign all outbound email with the private key. This
signature, along with some simple metadata, is embedded in your email using the
**DKIM-Signature** header. Upon receipt, the service you emailed will read this
header, and validate the signature by checking your public key, which you set
up in DNS under a TXT record.

NOTE: You can have more than one key doing singing (like one key per server),
      and these keys are identified by something called a selector. The default
      selector is called **default** (creative, I know). 

To check the domain signing key, you can do a dig command with...

  dig +short -t txt default._domainkey.oper.io

That won't return anything for my website as oper.io uses a randomly generated
selector rather than default, which is set in my mail server configs. Were that
to work though, the dig command would return something like...

  $ dig +short -t txt default._domainkey.oper.io
  v=DKIM1; k=rsa; p=MlGfMa03Rw+ZRaxSEjL67BkNZzsX8BPQ5tz11pN\010hivr...

The retrieved public key is then used to validate the from and subject of the
email with the signature in the header.

While this spec isn't great, many major email providers expect it, so it's
probably best to set it up to avoid being sent to spam.


Summary
-------

Hopefully this post helps you if you are one of the rare hoomans who wishes to
brave the perils of running your own email server. Though the task list to set
up a reputable mail server is large at first, maintaining is is relatively
simple once you have it.

If you have any questions or would like me to write more about a particular
topic, send me an email. It likely won't go to spam since my server only
analyzes emails for smamminess based on content, rather than a long list of
complex factors.  :)

[role="datelastedit"]
Last edited: {docdate} {doctime}
