This is the third post in a series for developers working on email-enabled applications. View Part 1 and Part 2.

In the first part of this series, we covered the components and protocol that comprise email. In the second post, we explored the anatomy of an email message, MIME, and send email using SMTP.

In this post, let’s cover advanced topics of SMTP. In future posts, we’ll look at retrieving and processing messages using POP3 and IMAP.

7. Anti-Spam Controls

When email was designed in the 1970s, security and authenticity weren’t top of mind. It didn’t take long for someone to co-opt the medium to send unsolicited junk called “spam”. The first spam dates all the way back to 1978. Since then, spam has grown into such a problem that it accounts for up to 95% of all messages.

Over the years, multiple initiatives have been tested, including radical solutions like charging a fraction of a penny per message. Since around 2000, the following anti-spam technologies have become standard:

1. Sender Policy Framework (SPF) is a simple validation system that has recipients confirm that senders used an authorized server to send the message from their domain. It’s one of the simplest safeguards to implement, consisting of a DNS TXT record listing authorized IP ranges to send on behalf of the domain. Best of all, the change can be implemented once by an administrator and requires no changes on individual email clients.
2. DomainKeys Identified Mail (DKIM) is similar to SPF, but adds assymetric cryptography to authenticate that the sender used an authorized server for its domain. A special header called “DKIM-Signature” is computed by the sending SMTP server, hashing the message and headers using the recipient domain’s public key. DKIM is an effective tool when whitelisting, blacklisting, and validating senders. It requires no change on clients, but does take special configuration on the sending SMTP server and DNS records.
3. Domain-based Message Authentication, Reporting & Conformance (DMARC) is an anti-spoofing technique built on top of SPF and DKIM. It is a newer standard that is not widely adopted yet. Its implementation is a superset of SPF and DKIM.
4. DNS-based Blackhole List (DNSBL) or Real-time Blackhole List (RBL) is an effort to stop spamming by publicly cataloging a list of the worst offenders. These tend to be controversial and have become less necessary as SPF, DKIM, and machine learning have improved.

The good news is that all of these protections are handled by your outbound SMTP server and validated by the recipient’s inbound server. No special configurations are usually needed on email clients.

To ensure that your message is not errantly blocked, we recommend always including the standard headers (From, To, Subject, Content-Type, Content-Transfer-Encoding, and Date).

8. Email Encryption

Email is an inherently transparent protocol, partially by design. Messages are transmitted in plaintext and not encrypted by default. This is why it’s always important to run STARTSSL and ensure proper encryption between all computers in your mailflow.

One of the benefits of having email unencrypted is that servers can read, search, and learn from messages. Instead of downloading all of your messages locally and running a full-text indexer in order to search for a keyword, your IMAP server can handle that for you. Moreover, the server can de-duplicate large messages on distribution lists instead of storing hundreds or thousands of large redundant files. Service providers can also learn from all of the inbound mail to detect spam, malware, and other problematic messages to block them.

That said, most of us would prefer our communications stay private. Because of that, there are three email privacy standards. Each has major usability challenges and none are widely adopted, but they make sense for limited use cases.

1. Secure MIME (S/MIME) is a method for securing email based on Public Key Infrastructure (PKI). It relies on each mailbox having a public key which is shared with all potential senders. The corresponding private key is then secured in individual email clients, allowing only the recipient to decrypt messages. S/MIME messages contain additional MIME body parts for encryption and/or signing. When sending to multiple recipients, the sender needs to have each recipient’s public key or else the message cannot be sent securely. S/MIME sounds good in theory, but rarely works in practice due to the difficulty of sharing keys, costs in renewing keys, and inability to search messages without decrypting them first.
2. Pretty Good Privacy (PGP) is similar to S/MIME in that it relies on public and private keys for securing messages. Instead of an hierarchical PKI involving certificate authorities, PGP utilizes a “web of trust” where users vouch for each other based on signing each others’ keys. PGP is less rigid than S/MIME in that certificates are self-managed and don’t expire within two years. PGP is available in many open source projects, but has not caught on in commercial applications. It faces the same usability and distribution challenges of S/MIME.
3. Dark Mail is an ongoing initiative to replace SMTP and IMAP with always-encrypted alternatives called Dark Mail Transfer Protocol (DMTP) and Dark Mail Access Protocol (DMAP). The project was crowdfunded in 2013 by founders of Silent Circle and Lavabit, largely in response to the Edward Snowden PRISM leaks. The project is ambitious and backed by proven technologists, but is now on its fourth year with no production release. Keep an eye out for it.

Email encryption is not for everyone. S/MIME is most practical in organizations hosting their own certificate authority and email servers, which can automate key creation and distribution. PGP may be useful for tightknit groups who rely on confidentiality. Dark Mail is entirely experimental as of 2016.

If you choose to add encryption to your email application, OpaqueMail may be a big timesaver. It attempts to abstract the complexities involved with certificates, encryption, signing, and validation.

Check back for future posts about processing messages using POP3 and IMAP.