How to send emails with Django via SES with SNS

This article covers an advanced approach of sending emails.

Not only we will setup sending though AWS Simple Email Service, but we'll make dashboards with metrics and put some effort into improving deliverability of your emails.

While you can use this tutorial for any Django project, it is recommended that you follow our Django Project Tutorial for beginners

In this article you will learn how to:

Setup AWS SES Account¶

In order to setup SES sending you need to have AWS account. If you don't have one you should get one here: https://aws.amazon.com/

Go to AWS SES Domains Management page.

Click "Verify a New Domain" button.

This will open a modal window. There you should type your domain name and check "Generate DKIM Settings" checkbox, click "Verify This Domain" button.

Create these records in DNS management for your domain.

In AWS interface you can close the modal window with DNS instructions and you will see that your domain is pending verification.

If you click on the domain name you will see the status of the domain in AWS SES:

At this example they have already checked DKIM records, but still haven't finished checking TXT record.

It can take a while, let's go ahead and set up a Configuration Set.

SES Configuration Set¶

In AWS SES interface click the "Configuration Sets" link.

Click the "Create Configuration Set" button.

A modal window should appear. Enter the name "Engagement" and click "Create Configuration Set" button.

You will see a modal window saying that Configuration Set was created. Click OK.

On the list of Configuration Sets you can see the one we just created. Click on its name.

We need to add a destination, where data from this Configuration Set will be sent.

First we'll setup sending to CloudWatch:

First specify the Name, Value source: Message Tag, name ses:configuration-set and Default Value Engagement.

This allows using this Configuration Set by specifying configuration set in django-ses settings. We'll talk about it later.

Then select events types, all except Rendering Failure.

The "Domain" section will show up.

For radio buttons under "Which domain do you want to use for open and click tracking?" select "Use you own subdomain".

It is an important step because otherwise it will use the default awstrack.me domain and it is blacklisted in a lot of email providers.

Specify o. as the subdomain and select your domain from the dropdown.

If SES hasn't verified your domain yet, then you will have to wait to complete this step.

Click "Save".

The configuration set should look like this now:

Setup SNS to track bounces and complaints¶

Let's go back to our domains list.

https://console.aws.amazon.com/ses/home?region=us-east-1#verified-senders-domain:

As you can see it is verified.

Click on the domain name.

Expand the "Notifications" section.

Click "Edit Configuration" button.

In the modal window, click "Click here to create a new Amazon SNS topic".

We need to create 2 SNS topics. One for bounces, another for complaints.

I suggest that name consists of domain name + purpose.

So for our example here we'll make withfeedback_bounces and withfeedback_complaints.

I use the same values for both "Topic Name" and "Display Name" fields.

When you have both SNS topics created, choose them in dropdowns Bounces and Complaints.

Now you can click "Save Config" button.

You will see your domain page with "Notifications" section filled with SNS topics:

Get credentials for sending emails¶

If you don't use AWS services in your app yet, let's create a user.

In order to do that go to IAM Management Console https://console.aws.amazon.com/iam/home?region=us-east-1#/users$new?step=details

Fill the name, check "Programmatic Access" checkbox and click the button below: "Next: Permissions"

On the permissions page switch to "Attach existing policies directly", search for "ses" and check "AmazonSESFullAccess", click Next.

On the tags page, don't change anything, click "Next: Review".

Review the new user and click "Create User".

On the Success page you will be provided with credentials that you need to put into environment variables for your app.

Send Emails from Django Project with django-post_office vis SES¶

In order to send emails from your Django project you should have django-ses==1.0.3 and django-post-office==3.5.3 installed.

If you don't have them, add to requirements.txt of your project and pip install -r requirements.txt to install everything from your requirements.txt

Now, in settings.py of your project add the following lines. If you were following the tutorial of starting Django Project from scratch, you will find some lines familiar.

Add post_office to your INSTALLED_APPS (or THIRD_PARTY_APPS for those who uses Djangitos template)

Make sure you have EMAIL_BACKEND and POST_OFFICE variables defined as follows:

EMAIL_BACKEND = 'post_office.EmailBackend'

POST_OFFICE = {
    'BACKENDS': {
        'default': 'django_ses.SESBackend',
    },
    'DEFAULT_PRIORITY': 'now',
}

Now finally SES specific settings:

AWS_SES_REGION_NAME = env('AWS_SES_REGION_NAME', default='us-east-1')
AWS_SES_REGION_ENDPOINT = env('AWS_SES_REGION_ENDPOINT', default='email.us-east-1.amazonaws.com')
AWS_SES_CONFIGURATION_SET = env('AWS_SES_CONFIGURATION_SET', default=None)

What also will be required is that you pass environment variables with AWS credentials: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

If you are running Django under AWS Lambda (with Zappa) then you must make sure that these variables are not set in your settings.py, otherwise credentials will be incorrect and your lambda will not have access to AWS services.

In order to use our Configuration Set "Engagement" you should set environment variable AWS_SES_CONFIGURATION_SET with value Engagement. If you don't do it, then links will not be wrapped into tracking domain, open tracking pixel will not be in the email and nothing will appear in the CloudWatch dashboard(more on this later).

Now this should be enough Django to send emails. For example Django AllAuth will send email address verification emails and password reset links.

If emails don't go out, go to Django Admin interface, find "POST OFFICE" -> "Emails" and find the emails there that have failed. Click on the email message and scroll down. At the end of the page you will find logs of attempted send. This should give you idea why sending has failed.

To send an email programmatically let's use example from Django Post Office readme:

from post_office import mail

mail.send(
    'recipient@example.com', # List of email addresses also accepted
    'from@example.com',
    subject='My email',
    message='Hi there!',
    html_message='Hi <strong>there</strong>!',
)

Couple things to add. If you don't specify priority keyword then the priority from settings.py will be used.

One of the benefits of django-post-office is that you can create an email in request/response cycle and send it later asynchronously. In order to do that you should have priority set to low, medium or 'high' and in your celery task or otherwise send it with email_instance.dispatch().

django-post-office also comes with management command and celery tasks for sending queued messages.

For more information about Django Post Office click here: https://pypi.org/project/django-post-office/

Receive bounces and complaints from SNS into the Django¶

Sending emails is great, but having your emails delivered is better.

We already made one good thing – not using standard awstrack.me domain for tracking clicks and opens.

Next thing – we should stop sending emails to addresses we know don't get delivered, or emails addresses that complained about spam from us.

You should pay attention to % of bounces and complaints.

You can check your current stats here https://console.aws.amazon.com/ses/home?region=us-east-1#reputation-dashboard:

Quotes from that link:

We expect our senders' bounce rates to remain below 5%. Senders with a bounce rate exceeding 10% risk a sending Pause.

We expect our senders' complaint rates to remain below 0.1%. Senders with a complaint rate exceeding 0.5% risk a sending Pause.

Let me go over this again, in simple words.

Your app sends an email, for some reason it doesn't get delivered – it is recorded as bounce. It can happen because such email doesn't exist or remote server doesn't want to accept an email from us.

Your app sends an email, but recipient marks it as spam.

If you keep sending those emails again – you can get blacklisted.

This will affect not only you, but Email Service Provider too since you are using their infrastracture. In this case SES.

If you hurt their reputation enough, they will disable your sending capabilities.

Good news, SES will report you about bounces and complaints. They will do it via SNS topics that we created earlier.

What we need to do now is to teach our Django project to receive these notifications, store those bad email addresses and set up SNS to send notifications to our app via HTTPS webhook.

Let's start with an app that will receive and process notifications.

In the root of your project run this command to create a new app:

python manage.py startapp ses_sns

Or, if you are running your app locally via docker-compose:

docker-compose run web python manage.py startapp ses_sns

Add ses_sns to your INSTALLED_APPS (PROJECT_APPS if you are using Djangitos or following our beginner tutorial).

In root urls.py (djangito/urls.py) you need to add a path to urlpatterns list:

from django.urls import path, include


urlpatterns = [
...

    path('sns/', include('ses_sns.urls')),

...
]

In the folder of our new app, create urls.py and put the following code:

# ses_sns/urls.py
from django.urls import path

from django.views.decorators.csrf import csrf_exempt

from ses_sns.views import ReceiveSNSNotification

urlpatterns = [
    path("sns_url", csrf_exempt(ReceiveSNSNotification.as_view()), name="sns_url"),
]

In ses_sns/models.py put the model to store bad email addresses and SNS notifications that app received:

import json
import logging
from collections import namedtuple

import arrow

from django.db import models

from ses_sns.utils import get_permanent_bounced_emails_from_bounce_obj, get_emails_from_complaint_obj

logger = logging.getLogger(__name__)

NOTIFICATION_STATUSES = namedtuple('NOTIFICATION_STATUSES', 'new processed failed')._make(range(3))


class BlacklistedEmail(models.Model):
    email = models.CharField(max_length=255, unique=True)

    def __str__(self):
        return self.email

    class Meta:
        verbose_name = 'Blacklisted Email'
        verbose_name_plural = 'Blacklisted Emails'


class SNSNotification(models.Model):
    """Stores incoming notifications from SNS for later processing of bounces and complaints"""
    STATE_CHOICES = (
        (NOTIFICATION_STATUSES.new, "New"),
        (NOTIFICATION_STATUSES.processed, "Processed"),
        (NOTIFICATION_STATUSES.failed, "Failed"),

    )
    headers = models.JSONField(default=dict, )
    data = models.JSONField(default=dict, )
    added_dt = models.DateTimeField(auto_now_add=True, db_index=True)
    state = models.SmallIntegerField(default=NOTIFICATION_STATUSES.new, choices=STATE_CHOICES, db_index=True)
    last_processed_dt = models.DateTimeField(null=True, blank=True, db_index=True)
    processing_error = models.CharField(max_length=255, blank=True, null=True)

    class Meta:
        """Settings"""
        verbose_name = "SNS Notification"
        verbose_name_plural = "SNS Notifications"

    def __str__(self):
        return str(self.pk)

    def process(self):
        """Attempt to see if this notification is any of use (Bounce or complaint).
        If so - try to find emails and send to GSL in platform via celery task"""
        try:
            if self.data.get('Type') == "Notification":
                message = json.loads(self.data['Message'])
                event_type = message.get('notificationType')
                if event_type == 'Bounce':
                    bounce_obj = message.get('bounce', {})
                    bounced_recipients = get_permanent_bounced_emails_from_bounce_obj(bounce_obj)
                    for email in bounced_recipients:
                        BlacklistedEmail.objects.get_or_create(email=email)
                elif event_type == 'Complaint':
                    complaint_obj = message.get('complaint', {})
                    complaint_emails = get_emails_from_complaint_obj(complaint_obj)
                    for email in complaint_emails:
                        BlacklistedEmail.objects.get_or_create(email=email)
                else:
                    raise ValueError("Wrong type of notification")
            else:
                raise ValueError("Not a Notification")
        except Exception as e:
            logger.debug(f"Processing SNS Notification failed with {e}")
            self.state = NOTIFICATION_STATUSES.failed
            self.processing_error = str(e)
            self.last_processed_dt = arrow.now().datetime
        self.save(update_fields=['state', 'last_processed_dt'])

Create the ses_sns/utils.py with the following code:

def get_permanent_bounced_emails_from_bounce_obj(bounce_obj: dict) -> list:
    """Extracts permanent bounced email addresses only as a list of strings.
    https://docs.aws.amazon.com/ses/latest/DeveloperGuide/notification-contents.html#bounce-object
    """
    bounced_recipients = bounce_obj.get("bouncedRecipients", list())
    return [br.get("emailAddress") for br in bounced_recipients if br.get("status").startswith("5")]


def get_emails_from_complaint_obj(complaint_obj: dict) -> list:
    """Extracts complaint email addresses from complaint_obj
    https://docs.aws.amazon.com/ses/latest/DeveloperGuide/notification-contents.html#complaint-object
    """
    return [cr.get("emailAddress") for cr in complaint_obj.get("complainedRecipients", list())]

Finally, ses_sns/views.py:

import json
import logging
from django.http import HttpRequest, HttpResponse

from django.views.generic.base import View
from rest_framework import status

from ses_sns.models import SNSNotification

logger = logging.getLogger(__name__)


class ReceiveSNSNotification(View):
    """Receives a message from SNS and stores to db"""

    def post(self, request: HttpRequest, *args, **kwargs):
        http_headers = {i[0]: i[1] for i in request.META.items() if i[0].startswith('HTTP_')}
        body_unicode = request.body.decode('utf-8')
        body = json.loads(body_unicode)
        try:
            SNSNotification.objects.create(data=body, headers=http_headers).process()
            return HttpResponse(status=status.HTTP_202_ACCEPTED)
        except Exception as e:
            print(f"Exception during processing SNS Notification {e}")
        return HttpResponse(status=status.HTTP_500_INTERNAL_SERVER_ERROR)

Let's add new models to ses_sns/urls.py:

from django.contrib import admin
from ses_sns.models import SNSNotification, BlacklistedEmail


class SNSNotificationAdmin(admin.ModelAdmin):
    pass


admin.site.register(SNSNotification, SNSNotificationAdmin)


class BlacklistedEmailAdmin(admin.ModelAdmin):
    pass


admin.site.register(BlacklistedEmail, BlacklistedEmailAdmin)

Now what's left is to create migrations for those new models.

python manage.py makemigrations

Or if you are using docker-compose:

docker-compose run web python manage.py makemigrations

Setup SNS HTTPS Subscriptions for bounces and complaints¶

Now in order to notify our app about bounces and complaints we need to make SNS to send HTTPS requests to our app.

Go to Simple Notification Service: https://console.aws.amazon.com/sns/v3/home?region=us-east-1#/dashboard

Click "Create Subscription"

Click on the "Topic ARN" and your existing topics will be listed in the dropdown. Let's start from the bounces one. Select the option having "bounces" in it.

Specify HTTPS as the protocol, since Appliku provides every app with Let's Encrypt certificate.

Endpoint must consist of https:// + domain name + /sns/sns_url

Click the button below "Create Subscription".

You will be taken to the subscription page, where it says "Status: Pending confirmation"

Let's go to our app's admin panel to the SNS Notifications model.

Click on the only notification that there is:

You will see the object that came in the notification. Find the Subscribe URL and copy the link to clipboard and open it in the browser.

And you will get a ConfirmSubscriptionResponse from AWS.

Back at AWS SNS page with our subscription, refresh the page and you will see that subscription is confirmed.

Let's do the same with Complaints.

Go to subscriptions https://console.aws.amazon.com/sns/v3/home?region=us-east-1#/subscriptions

Click "Create Subscription" button.

Select complaints topic

Protocol is HTTPS and Endpoint is the same https:// + domain name + /sns/sns_url

Back to our Django Admin we will have another notification where we need to get Subscribe URL and open it in browser to confirm subscription.

Open URL in browser and get confirmation.

Back to SNS Subscription page we can see subscription is confirmed.

In our tutorial we have no functionality to send emails with.

If you project has already any functionality that sends emails, for exaple registration then you can test that bad emails are being added to the Blacklist.

SES has testing simulator, read more about it here: https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-simulator.html

What we want to test is bounces and complaints so in order to test emails being added to blacklist you can make your app send an email to bounce@simulator.amazonses.com and complaint@simulator.amazonses.com.

Filter email addresses that are know for bounces and complaints¶

So we have educated our app how to receive and store bad email addresses. Now we need to make email sending process aware of this blacklist.

In order to filter email addresses we need to update the sending backend to exclude recipients from To, Cc, Bcc fields if they are in the blacklist. On top of that if there are no email addresses left – skip sending email at all.

Let's create a file ses_sns/backend.py and put our modified version of post_office.backends.EmailBackend here:

from collections import OrderedDict
from email.mime.base import MIMEBase
from django.core.files.base import ContentFile
from post_office import EmailBackend

from post_office.settings import get_default_priority
from ses_sns.models import BlacklistedEmail

def filter_blacklisted_recipients(addresses):
    """Remove blacklisted emails from addresses"""
    if type(addresses) is str:
        addr = parseaddr(addresses)[1]
        if not BlacklistedEmail.objects.filter(email=addr).exists():
            return addresses
        return []
    if type(addresses) is list:
        filtered_addresses = []
        for recipient in addresses:
            addr = parseaddr(recipient)[1]
            logger.debug(f"Address in check for blacklist: {addr}")
            if BlacklistedEmail.objects.filter(email=addr).exists():
                continue
            filtered_addresses.append(recipient)
        return filtered_addresses


class FilteringEmailBackend(EmailBackend):

    def open(self):
        pass

    def close(self):
        pass

    def send_messages(self, email_messages):
        """
        Queue one or more EmailMessage objects and returns the number of
        email messages sent.
        """
        from post_office.mail import create
        from post_office.utils import create_attachments

        if not email_messages:
            return

        for email_message in email_messages:
            subject = email_message.subject
            from_email = email_message.from_email
            headers = email_message.extra_headers
            message = email_message.message()

            # Look for first 'text/plain' and 'text/html' alternative in email
            plaintext_body = html_body = ''
            for part in message.walk():
                if part.get_content_type() == 'text/plain':
                    plaintext_body = part.get_payload()
                    if html_body:
                        break
                if part.get_content_type() == 'text/html':
                    html_body = part.get_payload()
                    if plaintext_body:
                        break

            attachment_files = {}
            for attachment in email_message.attachments:
                if isinstance(attachment, MIMEBase):
                    attachment_files[attachment.get_filename()] = {
                        'file': ContentFile(attachment.get_payload()),
                        'mimetype': attachment.get_content_type(),
                        'headers': OrderedDict(attachment.items()),
                    }
                else:
                    attachment_files[attachment[0]] = ContentFile(attachment[1])
            recipients = filter_blacklisted_recipients(email_message.to)
            cc = filter_blacklisted_recipients(email_message.cc)
            bcc = filter_blacklisted_recipients(email_message.bcc)
            if not len(recipients + cc + bcc):
                continue
            email = create(sender=from_email,
                           recipients=recipients,
                           cc=cc,
                           bcc=bcc,
                           subject=subject,
                           message=plaintext_body,
                           html_message=html_body,
                           headers=headers)

            if attachment_files:
                attachments = create_attachments(attachment_files)

                email.attachments.add(*attachments)

            if get_default_priority() == 'now':
                email.dispatch()

Go to project settings.py and update EMAIL_BACKEND:

EMAIL_BACKEND = 'ses_sns.backend.FilteringEmailBackend'

Now all set. When we send an email its recipients will be checked against the blacklist and bad ones will be removed. If the email message has no more recipients after filtering, then email will not be sent.