Wednesday, May 11, 2022
HomeBig DataAnalyze Amazon SES occasions at scale utilizing Amazon Redshift

Analyze Amazon SES occasions at scale utilizing Amazon Redshift

E mail is likely one of the most necessary strategies for enterprise communication throughout many organizations. It’s additionally one of many major strategies for a lot of companies to speak with their prospects. With the ever-increasing necessity to ship emails at scale, monitoring and evaluation has develop into a significant problem.

Amazon Easy E mail Service (Amazon SES) is a cheap, versatile, and scalable electronic mail service that lets you ship and obtain emails out of your functions. You should utilize Amazon SES for a number of use circumstances, reminiscent of transactional, advertising and marketing, or mass electronic mail communications.

An necessary advantage of Amazon SES is its native integration with different AWS providers, reminiscent of Amazon CloudWatch and Amazon Redshift, which lets you monitor and analyze your emails sending at scale seamlessly. You’ll be able to retailer your electronic mail occasions in Amazon Redshift, which is a extensively used, quick, and absolutely managed cloud information warehouse. You’ll be able to then analyze these occasions utilizing SQL to realize enterprise insights reminiscent of advertising and marketing marketing campaign success, electronic mail bounces, complaints, and so forth.

On this submit, you’ll discover ways to implement an end-to-end resolution to automate this electronic mail evaluation and monitoring course of.

Answer overview

The next structure diagram highlights the end-to-end resolution, which you’ll provision mechanically with an AWS CloudFormation template.

On this resolution, you publish Amazon SES electronic mail occasions to an Amazon Kinesis Information Firehose supply stream that publishes information to Amazon Redshift. You then hook up with the Amazon Redshift database and use a SQL question device to investigate Amazon SES electronic mail occasions that meet the given standards. We use the Amazon Redshift SUPER information sort to retailer the occasion (JSON information) in Amazon Redshift. The SUPER information sort handles semi-structured information, which might have various desk attributes and kinds.

The alarm system makes use of Amazon CloudWatch logs that Kinesis Information Firehose generates when an information load to Amazon Redshift fails. We’ve got arrange a metric filter that sample matches the CloudWatch log occasions to find out the error situation and triggers a CloudWatch alarm. This in flip sends out electronic mail notifications utilizing Amazon Easy Notification Service (Amazon SNS).


As a prerequisite for deploying the answer on this submit, you want to arrange Amazon SES in your account. For extra data, see Getting Began with Amazon Easy E mail Service.

Answer sources and options

The structure constructed by AWS CloudFormation helps AWS greatest practices for top availability and safety. The CloudFormation template takes care of the next key sources and options:

  • Amazon Redshift cluster – An Amazon Redshift cluster with encryption at relaxation enabled utilizing an AWS Key Administration Service (AWS KMS) buyer managed key (CMK). This cluster acts because the vacation spot for Kinesis Information Firehose and shops all of the Amazon SES electronic mail sending occasions within the desk ses, as proven within the following screenshot.
  • Kinesis Information Firehose configuration – A Kinesis Information Firehose supply stream that acts because the occasion vacation spot for all Amazon SES electronic mail sending metrics. The supply stream is about up with Amazon Redshift because the vacation spot. Server-side encryption is enabled utilizing an AWS KMS CMK, and vacation spot error logging has been enabled as per greatest practices.
  • Amazon SES configuration – A configuration set in Amazon SES that’s used to map Kinesis Information Firehose because the occasion vacation spot to publish electronic mail metrics.

To make use of the configuration set when sending emails, you possibly can specify a default configuration set on your verified identification, or embrace a reference to the configuration set within the headers of the e-mail.

  • Exploring and analyzing the information – We use Amazon Redshift question editor v2 for exploring and analyzing the information.
  • Alarms and notifications for ingestion failures – A knowledge load error notification system utilizing CloudWatch and Amazon SNS generates email-based notifications within the occasion of a failure throughout information load from Kinesis Information Firehose to Amazon Redshift. The setup creates a CloudWatch log metric filter, as proven within the following screenshot.

A CloudWatch alarm based mostly on the metric filter triggers an SNS notification when in alarm state. For extra data, see Utilizing Amazon CloudWatch alarms.

Deploy the CloudFormation template

The offered CloudFormation template mechanically creates all of the required sources for this resolution in your AWS account. For extra data, see Getting began with AWS CloudFormation.

  1. Sign up to the AWS Administration Console.
  2. Select Launch Stack to launch AWS CloudFormation in your AWS account:
  3. For Stack title, enter a significant title for the stack, for instance, ses_events.
  4. Present the next values for the stack parameters:
    1. ClusterName – The title of the Amazon Redshift cluster.
    2. DatabaseName – The title of the primary database to be created when the Amazon Redshift cluster is created.
    3. DeliveryStreamName – The title of the Firehose supply stream.
    4. MasterUsername – The consumer title that’s related to the first consumer account for the Amazon Redshift cluster.
    5. NodeType – The kind of node to be provisioned. (Default dc2.massive)
    6. NotificationEmailId – The e-mail notification listing that’s used to configure an SNS subject for sending CloudWatch alarm and occasion notifications.
    7. NumberofNodes – The variety of compute nodes within the Amazon Redshift cluster. For multi-node clusters, the NumberofNodes parameter have to be larger than 1.
    8. OnPremisesCIDR – IP vary (CIDR notation) on your current infrastructure to entry the goal and reproduction Amazon Redshift clusters.
    9. SESConfigSetName – Identify of the Amazon SES configuration set.
    10. SubnetId – Subnet ID the place supply Amazon Redshift cluster is created.
    11. Vpc – VPC by which Amazon Redshift cluster is launched.
  5. Select Subsequent.
  6. Overview all the data and choose I acknowledge that AWS CloudFormation may create IAM sources.
  7. Select Create stack.

You’ll be able to observe the progress of the stack creation on the Occasions tab. Anticipate the stack to finish and present the standing CREATE_COMPLETE.

Take a look at the answer

To ship a take a look at electronic mail, we use the Amazon SES mailbox simulator. Set the configuration-set header to the one created by the CloudFormation template.

We use the Amazon Redshift question editor V2 to question the Amazon Redshift desk (created by the CloudFormation template) and see if the occasions have proven up.

If the information load of the occasion stream fails from Kinesis Information Firehose to Amazon Redshift, the failure notification system is triggered, and also you obtain an electronic mail notification through Amazon SNS.

Clear up

A number of the AWS sources deployed by the CloudFormation stacks on this submit incur a price so long as you proceed to make use of them.

You’ll be able to delete the CloudFormation stack to delete all AWS sources created by the stack. To scrub up all of your stacks, use the AWS CloudFormation console to take away the stacks that you just created in reverse order.

  1. On the Stacks web page on the AWS CloudFormation console, select the stack to delete.
  2. Within the stack particulars pane, select Delete.
  3. Select Delete stack when prompted.

After stack deletion begins, you possibly can’t cease it. The stack proceeds to the DELETE_IN_PROGRESS state. When the stack deletion is full, the stack modifications to the DELETE_COMPLETE state. The AWS CloudFormation console doesn’t show stacks within the DELETE_COMPLETE state by default. To show deleted stacks, you could change the stack view filter. For extra data, see Viewing deleted stacks on the AWS CloudFormation console.

If the delete fails, the stack enters the DELETE_FAILED state. For options, see Delete stack fails.


On this submit, we walked by way of the method of organising Amazon SES and Amazon Redshift to deploy an electronic mail reporting service that may scale to assist tens of millions of occasions. We used Amazon Redshift to retailer semi-structured messages utilizing the SUPER information sort in database tables to assist various message sizes and codecs. With this resolution, you possibly can simply run analytics at scale and analyze your electronic mail occasion information for deliverability-related points reminiscent of bounces or complaints.

Use the CloudFormation template offered to hurry up provisioning of the cloud sources required for the answer (Amazon SES, Kinesis Information Firehose, and Amazon Redshift) in your account whereas following safety greatest practices. Then you possibly can analyze Amazon SES occasions at scale utilizing Amazon Redshift.

In regards to the Authors

Manash Deb is a Software program Improvement Engineer within the AWS Listing Service group. He has labored on constructing end-to-end functions in numerous database and applied sciences for over 15 years. He likes to study new applied sciences and fixing, automating, and simplifying buyer issues on AWS.

Arnab Ghosh is a Options Architect for AWS in North America serving to enterprise prospects construct resilient and cost-efficient architectures. He has over 13 years of expertise in architecting, designing, and creating enterprise functions fixing advanced enterprise issues.

Sanjoy Thanneer is a Sr. Technical Account Supervisor with AWS based mostly out of New York. He has over 20 years of expertise working in Database and Analytics Domains.  He’s obsessed with serving to enterprise prospects construct scalable , resilient and price environment friendly Functions.

Justin Morris is a E mail Deliverability Supervisor for the Easy E mail Service group. With over 10 years of expertise within the IT business, he has developed a pure expertise for diagnosing and resolving buyer points and repeatedly seems to be for progress alternatives to study new applied sciences and providers.



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments