Monday, April 4, 2022
HomeIoTThe way to combine AWS IoT Core with Amazon MSK

The way to combine AWS IoT Core with Amazon MSK

Put up by Milo Oostergo, Principal Options Architect and Doron Bleiberg, Senior Answer Architect, AWS Startups


Monitoring IoT gadgets in actual time can present beneficial insights that may make it easier to preserve the reliability, availability, and efficiency of your IoT gadgets. AWS IoT Core gives integrations with Amazon Kinesis Knowledge Streams and Amazon Managed Streaming for Apache Kafka (“Amazon MSK”) to arrange real-time streaming knowledge pipelines. Amazon MSK is in style alternative for patrons who’re aware of Kafka, want infinite message retention, and are searching for the bottom latency. On this weblog publish, we describe learn how to arrange AWS IoT Core to stream occasions to Amazon MSK and customary asks from our prospects.


The diagram beneath illustrates the elements and performance you possibly can construct following this weblog publish or utilizing this pattern AWS CloudFormation template. As a part of this answer, MQTT messages streamed to AWS IoT Core are routed to Amazon Managed Streaming for Apache Kafka (Amazon MSK) utilizing AWS IoT Guidelines actions. Entry to the Amazon MSK cluster is managed utilizing username and password which are securely saved in AWS Secrets and techniques Supervisor and encrypted utilizing AWS Key Administration Service.


Solution overview


Step 1: Organising an Amazon MSK cluster

To ship messages from IoT gadgets to Amazon MSK utilizing AWS IoT Core rule actions, it’s essential allow authentication in your Amazon MSK cluster. IoT rule actions can authenticate along with your Amazon MSK cluster with username and password authentication utilizing the SASL framework or by utilizing TLS consumer authentication by means of AWS Certificates Supervisor. On this weblog publish, we arrange the cluster utilizing SASL/SCRAM authentication methodology. As soon as a cluster is created, you possibly can’t modify the the authentication settings.

To create the Amazon MSK cluster with authentication enabled

  1. From the Amazon MSK console, select Create Cluster.
  2. Choose, enter a cluster identify, and hold the beneficial Apache Kafka model.Amazon MSK create cluster
  3. In Networking, choose your VPC and select “2” for Variety of Availability Zones. From the drop-downs, choose the 2 Availability Zones within the VPC, and select the non-public subnets for every.
  4. In Entry management methodology, select SCRAM/SASL authentication.Selecting the SASL/SCRAM authentication method for your Amazon MSK cluster
  5. Maintain the prevailing defaults and select Create cluster. It takes as much as quarter-hour to create the cluster and the standing is displayed within the Cluster Abstract panel.

Step 2: Create credentials in AWS Secrets and techniques Supervisor  

After the cluster is efficiently created, we create a set of credentials that can be utilized by the IoT rule to attach with the Amazon MSK cluster. The credentials should be saved in AWS Secrets and techniques Supervisor and related to the cluster. Earlier than we create the credentials in AWS Secrets and techniques Supervisor, we first create a customer-managed key in AWS Key Administration Service (KMS). Secrets and techniques encrypted with a AWS managed CMK can’t be used with an Amazon MSK cluster.

  1. Open the AWS KMS console and select Create key.
  2. Select symmetric key and observe the wizard to create the important thing. You don’t must outline the important thing administrative permissions or key utilization permissions at this level. We set this up later.Now that the KMS secret’s created, we are able to retailer the credentials in AWS Secrets and techniques Supervisor.
  1. Open the AWS Secrets and techniques Supervisor console and select Retailer a brand new credential.
  2. Select Different sort of secrets and techniques (e.g. API key) for the key sort.Store your secret in AWS Secrets Manager
  3. Enter the consumer and password knowledge, which should be within the following format:
       "username": "msk",
       "password": "msk-secret"

  4. Choose the shopper managed key you created in earlier step.
  5. To affiliate secrets and techniques with the Amazon MSK cluster, the key identify will need to have the prefix AmazonMSK_. On this instance, we use the identify AmazonMSK_secret.
  6. Report the ARN (Amazon Useful resource Identify) worth to your secret.

Step 3: Affiliate secret with Amazon MSK cluster

As soon as the key is created in AWS Secrets and techniques Supervisor, we are able to affiliate the key with our Amazon MSK cluster.

  1. Return to the Amazon MSK console and choose your cluster.
  2. Select Affiliate secrets and techniques and copy-paste the ARN of the key you created in earlier step.

Associate secret with Amazon MSK cluster

Step 4: Arrange AWS Identification and Entry Administration (IAM) position and coverage for AWS IoT rule

To grant AWS IoT entry to stream knowledge to our Amazon MSK cluster, you could create an IAM position with a coverage that enables entry to the required AWS sources.

To create an IAM position utilizing AWS CLI

  1. Save the next belief coverage doc, which grants AWS IoT permission to imagine the position, to a file named iot-role-trust.json:
          "Effect": "Allow",
          "Principal": {
            "Service": ""
          "Action": "sts:AssumeRole"

  2. Use the create-role command to create an IAM position specifying the iot-role-trust.json file. Be sure to exchange the AWS account id and area iam create-role --role-name IoT-Rule-MSK-Function --assume-role-policy-document file://iot-role-trust.json
  3. Save the next JSON right into a file named iot-msk-policy.json.
             "Useful resource":"*"
             "Useful resource":" ""arn:aws:secretsmanager:area:123456789012:AmazonMSK_*"

    This JSON is an instance coverage doc that gives entry to create and handle elastic community interfaces in your Amazon Digital Non-public Cloud and retrieve the credentials to achieve your Kafka brokers.

  4. Use the create-policy command to outline the actions and sources that AWS IoT Core can entry upon assuming the position, by passing within the iot-msk-policy.json file:aws iam create-policy --policy-name IoT-Rule-MSK-policy --policy-document file://iot-msk-policy.json
  5. Use the attach-role-policy command to connect your coverage and grant AWS IoT entry. Substitute the placeholder ARN by the coverage ARN returned within the earlier iam attach-role-policy --role-name IoT-Rule-MSK-Function --policy-arn "arn:aws:iam::123456789012:coverage/IoT-Rule-MSK-policy"

    To grant the IAM position entry to the KMS key
    As a way to decrypt the key saved in AWS Secrets and techniques Supervisor, we should add the IAM position to the listing of key customers for the Buyer Managed KMS key we earlier created.
    1. Go to the AWS KMS console and choose the KMS key you created within the earlier step.
    2. In Key customers add the IAM position IoT-Rule-MSK-Function.

Step 5 – Create VPC vacation spot for AWS IoT core

Create a vacation spot to your VPC the place Apache Kafka clusters reside. This vacation spot is used to route messages from gadgets to your Amazon MSK cluster.

  1. Go to AWS IoT console, select Act, after which select Locations.
  2. Select Create a VPC vacation spot.
  3. Choose the VPC and identical subnets which are used to your Amazon MSK cluster.
  4. Choose safety group that’s used to your Amazon MSK cluster.
  5. Choose the IoT-Rule-MSK-Function you created within the earlier step.

Step 6 – Create AWS IoT rule

  1. Go to AWS IoT console, select Act, select Guidelines, and create a brand new rule.
  2. In Actions select Add motion and choose Kafka.
  3. Choose the VPC vacation spot you created within the earlier step.
  4. Specify the Kafka subject.
  5. Specify the TLS bootstrap servers of your Amazon MSK cluster. You’ll be able to view the bootstrap server URLs in consumer info of your MSK cluster particulars.View client information to connect to your Amazon MSK cluster
  6. As we arrange our Amazon MSK cluster with the SCRAM SASL authentication methodology, choose SSL_SASL as safety.protocol and choose SCRAM-SHA512 as sasl.mechanism.
  7. Specify the next variable in sasl.scram.username and exchange the identify AmazonMSK_secret with the identify of the key you saved in step 2.${get_secret(‘AmazonMSK_secret', 'SecretString', ‘username’, 'arn:aws:iam::123456789012:position/iot-msk-role)}
  8. Specify the next variable in sasl.scram.password and save the IoT rule motion.${get_secret(‘AmazonMSK_secret', 'SecretString', ‘password, 'arn:aws:iam::123456789012:position/iot-msk-role)}Testing the AWS IoT rule
    At this level, you’ve got created the Amazon MSK cluster and arrange an AWS IoT Core rule with the mandatory IAM permissions. To confirm IoT occasions are streaming to your Amazon MSK cluster, you possibly can join a Kafka client to your bootstrap servers and ship an occasion to your IoT subject utilizing the MQTT check consumer obtainable within the AWS IoT console.Sending a MQTT test messageThe Kafka client linked to your cluster can now obtain messages on the Amazon MSK subject. To study how one can connect with your Amazon MSK cluster, see the part Connecting to your cluster with a username and password within the Amazon MSK developer information.Receiving IoT messages in your Kafka consumerOrganising the permissions incorrectly is a standard situation leading to prospects not receiving occasions on their Amazon MSK cluster.  When AWS IoT is unable to ship occasions, the foundations engine triggers an Error motion. For instance, you possibly can arrange an error motion to ship the occasions to Amazon CloudWatch Logs and specify the CloudWatch log group to which the IoT rule motion sends the info. When an error happens whereas processing your rule, you possibly can view the stream of log occasions within the log group in CloudWatch Logs.

Cleansing up

In case you adopted together with this answer, full the next steps to keep away from incurring undesirable fees to your AWS account.

AWS IoT Core

  • Within the Act part, delete the rule and VPC vacation spot.

Amazon MSK


  • Delete the Buyer Managed Key used to encrypt the secrets and techniques saved in AWS Secrets and techniques Supervisor.

AWS Secrets and techniques Supervisor

  • Delete the key created to authenticate along with your Amazon MSK cluster.


  • Delete the insurance policies and roles created alongside the way in which.

Amazon CloudWatch

  • Delete the related Log teams.


On this publish, we gave you an outline of learn how to construct a real-time streaming knowledge pipeline to your IoT gadgets by integrating AWS IoT Core with Amazon MSK. We confirmed the way you setup Amazon MSK and IoT rule actions to ship messages to Apache Kafka.

In regards to the authors

Milo Oostergo
Milo is a Principal Answer Architect for AWS Startups group in Amsterdam. Earlier than becoming a member of the Startup group, he labored as Principal Product Supervisor on numerous AWS providers.

Doron Bleiberg
Doron is a senior Answer Architect for AWS Startups group in Israel, specializing in AWS IoT providers



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments