Private MSK Proxy

Private MSK Proxy

Estimated time to complete 20-30 mins.

Overview

The Aklivity Private MSK Proxyopen in new window enables cross-account connectivity between Kafka clients and Amazon MSK clusters.

Bundled CloudFormation templates provide automated configuration of a VPC Endpoint Service for your Amazon MSK cluster as well as corresponding VPC Endpoints. These VPC endpoints enable secure access to your MSK cluster via AWS PrivateLinkopen in new window from an outside VPC, even if that VPC is owned by a different AWS account. Authorized Kafka clients in each consuming VPC will be able to connect, publish messages and subscribe to topics in your Amazon MSK cluster.

In this guide we will deploy the Aklivity Private MSK Proxy and showcase cross VPC connectivity between an MSK cluster and a Kafka client.

The following AWS services are used by Aklivity Private MSK Proxyopen in new window for this deployment.

ServiceRequiredUsageQuota
Marketplace MeteringYesStartup, then hourlyNoneopen in new window
Resource Groups and TaggingYesStartup onlyNoneopen in new window
Secrets ManagerYesStartup onlyNot reachedopen in new window
Certificate ManagerNo

Private key and certificate can be inlined in Secrets Manager instead		Startup onlyNot reachedopen in new window
Private Certificate ManagerNo

Private key and certificate can be inlined in Secrets Manager instead		Startup onlyNot reachedopen in new window

The default AWS Service Quotas are sufficient.

Info

Check out the Troubleshooting guide if you run into any issues.

Prerequisites

Before setting up cross-VPC access to your MSK Cluster, you will need the following:

  • an MSK Cluster configured for TLS encrypted client access
  • an VPC security group for MSK Proxy instances
  • an IAM security role for MSK Proxy instances
  • subscription to Aklivity Private MSK Proxy via AWS Marketplace

Create MSK Cluster

We need to create an MSK cluster in preparation for secure remote access outside the VPC. You can skip this step if you have already created an MSK cluster with equivalent configuration.

Follow the Create VPC guide to create a VPC for your MSK cluster with the following parameters.

Name tag: my-msk-cluster
IPv4 CIDR block: 10.0.0.0/16
Region: us-east-1

Then follow the Create MSK Cluster guide to create your MSK cluster with the following parameters.

Name: aklivity
VPC: my-msk-cluster
Subnets: my-msk-cluster-1a my-msk-cluster-1b my-msk-cluster-1c

Tips

This creates your MSK cluster in preparation for secure access outside the VPC.

Create the MSK Proxy security group

We need to create a VPC security group that will be used by the Private MSK Proxy instances when they are launched.

Follow the Create Security Group guide with the following parameters to create a security group in the same VPC as your MSK cluster.

VPC: my-msk-cluster
Name: my-msk-proxy
Description: Kafka clients and SSH access

Inbound Rule

Type: TCP 9094
Source: <Any IPv4>

Inbound Rule

Type: SSH
Source: <My IP>

Tips

This creates your Private MSK proxy security group to allow Kafka clients and SSH access.

Update your MSK Cluster security group rules

Follow the Update Security Group guide with the following parameters to allow the MSK Proxy instances to communicate with the MSK cluster.

VPC: vpc-xxx (my-msk-cluster)
Security Group: default (MSK security group)

Inbound Rule

Type: TCP 9094
Source: Custom Security groups: my-msk-proxy

Tips

This allows the MSK Proxy instances to access your MSK cluster.

Create the MSK Proxy IAM security role

Follow the Create IAM Role guide to create an IAM security role with the following parameters:

Name: aklivity-private-msk-proxy

Managed Policy

Name: AWSMarketplaceMeteringFullAccess

Tips

This creates an IAM security role to enable the required AWS services for the MSK Proxy instances.

Subscribe via AWS Marketplace

The Aklivity Private MSK Proxy is availableopen in new window through the AWS Marketplace. You can skip this step if you have already subscribed to Aklivity Private MSK Proxy via AWS Marketplace.

To get started, visit the Proxy's Marketplace Product Pageopen in new window and Subscribe to the offering.

Info

You should now see Aklivity Private MSK Proxy listed in your AWS Marketplace Subscriptionsopen in new window.

Create the VPC Endpoint Service

Navigate to your AWS Marketplace Subscriptionsopen in new window and select Aklivity Private MSK Proxy to show the details page. Then select Launch CloudFormation stack from the Actions menu in the Agreement section.

Make sure you have selected the desired region, such as US East (N. Virginia) us-east-1, and then click Continue to Launch. Choose the action Launch CloudFormation, then click Launch to complete the Create stack wizard with the following details:

Step 1. Specify template

Prepare template: Template is ready
Specify template: (auto-filled)

Step 2. Specify stack details

Stack name: my-msk-endpoint-service

Parameters

Network Configuration

VPC: my-msk-cluster
Subnets: my-msk-cluster-1a my-msk-cluster-1b my-msk-cluster-1c

MSK Configuration

Wildcard DNS pattern [1]: *.aklivity.[...].amazonaws.com
Port number: 9094

MSK Proxy Configuration

Instance count: 2
Instance type [2]: t3.small
Role: aklivity-private-msk-proxy
Security Groups: my-msk-proxy
Key pair for SSH access: <key pair>

Step3. Configure stack options: (defaults)

Step4. Review: (review)

[1] Follow the Lookup MSK Server Names guide to discover the wildcard DNS pattern for your MSK cluster.

[2] Consider the network throughput characteristics of the AWS instance type as that will impact the upper bound on network performance.

Click Create Stack.

Tips

This initiates creation of a VPC Endpoint Service using the Aklivity Private MSK Proxy stack via CloudFormation.

Info

When your VPC Endpoint Service is ready, the CloudFormation consoleopen in new window will show CREATE_COMPLETE for the newly created stack.

Verify Private MSK Proxy Service

Navigate to the EC2 Management Consoleopen in new window and make sure you have selected the desired region in the upper right corner, such as US East (N. Virginia) us-east-1.

Under the Resources by Region section, select the Instances resource box to show your Instances. Select either of the Private MSK Proxy instances launched by the CloudFormation template to show the details.

Info

They each have the IAM Role name aklivity-private-msk-proxy.

Find the Public IPv4 Address and then SSH into the instance.

ssh -i ~/.ssh/<key-pair.cer> ec2-user@<instance-public-ip-address>

After logging in via SSH, check the status of the msk-proxy system service.

systemctl status msk-proxy.service

Verify that the msk-proxy service is active and logging output similar to that shown below.

● msk-proxy.service - MSK Proxy
   Loaded: loaded (/etc/systemd/system/msk-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-08-24 20:56:51 UTC; 1 day 19h ago
 Main PID: 1803 (java)
   CGroup: /system.slice/msk-proxy.service
           └─...

Aug 26 06:56:54 ip-10-0-3-104.ec2.internal zilla[1803]: Recorded usage for record id ...

Info

Repeat these steps for each of the other Private MSK Proxy instances launched by the CloudFormation template.

Create the MSK VPC Endpoint

We can now create a VPC Endpoint to access your MSK cluster from Kafka clients in a different VPC via the newly created VPC Endpoint Service.

Create the VPC

Follow the Create VPC guide to create a VPC for your Kafka clients with the following parameters.

Name tag: my-msk-client
IPv4 CIDR block: 10.1.0.0/16
Region: us-east-1

Tips

This creates your client VPC in preparation for secure cross-VPC access to your MSK cluster.

Enable DNS Hostnames

Navigate to the VPC Management Consoleopen in new window and make sure you have selected the desired region in the upper right corner, such as US East (N. Virginia) us-east-1.

Under the Resources by Region section, select the VPCs resource box to show Your VPCs. Then select the VPC named my-msk-client to show the details.

Choose Edit DNS hostnames from the Actions menu and fill out the Edit DNS hostnames form as follows:

DNS hostnames: Enable
This is required to let the Kafka clients properly resolve your MSK cluster bootstrap server names.

Warning

Make sure to enable DNS Hostnames for this VPC.

Create the VPC Endpoint

Navigate to the CloudFormation Management Consoleopen in new window and click the Create Stack button to complete the Create stack wizard with the following details:

Step 1. Specify template

Prepare template: Template is ready

Specify template

Template source: Amazon S3 URL
Amazon S3 URL: https://s3.amazonaws.com/marketplace.aklivity.io/private-msk-proxy/PrivateMskEndpoint.templateopen in new window

Step 2. Specify stack details

Stack name: my-msk-endpoint

Parameters

Network Configuration

VPC: my-msk-client
Subnets: my-msk-client-1a my-msk-client-1b my-msk-client-1c

Endpoint Configuration

Endpoint Service Name [1]: com.amazonaws.vpce.[...]
Wildcard DNS pattern [2]: *.aklivity.[...].amazonaws.com\

Step3. Configure stack options: (defaults)\

Step4. Review: (review)

[1] Locate the Endpoint Service Name from the Outputs tab of the my-msk-endpoint-service stack previously created in the same VPC as your MSK Cluster.

[2] Follow the Lookup MSK Server Names guide to discover the wildcard DNS pattern for your MSK cluster.

Click Create Stack.

Tips

This initiates creation of a VPC Endpoint using the CloudFormation stack.

Info

When your VPC Endpoint is ready, the CloudFormation consoleopen in new window will show CREATE_COMPLETE for the newly created stack.

Verify Kafka Client Connectivity

Now we must prepare a Kafka client running in the client VPC to verify connectivity to your MSK cluster via the VPC Endpoint.

Launch the EC2 Instance

Follow the Launch EC2 Instance guide with the following parameters to launch an EC2 instance with remote SSH access and login to the instance via SSH.

VPC: my-msk-client
Region: us-east-1

Instance

AMI: Amazon Linux 2 AMI (HVM), SSD Volume Type, 64-bit (x86)

Type: t3.small

Tips

This launches an EC2 instance that you can access remotely via SSH.

Install the Kafka Client

First, we must install a Java runtime that can be used by the Kafka client.

After logging into the instance via SSH, run the following command:

sudo yum install java-1.8.0

Now we are ready to install the Kafka client:

wget https://archive.apache.org/dist/kafka/2.8.0/kafka_2.13-2.8.0.tgz
tar -xzf kafka_2.13-2.8.0.tgz
cd kafka_2.13-2.8.0

After changing the directory to kafka_2.13-2.8.0 we must copy the Kafka clients trustore:

cp /usr/lib/jvm/<JDKFolder>/lib/security/cacerts /tmp/kafka.client.truststore.jks

You can get the value for **** <JDKFolder>**** by typing

cp /usr/lib/jvm/j **double tap TAB**

and the selecting the longer entry that starts with "jre-1.8.0-openjdk...."

A sample full copy command command of a Kafka client's trustore will appear as follows:

cp /usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.302.b08-0.amzn2.0.1.x86_64/lib/security/cacerts /tmp/kafka.client.truststore.jks

Configure the Kafka Client

With the Kaka client now installed we are ready to configure it and point it at the MSK VPC Endpoint.

The MSK Proxy relies on TLS so we need to create a file called client.properties that tells the Kafka client to use SSL and were the truststore is:

security.protocol=SSL
ssl.truststore.location=/tmp/kafka.client.truststore.jks

Tips

Use the Vim to create client.properties.Type vi client.properties in the SSH terminal, paste the above contents, then enter the:wq command to save the file..

Test the Kafka Client

We can now verify that the Kafka client can successfully communicate with your MSK cluster via the VPC Endpoint in a different VPC to create a topic, then publish and subscribe to the same topic.

Follow the Lookup MSK Server Names guide to find the TLS bootstrap server names for your MSK cluster.

Create a Topic

Use the Kafka client to create a topic called vpce-test, replacing<tls-bootstrap-server-names> **** in the command below with the TLS bootstrap server names of your MSK cluster:

bin/kafka-topics.sh --create --topic vpce-test --partitions 3 --replication-factor 3 --command-config client.properties --bootstrap-server <tls-bootstrap-server-names>

A quick summary of what just happened:

  1. The Kafka client in the my-msk-client VPC issued a request to create a new topic
  2. This request was directed to the MSK VPC Endpoint
  3. The request was then relayed across PrivateLink to the MSK VPC Endpoint Service in the my-msk-cluster VPC
  4. The Endpoint Service forwarded the request to the Network Load Balancer
  5. The Network Load Balancer forwarded the request to the Aklivity MSK Private Proxy
  6. The Aklivity MSK Private Proxy look at the info in the request and relayed it through to the appropriate MSK broker
  7. The topic was created in the MSK broker
  8. Cross VPC connectivity was verified

Publish messages

Publish two messages to the newly created topic via the following producer command:

bin/kafka-console-producer.sh --topic vpce-test --producer.config client.properties --broker-list <tls-bootstrap-server-names>

A prompt will appear for you to type in the messages:

>This is my first event
>This is my second event

Receive messages

Read these messages back via the following consumer command:

bin/kafka-console-consumer.sh --topic vpce-test --from-beginning --consumer.config client.properties --bootstrap-server <tls-bootstrap-server-names>

You should see the This is my first event and This is my second event messages.

This is my first event
This is my second event

Tips

This verifies cross-VPC connectivity to your MSK cluster via Aklivity Private MSK Proxyopen in new window!

Monitor the VPC Endpoint Service

The CloudFormation template used to deploy the VPC Endpoint Service includes a Network Load Balancer that can be monitored via CloudWatchopen in new window to verify continuous health.

Network Load Balancers have many available metricsopen in new window, including the following.

MetricDescription
TCP_Target_Reset_CountThe total number of reset (RST) packets sent from a target to a client. These resets are generated by the target and forwarded by the load balancer.
UnHealthyHostCountThe number of targets that are considered unhealthy.

You can use CloudWatchopen in new window to create a dashboard to monitor these metrics and set alarms to alert you when specific metric thresholds are reached.

Upgrade the VPC Endpoint Service

Navigate to your AWS Marketplace Subscriptionsopen in new window and select Aklivity Private MSK Proxy to show the details page. Then select Launch CloudFormation stack from the Actions menu in the Agreement section.

Make sure you have selected the desired region, such as US East (N. Virginia) us-east-1, and then click Continue to Launch. Choose the action Launch CloudFormation, then click Launch to show the URL of the CloudFormation template.

Copy the CloudFormation template Amazon S3 URL and then select your existing CloudFormation Stack from a previous deployment of Aklivity Private MSK Proxy. Click Update and Replace current template with the copied Amazon S3 URL. Then complete the wizard to deploy the updated stack.

CloudFormation will incrementally deploy the MSK Proxy instances for the new version behind the same Network Load Balancer, checking for successful deployment before terminating the MSK Proxy instances for the previous version.

Connected clients will see their connections drop, and when they reconnect automatically, the Network Load Balancer will direct them to the new MSK Proxy instances. If the update is unsuccessful, then CloudFormation will rollback to use the previous stack deployment.

Last update: