Private MSK Proxy
Securely access an Amazon MSK cluster across VPCs via AWS PrivateLink.
Estimated time to complete 20-30 mins.

Overview

The Aklivity Private MSK Proxy enables cross-account connectivity between Kafka clients and Amazon MSK clusters.
Bundled CloudFormation templates provide automated configuration of a VPC Endpoint Service for your Amazon MSK cluster as well as corresponding VPC Endpoints. These VPC endpoints enable secure access to your MSK cluster via AWS PrivateLink from an outside VPC, even if that VPC is owned by a different AWS account. Authorized Kafka clients in each consuming VPC will be able to connect, publish messages and subscribe to topics in your Amazon MSK cluster.
In this guide we will deploy the Aklivity Private MSK Proxy and showcase cross VPC connectivity between an MSK cluster and a Kafka client.
The following AWS services are used by Aklivity Private MSK Proxy for this deployment.
Service
Required
Usage
Quota
Marketplace Metering
Yes
Startup, then hourly
None
Resource Groups and Tagging
Yes
Startup only
None
Secrets Manager
Yes
Startup only
Certificate Manager
No Private key and certificate can be inlined in Secrets Manager instead
Startup only
Private Certificate Manager
No Private key and certificate can be inlined in Secrets Manager instead
Startup only
The default AWS Service Quotas are sufficient.
Check out the Troubleshooting guide if you run into any issues.

Prerequisites

Before setting up cross-VPC access to your MSK Cluster, you will need the following:
  • an MSK Cluster configured for TLS encrypted client access
  • an VPC security group for MSK Proxy instances
  • an IAM security role for MSK Proxy instances
  • subscription to Aklivity Private MSK Proxy via AWS Marketplace

Create MSK Cluster

We need to create an MSK cluster in preparation for secure remote access outside the VPC. You can skip this step if you have already created an MSK cluster with equivalent configuration.
Follow the Create VPC guide to create a VPC for your MSK cluster with the following parameters.
Name tag
my-msk-cluster
IPv4 CIDR block
10.0.0.0/16
Region
us-east-1
Then follow the Create MSK Cluster guide to create your MSK cluster with the following parameters.
Name
aklivity
VPC
my-msk-cluster
Subnets
my-msk-cluster-1a
my-msk-cluster-1b
my-msk-cluster-1c
This creates your MSK cluster in preparation for secure access outside the VPC.

Create the MSK Proxy security group

We need to create a VPC security group that will be used by the Private MSK Proxy instances when they are launched.
Follow the Create Security Group guide with the following parameters to create a security group in the same VPC as your MSK cluster.
VPC
my-msk-cluster
Name
my-msk-proxy
Description
Kafka clients and SSH access
Inbound Rule
Type
TCP 9094
Source
<Any IPv4>
Inbound Rule
Type
SSH
Source
<My IP>
This creates your Private MSK proxy security group to allow Kafka clients and SSH access.

Update your MSK Cluster security group rules

Follow the Update Security Group guide with the following parameters to allow the MSK Proxy instances to communicate with the MSK cluster.
VPC
vpc-xxx (my-msk-cluster)
Security Group
default (MSK security group)
Inbound Rule
Type
TCP 9094
Source
Custom
Security groups:
my-msk-proxy
This allows the MSK Proxy instances to access your MSK cluster.

Create the MSK Proxy IAM security role

Follow the Create IAM Role guide to create an IAM security role with the following parameters:
Name
aklivity-private-msk-proxy
Managed Policy
Name
AWSMarketplaceMeteringFullAccess
This creates an IAM security role to enable the required AWS services for the MSK Proxy instances.

Subscribe via AWS Marketplace

The Aklivity Private MSK Proxy is available through the AWS Marketplace. You can skip this step if you have already subscribed to Aklivity Private MSK Proxy via AWS Marketplace.
To get started, visit the Proxy's Marketplace Product Page and Subscribe to the offering.
You should now see Aklivity Private MSK Proxy listed in your AWS Marketplace Subscriptions.

Create the VPC Endpoint Service

Navigate to your AWS Marketplace Subscriptions and select Aklivity Private MSK Proxy to show the details page. Then select Launch CloudFormation stack from the Actions menu in the Agreement section.
Make sure you have selected the desired region, such as US East (N. Virginia) us-east-1, and then click Continue to Launch. Choose the action Launch CloudFormation, then click Launch to complete the Create stack wizard with the following details:
Step 1. Specify template
Prepare template
Template is ready
Specify template
(auto-filled)
Step 2. Specify stack details
Stack name
my-msk-endpoint-service
Parameters
Network Configuration
VPC
my-msk-cluster
Subnets
my-msk-cluster-1a
my-msk-cluster-1b
my-msk-cluster-1c
MSK Configuration
Wildcard DNS pattern [1]
*.aklivity.[...].amazonaws.com
Port number
9094
MSK Proxy Configuration
Instance count
2
Instance type [2]
t3.small
Role
aklivity-private-msk-proxy
Security Groups
my-msk-proxy
Key pair for SSH access
<key pair>
Step 3. Configure stack options
(defaults)
Step 4. Review
(review)
[1] Follow the Lookup MSK Server Names guide to discover the wildcard DNS pattern for your MSK cluster.
[2] Consider the network throughput characteristics of the AWS instance type as that will impact the upper bound on network performance.
Click Create Stack.
This initiates creation of a VPC Endpoint Service using the Aklivity Private MSK Proxy stack via CloudFormation.
When your VPC Endpoint Service is ready, the CloudFormation console will show CREATE_COMPLETE for the newly created stack.

Verify Private MSK Proxy Service

Navigate to the EC2 Management Console and make sure you have selected the desired region in the upper right corner, such as US East (N. Virginia) us-east-1.
Under the Resources by Region section, select the Instances resource box to show your Instances. Select either of the Private MSK Proxy instances launched by the CloudFormation template to show the details.
They each have the IAM Role name aklivity-private-msk-proxy.
Find the Public IPv4 Address and then SSH into the instance.
ssh -i ~/.ssh/<key-pair.cer> [email protected]<instance-public-ip-address>
After logging in via SSH, check the status of the msk-proxy system service.
systemctl status msk-proxy.service
Verify that the msk-proxy service is active and logging output similar to that shown below.
● msk-proxy.service - MSK Proxy
Loaded: loaded (/etc/systemd/system/msk-proxy.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2021-08-24 20:56:51 UTC; 1 day 19h ago
Main PID: 1803 (java)
CGroup: /system.slice/msk-proxy.service
└─...
Aug 26 06:56:54 ip-10-0-3-104.ec2.internal zilla[1803]: Recorded usage for record id ...
Repeat these steps for each of the other Private MSK Proxy instances launched by the CloudFormation template.

Create the MSK VPC Endpoint

We can now create a VPC Endpoint to access your MSK cluster from Kafka clients in a different VPC via the newly created VPC Endpoint Service.

Create the VPC

Follow the Create VPC guide to create a VPC for your Kafka clients with the following parameters.
Name tag
my-msk-client
IPv4 CIDR block
10.1.0.0/16
Region
us-east-1
This creates your client VPC in preparation for secure cross-VPC access to your MSK cluster.

Enable DNS Hostnames

Navigate to the VPC Management Console and make sure you have selected the desired region in the upper right corner, such as US East (N. Virginia) us-east-1.
Under the Resources by Region section, select the VPCs resource box to show Your VPCs. Then select the VPC named my-msk-client to show the details.
Choose Edit DNS hostnames from the Actions menu and fill out the Edit DNS hostnames form as follows:
DNS hostnames
Enable
This is required to let the Kafka clients properly resolve your MSK cluster bootstrap server names.
Make sure to enable DNS Hostnames for this VPC.

Create the VPC Endpoint

Navigate to the CloudFormation Management Console and click the Create Stack button to complete the Create stack wizard with the following details:
Step 1. Specify template
Prepare template
Template is ready
Specify template
Template source
Amazon S3 URL
Step 2. Specify stack details
Stack name
my-msk-endpoint
Parameters
Network Configuration
VPC
my-msk-client
Subnets
my-msk-client-1a
my-msk-client-1b
my-msk-client-1c
Endpoint Configuration
Endpoint Service Name [1]
com.amazonaws.vpce.[...]
Wildcard DNS pattern [2]
*.aklivity.[...].amazonaws.com
Step 3. Configure stack options
(defaults)
Step 4. Review
(review)
[1] Locate the Endpoint Service Name from the Outputs tab of the my-msk-endpoint-service stack previously created in the same VPC as your MSK Cluster.
[2] Follow the Lookup MSK Server Names guide to discover the wildcard DNS pattern for your MSK cluster.
Click Create Stack.
This initiates creation of a VPC Endpoint using the CloudFormation stack.
When your VPC Endpoint is ready, the CloudFormation console will show CREATE_COMPLETE for the newly created stack.

Verify Kafka Client Connectivity

Now we must prepare a Kafka client running in the client VPC to verify connectivity to your MSK cluster via the VPC Endpoint.

Launch the EC2 Instance

Follow the Launch EC2 Instance guide with the following parameters to launch an EC2 instance with remote SSH access and login to the instance via SSH.
VPC
my-msk-client
Region
us-east-1
Instance
AMI
Amazon Linux 2 AMI (HVM),
SSD Volume Type,
64-bit (x86)
Type
t3.small
This launches an EC2 instance that you can access remotely via SSH.

Install the Kafka Client

First, we must install a Java runtime that can be used by the Kafka client.
After logging into the instance via SSH, run the following command:
sudo yum install java-1.8.0
Now we are ready to install the Kafka client:
wget https://archive.apache.org/dist/kafka/2.8.0/kafka_2.13-2.8.0.tgz
tar -xzf kafka_2.13-2.8.0.tgz
cd kafka_2.13-2.8.0
After changing the directory to kafka_2.13-2.8.0 we must copy the Kafka clients trustore:
cp /usr/lib/jvm/<JDKFolder>/lib/security/cacerts /tmp/kafka.client.truststore.jks
You can get the value for <JDKFolder> by typing
cp /usr/lib/jvm/j **double tap TAB**
and the selecting the longer entry that starts with "jre-1.8.0-openjdk...."
A sample full copy command command of a Kafka client's trustore will appear as follows:
cp /usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.302.b08-0.amzn2.0.1.x86_64/lib/security/cacerts /tmp/kafka.client.truststore.jks

Configure the Kafka Client

With the Kaka client now installed we are ready to configure it and point it at the MSK VPC Endpoint.
The MSK Proxy relies on TLS so we need to create a file called client.properties that tells the Kafka client to use SSL and were the truststore is:
client.properties
security.protocol=SSL
ssl.truststore.location=/tmp/kafka.client.truststore.jks
Hint: use the Vim to createclient.properties.Type vi client.properties in the SSH terminal, paste the above contents, then enter the:wq command to save the file..

Test the Kafka Client

We can now verify that the Kafka client can successfully communicate with your MSK cluster via the VPC Endpoint in a different VPC to create a topic, then publish and subscribe to the same topic.
Follow the Lookup MSK Server Names guide to find the TLS bootstrap server names for your MSK cluster.

Create a Topic

Use the Kafka client to create a topic called vpce-test, replacing<tls-bootstrap-server-names> in the command below with the TLS bootstrap server names of your MSK cluster:
bin/kafka-topics.sh --create --topic vpce-test --partitions 3 --replication-factor 3 --command-config client.properties --bootstrap-server <tls-bootstrap-server-names>
A quick summary of what just happened:
  1. 1.
    The Kafka client in the my-msk-client VPC issued a request to create a new topic
  2. 2.
    This request was directed to the MSK VPC Endpoint
  3. 3.
    The request was then relayed across PrivateLink to the MSK VPC Endpoint Service in the my-msk-cluster VPC
  4. 4.
    The Endpoint Service forwarded the request to the Network Load Balancer
  5. 5.
    The Network Load Balancer forwarded the request to the Aklivity MSK Private Proxy
  6. 6.
    The Aklivity MSK Private Proxy look at the info in the request and relayed it through to the appropriate MSK broker
  7. 7.
    The topic was created in the MSK broker
  8. 8.
    Cross VPC connectivity was verified

Publish messages

Publish two messages to the newly created topic via the following producer command:
bin/kafka-console-producer.sh --topic vpce-test --producer.config client.properties --broker-list <tls-bootstrap-server-names>
A prompt will appear for you to type in the messages:
>This is my first event
>This is my second event

Receive messages

Read these messages back via the following consumer command:
bin/kafka-console-consumer.sh --topic vpce-test --from-beginning --consumer.config client.properties --bootstrap-server <tls-bootstrap-server-names>
You should see the This is my first event and This is my second event messages.
This is my first event
This is my second event
This verifies cross-VPC connectivity to your MSK cluster via Aklivity Private MSK Proxy!

Monitor the VPC Endpoint Service

The CloudFormation template used to deploy the VPC Endpoint Service includes a Network Load Balancer that can be monitored via CloudWatch to verify continuous health.
Network Load Balancers have many available metrics, including the following.
Metric
Description
TCP_Target_Reset_Count
The total number of reset (RST) packets sent from a target to a client. These resets are generated by the target and forwarded by the load balancer.
UnHealthyHostCount
The number of targets that are considered unhealthy.
You can use CloudWatch to create a dashboard to monitor these metrics and set alarms to alert you when specific metric thresholds are reached.

Upgrade the VPC Endpoint Service

Navigate to your AWS Marketplace Subscriptions and select Aklivity Private MSK Proxy to show the details page. Then select Launch CloudFormation stack from the Actions menu in the Agreement section.
Make sure you have selected the desired region, such as US East (N. Virginia) us-east-1, and then click Continue to Launch. Choose the action Launch CloudFormation, then click Launch to show the URL of the CloudFormation template.
Copy the CloudFormation template Amazon S3 URL and then select your existing CloudFormation Stack from a previous deployment of Aklivity Private MSK Proxy. Click Update and Replace current template with the copied Amazon S3 URL. Then complete the wizard to deploy the updated stack.
CloudFormation will incrementally deploy the MSK Proxy instances for the new version behind the same Network Load Balancer, checking for successful deployment before terminating the MSK Proxy instances for the previous version.
Connected clients will see their connections drop, and when they reconnect automatically, the Network Load Balancer will direct them to the new MSK Proxy instances. If the update is unsuccessful, then CloudFormation will rollback to use the previous stack deployment.