Private MSK Proxy
Securely access an Amazon MSK cluster across VPCs via AWS PrivateLink.
Estimated time to complete 20-30 mins.
The Aklivity Private MSK Proxy enables cross-account connectivity between Kafka clients and Amazon MSK clusters.
Bundled CloudFormation templates provide automated configuration of a VPC Endpoint Service for your Amazon MSK cluster as well as corresponding VPC Endpoints. These VPC endpoints enable secure access to your MSK cluster via AWS PrivateLink from an outside VPC, even if that VPC is owned by a different AWS account. Authorized Kafka clients in each consuming VPC will be able to connect, publish messages and subscribe to topics in your Amazon MSK cluster.
In this guide we will deploy the Aklivity Private MSK Proxy and showcase cross VPC connectivity between an MSK cluster and a Kafka client.
Service | Required | Usage | Quota |
|---|---|---|---|
Marketplace Metering | Yes | Startup, then hourly | |
Resource Groups and Tagging | Yes | Startup only | |
Secrets Manager | Yes | Startup only | |
Certificate Manager | No
Private key and certificate can be inlined in Secrets Manager instead | Startup only | |
Private Certificate Manager | No
Private key and certificate can be inlined in Secrets Manager instead | Startup only |
The default AWS Service Quotas are sufficient.
Before setting up cross-VPC access to your MSK Cluster, you will need the following:
- an MSK Cluster configured for TLS encrypted client access
- an VPC security group for MSK Proxy instances
- an IAM security role for MSK Proxy instances
- subscription to Aklivity Private MSK Proxy via AWS Marketplace
We need to create an MSK cluster in preparation for secure remote access outside the VPC. You can skip this step if you have already created an MSK cluster with equivalent configuration.
Name tag | my-msk-cluster |
IPv4 CIDR block | 10.0.0.0/16 |
Region | us-east-1 |
Name | aklivity |
VPC | my-msk-cluster |
Subnets | my-msk-cluster-1amy-msk-cluster-1bmy-msk-cluster-1c |
This creates your MSK cluster in preparation for secure access outside the VPC.
We need to create a VPC security group that will be used by the Private MSK Proxy instances when they are launched.
Follow the Create Security Group guide with the following parameters to create a security group in the same VPC as your MSK cluster.
VPC | my-msk-cluster |
Name | my-msk-proxy |
Description | Kafka clients and SSH access |
Inbound Rule | |
Type | TCP 9094 |
Source | <Any IPv4> |
Inbound Rule | |
Type | SSH |
Source | <My IP> |
This creates your Private MSK proxy security group to allow Kafka clients and SSH access.
Follow the Update Security Group guide with the following parameters to allow the MSK Proxy instances to communicate with the MSK cluster.
VPC | vpc-xxx (my-msk-cluster) |
Security Group | default (MSK security group) |
Inbound Rule | |
Type | TCP 9094 |
Source | CustomSecurity groups:my-msk-proxy |
This allows the MSK Proxy instances to access your MSK cluster.
Name | aklivity-private-msk-proxy |
Managed Policy | |
Name | AWSMarketplaceMeteringFullAccess |
This creates an IAM security role to enable the required AWS services for the MSK Proxy instances.
The Aklivity Private MSK Proxy is available through the AWS Marketplace. You can skip this step if you have already subscribed to Aklivity Private MSK Proxy via AWS Marketplace.
Navigate to your AWS Marketplace Subscriptions and select
Aklivity Private MSK Proxy to show the details page. Then select Launch CloudFormation stack from the Actions menu in the Agreement section.Make sure you have selected the desired region, such as
US East (N. Virginia) us-east-1, and then click Continue to Launch. Choose the action Launch CloudFormation, then click Launch to complete the Create stack wizard with the following details:Step 1. Specify template | |
Prepare template | Template is ready |
Specify template | (auto-filled) |
Step 2. Specify stack details | |
Stack name | my-msk-endpoint-service |
Parameters | |
Network Configuration | |
VPC | my-msk-cluster |
Subnets | my-msk-cluster-1amy-msk-cluster-1bmy-msk-cluster-1c |
MSK Configuration | |
Wildcard DNS pattern [1] | *.aklivity.[...].amazonaws.com |
Port number | 9094 |
MSK Proxy Configuration | |
Instance count | 2 |
Instance type [2] | t3.small |
Role | aklivity-private-msk-proxy |
Security Groups | my-msk-proxy |
Key pair for SSH access | <key pair> |
Step 3. Configure stack options | (defaults) |
Step 4. Review | (review) |
[1] Follow the Lookup MSK Server Names guide to discover the wildcard DNS pattern for your MSK cluster.
[2] Consider the network throughput characteristics of the AWS instance type as that will impact the upper bound on network performance.
Click
Create Stack. This initiates creation of a VPC Endpoint Service using the Aklivity Private MSK Proxy stack via CloudFormation.
When your VPC Endpoint Service is ready, the CloudFormation console will show
CREATE_COMPLETE for the newly created stack.Navigate to the EC2 Management Console and make sure you have selected the desired region in the upper right corner, such as
US East (N. Virginia) us-east-1.Under the
Resources by Region section, select the Instances resource box to show your Instances. Select either of the Private MSK Proxy instances launched by the CloudFormation template to show the details.They each have the IAM Role name
aklivity-private-msk-proxy. Find the
Public IPv4 Address and then SSH into the instance.ssh -i ~/.ssh/<key-pair.cer> [email protected]<instance-public-ip-address>
After logging in via SSH, check the status of the
msk-proxy system service.systemctl status msk-proxy.service
Verify that the
msk-proxy service is active and logging output similar to that shown below.● msk-proxy.service - MSK Proxy
Loaded: loaded (/etc/systemd/system/msk-proxy.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2021-08-24 20:56:51 UTC; 1 day 19h ago
Main PID: 1803 (java)
CGroup: /system.slice/msk-proxy.service
└─...
Aug 26 06:56:54 ip-10-0-3-104.ec2.internal zilla[1803]: Recorded usage for record id ...
Repeat these steps for each of the other Private MSK Proxy instances launched by the CloudFormation template.
We can now create a VPC Endpoint to access your MSK cluster from Kafka clients in a different VPC via the newly created VPC Endpoint Service.
Name tag | my-msk-client |
IPv4 CIDR block | 10.1.0.0/16 |
Region | us-east-1 |
This creates your client VPC in preparation for secure cross-VPC access to your MSK cluster.
Navigate to the VPC Management Console and make sure you have selected the desired region in the upper right corner, such as
US East (N. Virginia) us-east-1.Under the
Resources by Region section, select the VPCs resource box to show Your VPCs. Then select the VPC named my-msk-client to show the details.Choose
Edit DNS hostnames from the Actions menu and fill out the Edit DNS hostnames form as follows:DNS hostnames | Enable |
This is required to let the Kafka clients properly resolve your MSK cluster bootstrap server names.
Make sure to enable DNS Hostnames for this VPC.
Navigate to the CloudFormation Management Console and click the
Create Stack button to complete the Create stack wizard with the following details:Step 1. Specify template | |
Prepare template | Template is ready |
Specify template | |
Template source | Amazon S3 URL |
Amazon S3 URL | |
Step 2. Specify stack details | |
Stack name | my-msk-endpoint |
Parameters | |
Network Configuration | |
VPC | my-msk-client |
Subnets | my-msk-client-1amy-msk-client-1bmy-msk-client-1c |
Endpoint Configuration | |
Endpoint Service Name [1] | com.amazonaws.vpce.[...] |
Wildcard DNS pattern [2] | *.aklivity.[...].amazonaws.com |
Step 3. Configure stack options | (defaults) |
Step 4. Review | (review) |
[1] Locate the
Endpoint Service Name from the Outputs tab of the my-msk-endpoint-service stack previously created in the same VPC as your MSK Cluster.[2] Follow the Lookup MSK Server Names guide to discover the wildcard DNS pattern for your MSK cluster.
Click
Create Stack. This initiates creation of a VPC Endpoint using the CloudFormation stack.
When your VPC Endpoint is ready, the CloudFormation console will show
CREATE_COMPLETE for the newly created stack.Now we must prepare a Kafka client running in the client VPC to verify connectivity to your MSK cluster via the VPC Endpoint.
Follow the
Launch EC2 Instance guide with the following parameters to launch an EC2 instance with remote SSH access and login to the instance via SSH.VPC | my-msk-client |
Region | us-east-1 |
Instance | |
AMI | Amazon Linux 2 AMI (HVM), SSD Volume Type, 64-bit (x86) |
Type | t3.small |
This launches an EC2 instance that you can access remotely via SSH.
First, we must install a Java runtime that can be used by the Kafka client.
After logging into the instance via SSH, run the following command:
sudo yum install java-1.8.0
Now we are ready to install the Kafka client:
wget https://archive.apache.org/dist/kafka/2.8.0/kafka_2.13-2.8.0.tgz
tar -xzf kafka_2.13-2.8.0.tgz
cd kafka_2.13-2.8.0
After changing the directory to
kafka_2.13-2.8.0 we must copy the Kafka clients trustore:cp /usr/lib/jvm/<JDKFolder>/lib/security/cacerts /tmp/kafka.client.truststore.jks
You can get the value for
<JDKFolder> by typing cp /usr/lib/jvm/j **double tap TAB**
and the selecting the longer entry that starts with "jre-1.8.0-openjdk...."
A sample full copy command command of a Kafka client's trustore will appear as follows:
cp /usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.302.b08-0.amzn2.0.1.x86_64/lib/security/cacerts /tmp/kafka.client.truststore.jks
With the Kaka client now installed we are ready to configure it and point it at the MSK VPC Endpoint.
The MSK Proxy relies on TLS so we need to create a file called
client.properties that tells the Kafka client to use SSL and were the truststore is:client.properties
security.protocol=SSL
ssl.truststore.location=/tmp/kafka.client.truststore.jks
Hint: use the Vim to create
client.properties.Type vi client.properties in the SSH terminal, paste the above contents, then enter the:wq command to save the file..We can now verify that the Kafka client can successfully communicate with your MSK cluster via the VPC Endpoint in a different VPC to create a topic, then publish and subscribe to the same topic.
Follow the Lookup MSK Server Names guide to find the TLS bootstrap server names for your MSK cluster.
Use the Kafka client to create a topic called
vpce-test, replacing<tls-bootstrap-server-names> in the command below with the TLS bootstrap server names of your MSK cluster:bin/kafka-topics.sh --create --topic vpce-test --partitions 3 --replication-factor 3 --command-config client.properties --bootstrap-server <tls-bootstrap-server-names>
A quick summary of what just happened:
- 1.The Kafka client in the my-msk-client VPC issued a request to create a new topic
- 2.This request was directed to the MSK VPC Endpoint
- 3.The request was then relayed across PrivateLink to the MSK VPC Endpoint Service in the my-msk-cluster VPC
- 4.The Endpoint Service forwarded the request to the Network Load Balancer
- 5.The Network Load Balancer forwarded the request to the Aklivity MSK Private Proxy
- 6.The Aklivity MSK Private Proxy look at the info in the request and relayed it through to the appropriate MSK broker
- 7.The topic was created in the MSK broker
- 8.Cross VPC connectivity was verified
Publish two messages to the newly created topic via the following producer command:
bin/kafka-console-producer.sh --topic vpce-test --producer.config client.properties --broker-list <tls-bootstrap-server-names>
A prompt will appear for you to type in the messages:
>This is my first event
>This is my second event
Read these messages back via the following consumer command:
bin/kafka-console-consumer.sh --topic vpce-test --from-beginning --consumer.config client.properties --bootstrap-server <tls-bootstrap-server-names>
You should see the
This is my first event and This is my second event messages.This is my first event
This is my second event
The CloudFormation template used to deploy the VPC Endpoint Service includes a Network Load Balancer that can be monitored via CloudWatch to verify continuous health.
Metric | Description |
|---|---|
TCP_Target_Reset_Count | The total number of reset (RST) packets sent from a target to a client. These resets are generated by the target and forwarded by the load balancer. |
UnHealthyHostCount | The number of targets that are considered unhealthy. |
You can use CloudWatch to create a dashboard to monitor these metrics and set alarms to alert you when specific metric thresholds are reached.
Navigate to your AWS Marketplace Subscriptions and select
Aklivity Private MSK Proxy to show the details page. Then select Launch CloudFormation stack from the Actions menu in the Agreement section.Make sure you have selected the desired region, such as
US East (N. Virginia) us-east-1, and then click Continue to Launch. Choose the action Launch CloudFormation, then click Launch to show the URL of the CloudFormation template.Copy the CloudFormation template Amazon S3 URL and then select your existing CloudFormation Stack from a previous deployment of
Aklivity Private MSK Proxy. Click Update and Replace current template with the copied Amazon S3 URL. Then complete the wizard to deploy the updated stack.CloudFormation will incrementally deploy the MSK Proxy instances for the new version behind the same Network Load Balancer, checking for successful deployment before terminating the MSK Proxy instances for the previous version.
Connected clients will see their connections drop, and when they reconnect automatically, the Network Load Balancer will direct them to the new MSK Proxy instances. If the update is unsuccessful, then CloudFormation will rollback to use the previous stack deployment.
Last modified 1yr ago