Troubleshooting Guides

These guides cover common issues encountered while deploying and operating Zilla Plus on AWS. Each entry lists the Symptom you'll observe, the likely Cause, and the Fix to apply.

Quick reference

Match what you're seeing to jump straight to the fix:

Symptom	Guide
CloudFormation stack rolls back (`ROLLBACK_COMPLETE`) during deployment	Why does CloudFormation stack creation timeout and rollback?
SSH connection is actively refused (`Permission denied`)	Why does my SSH client reject access to the Zilla proxy instances?
SSH connection hangs and times out	Why does my SSH client timeout when attempting to access the Zilla proxy instances?
`zilla-plus` service keeps restarting	Why does the `zilla-plus` service keep restarting on the Zilla proxy instances?
Kafka client can't connect via Zilla Plus	Why does my Kafka client fail to connect via Zilla Plus?

Deployment

Why does CloudFormation stack creation timeout and rollback?

Symptom: the CloudFormation stack creation fails and the stack rolls back to ROLLBACK_COMPLETE.

Cause: the launched instances are unable to call back to CloudFormation over the internet. This happens when the target VPC has no attached Internet Gateway, or the main Route Table for the VPC has no default route to the Internet Gateway.

Fix: attach the Internet Gateway and route to the Internet Gateway, then redeploy the stack.

Instance access

Why does my SSH client reject access to the Zilla proxy instances?

Symptom: the SSH client actively rejects the connection attempt to a Zilla proxy instance with Permission denied.

Cause: you're not using the same Launch Key specified during CloudFormation stack creation, or not logging in as the ec2-user username.

Fix: try again using the same Launch Key as specified during CloudFormation stack creation, and the username ec2-user.

Why does my SSH client timeout when attempting to access the Zilla proxy instances?

Symptom: the SSH client times out connecting to a Zilla proxy instance instead of being actively rejected.

Cause: network traffic is being dropped between the launched Zilla proxies and the SSH client. This happens when the target VPC has no attached Internet Gateway, or the main Route Table for the VPC has no default route to the Internet Gateway.

Fix: attach the Internet Gateway and route to the Internet Gateway, then try again.

Service health

Why does the `zilla-plus` service keep restarting on the Zilla proxy instances?

Symptom: the zilla-plus service keeps restarting on the Zilla proxy instances.

Cause: the IAM Role associated with your Zilla proxies has insufficient privileges to use the AWS services needed by Zilla proxy.

Fix: check the policies attached to your Zilla proxy instance IAM Role, based on the deployment type, then try again:

Zilla proxy

Kafka connectivity

Why does my Kafka client fail to connect via Zilla Plus?

Symptom: the Kafka client can't connect via Zilla Plus.

Cause: the DNS names aren't set up correctly to point to the NLB Load Balancer deployed via the provided CloudFormation template, or the client.properties file isn't configured with the correct keystore or truststore.

Fix: confirm your DNS and client.properties are configured correctly, then work through the following checks in order.

1. Verify TCP connectivity

nc -v <b-1-broker-dns-name> 9094

The nc output should be as shown below, and nc should stay connected:

Connection to <b-1-broker-dns-name> port 9094 [tcp/*] succeeded!

2. Verify TLS connectivity

Verify TLS connectivity with your client key and signed client certificate:

openssl s_client \
-connect <b-1-broker-dns-name>:9094 \
-servername <b-1-broker-dns-name> \
-cert client.cert \
-key client.key.pem

Note: if you followed Create Server Certificate to create the server certificate instead of Create Server Certificate (LetsEncrypt), then you will need to export the CA certificate and have openssl trust the exported CA certificate:

openssl s_client \
-connect <b-1-broker-dns-name>:9094 \
-servername <b-1-broker-dns-name> \
-cert client.cert \
-key client.key.pem
-CAfile Certificate.pem

The openssl output should be as shown below:

...
    Verify return code: 0 (ok)
---

Info

A successful handshake means openssl shows no error and stays connected — the client-to-proxy leg is fine. If instead the backend TLS handshake from the Zilla Plus proxy to MSK fails, openssl disconnects right after the handshake with no openssl error code — that points at the proxy-to-MSK leg, not your client.

3. Verify Kafka connectivity

kcat \
-L \
-b <b-1-broker-dns-name>:9094 \
-X security.protocol=ssl \
-X ssl.certificate.location=client.cert \
-X ssl.key.location=client.key.pem

kcat \
-L \
-b <b-1-broker-dns-name>:9094 \
-X security.protocol=ssl \
-X ssl.certificate.location=client.cert \
-X ssl.key.location=client.key.pem \
-X ssl.ca.location=Certificate.pem

The kcat output should show the list of brokers and topics accessible to the client.