Deploying Redpanda Kafka-Compatible Streaming Platform on Ubuntu 24.04

Redpanda is a Kafka-API-compatible streaming platform written in C++ with no JVM and no ZooKeeper. This guide installs Redpanda on Ubuntu 24.04, secures it with a Let’s Encrypt certificate and SASL/SCRAM authentication, tunes the kernel for production, verifies with a producer/consumer test, and exposes Redpanda Console behind Nginx basic auth. By the end, you’ll have a secured, production-tuned single-node Redpanda cluster with a web console.

Prerequisite: Ubuntu 24.04 server sized per Redpanda’s CPU/memory requirements, non-root sudo user, and a domain A record (e.g. redpanda.example.com).

Install Redpanda

$ sudo apt update
$ curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.deb.sh' | sudo -E bash

Enter fullscreen mode Exit fullscreen mode

Warning: Only run vendor setup scripts you trust — piped curl | sudo bash runs with root privileges.

$ sudo apt install redpanda -y
$ rpk --version

Enter fullscreen mode Exit fullscreen mode

Open the Firewall

Port Service Purpose 9092 Kafka API Producer/consumer traffic 8082 Pandaproxy (HTTP) REST access for non-Kafka clients 8081 Schema Registry Avro/Protobuf schema versioning 9644 Admin API Monitoring, config, health checks 33145 Internal RPC Inter-node communication

$ sudo ufw allow 9092,8082,8081,9644,33145/tcp
$ sudo ufw allow 80/tcp
$ sudo ufw allow 443/tcp
$ sudo ufw reload

Enter fullscreen mode Exit fullscreen mode

Issue a Let’s Encrypt Certificate

Redpanda ships with plaintext networking by default, fine for a lab, not for anything else.

$ sudo apt install certbot -y
$ DOMAIN=redpanda.example.com
$ [email protected]
$ sudo certbot certonly --standalone -d $DOMAIN --non-interactive --email $EMAIL

Enter fullscreen mode Exit fullscreen mode

Certbot stores certs under /etc/letsencrypt/live, readable only by root. Redpanda runs as its own redpanda user, so copy the certs into a dedicated directory:

$ sudo mkdir /etc/redpanda/certs
$ sudo cp /etc/letsencrypt/live/$DOMAIN/fullchain.pem /etc/redpanda/certs/node.crt
$ sudo cp /etc/letsencrypt/live/$DOMAIN/privkey.pem /etc/redpanda/certs/node.key
$ sudo cp /etc/letsencrypt/live/$DOMAIN/chain.pem /etc/redpanda/certs/ca.crt
$ sudo chown -R redpanda:redpanda /etc/redpanda/certs/
$ sudo chmod 400 /etc/redpanda/certs/node.key
$ sudo chmod 444 /etc/redpanda/certs/node.crt /etc/redpanda/certs/ca.crt

Enter fullscreen mode Exit fullscreen mode

Automate renewal — Let’s Encrypt certs expire every 90 days, so wire a deploy hook that refreshes Redpanda’s copies and restarts the service:

$ sudo nano /etc/letsencrypt/renewal-hooks/deploy/redpanda-renewal.sh

Enter fullscreen mode Exit fullscreen mode

#!/bin/bash
DOMAIN="redpanda.example.com"

cp /etc/letsencrypt/live/$DOMAIN/fullchain.pem /etc/redpanda/certs/node.crt
cp /etc/letsencrypt/live/$DOMAIN/privkey.pem /etc/redpanda/certs/node.key
cp /etc/letsencrypt/live/$DOMAIN/chain.pem /etc/redpanda/certs/ca.crt

chown -R redpanda:redpanda /etc/redpanda/certs/
chmod 400 /etc/redpanda/certs/node.key
chmod 444 /etc/redpanda/certs/node.crt /etc/redpanda/certs/ca.crt

systemctl restart redpanda

Enter fullscreen mode Exit fullscreen mode

$ sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/redpanda-renewal.sh
$ sudo certbot renew --dry-run

Enter fullscreen mode Exit fullscreen mode

Tune for Production

$ sudo rpk redpanda mode production
$ sudo rpk redpanda tune all
$ sudo systemctl start redpanda-tuner

Enter fullscreen mode Exit fullscreen mode

tune all optimizes I/O schedulers and CPU settings, takes ~30 seconds. The tuner service reapplies these on every reboot since kernel tuning doesn’t persist otherwise.

Bootstrap Authentication

Set security policy before the first startup so it’s enforced from the very first boot.

$ sudo nano /etc/redpanda/.bootstrap.yaml

Enter fullscreen mode Exit fullscreen mode

enable_sasl: true
admin_api_require_auth: true
superusers:
  - admin

Enter fullscreen mode Exit fullscreen mode

Note: This file is only read on first startup. Later changes go through the Admin API or rpk cluster config.

Set the superuser’s password:

$ echo "RP_BOOTSTRAP_USER=admin:STRONG_PASSWORD:SCRAM-SHA-256" | sudo tee -a /etc/default/redpanda

Enter fullscreen mode Exit fullscreen mode

Configure the Node

$ sudo cp /etc/redpanda/redpanda.yaml /etc/redpanda/redpanda-backup.yaml
$ sudo nano /etc/redpanda/redpanda.yaml

Enter fullscreen mode Exit fullscreen mode

Replace the contents (swap redpanda.example.com for your domain):

redpanda:
  data_directory: /var/lib/redpanda/data
  seed_servers: []

  rpc_server:
    address: 0.0.0.0
    port: 33145
  advertised_rpc_api:
    address: redpanda.example.com
    port: 33145

  kafka_api:
    - name: tls_listener
      address: 0.0.0.0
      port: 9092
      authentication_method: sasl
  advertised_kafka_api:
    - name: tls_listener
      address: redpanda.example.com
      port: 9092
  kafka_api_tls:
    - name: tls_listener
      key_file: /etc/redpanda/certs/node.key
      cert_file: /etc/redpanda/certs/node.crt
      truststore_file: /etc/redpanda/certs/ca.crt
      enabled: true
      require_client_auth: false

  admin:
    - address: 0.0.0.0
      port: 9644
  admin_api_tls:
    - enabled: true
      key_file: /etc/redpanda/certs/node.key
      cert_file: /etc/redpanda/certs/node.crt
      truststore_file: /etc/redpanda/certs/ca.crt

rpk:
  tune_network: true
  tune_disk_scheduler: true
  tune_disk_nomerges: true
  tune_disk_write_cache: true
  tune_disk_irq: true
  tune_cpu: true
  tune_aio_events: true
  tune_clocksource: true
  tune_swappiness: true
  coredump_dir: /var/lib/redpanda/coredump
  tune_ballast_file: true

Enter fullscreen mode Exit fullscreen mode

This enables SASL-over-TLS on the Kafka API (9092), TLS on the Admin API (9644), advertises both under your public domain, and applies the full rpk tuning profile.

$ sudo systemctl start redpanda
$ sudo systemctl status redpanda

Enter fullscreen mode Exit fullscreen mode

Create an rpk Profile

Run rpk as your regular user (not sudo) so it picks up your profile.

$ nano ~/rpk-profile.yaml

Enter fullscreen mode Exit fullscreen mode

kafka_api:
  sasl:
    user: admin
    password: STRONG_PASSWORD
    mechanism: SCRAM-SHA-256
  tls:
    enabled: true
    ca_file: /etc/redpanda/certs/ca.crt
  brokers:
    - redpanda.example.com:9092

admin_api:
  tls:
    enabled: true
    ca_file: /etc/redpanda/certs/ca.crt
  addresses:
    - redpanda.example.com:9644

Enter fullscreen mode Exit fullscreen mode

$ rpk profile create production --from-profile ~/rpk-profile.yaml
$ rpk cluster health

Enter fullscreen mode Exit fullscreen mode

CLUSTER HEALTH OVERVIEW
=======================
Healthy:                          true
Unhealthy reasons:                []
Controller ID:                    0
All nodes:                        [0]
Nodes down:                       []
Cluster UUID:                     65e28fbb-c1b4-42c3-b0b7-fd82608699f3

Enter fullscreen mode Exit fullscreen mode

Test with a Producer and Consumer

$ rpk topic create test-topic

Enter fullscreen mode Exit fullscreen mode

TOPIC       STATUS
test-topic  OK

Enter fullscreen mode Exit fullscreen mode

$ echo "Hello Redpanda" | rpk topic produce test-topic

Enter fullscreen mode Exit fullscreen mode

Produced to partition 0 at offset 0 with timestamp 1699618845123.

Enter fullscreen mode Exit fullscreen mode

$ rpk topic consume test-topic --num 1

Enter fullscreen mode Exit fullscreen mode

Deploy Redpanda Console

$ sudo apt install redpanda-console -y
$ sudo nano /etc/redpanda/redpanda-console-config.yaml

Enter fullscreen mode Exit fullscreen mode

server:
  listenPort: 8080
  listenAddress: 127.0.0.1

kafka:
  brokers:
    - redpanda.example.com:9092
  tls:
    enabled: true
    caFilepath: /etc/redpanda/certs/ca.crt
  sasl:
    enabled: true
    username: admin
    password: STRONG_PASSWORD
    mechanism: SCRAM-SHA-256

redpanda:
  adminApi:
    enabled: true
    urls:
      - https://redpanda.example.com:9644
    tls:
      enabled: true
      caFilepath: /etc/redpanda/certs/ca.crt
    authentication:
      basic:
        username: admin
        password: STRONG_PASSWORD

Enter fullscreen mode Exit fullscreen mode

$ sudo systemctl enable redpanda-console
$ sudo systemctl start redpanda-console

Enter fullscreen mode Exit fullscreen mode

Front the Console with Nginx Basic Auth

Redpanda Console CE has no built-in login — put Nginx basic auth in front of it.

$ sudo apt install nginx -y
$ sudo apt install apache2-utils -y
$ sudo htpasswd -c /etc/nginx/.htpasswd admin

Enter fullscreen mode Exit fullscreen mode

$ sudo nano /etc/nginx/sites-available/redpanda-console

Enter fullscreen mode Exit fullscreen mode

server {
    listen 80;
    server_name redpanda.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name redpanda.example.com;

    ssl_certificate /etc/letsencrypt/live/redpanda.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/redpanda.example.com/privkey.pem;

    location / {
        auth_basic "Restricted Access";
        auth_basic_user_file /etc/nginx/.htpasswd;
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Enter fullscreen mode Exit fullscreen mode

$ sudo ln -s /etc/nginx/sites-available/redpanda-console /etc/nginx/sites-enabled/
$ sudo nginx -t
$ sudo systemctl restart nginx

Enter fullscreen mode Exit fullscreen mode

Open https://redpanda.example.com, enter the basic-auth credentials to reach the console dashboard.

Next Steps

Redpanda is running with TLS, SASL auth, production tuning, and a secured console. From here you can:

• Add more brokers and set seed_servers for a multi-node cluster
• Wire Kafka Connect or a schema registry client against port 8081
• Point existing Kafka producers/consumers at Redpanda. The wire protocol is compatible, no client changes needed

For the full guide with additional tips, visit the original article on Vultr Docs.