Hand-On Beginners — Getting started with Kafka

We’ll divide this article into two parts:

1. Installing Kafka on macOS

2. Building a beginner-friendly Kafka project where a Kafka producer sends data to a Kafka topic and a consumer reads from it.

In this hands-on guide, we’ll use the following tools and technologies:

• Docker

• Python

• Zookeeper

• Kafka Broker Service

• Kafka UI for monitoring

Installing Kafka on macOS

Step 1.

Create a dedicated project directory and docker-compose YAML file

# Below goes in your terminal
mkdir ~/kafka-docker
cd ~/kafka-docker


# Create the docker-compose.yml file inside this directory
touch docker-compose.yml

Step 2.

Edit this YAML file as per your setting. We are providing a YAML file that works with our system settings and docker version as below.

# Edit the File
# Using nano
nano docker-compose.yml

# Using vim
vim docker-compose.yml

version: '3.8'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.4.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:7.4.0
    hostname: kafka
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    depends_on:
      - kafka
    ports:
      - "8080:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092

Lets understand this code first

version: '3.8': Version 3.8 is compatible with newer Docker features and Swarm mode.

Zookeeper: Zookeeper is required by Kafka to manage brokers and metadata.

zookeeper:
    image: confluentinc/cp-zookeeper:7.4.0

hostname / container_name: Sets a fixed name.

ports:2181:2181 — Exposes Zookeeper on the host machine.

environment:

ZOOKEEPER_CLIENT_PORT: 2181 — Port for clients (like Kafka) to connect.

ZOOKEEPER_TICK_TIME: 2000 — Basic time unit for internal Zookeeper operations.

kafka

kafka:
  image: confluentinc/cp-kafka:7.4.0

This is the actual Kafka broker service.

depends_on: Waits for Zookeeper to start first.

ports:

9092:9092 — Port exposed to your local machine (used by client apps).

9101:9101 — Port for JMX (monitoring Kafka internals).

kafka-ui

kafka-ui:
  image: provectuslabs/kafka-ui:latest

depends_on: Waits for Kafka to start.

ports:

8080:8080 — Exposes the Kafka UI dashboard to your local machine.

Side Note:(Optional Read)

Photo by AbsolutVision on Unsplash

Some things to remember docker-compose.yml file can vary depending on:

a. Docker Compose File Version (version:)

b. Host Environment Differences (OS or Machine Specific)

c. Docker Engine Capabilities

• deploy: is only used with Docker Swarm, not with plain docker-compose.

• GPU support via runtime: nvidia needs special Docker setup.

Step 3.

Complete Directory Structure

~/kafka-docker/ 
├── docker-compose.yml 
├── data/ # (optional) for persistent data 
├── kafka/ 
└── zookeeper/
└── config/ # (optional) for custom configs 
├── server.properties 
└── zookeeper.properties

Start the service

docker-compose up -d

Check if containers are running

docker-compose ps

Access Kafka UI at http://localhost:8080

Testing Your Installation

Create and Test a Topic

# Create a topic
kafka-topics - create - topic my-test-topic \
 - bootstrap-server localhost:9092 \
 - partitions 3 \
 - replication-factor 1

# List all topics
kafka-topics - list - bootstrap-server localhost:9092

# Describe the topic
kafka-topics - describe - topic my-test-topic - bootstrap-server localhost:9092

Side Note:(Optional Read)

Photo by AbsolutVision on Unsplash

If you get error “zsh: command not found: kafka-topics”, means that your terminal (zsh shell) can’t find the kafka-topics command, likely because it’s not in your system’s PATH.

You can get around this by using

# Instead of: kafka-topics --list --bootstrap-server localhost:9092
# Use this:
docker exec kafka kafka-topics --list --bootstrap-server localhost:9092

# Create a topic
docker exec kafka kafka-topics --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

# List topics
docker exec kafka kafka-topics --list --bootstrap-server localhost:9092

But to add this permanently

# Add to ~/.zshrc
cat >> ~/.zshrc << 'EOF'

# Kafka Docker aliases
alias kafka-topics='docker exec kafka kafka-topics'
alias kafka-console-producer='docker exec -it kafka kafka-console-producer'
alias kafka-console-consumer='docker exec -it kafka kafka-console-consumer'
alias kafka-consumer-groups='docker exec kafka kafka-consumer-groups'
alias kafka-configs='docker exec kafka kafka-configs'

EOF

Reload configuration

source ~/.zshrc

After this below test should work

# Create a topic
kafka-topics --create --topic my-test-topic \
  --bootstrap-server localhost:9092 \
  --partitions 3 \
  --replication-factor 1

# List all topics
kafka-topics --list --bootstrap-server localhost:9092

# Describe the topic
kafka-topics --describe --topic my-test-topic --bootstrap-server localhost:9092

Producer and Consumer Test

Start a consumer (in one terminal)

kafka-console-consumer 
--topic my-test-topic 
--from-beginning 
--bootstrap-server localhost:9092

Start a producer (in another terminal)

kafka-console-producer 
--topic my-test-topic 
--bootstrap-server localhost:9092

Type messages in producer terminal and see them appear in consumer terminal