Back to all blogs
    3 min read

    ⚙️ Kafka Cluster Configuration

    #kafka#configuration#admin

    🛠️ Broker Configurations (server.properties)

    Anatomy Breakdown: A Kafka broker is configured using the server.properties file. Two critical configurations dictate durability and scalability:

    • log.retention.hours: How long Kafka keeps your data before permanently deleting it. (Default is often 168 hours / 7 days).
    • num.partitions: The default number of partitions created when someone makes a new topic without specifying a count. (More partitions = more parallel processing).
    • min.insync.replicas: The minimum number of replica brokers that must acknowledge a write (when acks=all) for it to be considered successful. This ensures data durability even if a node crashes.
    Disk Space vs Retention

    Kafka does not keep data forever by default! It is a streaming platform, not a permanent database. Adjusting log.retention.hours is critical to prevent your brokers from running out of disk space.

    🧩 Metadata Management

    What Existed Previously: Historically, Kafka absolutely required a separate, external ZooKeeper cluster to manage metadata (e.g., which broker is the leader, where partitions are located).

    Problems Faced: Running two entirely different distributed systems (Kafka + ZooKeeper) was an operational nightmare. It doubled the security setup, monitoring overhead, and point of failures.

    How Present Technology Solves It: Kafka introduced KRaft (Kafka Raft Metadata mode). It completely removes ZooKeeper, allowing Kafka to manage its own metadata internally. This makes Kafka much faster, easier to scale, and dramatically simpler to operate.

    💻 Practical: AdminClient

    The Long Way: SSHing into the Kafka broker server and running raw terminal commands to create topics or change configurations.

    The Smart Way: Using the Kafka AdminClient API to programmatically create topics, change partition counts, and manage the cluster directly from your code.

    python
    from kafka import KafkaAdminClient
from kafka.admin import NewTopic

# 1. Connect to the cluster as an Administrator
admin = KafkaAdminClient(bootstrap_servers='localhost:9092')

# 2. Define a new topic with custom properties
new_topic = NewTopic(
    name='cfg-test', 
    num_partitions=6,       # Override the default partitions
    replication_factor=1    # No backup copies (just for local testing)
)

# 3. Create the topic programmatically
admin.create_topics([new_topic])

# 4. Read the metadata back to verify
metadata = admin.describe_topics(['cfg-test'])
print(metadata)
    Expected Output

    You will see a metadata dictionary confirming the topic cfg-test was successfully created with exactly 6 partitions.

    🧪 Practice Drill

    text
    // Try answering these:
Q1. What file is used to configure a Kafka broker's settings?

Q2. Which setting ensures your broker doesn't run out of disk space by deleting old messages?

Q3. Why is the Kafka community aggressively moving away from ZooKeeper?

Q4. If you want to write a Python script to automatically create topics when your app boots, which Kafka class should you use?
    💡 Click for Solutions

    A1. server.properties.

    A2. log.retention.hours (or .bytes).

    A3. Because running two separate distributed systems (Kafka and ZooKeeper) is operationally complex. KRaft replaces ZooKeeper for internal metadata management.

    A4. KafkaAdminClient.

    ← 📬 Introduction to Apache Kafka | Next Topic → 📤 Kafka Producers & Advanced Features