Back to all blogs
    2 min read

    📚 Kafka & Spark Streaming Syllabus

    #kafka#spark#syllabus

    Apache Kafka & Spark Streaming

    This syllabus covers real-time data streaming concepts, focusing on Spark Streaming and Apache Kafka in the exact chronological order of the original presentation.

    01 - Foundations of Spark Streaming.md

    • Batch vs Streaming architectures
    • Micro-batch Architecture and DStreams
    • Continuous data processing and real-time decision making
    • Practical: Socket Word Count

    02 - Advanced Spark Streaming.md

    • Streaming workflow and Uber Use Case
    • Windowed and Stateful Operations (reduceByKeyAndWindow)
    • Slice, ReduceByWindow, and Checkpointing
    • Practical: Real-time Hashtag extraction

    03 - Introduction to Apache Kafka.md

    • Need for Kafka (Real-time, Decoupled Architecture)
    • Core Concepts: Topics, Partitions, Brokers, and Consumer Groups
    • Kafka Architecture: Distributed Broker Cluster and Zookeeper/KRaft Coordination
    • Message Ordering and Offsets

    04 - Kafka Cluster Configuration.md

    • Cluster Metadata Management and Replication
    • Performance and Durability settings (server.properties)
    • Security, Monitoring, and Administration (AdminClient API)
    • Configuring Log Retention and Default Partitions

    05 - Kafka Producers & Advanced Features.md

    • Message Publishing, Partitioning, and Ordering
    • Acknowledgment (acks) and Retries
    • Tuning for Throughput (batch.size, linger.ms, compression)
    • Serializers (String, Integer, Avro, Custom)
    • RoundRobin and Custom Partitioners
    • Headers, Interceptors, Quotas, and Throttling

    06 - Spark Streaming & Kafka Integration.md

    • Setting up StreamingContext with KafkaUtils
    • Direct Stream approach vs Receiver-based approach
    • Subscribing to Topics and parsing JSON/Avro streams
    • Writing Streaming DataFrames to Delta Lake with ACID guarantees

    07 - Kafka Consumer API & Monitoring.md

    • Pull-Based Consumer Model (Auto vs Manual Commits)
    • Single, Multi-Topic, and Pattern-Based Subscriptions
    • Continuous vs Batch Polling Strategies
    • Handling Partition Reassignments (ConsumerRebalanceListener)
    • Monitoring Metrics (Prometheus, Grafana, Consumer Lag)

    08 - Advanced Streaming Architectures.md

    • Cross-Cluster Data Mirroring (MirrorMaker)
    • High Availability and Disaster Recovery (RTO/RPO)
    • Aggregation from Edge Clusters to Central Clusters
    • End-to-End IoT Use Case: Faulty Cell Tower Detection

    Course Contents: