What is Kafka

In today’s fast-paced and data-driven world, the need for scalable and high-performance data streaming solutions has never been greater. Apache Kafka, a distributed streaming platform, has emerged as a go-to choice for organizations looking to handle massive amounts of data with efficiency and reliability.

Apache Kafka is an open-source distributed streaming platform that is designed for building real-time data pipelines and streaming applications. It was originally developed by LinkedIn and later open-sourced as an Apache project. Kafka is widely used in the industry to handle large volumes of data in a scalable and fault-tolerant manner.

Here are some key features and concepts associated with Apache Kafka:

• Publish-Subscribe Messaging System: Kafka operates as a publish-subscribe messaging system, where producers send messages (known as “events” or “records”) to topics, and consumers subscribe to topics to receive these messages in real-time.
  
• Distributed: Kafka is designed to be distributed and can run on a cluster of servers. This allows for horizontal scalability, fault tolerance, and high throughput.

• Topics: Messages in Kafka are organized into topics. Producers publish messages to specific topics, and consumers subscribe to one or more topics to receive messages. Topics act as logical channels for data.

• Partitions: Each topic can be divided into partitions, which are the basic unit of parallelism and distribution in Kafka. Partitions allow Kafka to distribute data across multiple servers and enable high throughput and parallel processing.

• Brokers: Kafka brokers are the servers in the Kafka cluster that store and manage the data. They receive messages from producers, store them, and serve them to consumers. Kafka clusters can consist of multiple brokers.
  
• Replication: Kafka supports data replication to ensure fault tolerance. Each partition can have multiple replicas, and it automatically replicates data across brokers. If one broker fails, another can take over.
  
• Consumer Groups: Kafka allows consumers to be organized into consumer groups. Each consumer group can have multiple consumers, and it ensures that each message is consumed by only one consumer within a group. This enables load balancing and parallel processing of data.
• Retention: Kafka allows you to configure data retention policies. You can specify how long data should be retained in topics before it is automatically deleted.
  
• Stream Processing: Kafka’s stream processing capabilities make it suitable for building real-time data processing and analytics applications. It Streams and other stream processing libraries integrate with it to enable real-time data processing.
  
• Ecosystem: Kafka has a rich ecosystem of connectors and tools that make it easy to integrate with various data sources, sinks, and processing frameworks.

Kafka is commonly used in scenarios such as log aggregation, event sourcing, real-time analytics, and building data pipelines for applications that require processing and reacting to data in real-time. Its ability to handle large volumes of data reliably and efficiently has made it a popular choice in the world of data streaming and event-driven architecture.

Where we should use Kafka?

It is a versatile and powerful distributed streaming platform that can be used in various scenarios where real-time data processing, event streaming, and reliable data transportation are required. Here are some common use cases:

• Log Aggregation: Kafka can be used to collect and centralize logs generated by various components of a distributed system. It provides a reliable and scalable way to store logs, making them available for analysis and monitoring.
• Real-time Data Processing: Kafka is often used as a real-time data pipeline to ingest, process, and transform data in real-time. It allows you to build real-time analytics, recommendation engines, and fraud detection systems.
• Event Sourcing: Kafka’s durable and ordered event log makes it a suitable choice for implementing event sourcing patterns. It allows you to capture and store all changes to the state of an application, making it possible to reconstruct the application’s state at any point in time.
• Microservices Communication: Kafka can be used as a communication channel between microservices in a distributed system. It allows microservices to exchange events and messages, enabling decoupled and asynchronous communication.
• Stream Processing: Kafka Streams and other stream processing libraries integrate with Kafka to enable real-time stream processing. This is useful for tasks like real-time aggregations, filtering, and complex event processing.
• Data Integration: Kafka connectors and the Kafka Connect framework make it easy to integrate with various data sources and sinks, including databases, data warehouses, cloud services, and more. It simplifies data movement and synchronization between systems.
• Metrics and Monitoring: Kafka can be used to collect and transport metrics and monitoring data from various parts of a system. This data can then be processed and visualized for performance monitoring and troubleshooting.
• Clickstream Analysis: Websites and applications can use Kafka to capture user interactions (clickstream data) in real-time. This data can be analyzed to gain insights into user behavior and make real-time recommendations.
• Event-driven Architecture: Kafka is at the core of event-driven architectures, allowing systems to react to events and trigger actions in response to specific conditions or events.
• Data Replication: Kafka’s data replication and fault-tolerant features make it suitable for data replication across data centers or cloud regions, ensuring data availability and reliability.
• Data Archiving: Kafka’s retention policies can be configured to store data for historical analysis and compliance purposes.

These are just a few examples of where Kafka can be used. Its flexibility, scalability, and reliability make it a valuable tool for a wide range of real-time data processing and event-driven applications across industries such as finance, e-commerce, healthcare, and more. It’s ability to handle high-throughput data streams with low latency has contributed to its popularity in modern data architectures.

Shiprocket Use Cases

In the Shiprocket ecosystem, Apache Kafka plays a pivotal role in optimizing and enhancing various processes. Here are three key use cases demonstrating how it is leveraged to address specific challenges:

Bulk Assignment of Courier AWB Numbers

Challenge: Processing a high volume of AWB (Air Waybill) assignment requests, potentially reaching 40,000 requests per second, while ensuring uniqueness and adherence to business rules.

Solution: Implemented Kafka as a message broker. All AWB assignment requests are sent to a Kafka queue for asynchronous processing.

How it Works: Multiple worker processes concurrently consume messages from the Kafka queue. Each worker interacts with the courier API to obtain unique AWB numbers in parallel. The workers then handle additional business rules and validations.

Scalability: Kafka’s parallel processing capabilities allow the system to scale and handle increased loads, accommodating up to 60,000 requests per second during peak sales periods.

Communication Service for Tracking Updates

Challenge: Efficiently sending tracking updates via SMS/Email communications to customers based on changes in tracking status, without relying on constant API calls.

Solution: Utilized Kafka to streamline event-driven tracking updates.

How it Works: Instead of making HTTP calls for every event, Kafka eliminates the need for direct API calls. Changes in tracking status are captured by consuming the MySQL Binlog using KSQL. A Kafka consumer then processes this data, triggering tracking notifications to customers.

Benefits: This approach reduces unnecessary API calls and ensures that tracking notifications are sent promptly whenever there is a status change, enhancing the overall customer experience.

Support System Ticket Aggregation

Challenge: Efficiently aggregating and presenting support tickets, which come in various categories with differing table structures, into a single view for sellers.

Solution: Synchronized all ticket requests to Kafka and implemented a microservice for streamlined data access.

How it Works: Support ticket requests from various sources are synced to Kafka, where they are uniformly structured. A dedicated microservice retrieves the relevant details based on the category of the ticket.

Benefits: By leveraging Kafka for data synchronization and maintaining a single, standardized data source, the system significantly improves performance and simplifies the presentation of support tickets to sellers. This approach enables efficient querying and reporting while accommodating various ticket categories.

In each of these use cases, Apache Kafka facilitates real-time data processing, scalability, and efficient communication, ultimately enhancing the Shiprocket ecosystem’s performance and user experience.

Wallet System

Challenge: Managing and processing changelog events from multiple databases, such as Multichannel, SRBS, WHMCS, and Engage, and efficiently directing this data to specific consumers within the sr_wallet application for various charge types.

Solution: 

Kafka Connectors: Utilize Kafka connectors to capture changelog events from the databases.

Kafka Topics: Store these events in Kafka topics, preserving both before and after-update entries.

KSQL Processing: Apply KSQL to filter and transform the events, funneling them into new Kafka topics.

Consumption by sr_wallet: Have the sr_wallet application consume the final topics.

Multiple Consumers: Implement multiple consumers within sr_wallet for distinct charge categories, such as Engage charges, freight charges, WHMCS balance, and more.

How it Works: Kafka connectors continuously monitor binlog files in the databases, capturing any data changes.

The captured events are stored in Kafka topics, preserving the original and updated data.

KSQL is employed to process these events, applying filtering and data transformation logic as required.

Processed events are directed to new Kafka topics for each charge category.

Within sr_wallet, dedicated consumers handle specific charge types, ensuring proper data routing and processing.

Scalability:

Kafka’s distributed nature allows for horizontal scalability, accommodating increasing data volumes from multiple databases.

The modular structure of consumers within sr_wallet enables easy scalability by adding more consumers as needed for additional charge categories.

Kafka’s inherent ability to handle high-throughput data streams ensures that the solution can scale seamlessly with growing data demands and evolving business needs.

Conclusion

In each of these use cases, Apache Kafka facilitates real-time data processing, scalability, and efficient communication, ultimately enhancing the Shiprocket ecosystem’s performance and user experience.

In summary, Kafka topics are essential for organizing, distributing, and processing real-time data streams efficiently. They enable scalability, parallel processing, data retention, and fault tolerance. Kafka topics form the backbone of event-driven architectures, analytics pipelines, and seamless data integration within the Kafka ecosystem.