EXPLORING THE CAPABILITIES OF KAFKA STREAMS FOR REAL-TIME STREAM PROCESSING: A PRACTICAL IMPLEMENTATION

Abstract

The ultimate goal of my research is to explore the capabilities of Apache Kafka, Kafka Streams API and Java 8 to process data streams in real time. As you know, the volume of data generated by modern applications is growing faster than traditional processing methods can process it. Many IT companies, including those in Kazakhstan, have problems processing big data in real time, especially in the financial sector and payroll departments, where calculation errors can have serious consequences as financial losses due to incorrect data processing. As part of the study, I studied the architecture, functionality and capabilities of Kafka Streams, as well as studied the literature and examples of using Kafka Streams in real projects. As a result, several demonstrative projects have been developed and implemented, as well as a practical system for calculating the payroll of employees, which shows the benefits of using Kafka Streams to process data streams in real time. To evaluate the effectiveness of the system, several experiments were carried out and the corresponding metrics were analyzed. In particular, I compared the performance of batch processing and real-time processing using several metrics: number of messages processed along with latency, throughput, and fetch requests. These metrics allow to evaluate the performance of messaging system and understand how well it is performing. Overall, this paper is a valuable resource for researchers and practitioners interested in using Apache Kafka, the Kafka Streams API, and Java 8 to process real-time data streams and solve real-time big data processing problems.