Kafka is a real time data stream processing tool. In an understandable way. You can add a message onto Kafka and others can have a subscription on a messag for processing it. The sender is the Producer that publishes a message on a topic and the receiver is an Consumer that subsribes on a topic. For some basic info read the wiki page about Apache Kafka. As you know I’m Redwood RunMyJobs (RMJ) administrator and Jave developer (formerly plsql dev) at ASWatson. From the business it was required to als support Kafka processes.
RMJ in itself is an eventbased processautomation tool build in Java. Custom made Java code can be hooked into RMJ by creating a library and link that to processdefinitions. So there I had a base to start developing. First attempt where huge piles of serial code. No object no nothing ugly to the base. A mixed bag op repetitive code for consumer producers with and withou AVRO serializing. And worst of all. One of the consumers created piles of extra processes. Blocking the whole system. As we say in Dutch ‘Dit moest anders’.
So I started to think about building a Factory. A DTO (Pojo) that can be fed with a mixbag of parameters on which a producer or consumer can be started. Anything is in there. From base Credentials to kafka broker url string. From a Oracle jdbc connection that can handle stored procedure returning ref cursors to retreive data from a database to stored procedures to process the messages from a consumer. And of course the properties for AVRO or String based (de-)serializing.
This whole KafkaClientFactory can by found on my GitHub.
https://github.com/JohannesKalma/KafkaClientFactory
This KafkaClientFactory can run from any (maven based) java project. For linking this library to the RMJ way of working I wrote a Parameter Wrapper, that maps the parameters from a job onto the KafkaClientFactory. I create a seperate gitHub project for that.
https://github.com/JohannesKalma/RunMyJobsKafkaClientFactoryParameterMapper