Using Apache Kafka for fault tolerant systems

I work in finance. Losing data equals death. I had never heard of a messaging queue system about 4 months before (though people in my circles did talk about RabbitMQ but I didn’t know much about it). I use Kafka (parallel/competitor to RabbitMQ) for logging data. As I mentioned, there is not a possibility for my system to lose data.

In our case, we have a 3 node Kafka cluster which we use to collect data from a time intensive financial application. Data loss = Huge liabilities. So we ensure that all three machines (hosting our Kafka nodes and consumer scripts) are working all the time. Say if the consumer script dies or Kafka (or underlying zookeeper) crashes for some reason, or there is an earthquake due to which one of the machine dies, the other two machines will automatically take care of the load balancing part for incoming stream of data (isn’t it awesome!!).

Say the second machine dies too, the 3rd machine can take all the load (not a good scenario, but still our a** is covered). God forbid, even the 3rd machine dies, Kafka still logs the data it receives. By default the log duration is 7 days, which means that after 7 days the old data is cleared in the manner of a ‘queue’. I read somewhere that this is one of the big differences b/w Kafka and RabbitMQ. So, I can still go back to the logs and pick out missing data from the message queue.

The good as well as bad thing about Kafka are the config files. In particular, zookeeper.properties and server.properties file within the config folder. It takes some time to understand what different settings inside these files do, but once done, life becomes a breeze. I recommend spending some time playing around different configs to get an idea about it.

Feeling stuck? Use ps -ef | grep zookeeper.properties or ps -ef | grep server.properties to kill zookeeper and kafka server respectively.

That’s it folks. Kafka provides you High Availability and fault tolerance for different scenarios and I feel that it’s a must have for all modern data intensive as well as data critical systems.

Related posts:

The first Virus World NFT illustration starts in human form. However, as the level of the NFT increases through the holder’s V2E activities, it gradually changes into a zombie or monster. This…

We seem to hear about a major cybersecurity breach or a ransom-ware attack almost on a daily basis. And this isn’t going to change anytime soon and the main reason for that is the Internet was never…

Definitely one of my pals the sad thing is is suffering from schizophrenia. It engineered during his overdue young people, and regrettably he is in a family members with moms and dads who struggled…