How to install Apache Kafka | Kafdrop on WindowsFor and integrate with Drupal

Apache Kafka is an open-source distributed event streaming platform. That uses Publish & subscribe mechanism to stream the records.​

Download & Installing the Apache Kafka:

​Prerequisite: You should have Java (JDK) installed on your Windows machine.

Step -1: Download the binary version from the kafka official download page ( https://kafka.apache.org/downloads )

Step -2: Unzip it and and rename the folder like kafka_binary & paste the folder in c drive like C:\dev\kafka_binary

Step -3: Change the log path setting in the below files

filename : C:\dev\kafka_binary\config\server.properties​

Change :  log.dirs=/tmp/kafka-logs  =>  log.dirs=C:/dev/kafka_binary/kafka-logs

​               filename : C:\dev\kafka_binary\config\zookeeper.properties

Change : dataDir=C:/tmp/zookeeper  => dataDir=C:/dev/kafka_binary/zookeeper

Now Kafka is installed in your window machine & we can start the Kafka server. 

Kafka has a dependency on the zookeeper service, and this zookeeper is already inbuilt with our server; so first we need to start the zookeeper, then we can start the Kafka server.

Kafka command uses 2 files as a key-value pair to execute the command, It means for every task, there is 1 file in .bin\windows folder, which will work with the combination of .\config folder on the right side bar.

like

 
C:\dev\kafka_binary.\bin\windows\  => .\config\

Kafka Uses with the Help of CLI :

Open the CMD command prompt & perform the below commands,

Step -1: Start the ZooKeeper service by the below command, so it will start the Zookeeper server keep it running in this terminal; for performing any other cmd command, use another command prompt.​

C:\dev\kafka_binary.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

Start zookeeper server

Step -2 : Start the Kafka broker service by the below command. So it will start the Kafka server keep it running in this terminal; for performing any other cmd command use the another command prompt.

C:\dev\kafka_binary\.\bin\windows\kafka-server-start.bat .\config\server.properties​

Start Kafka Server

​Step -3 : Now we can create a topic to store your event,

​C:\dev\kafka_binary\.\bin\windows\kafka-topics.bat --create --topic topic_part2 --bootstrap-server localhost:9092

​

Step -4 : Write some events into the topics

C:\dev\kafka_binary\.\bin\windows\kafka-console-producer.bat --topic topic_part2 --bootstrap-server localhost:9092

​Step -5 : Read the messages from the topic which we save in step -4, 

C:\dev\kafka_binary\.\bin\windows\kafka-console-consumer.bat --topic topic_part1 --from-beginning --bootstrap-server localhost:9092

That is the whole process to create message in the topic and read the message from the topic.

Download & Install PHP rd-Kafka Extension on Wamp :

I am using the Wamp & PHP version 8.0.26, if you have any other PHP version, update the command accordingly.

To enable the rd-kafka extension on wamp we need to download the DLL files from the below URL,

Reference : https://arnaud.le-blanc.net/php-rdkafka-doc/phpdoc/rdkafka.installation.windows.html

Download Link : http://pecl.php.net/package/rdkafka

Before downloading the correct version please check your system information like below

I am using the 5.0.2 version,

http://pecl.php.net/package/rdkafka/5.0.2/windows

I have Below configuration on my system please check all these config on your system if it differ then download the DLL based on that, otherwise it will not work properly.

php -v = 8.0.26
rdkafka version = 5.0.2
OS Type = 64-bit
Thread sefe = Enabled

Steps to configure the rd-kafka extension on Wamp​ Server

Step -1 : Paste the DLL files which we download on below Paths

C:\wamp64\bin\php\php8.0.26\   Paste the librdkafka.dll​

C:\wamp64\bin\php\php8.0.26\ext\ Paste the php_rdkafka.dll​

Step -2 : Enable the rd-kafka extension in below files

C:\wamp64\bin\php\php8.0.26\php.ini  => Add  extension=php_rdkafka.dll​

C:\wamp64\bin\apache\apache2.4.54.2\bin\php.ini   => Add  extension=php_rdkafka.dll​

Restart the server and verify with php –m , It should display the rdkafka extension in the list

Install & configure the Kafka Web UI – Kafdrop​

Requirement: ​

    Java 11 or newer​

   Kafka (version 0.11.0 or newer)

Reference to download the kafdrop : https://github.com/obsidiandynamics/kafdrop/releases​

Step -1: Download the jar files from  below URL 

curl -LO https://github.com/obsidiandynamics/kafdrop/releases/download/3.30.0/kafdrop-3.30.0.jar​

Step -2: Rename the jar file

mv kafdrop-3.30.0.jar kafdrop.jar​

Step -3: Running/Executing the jar file

java –add-opens=java.base/sun.nio.ch=ALL-UNNAMED \​

    -jar /c/kafdrop/kafdrop.jar \​

    —kafka.brokerConnect=localhost:9092​

Step -4: visit http://localhost:9000/​

It should display the UI like below,  Make sure the Zookeeper & Kafka server is running.

Install & configure Drupal Module :

We can download the kafka drupal module from the https://www.drupal.org/project/kafka

Use composer to download the module

composer require ‘drupal/kafka:^1.0@beta’​

After download Enable the Module & update below setting in settings.php file to connect to the kafka server

$settings['kafka'] = [​
'topic' => 'kafka-demo',​
'consumer' => [​
'brokers' => ['127.0.0.1:9092'],​
],​
'producer' => [​
'brokers' => ['127.0.0.1:9092'],​
],​
];

Now From the Drupal Menu you can view the Kafka Status Reports => Status report => Kafka

It should be something like Below,

That’s It,  Congratulations !!