How to install Apache Kafka | Kafdrop on WindowsFor and integrate with Drupal

25 / Jun / 2023 by Dharmendra Singh 0 comments

Apache Kafka is an open-source distributed event streaming platform. That uses Publish & subscribe mechanism to stream the records.​

Download & Installing the Apache Kafka:

Prerequisite: You should have Java (JDK) installed on your Windows machine.

Step -1: Download the binary version from the kafka official download page ( https://kafka.apache.org/downloads )

Step -2: Unzip it and and rename the folder like kafka_binary & paste the folder in c drive like C:\dev\kafka_binary

Step -3: Change the log path setting in the below files

filename : C:\dev\kafka_binary\config\server.properties
Change :  log.dirs=/tmp/kafka-logs  =>  log.dirs=C:/dev/kafka_binary/kafka-logs

​               filename : C:\dev\kafka_binary\config\zookeeper.properties
Change : dataDir=C:/tmp/zookeeper  => dataDir=C:/dev/kafka_binary/zookeeper

Now Kafka is installed in your window machine & we can start the Kafka server. 

Kafka has a dependency on the zookeeper service, and this zookeeper is already inbuilt with our server; so first we need to start the zookeeper, then we can start the Kafka server.

Kafka command uses 2 files as a key-value pair to execute the command, It means for every task, there is 1 file in .bin\windows folder, which will work with the combination of .\config folder on the right side bar.

like

 
C:\dev\kafka_binary.\bin\windows\  => .\config\

Kafka Uses with the Help of CLI :

Open the CMD command prompt & perform the below commands,

Step -1: Start the ZooKeeper service by the below command, so it will start the Zookeeper server keep it running in this terminal; for performing any other cmd command, use another command prompt.

C:\dev\kafka_binary.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Start zooker server

Start zookeeper server

Step -2 : Start the Kafka broker service by the below command. So it will start the Kafka server keep it running in this terminal; for performing any other cmd command use the another command prompt.

C:\dev\kafka_binary\.\bin\windows\kafka-server-start.bat .\config\server.properties
Start Kafka Server

Start Kafka Server

Step -3 : Now we can create a topic to store your event,

C:\dev\kafka_binary\.\bin\windows\kafka-topics.bat --create --topic topic_part2 --bootstrap-server localhost:9092

Create Topic

Step -4 : Write some events into the topics

C:\dev\kafka_binary\.\bin\windows\kafka-console-producer.bat --topic topic_part2 --bootstrap-server localhost:9092

Write message in topic

Step -5 : Read the messages from the topic which we save in step -4, 

C:\dev\kafka_binary\.\bin\windows\kafka-console-consumer.bat --topic topic_part1 --from-beginning --bootstrap-server localhost:9092

Read message from kafka

 

That is the whole process to create message in the topic and read the message from the topic.

Download & Install PHP rd-Kafka Extension on Wamp :

I am using the Wamp & PHP version 8.0.26, if you have any other PHP version, update the command accordingly.

To enable the rd-kafka extension on wamp we need to download the DLL files from the below URL,

Reference : https://arnaud.le-blanc.net/php-rdkafka-doc/phpdoc/rdkafka.installation.windows.html

Download Link : http://pecl.php.net/package/rdkafka

Before downloading the correct version please check your system information like below

I am using the 5.0.2 version,

http://pecl.php.net/package/rdkafka/5.0.2/windows

I have Below configuration on my system please check all these config on your system if it differ then download the DLL based on that, otherwise it will not work properly.

php -v = 8.0.26
rdkafka version = 5.0.2
OS Type = 64-bit
Thread sefe = Enabled

Dll-download-for rd-kafka

 

Steps to configure the rd-kafka extension on Wamp​ Server

Step -1 : Paste the DLL files which we download on below Paths

C:\wamp64\bin\php\php8.0.26\   Paste the librdkafka.dllC:\wamp64\bin\php\php8.0.26\ext\ Paste the php_rdkafka.dll

Step -2 : Enable the rd-kafka extension in below files

C:\wamp64\bin\php\php8.0.26\php.ini  => Add  extension=php_rdkafka.dllC:\wamp64\bin\apache\apache2.4.54.2\bin\php.ini   => Add  extension=php_rdkafka.dll

Restart the server and verify with php –m , It should display the rdkafka extension in the list

Install & configure the Kafka Web UI – Kafdrop​

Requirement: 

    Java 11 or newer

   Kafka (version 0.11.0 or newer)

Reference to download the kafdrop : https://github.com/obsidiandynamics/kafdrop/releases

Step -1: Download the jar files from  below URL 

curl -LO https://github.com/obsidiandynamics/kafdrop/releases/download/3.30.0/kafdrop-3.30.0.jar

Step -2: Rename the jar file

mv kafdrop-3.30.0.jar kafdrop.jar

Step -3: Running/Executing the jar file

java –add-opens=java.base/sun.nio.ch=ALL-UNNAMED \

    -jar /c/kafdrop/kafdrop.jar \

    —kafka.brokerConnect=localhost:9092

Step -4: visit http://localhost:9000/

It should display the UI like below,  Make sure the Zookeeper & Kafka server is running.

kafdrop

Install & configure Drupal Module :

We can download the kafka drupal module from the https://www.drupal.org/project/kafka

Use composer to download the module

composer require ‘drupal/kafka:^1.0@beta’

After download Enable the Module & update below setting in settings.php file to connect to the kafka server

$settings['kafka'] = [​
'topic' => 'kafka-demo',​
'consumer' => [​
'brokers' => ['127.0.0.1:9092'],​
],​
'producer' => [​
'brokers' => ['127.0.0.1:9092'],​
],​
];

Now From the Drupal Menu you can view the Kafka Status Reports => Status report => Kafka

It should be something like Below,

drupal-kafka

That’s It,  Congratulations !!

FOUND THIS USEFUL? SHARE IT

Leave a Reply

Your email address will not be published. Required fields are marked *