Category : apache-kafka

I build a docker container based on alpine linux. I try to send messages to an external kafka broker using the symfony messenger. This is my messenger config: messenger: transports: de_fadi_criminal_charges_public_criminal_charge_created: dsn: ‘%env(KAFKA_DSN)%’ serializer: AppSerializerAvroCriminalChargeCreatedSerializer options: flushTimeout: 10000 flushRetries: 5 topic: name: ‘de.fadi.criminal_charges.public.criminal_charge_created’ kafka_conf: security.protocol: ‘sasl_ssl’ ssl.ca.location: ‘%kernel.project_dir%/config/kafka/ca.pem’ sasl.username: ‘%env(KAFKA_SASL_USERNAME)%’ sasl.password: ‘%env(KAFKA_SASL_PASSWORD)%’ sasl.mechanism: ‘PLAIN’ and ..

Read more

When I launch Docker container with Kafka broker it fails sometimes, but I can’t understand by logs what exactly happens, logs always are: # docker-compose up broker-1 zookeeper-1 is up-to-date Starting broker-1 … done Attaching to broker-1 broker-1 | ===> User broker-1 | uid=1000(appuser) gid=1000(appuser) groups=1000(appuser) broker-1 | ===> Configuring … broker-1 | ===> Running ..

Read more

Given this doker-compose.yml version: ‘2’ services: zookeeper: image: confluentinc/cp-zookeeper:latest environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 kafka: image: confluentinc/cp-kafka:latest depends_on: – zookeeper ports: – 9092:9092 environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 I tried to send some data to docker consumer : val props = new Properties() props.put("bootstrap.servers", "kafka:9092") props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer") ..

Read more

I am new to Kafka. I tried to create Kafak topic but I am getting the error. docker exec -ti kafka-server1 > /opt/bitnami/kafka/bin/kafka-topics.sh –create > –zookeeper zookeeper-server:2181 > –replication-factor 1 > –partitions 1 > –topic topic_model_local Exception in thread "main" joptsimple.UnrecognizedOptionException: zookeeper is not a recognized option at joptsimple.OptionException.unrecognizedOption(OptionException.java:108) at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510) at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56) at joptsimple.OptionParser.parse(OptionParser.java:396) ..

Read more

When I change image in docker-compose.yml from enterprisec cp-enterprise-kafka: https://hub.docker.com/layers/confluentinc/cp-enterprise-kafka/6.2.1/images/sha256-8a1b5bdeacc1a4a3bad6353caea9bfc0cacfe4a7ac3e48cd0797c3ab36032f39?context=explore to regular cp-kafka: https://hub.docker.com/layers/confluentinc/cp-kafka/6.2.1/images/sha256-870c760a9e184260c159e4f7e5b4e715be0b89726b397f709fd6ce43add34456?context=explore it fails to start with: enter image description here My config in docker-compose.yml is: version: ‘3’ services: zookeeper-1: image: confluentinc/cp-zookeeper:6.2.1 hostname: zookeeper-1 container_name: zookeeper-1 ports: – "2181:2181" – "2888:2888" – "3888:3888" environment: ZOOKEEPER_SERVER_ID: 1 ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 volumes: – ./kafka-data/zookeeper-1:/var/lib/zookeeper/data ..

Read more

Hi I’m trying to run kafka with unique broker hosts for each node the main goal is to make sure that after restarting the Kafka service, broker hosts do not change i try to do something like this version: ‘3.7’ services: zookeeper: image: wurstmeister/zookeeper ports: – "2181:2181" kafka: image: wurstmeister/kafka:2.12-2.5.0 hostname: "{{.Node.Hostname}}" deploy: mode: global ..

Read more