Questions tagged apache-kafka-connect

Explore the latest questions and answers asked by our top developers.

Kafka Connect Error : java.lang.NoClassDefFoundError: org/apache/http/conn/HttpClientConnectionManager

I’m using docker with kafka and clickhouse. I want to connect ‘KsqlDB table’ and ‘clickhouse’ using ‘kafka connect’. So I referred to this document and modified ‘docker composite’. here is my docker-compose — version: ‘2’ services: zookeeper: image: confluentinc/cp-zookeeper:6.2.1 hostname: zookeeper container_name: zookeeper ports: – "2181:2181" environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 broker: image: confluentinc/cp-server:6.2.1 hostname: […]

Using Debezium, but cannot see changes in sink Postgres Docker container

I am learning Debezium, and I am new to Docker as well. I want to replicate changes from a source Postgres database to a destination Postgres (empty). Both are in separate Docker containers. I am using Debezium to replicate the changes. When I run the Docker stack, and make changes in source Postgres, I can […]

Add MongoDB Sink Connector on docker?

I have a Ubuntu machine, where I followed this steps in order to run Confluent Platform with docker. https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html I can produce and subscribe to messages just fine. I’m trying to add a MongoDB Sink Connector, in order to sync data with a mongo database. I’ve downloaded this zip file https://www.confluent.io/hub/hpgrahsl/kafka-connect-mongodb I’ve edited the etc/MongoDbSinkConnector.properties […]

Running docker containers for Kafka-MQTT-MongoDB; LEADER_NOT_AVAILABLE for kafka topic

I am running Docker containers for kafka, zookeeper, kafka-connect, mongoDB & mosquitto within a network. When i register connector for mqtt source and send a message to bootstrap server: kafka:29092, I get the following warning when I listen to the topic. docker exec kafka kafka-console-consumer –bootstrap-server kafka:29092 –topic connect-custom –from-beginning –max-messages 42 [2021-08-25 11:53:04,555] WARN […]

How to keep all the settings configured even after restarting a machine with confluent kafka docker-compose configured?

Here’s the docker-compose file I am using for kafka and ksqldb setup, — version: ‘2’ services: zookeeper: image: confluentinc/cp-zookeeper:6.2.0 hostname: zookeeper container_name: zookeeper ports: – "2181:2181" environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 broker: image: confluentinc/cp-server:6.2.0 hostname: broker container_name: broker depends_on: – zookeeper ports: – "9092:9092" – "9101:9101" environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: ‘zookeeper:2181’ KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092 […]

Debezium with Postgres | Kafka Consumer not able to consume any message

Here is my docker-compose file: version: ‘3.7’ services: postgres: image: debezium/postgres:12 container_name: postgres networks: – broker-kafka environment: POSTGRES_PASSWORD: admin POSTGRES_USER: antriksh ports: – 5499:5432 zookeeper: image: confluentinc/cp-zookeeper:latest container_name: zookeeper networks: – broker-kafka ports: – 2181:2181 environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 kafka: image: confluentinc/cp-kafka:latest container_name: kafka networks: – broker-kafka depends_on: – zookeeper ports: – 9092:9092 environment: […]

Mount (Volumes) not working when embedding Kafka connect into Ksqldb-server

The "volumes" mapping does not seem to work when trying to embed Kafka connector in Ksqldb Server Below is my docker file ksqldb-server: image: confluentinc/ksqldb-server:0.18.0 hostname: ksqldb-server container_name: ksqldb-server depends_on: – broker – schema-registry ports: – "8088:8088" volumes: – "/usr/local/share/kafka/connectors/debezium-debezium-connector-mysql/lib:/usr/share/kafka/plugins/" environment: KSQL_LISTENERS: "http://0.0.0.0:8088" KSQL_BOOTSTRAP_SERVERS: "broker:9092" KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:8081" KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true" KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true" # Configuration to embed […]

How to launch Docker Compose and run curl together from dockerfile

Hy guys, I’m newer in Docker. I have a problem with write a dockerfile for my application. My code is beloy: # define a imagem base FROM ubuntu:latest # define a owner image LABEL maintainer="MyCompany" # Update a image with packages RUN apt-get update && apt-get upgrade -y # Expose port 80 EXPOSE 8089 # […]

Debezium Kafka Connect – Error Loading Plugin From Confluent Hub via Docker

I’ve recently tried installing the Debezium PostgreSQL Connector v1.6.0 via confluent-hub. When the plugin is loaded, the connector fails to load, raised by loadProperties in IoUtil, per this error: [2021-07-13 00:45:06,715] INFO Loading plugin from: /usr/share/confluent-hub-components/debezium-debezium-connector-postgresql (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [2021-07-13 00:45:07,375] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed) java.lang.ExceptionInInitializerError at io.debezium.connector.postgresql.PostgresConnector.version(PostgresConnector.java:47) at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.versionFor(DelegatingClassLoader.java:387) at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.versionFor(DelegatingClassLoader.java:392) at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:362) at […]

Connection timeout using local kafka-connect cluster to connect on a remote database

I’m trying to run a local kafka-connect cluster using docker-compose. I need to connect on a remote database and i’m also using a remote kafka and schema-registry. I have enabled access to these remotes resources from my machine. To start the cluster, on my project folder in my Ubuntu WSL2 terminal, i’m running docker build […]

1 2 3 4 5

Still Have Questions?


Our dedicated development team is here for you!

We can help you find answers to your question for as low as 5$.

Contact Us
faq