Kafka Connect automatic sink/source initialization

This is something that somehow no search anywhere would turn up.

I’m using the Confluent Kafka Connect image, deployed into a K8S cluster via Helm. I’ve made a new image out of their base image so that I have certain Connector Plugins pre-installed.

FROM confluentinc/cp-kafka-connect-base:6.1.0

RUN confluent-hub install --no-prompt mongodb/kafka-connect-mongodb:1.5.0

When the container starts, I’d like it to initialize some connections using these plugins. Sink and Source both, if that matters.

I cannot for the life of me find any instruction on the best way to do that in some automated way when the container starts for the first time. Or even any way to do that.

It would seem like the best way might be to just do something like:

curl -X POST http://localhost:9092/connectors blah blah

… using a *-sink.properties file as the payload, but is that really how this is expected to be done?

How do I handle duplicates? curl http://localhost:9092/connectors, filter out pre-existing ones and POST the result?

This seems super clunky. Am I missing something?

Source: Docker Questions

LEAVE A COMMENT