Java HiveMQ Client reconnects every second from different port in docker swarm

  docker, docker-swarm, hivemq, mosquitto, mqtt

I have a Java Spring Service in which I recently added a connection to an eclipse-mosquitto-container with the hiveMQ-library. The mosquitto and java service are both running in a docker swarm with the following configs:

version: "3.7"

services:
  mqtt:
    image: eclipse-mosquitto:2.0.7
    volumes:
      - mqtt_data:/mosquitto/data
      - /etc/config/mosquitto.conf:/mosquitto/config/mosquitto.conf
    secrets:
      - mqtt_secret
    networks:
      mqtt_net:
        aliases:
          - mqtt

networks:
  mqtt_net:
    external: true
volumes:
  mqtt_data:
    external: true
secrets:
  mqtt_secret:
    external: true

and the Java service (I simplified it a little and left out env vars for the database etc. to make it more easily readable):

version: "3.7"

services:
  my-app:
    image: my-registry/my-app
    networks:
      - mysql_net
      - gateway_net
      - es_net
      - mqtt_net

networks:
  mysql_net: 
    external: true
  gateway_net:
    external: true
  notes_es_net:
    external: true
  mqtt_net:
    external: true

I tried running the java service locally against mqtt which worked without problems. It works on the server too but as soon as the java-container started up my mqtt-logs EXPLODE with tons of messages every second:

mqtt_mqtt.1   | 1617469247: New connection from 10.0.6.6:48830 on port 1883.
mqtt_mqtt.1   | 1617469247: Client 8nb40u8qa88t9f0qs1gt28 already connected, closing old connection.
mqtt_mqtt.1   | 1617469247: New client connected from 10.0.6.6:48830 as 8nb40u8qa88t9f0qs1gt28 (p5, c1, k60, u'notes-app').
mqtt_mqtt.1   | 1617469247: New connection from 10.0.6.6:48836 on port 1883.
mqtt_mqtt.1   | 1617469248: New connection from 10.0.6.6:48838 on port 1883.
mqtt_mqtt.1   | 1617469248: Client 8nb40u8qa88t9f0qs1gt28 already connected, closing old connection.
mqtt_mqtt.1   | 1617469248: New client connected from 10.0.6.6:48838 as 8nb40u8qa88t9f0qs1gt28 (p5, c1, k60, u'notes-app').
mqtt_mqtt.1   | 1617469248: New connection from 10.0.6.6:48842 on port 1883.
mqtt_mqtt.1   | 1617469248: Client 8nb40u8qa88t9f0qs1gt28 already connected, closing old connection.
mqtt_mqtt.1   | 1617469248: New client connected from 10.0.6.6:48836 as 8nb40u8qa88t9f0qs1gt28 (p5, c1, k60, u'notes-app').
mqtt_mqtt.1   | 1617469248: Client 8nb40u8qa88t9f0qs1gt28 already connected, closing old connection.
mqtt_mqtt.1   | 1617469248: New client connected from 10.0.6.6:48842 as 8nb40u8qa88t9f0qs1gt28 (p5, c1, k60, u'notes-app').
mqtt_mqtt.1   | 1617469248: New connection from 10.0.6.6:48848 on port 1883.
mqtt_mqtt.1   | 1617469248: Client 8nb40u8qa88t9f0qs1gt28 already connected, closing old connection.
mqtt_mqtt.1   | 1617469248: New client connected from 10.0.6.6:48848 as 8nb40u8qa88t9f0qs1gt28 (p5, c1, k60, u'notes-app').
mqtt_mqtt.1   | 1617469248: New connection from 10.0.6.6:48854 on port 1883.
mqtt_mqtt.1   | 1617469248: Client 8nb40u8qa88t9f0qs1gt28 already connected, closing old connection.
mqtt_mqtt.1   | 1617469248: New client connected from 10.0.6.6:48854 as 8nb40u8qa88t9f0qs1gt28 (p5, c1, k60, u'notes-app').
mqtt_mqtt.1   | 1617469248: New connection from 10.0.6.6:48856 on port 1883.
mqtt_mqtt.1   | 1617469248: Client 8nb40u8qa88t9f0qs1gt28 already connected, closing old connection.
mqtt_mqtt.1   | 1617469248: New client connected from 10.0.6.6:48856 as 8nb40u8qa88t9f0qs1gt28 (p5, c1, k60, u'notes-app').
mqtt_mqtt.1   | 1617469248: New connection from 10.0.6.6:48858 on port 1883.
mqtt_mqtt.1   | 1617469248: Client 8nb40u8qa88t9f0qs1gt28 already connected, closing old connection.
mqtt_mqtt.1   | 1617469248: New client connected from 10.0.6.6:48858 as 8nb40u8qa88t9f0qs1gt28 (p5, c1, k60, u'notes-app').
mqtt_mqtt.1   | 1617469249: New connection from 10.0.6.6:48860 on port 1883.
mqtt_mqtt.1   | 1617469249: Client 8nb40u8qa88t9f0qs1gt28 already connected, closing old connection.
mqtt_mqtt.1   | 1617469249: New client connected from 10.0.6.6:48860 as 8nb40u8qa88t9f0qs1gt28 (p5, c1, k60, u'notes-app').
mqtt_mqtt.1   | 1617469249: New connection from 10.0.6.6:48862 on port 1883.
mqtt_mqtt.1   | 1617469249: Client 8nb40u8qa88t9f0qs1gt28 already connected, closing old connection.
mqtt_mqtt.1    | 1617469249: New client connected from 10.0.6.6:48862 as 8nb40u8qa88t9f0qs1gt28 (p5, c1, k60, u'notes-app').

It looks to me as if the client reconnects multiple times a second with a different port which is a total mystery to me. Does docker swarm do something with networks internally which could explain this? Or does the hiveMQ-client change the port on reconnect?

I have no idea how to explain this behaviour or how to fix it. I tried playing around with the hiveMQ-reconnect-config but it always worked perfect normally and "broke" once deployed in docker swarm.

This is my hiveMQ-client in java (I am not sending messages or anything before the log starts spamming):

var builder = Mqtt5Client.builder()
                .identifier("notes-app-" + UUID.randomUUID().toString())
                .serverHost(host)
                .automaticReconnectWithDefaultConfig();
        if (user != null && password != null) {
            builder = builder.simpleAuth().username(user).password(password.getBytes(StandardCharsets.UTF_8)).applySimpleAuth();
        }
        this.publisher = builder.build();
        this.publisher.toBlocking().connect();

Other than the massive amount of logs, the functionality is not affected. This behaviour does however not look healthy to me and I am sure that this will run into problems once multiple clients connect or the port numbers get too large.

Can anyone point me in the right direction of what I am doing wrong?

Source: Docker Questions

LEAVE A COMMENT