GitLab pipeline strange failure, even on an old, previously "green" commit

I’m experiencing a very strange problem with GitLab CI: I have a JVM based pet project with end-to-end tests that use EventStore, MongoDb and RabbitMq, so a few months ago I configured .gitlab-ci-yml defining a couple of services: my e2e tests run against them and every worked well.

A few days ago I made some changes to my project, pushed to GitLab and the pipeline has failed, due the fact that my e2e test do not seem to receive the correct number of messages from RabbitMq.
After a few days of investigation I gathered some information:

  • the e2e test run without errors on two different local machines (in my IDE)
  • the same test fails systematically on the GitLab pipeline
  • the failing test fails due to a wrong number of received messages, but the wrong number never changes: the error seems to be deterministic
  • the e2e test run without errors launching it locally through gitlab-runner (same 12.5.0 version as in the remote pipeline execution)
  • if I start the pipeline execution from an old commit, for which the pipeline ran without errors months ago, the pipeline fails: same commit, same code, same services configuration in .gitlab-ci.yml, but the pipeline is now red (and it was green a few months ago)
  • in order to exclude any strange dependencies from the Docker latest tag, I fixed one of the services I’m using, specifying the version-tag explicitly: same result, remote pipeline red but local execution through gitlab-runner green
  • my .gitlab-ci.yml file is available here
  • example of pipeline execution for an old commit can be found here (green) and here(red): the main difference I see is the version of the GitLab runner involved, 12.5.0-rc1 (for the red case) vs. 11.10.1 (for the green one)

I realize that the question is complex and the variables involved are many, so… does anyone have any suggestions on how I can try to debug this scenario? Thank you very much in advance: any help will be appreciated!

Source: StackOverflow