Unable to push docker image to Nexus

  artifactory, continuous-integration, docker, java, nexus

I am running Nexus OSS version 3.29.2-02 and I am experiencing some weird behavior. I am building various images at a CI level (GitLab) and I am pushing them to a custom repository.

For the most part everything works OK and I have no issues tagging and pushing my produced images. Lately though, one of the projects that produces Docker images fails to push, with the following error:

$ TAGGED=${NEXUS_DOCKER_URL}/${BASE_IMAGE_NAME}:snapshot-MR${CI_MERGE_REQUEST_IID}
$ docker tag ${BASE_IMAGE_NAME}:latest ${TAGGED}
$ docker push ${TAGGED}
The push refers to repository [<custom-repository-url>/<image-name:tag>]
cade37b0f9c9: Preparing
578ec024f17c: Preparing
fe0b994190e8: Preparing
b24d08ca4359: Preparing
9a14db3b513b: Preparing
777b2c648970: Preparing
777b2c648970: Waiting
b24d08ca4359: Layer already exists
9a14db3b513b: Layer already exists
777b2c648970: Layer already exists
fe0b994190e8: Pushed
cade37b0f9c9: Pushed
578ec024f17c: Pushed
[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the <custom-repository-url> registry NOW to avoid future disruption.
errors:
blob unknown: blob unknown to registry
blob unknown: blob unknown to registry
ERROR: Job failed: exit code 1

I have tried debugging this behavior as well as search online for a solution but I have yet to find anything. It seems that for some reason, this specific Docker image cannot be uploaded. I have tried the same procedure from both a local machine as well as from stateless CI builders and the behavior is consistent i.e. I was able to push it only once and then the process kept failing.

For reference my Dockerfile is the following:

FROM <custom-repository-url>/adoptopenjdk/openjdk11:jre-11.0.10_9-alpine
WORKDIR /home/app
COPY build/libs/email-service.jar application.jar

# Set the appropriate timezone
RUN apk add --no-cache tzdata && 
    cp /usr/share/zoneinfo/America/New_York /etc/localtime && 
    echo "America/New_York" > /etc/timezone

EXPOSE 8080
CMD java -jar ${OPTS} application.jar

Which is quite straightforward and does not hide anything complicated. I initially thought that the problem could have been attributed to using a proxied based image (i.e FROM) but this is done of several other projects without any issues.

I have tried also checking Nexus’s logs and the only thing I see is the following:

2021-02-05 17:12:27,441+0000 ERROR [qtp1025847496-15765]  ci-deploy org.sonatype.nexus.repository.docker.internal.orient.V2ManifestUtilImpl - Manifest refers to missing layer: sha256:66db482b5034f8eda0b18533d4eddb0012f4940bf3d348b08ac3bac8486bb2ee for: fts/marketing/email-service/snapshot-MR40 in repository RepositoryImpl$$EnhancerByGuice$$4d5af99c{type=hosted, format=docker, name='docker-hosted-s3'}
2021-02-05 17:12:27,443+0000 ERROR [qtp1025847496-15765]  ci-deploy org.sonatype.nexus.repository.docker.internal.orient.V2ManifestUtilImpl - Manifest refers to missing layer: sha256:2ec25ba939258edb2e85293896c5126478d79fe416d3b60feb20426755bcea5a for: fts/marketing/email-service/snapshot-MR40 in repository RepositoryImpl$$EnhancerByGuice$$4d5af99c{type=hosted, format=docker, name='docker-hosted-s3'}
2021-02-05 17:12:27,445+0000 WARN  [qtp1025847496-15765]  ci-deploy org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: PUT /v2/fts/marketing/email-service/manifests/snapshot-MR40: 400 - org.sonatype.nexus.repository.docker.internal.V2Exception: Invalid Manifest

So my question are:

  1. What does this error really mean? I don’t find it very useful:
errors:
blob unknown: blob unknown to registry
blob unknown: blob unknown to registry
  1. What is really causing this behavior and how can I address the problem?

Note (not that it should make any difference), the image is a dockerized Micronaut application, using the latest version of the framework.

Source: Docker Questions

LEAVE A COMMENT