Traefik with Docker swarm not generating letsencrypt certificate

  docker, docker-swarm, traefik

"Warn" logs from traefik:

[email protected]:~# docker logs traefik_traefik.1.rt2qd68ainjp75mtzepzyx5mt
time="2021-10-27T13:35:34Z" level=info msg="Configuration loaded from flags."
time="2021-10-27T13:35:45Z" level=error msg="Unable to obtain ACME certificate for domains "traefik.[redacted].com": unable to generate a certificate for the domains [traefik.[redacted].com]: error: one or more domains had a problem:n[traefik.[redacted].com] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Cannot negotiate ALPN protocol "acme-tls/1" for tls-alpn-01 challengen" providerName=letsencrypt.acme [email protected] rule="Host(`traefik.[redacted].com`)"
time="2021-10-27T13:35:52Z" level=error msg="Unable to obtain ACME certificate for domains "traefik.[redacted].com": unable to generate a certificate for the domains [traefik.[redacted].com]: error: one or more domains had a problem:n[traefik.[redacted].com] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Cannot negotiate ALPN protocol "acme-tls/1" for tls-alpn-01 challengen" providerName=letsencrypt.acme rule="Host(`traefik.[redacted].com`)" [email protected]

traefik.yml file:

version: "3.8"

networks:
  t2_proxy:
    external: true
  default:
    driver: bridge
services:
  traefik:
    image: traefik:latest
    deploy:
      labels:
        - traefik.enable=true
        - traefik.http.services.traefik-https.loadbalancer.server.port=443
        - traefik.http.routers.traefik-http.entrypoints=http
        - traefik.http.routers.traefik-http.rule=Host(`traefik.[redacted].com`)
        - traefik.http.routers.traefik-http.middlewares=redirect-to-https
        - traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https
        - traefik.http.routers.traefik-https.entrypoints=https
        - traefik.http.routers.traefik-https.rule=Host(`traefik.[redacted].com`)
        - traefik.http.routers.traefik-https.tls=true
        - traefik.http.routers.traefik-https.tls.certresolver=letsencrypt
        - [email protected]
        - traefik.http.middlewares.auth.basicauth.usersfile=/var/data/secrets/htpasswd
      placement:
        constraints:
          - node.role == manager
      restart_policy:
        condition: on-failure
    command: # CLI arguments
      - --global.checkNewVersion=true
      - --global.sendAnonymousUsage=true
      - --entryPoints.http.address=:80
      - --entryPoints.https.address=:443
      - --entryPoints.ping.address=:8081
      - --entrypoints.https.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/12,172.64.0.0/13,131.0.72.0/22
      - --api=true
      - --api.insecure=false
      - --api.dashboard=true
      - --log=true
      - --log.level=WARN # (Default: error) DEBUG, INFO, WARN, ERROR, FATAL, PANIC
      - --accessLog=true
      - --accessLog.filePath=/traefik.log
      - --accessLog.bufferingSize=100
      - --accessLog.filters.statusCodes=400-499
      - --providers.docker=true
      - --providers.docker.endpoint=unix:///var/run/docker.sock
      - --providers.docker.exposedByDefault=false
      - --providers.docker.network=t2_proxy
      - --providers.docker.swarmMode=true
      - --providers.file.directory=/rules # Load dynamic configuration from one or more .toml or .yml files in a directory.
      - --providers.file.watch=true # Only works on top level files in the rules folder
      - --certificatesresolvers.letsencrypt.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory # LetsEncrypt Staging Server - uncomment when testing
      - --certificatesResolvers.letsencrypt.acme.email=[redacted]
      - --certificatesResolvers.letsencrypt.acme.storage=/acme.json
      - --certificatesResolvers.letsencrypt.acme.tlsChallenge=true
    networks:
      t2_proxy:
      socket_proxy
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    volumes:
      - /var/data/files/traefik/rules:/rules 
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/data/files/traefik/acme/acme.json:/acme.json 
      - /var/data/files/traefik/traefik.log:/traefik.log 
      - /var/data/files/shared:/shared
    environment:
      - CF_API_EMAIL=[redacted]
      - CF_API_KEY=[redacted]

Debug logs from traefik:

[email protected]:~# docker logs traefik_traefik.1.q9qvaqlczjxw6lm86yksghz0q
time="2021-10-28T08:43:58Z" level=info msg="Configuration loaded from flags."
time="2021-10-28T08:43:58Z" level=info msg="Traefik version 2.5.3 built on 2021-09-20T15:43:56Z"
time="2021-10-28T08:43:58Z" level=debug msg="Static configuration loaded {"global":{"checkNewVersion":true,"sendAnonymousUsage":true},"serversTransport":{"maxIdleConnsPerHost":200},"entryPoints":{"http":{"address":":80","transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s"}},"forwardedHeaders":{},"http":{},"udp":{"timeout":"3s"}},"https":{"address":":443","transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s"}},"forwardedHeaders":{"trustedIPs":["173.245.48.0/20","103.21.244.0/22","103.22.200.0/22","103.31.4.0/22","141.101.64.0/18","108.162.192.0/18","190.93.240.0/20","188.114.96.0/20","197.234.240.0/22","198.41.128.0/17","162.158.0.0/15","104.16.0.0/12","172.64.0.0/13","131.0.72.0/22"]},"http":{},"udp":{"timeout":"3s"}},"ping":{"address":":8081","transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s"}},"forwardedHeaders":{},"http":{},"udp":{"timeout":"3s"}}},"providers":{"providersThrottleDuration":"2s","docker":{"watch":true,"endpoint":"unix:///var/run/docker.sock","defaultRule":"Host(`{{ normalize .Name }}`)","swarmMode":true,"network":"t2_proxy","swarmModeRefreshSeconds":"15s"},"file":{"directory":"/rules","watch":true}},"api":{"dashboard":true},"log":{"level":"DEBUG","format":"common"},"accessLog":{"filePath":"/traefik.log","format":"common","filters":{"statusCodes":["400-499"]},"fields":{"defaultMode":"keep","headers":{"defaultMode":"drop"}},"bufferingSize":100},"certificatesResolvers":{"letsencrypt":{"acme":{"email":"[redacted]","caServer":"https://acme-staging-v02.api.letsencrypt.org/directory","storage":"/acme.json","keyType":"RSA4096","tlsChallenge":{}}}},"pilot":{"dashboard":true}}"
time="2021-10-28T08:43:58Z" level=info msg="Stats collection is enabled."
time="2021-10-28T08:43:58Z" level=info msg="Many thanks for contributing to Traefik's improvement by allowing us to receive anonymous information from your configuration."
time="2021-10-28T08:43:58Z" level=info msg="Help us improve Traefik by leaving this feature on :)"
time="2021-10-28T08:43:58Z" level=info msg="More details on: https://doc.traefik.io/traefik/contributing/data-collection/"
time="2021-10-28T08:43:58Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
time="2021-10-28T08:43:58Z" level=debug msg="Start TCP Server" entryPointName=ping
time="2021-10-28T08:43:58Z" level=debug msg="Start TCP Server" entryPointName=http
time="2021-10-28T08:43:58Z" level=debug msg="Start TCP Server" entryPointName=https
time="2021-10-28T08:43:58Z" level=info msg="Starting provider *file.Provider {"directory":"/rules","watch":true}"
time="2021-10-28T08:43:58Z" level=info msg="Starting provider *traefik.Provider {}"
time="2021-10-28T08:43:58Z" level=info msg="Starting provider *acme.Provider {"email":"[redacted]","caServer":"https://acme-staging-v02.api.letsencrypt.org/directory","storage":"/acme.json","keyType":"RSA4096","tlsChallenge":{},"ResolverName":"letsencrypt","store":{},"TLSChallengeProvider":{"Timeout":4000000000},"HTTPChallengeProvider":{}}"
time="2021-10-28T08:43:58Z" level=info msg="Testing certificate renew..." providerName=letsencrypt.acme
time="2021-10-28T08:43:58Z" level=info msg="Starting provider *acme.ChallengeTLSALPN {"Timeout":4000000000}"
time="2021-10-28T08:43:58Z" level=debug msg="Configuration received from provider file: {"http":{},"tcp":{},"udp":{},"tls":{}}" providerName=file
time="2021-10-28T08:43:58Z" level=debug msg="Configuration received from provider internal: {"http":{"services":{"api":{},"dashboard":{},"noop":{}},"serversTransports":{"default":{"maxIdleConnsPerHost":200}}},"tcp":{},"tls":{}}" providerName=internal
time="2021-10-28T08:43:58Z" level=debug msg="Configuration received from provider letsencrypt.acme: {"http":{},"tls":{}}" providerName=letsencrypt.acme
time="2021-10-28T08:43:58Z" level=debug msg="No default certificate, generating one" tlsStoreName=default
time="2021-10-28T08:43:58Z" level=info msg="Starting provider *docker.Provider {"watch":true,"endpoint":"unix:///var/run/docker.sock","defaultRule":"Host(`{{ normalize .Name }}`)","swarmMode":true,"network":"t2_proxy","swarmModeRefreshSeconds":"15s"}"
time="2021-10-28T08:43:58Z" level=debug msg="Provider connection established with docker 20.10.7 (API 1.41)" providerName=docker
time="2021-10-28T08:43:58Z" level=debug msg="Filtering disabled container" providerName=docker container=shepherd-shepherd-app-0eyd5cbldo09vov2eykpsv899
time="2021-10-28T08:43:58Z" level=debug msg="Configuration received from provider docker: {"http":{"routers":{"traefik-http":{"entryPoints":["http"],"middlewares":["redirect-to-https"],"service":"traefik-https","rule":"Host(`traefik.[redacted].com`)"},"traefik-https":{"entryPoints":["https"],"service":"[email protected]","rule":"Host(`traefik.[redacted].com`)","tls":{"certResolver":"letsencrypt"}}},"services":{"traefik-https":{"loadBalancer":{"servers":[{"url":"http://10.0.5.194:443"}],"passHostHeader":true}}},"middlewares":{"auth":{"basicAuth":{"usersFile":"/var/data/secrets/htpasswd"}},"redirect-to-https":{"redirectScheme":{"scheme":"https"}}}},"tcp":{},"udp":{}}" providerName=docker
time="2021-10-28T08:43:58Z" level=debug msg="http: panic serving 172.70.59.162:61878: runtime error: invalid memory address or nil pointer dereference"
time="2021-10-28T08:43:58Z" level=debug msg="goroutine 87 [running]:"
time="2021-10-28T08:43:58Z" level=debug msg="net/http.(*conn).serve.func1(0x40005b6500)"
time="2021-10-28T08:43:58Z" level=debug msg="t/usr/local/golang/1.10.8/go/src/net/http/server.go:1801 +0xe4"
time="2021-10-28T08:43:58Z" level=debug msg="panic({0x2823120, 0x5361030})"
time="2021-10-28T08:43:58Z" level=debug msg="t/usr/local/golang/1.10.8/go/src/runtime/panic.go:1052 +0x2b4"
time="2021-10-28T08:43:58Z" level=debug msg="crypto/tls.(*Conn).readClientHello(0x4000197500, {0x33c2b58, 0x40005b3940})"
time="2021-10-28T08:43:58Z" level=debug msg="t/usr/local/golang/1.10.8/go/src/crypto/tls/handshake_server.go:144 +0x68"
time="2021-10-28T08:43:58Z" level=debug msg="crypto/tls.(*Conn).serverHandshake(0x4000197500, {0x33c2b58, 0x40005b3940})"
time="2021-10-28T08:43:58Z" level=debug msg="t/usr/local/golang/1.10.8/go/src/crypto/tls/handshake_server.go:43 +0x40"
time="2021-10-28T08:43:58Z" level=debug msg="crypto/tls.(*Conn).handshakeContext(0x4000197500, {0x33c2c00, 0x40006604b0})"
time="2021-10-28T08:43:58Z" level=debug msg="t/usr/local/golang/1.10.8/go/src/crypto/tls/conn.go:1445 +0x388"
time="2021-10-28T08:43:58Z" level=debug msg="crypto/tls.(*Conn).HandshakeContext(...)"
time="2021-10-28T08:43:58Z" level=debug msg="t/usr/local/golang/1.10.8/go/src/crypto/tls/conn.go:1395"
time="2021-10-28T08:43:58Z" level=debug msg="net/http.(*conn).serve(0x40005b6500, {0x33c2c00, 0x40004c6630})"
time="2021-10-28T08:43:58Z" level=debug msg="t/usr/local/golang/1.10.8/go/src/net/http/server.go:1817 +0x210"
time="2021-10-28T08:43:58Z" level=debug msg="created by net/http.(*Server).Serve"
time="2021-10-28T08:43:58Z" level=debug msg="t/usr/local/golang/1.10.8/go/src/net/http/server.go:3033 +0x4ac"
time="2021-10-28T08:43:58Z" level=debug msg="No default certificate, generating one" tlsStoreName=default
time="2021-10-28T08:43:59Z" level=debug msg="No default certificate, generating one" tlsStoreName=default
time="2021-10-28T08:44:00Z" level=debug msg="Serving default certificate for request: "traefik.[redacted].com""
time="2021-10-28T08:44:00Z" level=debug msg="Creating middleware" serviceName=traefik-https middlewareName=pipelining middlewareType=Pipelining entryPointName=http [email protected]
time="2021-10-28T08:44:00Z" level=debug msg="Creating load-balancer" entryPointName=http [email protected] serviceName=traefik-https
time="2021-10-28T08:44:00Z" level=debug msg="Creating server 0 http://10.0.5.194:443" serviceName=traefik-https entryPointName=http [email protected] serverName=0
time="2021-10-28T08:44:00Z" level=debug msg="child http://10.0.5.194:443 now UP"
time="2021-10-28T08:44:00Z" level=debug msg="Propagating new UP status"
time="2021-10-28T08:44:00Z" level=debug msg="Added outgoing tracing middleware traefik-https" middlewareName=tracing middlewareType=TracingForwarder [email protected] entryPointName=http
time="2021-10-28T08:44:00Z" level=debug msg="Creating middleware" entryPointName=http [email protected] [email protected] middlewareType=RedirectScheme
time="2021-10-28T08:44:00Z" level=debug msg="Setting up redirection to https " [email protected] middlewareType=RedirectScheme entryPointName=http [email protected]
time="2021-10-28T08:44:00Z" level=debug msg="Adding tracing to middleware" entryPointName=http [email protected] [email protected]
time="2021-10-28T08:44:00Z" level=debug msg="Creating middleware" entryPointName=http middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2021-10-28T08:44:00Z" level=debug msg="Added outgoing tracing middleware [email protected]" middlewareName=tracing middlewareType=TracingForwarder [email protected] entryPointName=https
time="2021-10-28T08:44:00Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=https middlewareName=traefik-internal-recovery
time="2021-10-28T08:44:00Z" level=debug msg="Adding route for traefik.[redacted].com with TLS options default" entryPointName=https
time="2021-10-28T08:44:00Z" level=debug msg="Try to challenge certificate for domain [traefik.[redacted].com] found in HostSNI rule" providerName=letsencrypt.acme [email protected] rule="Host(`traefik.[redacted].com`)"
time="2021-10-28T08:44:00Z" level=debug msg="Looking for provided certificate(s) to validate ["traefik.[redacted].com"]..." [email protected] rule="Host(`traefik.[redacted].com`)" providerName=letsencrypt.acme
time="2021-10-28T08:44:00Z" level=debug msg="Domains ["traefik.[redacted].com"] need ACME certificates generation for domains "traefik.[redacted].com"." rule="Host(`traefik.[redacted].com`)" providerName=letsencrypt.acme [email protected]
time="2021-10-28T08:44:00Z" level=debug msg="Loading ACME certificates [traefik.[redacted].com]..." providerName=letsencrypt.acme [email protected] rule="Host(`traefik.[redacted].com`)"
time="2021-10-28T08:44:00Z" level=debug msg="Building ACME client..." providerName=letsencrypt.acme
time="2021-10-28T08:44:00Z" level=debug msg="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=letsencrypt.acme
time="2021-10-28T08:44:00Z" level=debug msg="Using TLS Challenge provider." providerName=letsencrypt.acme
time="2021-10-28T08:44:00Z" level=debug msg="legolog: [INFO] [traefik.[redacted].com] acme: Obtaining bundled SAN certificate"
time="2021-10-28T08:44:00Z" level=debug msg="Serving default certificate for request: "traefik.[redacted].com""
time="2021-10-28T08:44:01Z" level=debug msg="legolog: [INFO] [traefik.[redacted].com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/803593098"
time="2021-10-28T08:44:01Z" level=debug msg="legolog: [INFO] [traefik.[redacted].com] acme: use tls-alpn-01 solver"
time="2021-10-28T08:44:01Z" level=debug msg="legolog: [INFO] [traefik.[redacted].com] acme: Trying to solve TLS-ALPN-01"
time="2021-10-28T08:44:01Z" level=debug msg="TLS Challenge Present temp certificate for traefik.[redacted].com" providerName=tlsalpn.acme
time="2021-10-28T08:44:01Z" level=debug msg="Configuration received from provider tlsalpn.acme: {"http":{},"tls":{}}" providerName=tlsalpn.acme
time="2021-10-28T08:44:01Z" level=debug msg="No default certificate, generating one" tlsStoreName=default
time="2021-10-28T08:44:02Z" level=debug msg="Adding certificate for domain(s) acme challenge temp,traefik.[redacted].com"
time="2021-10-28T08:44:02Z" level=debug msg="Creating middleware" [email protected] serviceName=traefik-https entryPointName=http middlewareName=pipelining middlewareType=Pipelining
time="2021-10-28T08:44:02Z" level=debug msg="Creating load-balancer" entryPointName=http [email protected] serviceName=traefik-https
time="2021-10-28T08:44:02Z" level=debug msg="Creating server 0 http://10.0.5.194:443" entryPointName=http [email protected] serviceName=traefik-https serverName=0
time="2021-10-28T08:44:02Z" level=debug msg="child http://10.0.5.194:443 now UP"
time="2021-10-28T08:44:02Z" level=debug msg="Propagating new UP status"
time="2021-10-28T08:44:02Z" level=debug msg="Added outgoing tracing middleware traefik-https" middlewareType=TracingForwarder middlewareName=tracing entryPointName=http [email protected]
time="2021-10-28T08:44:02Z" level=debug msg="Creating middleware" middlewareType=RedirectScheme entryPointName=http [email protected] [email protected]
time="2021-10-28T08:44:02Z" level=debug msg="Setting up redirection to https " entryPointName=http [email protected] [email protected] middlewareType=RedirectScheme
time="2021-10-28T08:44:02Z" level=debug msg="Adding tracing to middleware" [email protected] [email protected] entryPointName=http
time="2021-10-28T08:44:02Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=http
time="2021-10-28T08:44:02Z" level=debug msg="Added outgoing tracing middleware [email protected]" [email protected] middlewareName=tracing middlewareType=TracingForwarder entryPointName=https
time="2021-10-28T08:44:02Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=https middlewareName=traefik-internal-recovery
time="2021-10-28T08:44:02Z" level=debug msg="Adding route for traefik.[redacted].com with TLS options default" entryPointName=https
time="2021-10-28T08:44:02Z" level=debug msg="Try to challenge certificate for domain [traefik.[redacted].com] found in HostSNI rule" rule="Host(`traefik.[redacted].com`)" providerName=letsencrypt.acme [email protected]
time="2021-10-28T08:44:02Z" level=debug msg="Looking for provided certificate(s) to validate ["traefik.[redacted].com"]..." providerName=letsencrypt.acme [email protected] rule="Host(`traefik.[redacted].com`)"
time="2021-10-28T08:44:02Z" level=debug msg="No ACME certificate generation required for domains ["traefik.[redacted].com"]." rule="Host(`traefik.[redacted].com`)" providerName=letsencrypt.acme [email protected]
time="2021-10-28T08:44:02Z" level=debug msg="TLS Challenge CleanUp temp certificate for traefik.[redacted].com" providerName=tlsalpn.acme
time="2021-10-28T08:44:02Z" level=debug msg="Configuration received from provider tlsalpn.acme: {"http":{},"tls":{}}" providerName=tlsalpn.acme
time="2021-10-28T08:44:02Z" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/803593098"
time="2021-10-28T08:44:03Z" level=error msg="Unable to obtain ACME certificate for domains "traefik.[redacted].com": unable to generate a certificate for the domains [traefik.[redacted].com]: error: one or more domains had a problem:n[traefik.[redacted].com] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Cannot negotiate ALPN protocol "acme-tls/1" for tls-alpn-01 challengen" providerName=letsencrypt.acme [email protected] rule="Host(`traefik.[redacted].com`)"

Further information:

  • Running on 4 x Raspberry Pi 4, on Ubuntu Server 20.04, using docker swarm, using Docker version 20.10.7, build 20.10.7-0ubuntu1~20.04.2
  • Domain name is registered and DNS is controlled by cloudflare
  • CLoudflare proxy is turned on to hide origin IP
  • API is acessible on traefik.[redacted].com domain from outside the network so there are no issues with domain/nameserver resolution.
  • I want to run traefik 2.5.* (latest), not an old version
  • There are currently no files in the /var/data/files/traefik/rules – I plan to use this to add non-docker services in the future.

For some reason traefik is not generating a letsencrypt certificate. I’m still using the letsencrypt staging service since it isn’t working. There are so many tutorials I’ve tried but this is the best I’ve gotten it to work so far. I’ve been able to use labels on other docker swarm stacks and have traefik serve them under the correct url, but can’t for the life of me get it to generate a letsencrypt certificate. Any help is very much appreciated as I’ve been trying this for weeks now!

Source: Docker Questions

LEAVE A COMMENT