Could not load type ‘Hangfire.HangfireEndpointRouteBuilderExtensions’ from assembly ‘Hangfire.AspNetCore when running as a docker container

I am unable to run a docker container with a .NET 5 application that uses Hangfire.
Locally it’s all good.
But then I build the image with Azure pipelines, save in Azure Container Registry and try to run it locally. The exception I get is:

Starting app for environment Production
crit: Microsoft.AspNetCore.Hosting.Diagnostics[6]
      Application startup exception
      System.TypeLoadException: Could not load type 'Hangfire.HangfireEndpointRouteBuilderExtensions' from assembly 'Hangfire.AspNetCore, Version=1.7.18.0, Culture=neutral, PublicKeyToken=null'.
         at MyCompany.Platform.Scheduler.WebApi.Startup.<>c.<Configure>b__5_1(IEndpointRouteBuilder endpoints)
         at Microsoft.AspNetCore.Builder.EndpointRoutingApplicationBuilderExtensions.UseEndpoints(IApplicationBuilder builder, Action`1 configure)
         at MyCompany.Platform.Scheduler.WebApi.Startup.Configure(IApplicationBuilder app, IWebHostEnvironment env) in /home/vsts/work/1/s/src/WebApi/Startup.cs:line 56
         at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
         at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
         at Microsoft.AspNetCore.Hosting.ConfigureBuilder.Invoke(Object instance, IApplicationBuilder builder)
         at Microsoft.AspNetCore.Hosting.ConfigureBuilder.<>c__DisplayClass4_0.<Build>b__0(IApplicationBuilder builder)
         at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass15_0.<UseStartup>b__1(IApplicationBuilder app)
         at Microsoft.AspNetCore.Mvc.Filters.MiddlewareFilterBuilderStartupFilter.<>c__DisplayClass0_0.<Configure>g__MiddlewareFilterBuilder|0(IApplicationBuilder builder)
         at Microsoft.AspNetCore.HostFilteringStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder app)
         at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
Unhandled exception. System.TypeLoadException: Could not load type 'Hangfire.HangfireEndpointRouteBuilderExtensions' from assembly 'Hangfire.AspNetCore, Version=1.7.18.0, Culture=neutral, PublicKeyToken=null'.
   at MyCompany.Platform.Scheduler.WebApi.Startup.<>c.<Configure>b__5_1(IEndpointRouteBuilder endpoints)
   at Microsoft.AspNetCore.Builder.EndpointRoutingApplicationBuilderExtensions.UseEndpoints(IApplicationBuilder builder, Action`1 configure)
   at MyCompany.Platform.Scheduler.WebApi.Startup.Configure(IApplicationBuilder app, IWebHostEnvironment env) in /home/vsts/work/1/s/src/WebApi/Startup.cs:line 56
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.AspNetCore.Hosting.ConfigureBuilder.Invoke(Object instance, IApplicationBuilder builder)
   at Microsoft.AspNetCore.Hosting.ConfigureBuilder.<>c__DisplayClass4_0.<Build>b__0(IApplicationBuilder builder)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass15_0.<UseStartup>b__1(IApplicationBuilder app)
   at Microsoft.AspNetCore.Mvc.Filters.MiddlewareFilterBuilderStartupFilter.<>c__DisplayClass0_0.<Configure>g__MiddlewareFilterBuilder|0(IApplicationBuilder builder)
   at Microsoft.AspNetCore.HostFilteringStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder app)
   at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at MyCompany.Platform.Scheduler.WebApi.Program.Main(String[] args) in /home/vsts/work/1/s/src/WebApi/Program.cs:line 10

This is my Dockerfile

FROM mcr.microsoft.com/dotnet/aspnet:5.0
COPY out .
EXPOSE 80
ENTRYPOINT ["dotnet", "WebApi.dll"]

This is my azure pipelines

trigger:
- master

pool:
  vmImage: ubuntu-20.04

variables:
  imageName: 'my-scheduler'

stages:
  - stage:
    displayName: 'Build'
    jobs:
      - job: 'Build'
        displayName: 'Restore and compile'
        steps:
          - task: [email protected]
            displayName: 'Use .NET Core sdk 5.0.x'
            inputs:
              version: '5.0.x'
              includePreviewVersions: true
          - script: dotnet restore --no-cache --force
            displayName: 'Restore dependencies'
          - script: dotnet build --configuration Release --no-restore
            displayName: 'Build with Release Configuration'
          - script: dotnet vstest test/*FunctionalTests/bin/Release/**/*FunctionalTests.dll
            displayName: 'Run functional tests'
          - script: dotnet publish -c Release -o out
            displayName: 'Publish to folder'
          - task: [email protected]
            inputs:
              containerRegistry: 'saswcr-service-connection'
              command: 'login'
          - task: [email protected]
            inputs:
              containerRegistry: 'saswcr-service-connection'
              repository: $(imageName)
              command: 'buildAndPush'
              dockerfile: './Dockerfile'
              tags: |
                latest
                $(Build.BuildNumber)

And this is how I run the container locally after logging into my ACR

docker run --name my-scheduler -p 8080:80 saswcr.azurecr.io/my-scheduler:20201124.4

I’m running out of ideas. The versions of hangfire I use:

<PackageReference Include="Hangfire.AspNetCore" Version="1.7.18" />
<PackageReference Include="Hangfire.Core" Version="1.7.18" />
<PackageReference Include="Hangfire.MemoryStorage" Version="1.7.0" />

Maybe there is something wrong with the image creation, but I cannot think what exactly.

If I do the following to see the contents of my container docker export $(docker ps -lq) | tar tf -
I see:

$ docker export $(docker ps -lq) | tar tf -
.dockerenv
Application.deps.json
Application.dll
Application.pdb
Azure.Core.dll
Azure.Identity.dll
Azure.Security.KeyVault.Certificates.dll
Azure.Security.KeyVault.Secrets.dll
Castle.Core.dll
CodeCoverage/
CodeCoverage/CodeCoverage.config
CodeCoverage/CodeCoverage.exe
CodeCoverage/amd64/
CodeCoverage/amd64/covrun64.dll
CodeCoverage/amd64/msdia140.dll
CodeCoverage/codecoveragemessages.dll
CodeCoverage/covrun32.dll
CodeCoverage/msdia140.dll
FluentAssertions.dll
Hangfire.AspNetCore.dll
Hangfire.Core.dll
Hangfire.MemoryStorage.dll
Hangfire.SqlServer.dll
Infra.Hangfire.deps.json
Infra.Hangfire.dll
Infra.Hangfire.pdb
Microsoft.AspNetCore.Antiforgery.dll
Microsoft.AspNetCore.Authentication.JwtBearer.dll
Microsoft.AspNetCore.Authentication.OpenIdConnect.dll
Microsoft.AspNetCore.Cryptography.Internal.dll
Microsoft.AspNetCore.DataProtection.Abstractions.dll
Microsoft.AspNetCore.DataProtection.dll
Microsoft.AspNetCore.Hosting.Abstractions.dll
Microsoft.AspNetCore.Hosting.Server.Abstractions.dll
Microsoft.AspNetCore.Http.Abstractions.dll
Microsoft.AspNetCore.Http.Extensions.dll
Microsoft.AspNetCore.Http.Features.dll
Microsoft.AspNetCore.Mvc.Testing.dll
Microsoft.AspNetCore.TestHost.dll
Microsoft.AspNetCore.WebUtilities.dll
Microsoft.Bcl.AsyncInterfaces.dll
Microsoft.CodeCoverage.props
Microsoft.CodeCoverage.targets
Microsoft.DotNet.PlatformAbstractions.dll
Microsoft.Extensions.Configuration.Abstractions.dll
Microsoft.Extensions.DependencyInjection.Abstractions.dll
Microsoft.Extensions.DependencyInjection.dll
Microsoft.Extensions.DependencyModel.dll
Microsoft.Extensions.FileProviders.Abstractions.dll
Microsoft.Extensions.FileSystemGlobbing.dll
Microsoft.Extensions.Hosting.Abstractions.dll
Microsoft.Extensions.Logging.Abstractions.dll
Microsoft.Extensions.ObjectPool.dll
Microsoft.Extensions.Options.dll
Microsoft.Extensions.Primitives.dll
Microsoft.Identity.Client.Extensions.Msal.dll
Microsoft.Identity.Client.dll
Microsoft.Identity.Web.UI.Views.dll
Microsoft.Identity.Web.UI.dll
Microsoft.Identity.Web.dll
Microsoft.IdentityModel.JsonWebTokens.dll
Microsoft.IdentityModel.Logging.dll
Microsoft.IdentityModel.Protocols.OpenIdConnect.dll
Microsoft.IdentityModel.Protocols.dll
Microsoft.IdentityModel.Tokens.dll
Microsoft.Net.Http.Headers.dll
Microsoft.OpenApi.dll
Microsoft.TestPlatform.CommunicationUtilities.dll
Microsoft.TestPlatform.CoreUtilities.dll
Microsoft.TestPlatform.CrossPlatEngine.dll
Microsoft.TestPlatform.PlatformAbstractions.dll
Microsoft.TestPlatform.Utilities.dll
Microsoft.VisualStudio.CodeCoverage.Shim.dll
Microsoft.VisualStudio.TestPlatform.Common.dll
Microsoft.VisualStudio.TestPlatform.ObjectModel.dll
Microsoft.VisualStudio.TraceDataCollector.dll
Microsoft.Win32.Registry.dll
Mono.Cecil.Mdb.dll
Mono.Cecil.Pdb.dll
Mono.Cecil.Rocks.dll
Mono.Cecil.dll
Moq.dll
Newtonsoft.Json.dll
NuGet.Frameworks.dll
Swashbuckle.AspNetCore.Swagger.dll
Swashbuckle.AspNetCore.SwaggerGen.dll
Swashbuckle.AspNetCore.SwaggerUI.dll
System.Buffers.dll
System.Configuration.ConfigurationManager.dll
System.Data.SqlClient.dll
System.Diagnostics.DiagnosticSource.dll
System.IdentityModel.Tokens.Jwt.dll
System.Runtime.CompilerServices.Unsafe.dll
System.Security.AccessControl.dll
System.Security.Cryptography.ProtectedData.dll
System.Security.Cryptography.Xml.dll
System.Security.Principal.Windows.dll
System.Text.Encoding.CodePages.dll
System.Text.Encodings.Web.dll
System.Threading.dll
UseCases/
UseCases/AddJob/
UseCases/AddJob/simple_job.json
UseCases/AddJob/simple_job_with_url_parameters.json
WebApi
WebApi.FunctionalTests.deps.json
WebApi.FunctionalTests.dll
WebApi.FunctionalTests.pdb
WebApi.FunctionalTests.runtimeconfig.json
WebApi.deps.json
WebApi.dll
WebApi.pdb
WebApi.runtimeconfig.json
appsettings.Development.json
appsettings.Local.json
appsettings.Production.json
appsettings.Qa.json
appsettings.Test.json
appsettings.json
bin/
etc.

The line that seems to be causing the issue is:

app.UseEndpoints(endpoints =>
{
    endpoints.MapControllers();
    endpoints
      .MapHangfireDashboard(HangfireConstants.DashboardUrl)
      .RequireAuthorization(PolicyConstants.DashboardPolicy);
});

What could be the reason? And why does it happen in docker container and not outside?

Source: Docker Questions