Unable to access api running in docker from host machine but works while debugging with VS2019

I am trying to run a docker container on my windows 10 machine using docker for desktop.
I am using sdk:3.1-alpine AS my base image.

My docker file is the default file generated by VS 2019 and my API is the default controller generated by VS2019 with dotnetcore 3.1.
What I see that I am able to access the API running in my docker container from host machine only when I am debugging using Visual studio, however as soon as I stop debugging I am not able to access the API anymore even though the docker container is still running.

What do I need to do to access my API without debugging it using VS2019.

Source: StackOverflow