nginx is not able to connect to uwsgi after 100 socket connection

We are using docker for flask application and there is nginx-uwsgi image tiangolo/uwsgi-nginx-flask which we are using.

Docker image works fine for most of the application, but one of the deployment gives error after some times.

Actual error is

2020/10/21 18:17:49 [error] 12#12: *142201 connect() to unix:///tmp/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.255.0.212, server: , request: "GET /healthcheck HTTP/1.0", upstream: "uwsgi://unix:///tmp/uwsgi.sock:"

Error says, there is issue with uwsgi socket. When I check the connection to uwsgi sockets, I get 101 connection with CONNECTING label.

[email protected]:/app# netstat -nap | grep uwsgi | grep CONNECTING | wc -l
101
[email protected]:/app# netstat -nap | grep uwsgi | grep CONNECTING
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock
unix  2      [ ]         STREAM     CONNECTING    0        -                    /tmp/uwsgi.sock

From this, it appears, nginx make connection to uwsgi and then its not able to release it.

There are other flask application running in same image, I check the how many CONNECTING socket I get, but other container return ZERO.

[email protected]:/app# ps aux | grep uwsgi
root         10  0.3  0.2 298340 44636 ?        Sl   Oct17  11:00 /usr/local/bin/uwsgi --ini /etc/uwsgi/uwsgi.ini
root         36  0.0  0.2 298340 42776 ?        S    Oct18   0:00 /usr/local/bin/uwsgi --ini /etc/uwsgi/uwsgi.ini
root         37  0.0  0.2 298340 43316 ?        S    Oct18   0:00 /usr/local/bin/uwsgi --ini /etc/uwsgi/uwsgi.ini
root        320  0.0  0.0  12848   960 pts/0    S+   17:23   0:00 grep uwsgi
[email protected]:/app# kill -9 10 36 37
[email protected]:/app# ps aux | grep uwsgi
root        321 43.0  0.3 222368 49872 ?        D    17:24   0:00 /usr/local/bin/uwsgi --ini /etc/uwsgi/uwsgi.ini
root        325  0.0  0.0  12848   960 pts/0    S+   17:24   0:00 grep uwsgi
[email protected]:/app# ps aux | grep uwsgi
root        321 23.5  0.3 298476 52380 ?        Sl   17:24   0:00 /usr/local/bin/uwsgi --ini /etc/uwsgi/uwsgi.ini
root        327  0.0  0.2 298476 43728 ?        S    17:24   0:00 /usr/local/bin/uwsgi --ini /etc/uwsgi/uwsgi.ini
root        328  0.0  0.2 298476 43728 ?        S    17:24   0:00 /usr/local/bin/uwsgi --ini /etc/uwsgi/uwsgi.ini
root        330  0.0  0.0  12848   960 pts/0    S+   17:24   0:00 grep uwsgi
[email protected]:/app# netstat -nap | grep uwsgi | grep CONNECTING | wc -l
0

There are some NGINX tuning parameter sets in this. printenv output is below.

[[email protected] ~]$ docker exec -it b2cff9ac29a2 printenv
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=b2cff9ac29a2
TERM=xterm
API_PASS=XXXXX
API_USER=XXXX
APP_ENVIRONMENT=bleeding
BCAPI_URL=https://example.com
BSAPI_APP=RpcApi
BSAPI_DB_NAME=RPC_Bleeding
BSAPI_DB_URI=mongodb://username:mongodb.instance:26200/database?ssl=true
BSAPI_ES_LOGGING=True
BSAPI_REDIS_URI=redis://:[email protected]:6379/0
BSNOTIFY_ENV=staging
CELERY_BROKER_URL=amqp://username:[email protected]/vhostg
NGINX_WORKER_CONNECTIONS=2048
NGINX_WORKER_OPEN_FILES= 2048
NGINX_WORKER_PROCESSES=8
LANG=C.UTF-8
GPG_KEY=0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D
PYTHON_VERSION=3.7.4
PYTHON_PIP_VERSION=19.2.1
PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/404c9418e33c5031b1a9ab623168b3e8a2ed8c88/get-pip.py
PYTHON_GET_PIP_SHA256=56bb63d3cf54e7444351256f72a60f575f6d8c7f1faacffae33167afc8e7609d
NGINX_VERSION=1.15.8-1~stretch
NJS_VERSION=1.15.8.0.2.7-1~stretch
UWSGI_INI=/app/uwsgi.ini
UWSGI_CHEAPER=2
UWSGI_PROCESSES=16
NGINX_MAX_UPLOAD=0
LISTEN_PORT=80
STATIC_URL=/static
STATIC_PATH=/app/static
STATIC_INDEX=0
PYTHONPATH=/app
HOME=/root

We are trying to understand why socket connection stuck in CONNECTING and not clear after request fulfill.

What make this socket queue to reach to 101 counts.

When it reach to 100+ its start giving socket error.

We can increase the queue size, but we are sure, it will full again.

Source: Docker Questions