New website serves glacially slow for external requests, but is fast internally and over VPN

I’m working on setting up a flask-gunicorn-docker stack behind nginx. So far I’ve got it up and running with https, and it works a treat in the office. Unfortunately, outside the office, it’s slow.

We have a contractor a state over who is getting speeds of 160kbps, meanwhile I was getting several mbps in the office. I’m working from home today and while I’m VPNed in, I’m getting solid speeds and things are working. The contractor’s got a 500mbps connection, so it’s not a connection issue. I’m also experiencing the slowdown when I’m not on the vpn.

The setup is a RHEL 7 box with an nginx proxying in front of a docker container running a gunicorn flask app. It’s running as a VPS on a hardware rack at one of our data centers elsewhere in the state, but I have full SSH and sudo access. The rest of the shop is windows, so the current sysadmin mostly hands the linux stuff off to me.

I’m not a sysadmin, but I’m pretty full stack, and I’m the only linux guy at the company, so I’m the one who needs to figure this out. The site isn’t live yet, so I have some leeway in fiddling around being a bit insecure while I debug.

I’ve searched across stack overflow, read through the gunicorn docs over 15 times, read a fair bit of the documentation available at http://nginx.org/en/docs/http/ngx_http_core_module.html

The problem is I don’t know what the issue is, nor how to determine it.

It seems like the images are loading the slowest, so I created an nginx location block to serve the images without invoking a gunicorn worker, enabled gzip, and then pretty much shoved everything I found online that stackoverflow recommended for slow static content.

That helped a bit, but it’s still slow.
I’ve double checked the firewall, added forwarded_allowed_ips=”*” to the gunicorn configuration, tried to check wireshark and traceroute for any abnormalities, and nslookup returns the proper domain.

If it weren’t serving externally at ALL, I’d have something to go on.

Here’s my nginx.conf:


pid /var/run/nginx.pid;

events {
  worker_connections 1024;
  accept_mutex off;
  use epoll;
}

http {
  include mime.types;
  # fallback in case we can't determine a type
  default_type application/octet-stream;
  access_log /var/log/nginx/access.log;
  error_log /var/log/nginx/error.log;

  server {
    listen 80 default_server;
    server_name REDACTED;
    return 301 https://$host$request_uri;
  }

  upstream app_server {
    server localhost:9898 fail_timeout=0; 
  }
  server {
    listen 443 ssl;
    client_max_body_size 100M;
    ssl_certificate  REDACTED;
    ssl_certificate_key      REDACTED;
    server_name REDACTED;
    keepalive_timeout 5;

    location ^~ /static-site/webimages/ {
      gzip on;
      sendfile on;
      tcp_nopush on;
      alias /etc/nginx/webimages/;     
      # checks for static file, if not found proxy to app
      try_files $uri /;
      sendfile_max_chunk 512k;
      open_file_cache          max=1000 inactive=20s;
      open_file_cache_valid    30s;
      open_file_cache_min_uses 2;
      open_file_cache_errors   on;
      proxy_buffers           32 4m;
      proxy_busy_buffers_size     25m;
      proxy_buffer_size 512k;
    }


    location / {
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header Host $http_host;
      proxy_redirect off;
      proxy_buffering off;
      proxy_pass http://app_server;
    }

  }
}

Source: StackOverflow