- My application’s deployed on a cluster.
- A single pod is deployed in Kubernetes.
- The pod is running two containers (ESP and my GPRC service)
…both are running;
kubectl get pods yields:
NAME READY STATUS RESTARTS AGE my-app-855d8447f-nnvz8 2/2 Running 0 24m
Need improved/HTTPS-centric GKE Ingress + ESP + gRPC example was very helpful in getting me this far.
Bad news; HTTP requests from my Angular app (running on
http://localhost:4200) result in 500 Internal Server Error.
I ran through Troubleshooting Cloud Endpoints in GKE and couldn’t see anything that applied (there is nothing obviously wrong).
OK. Time to debug.
Looking at the ESP (Docker container) logs with
kubectl logs -f my-app-855d8447f-nnvz8 -c esp, I can see the preflight request (
OPTIONS) and subsequent HTTP
INFO:Fetching an access token from the metadata service INFO:Fetching the service config ID from the rollouts service INFO:Fetching the service configuration from the service management service INFO:Attribute zone: us-central1-a INFO:Attribute project_id: my-cloud-project INFO:Attribute kube_env: KUBE_ENV nginx: [warn] Using trusted CA certificates file: /etc/nginx/trusted-ca-certificates.crt 188.8.131.52 - - [12/Jan/2020:17:34:11 +0000] "OPTIONS /v1/documents?key=Ei2ZvlV5zASIkuRf4LRl&format=JSON&showDefaults=false HTTP/1.1" 204 0 "http://localhost:4200/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36" 184.108.40.206 - - [12/Jan/2020:17:34:11 +0000] "POST /v1/documents?key=Ei2ZvlV5zASIkuRf4LRl&format=JSON&showDefaults=false HTTP/1.1" 500 242 "http://localhost:4200/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
The CORS (
OPTIONS) request results in HTTP 204, however the
POST request is HTTP 500. So there’s clearly a problem here.
Despite plenty of logging, there are no logs in my grpc server;
kubectl logs -f my-app-855d8447f-nnvz8 -c server yields nothing. So
- The Kubernetes pod is healthy and both containers arerunning.
- The ESP is starting and successfully using the metadata service
- CORS is working (it’s wide-open while I’m developing)
- The gRPC service is running but not being hit (no logs)
Somewhere between CORS and POST HTTP requests, it’s going wrong and the gRPC service never gets called.
I can’t find any errors; How can I debug what’s going on between ESP/nginx and my gRPC server?
For completeness, here’s my deployment yaml (with added comments):
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-app namespace: default spec: replicas: 1 template: metadata: labels: app: my-app spec: containers: - name: esp image: gcr.io/endpoints-release/endpoints-runtime:1 # The default value for the backend address is http://127.0.0.1:8080 args: [ "--http_port=8081", # My gPRC service is running on port 8080 on the same VM. "--backend=grpc://127.0.0.1:8080", "--service=my-project.endpoints.my-project-dev.cloud.goog", "--rollout_strategy=managed", # Makes ESP return code 200 for location /healthz, # instead of forwarding the request to the backend. "--healthz=healthz", # Wide-open CORS during development. "--cors_preset=basic", "--cors_allow_origin=*", ] # HEALTH CHECK START # Used by ingress to perform health check readinessProbe: httpGet: path: /healthz port: 8081 ports: - containerPort: 8081 # HEALTH CHECK END - name: server image: gcr.io/my-project-dev/server:latest imagePullPolicy: Always ports: - containerPort: 8080
The CORS response headers from the ESP/nginx container are:
Access-Control-Allow-Headers: DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization Access-Control-Allow-Methods: GET, POST, PUT, PATCH, DELETE, OPTIONS Access-Control-Allow-Origin: * Access-Control-Expose-Headers: Content-Length,Content-Range Content-Type: application/json Date: Sun, 12 Jan 2020 17:34:11 GMT Server: nginx Transfer-Encoding: chunked Via: 1.1 google
I don’t expect anyone to magic-up an answer, but right now I’m unsure of how to debug this any further when everthing looks good?