Google Endpoints on GKE (ESP + gRPC) – how to debug this further?

After following
Getting started with Cloud Endpoints on GKE and deploying the API and backend everything looks like it’s working (spoiler alert, it’s not);

  • My application’s deployed on a cluster.
  • A single pod is deployed in Kubernetes.
  • The pod is running two containers (ESP and my GPRC service)
    1. gcr.io/endpoints-release/endpoints-runtime:1
    2. gcr.io/my-app/server:latest

…both are running; kubectl get pods yields:

NAME                          READY   STATUS    RESTARTS   AGE
my-app-855d8447f-nnvz8        2/2     Running   0          24m

Need improved/HTTPS-centric GKE Ingress + ESP + gRPC example was very helpful in getting me this far.

Bad news; HTTP requests from my Angular app (running on http://localhost:4200) result in 500 Internal Server Error.

I ran through Troubleshooting Cloud Endpoints in GKE and couldn’t see anything that applied (there is nothing obviously wrong).

OK. Time to debug.

Looking at the ESP (Docker container) logs with kubectl logs -f my-app-855d8447f-nnvz8 -c esp, I can see the preflight request (OPTIONS) and subsequent HTTP POST request.

INFO:Fetching an access token from the metadata service
INFO:Fetching the service config ID from the rollouts service
INFO:Fetching the service configuration from the service management service
INFO:Attribute zone: us-central1-a
INFO:Attribute project_id: my-cloud-project
INFO:Attribute kube_env: KUBE_ENV

nginx: [warn] Using trusted CA certificates file: /etc/nginx/trusted-ca-certificates.crt

143.159.55.199 - - [12/Jan/2020:17:34:11 +0000] "OPTIONS /v1/documents?key=Ei2ZvlV5zASIkuRf4LRl&format=JSON&showDefaults=false HTTP/1.1" 204 0 "http://localhost:4200/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
143.159.55.199 - - [12/Jan/2020:17:34:11 +0000] "POST /v1/documents?key=Ei2ZvlV5zASIkuRf4LRl&format=JSON&showDefaults=false HTTP/1.1" 500 242 "http://localhost:4200/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"

The CORS (OPTIONS) request results in HTTP 204, however the POST request is HTTP 500. So there’s clearly a problem here.

Despite plenty of logging, there are no logs in my grpc server; kubectl logs -f my-app-855d8447f-nnvz8 -c server yields nothing. So

  • The Kubernetes pod is healthy and both containers arerunning.
  • The ESP is starting and successfully using the metadata service
  • CORS is working (it’s wide-open while I’m developing)
  • The gRPC service is running but not being hit (no logs)

Somewhere between CORS and POST HTTP requests, it’s going wrong and the gRPC service never gets called.

I can’t find any errors; How can I debug what’s going on between ESP/nginx and my gRPC server?

For completeness, here’s my deployment yaml (with added comments):

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: my-app
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: esp
        image: gcr.io/endpoints-release/endpoints-runtime:1
        # The default value for the backend address is http://127.0.0.1:8080
        args: [
          "--http_port=8081",

          # My gPRC service is running on port 8080 on the same VM.
          "--backend=grpc://127.0.0.1:8080",
          "--service=my-project.endpoints.my-project-dev.cloud.goog",
          "--rollout_strategy=managed",

          # Makes ESP return code 200 for location /healthz,
          # instead of forwarding the request to the backend.
          "--healthz=healthz",

          # Wide-open CORS during development.
          "--cors_preset=basic",
          "--cors_allow_origin=*",
        ]
        # HEALTH CHECK START
        # Used by ingress to perform health check
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8081
        ports:
          - containerPort: 8081
        # HEALTH CHECK END
      - name: server
        image: gcr.io/my-project-dev/server:latest
        imagePullPolicy: Always
        ports:
          - containerPort: 8080

The CORS response headers from the ESP/nginx container are:

Access-Control-Allow-Headers: DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization
Access-Control-Allow-Methods: GET, POST, PUT, PATCH, DELETE, OPTIONS
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Content-Length,Content-Range
Content-Type: application/json
Date: Sun, 12 Jan 2020 17:34:11 GMT
Server: nginx
Transfer-Encoding: chunked
Via: 1.1 google

I don’t expect anyone to magic-up an answer, but right now I’m unsure of how to debug this any further when everthing looks good?

Source: StackOverflow