Kubernetes readinessProbe returns "connection refuse"

environment: Ruby on Rails + Vue.js + Webpacker + Kubernetes

I added readinessProbe for healthcheck in my deploy of K8s.

But the pod cannot get ready to start,
so I checked the logs with th command

“kubectl describe po -n “.

Then I was able to get a message like

“Readiness probe failed: Get http://10.44.1.252:3000/health: dial tcp 10.44.1.252:3000: connect: connection refused.”

However, there was no error and the status of http is 200 when I check this route in localhost.
(I used the command curl http://localhost:3000/health in K8s pod)

I did my readinessProbe health check because I got an UNHEALTHY status in my ingress.

Here is my yamls.

mysql.yaml

apiVersion: v1
kind: Service
metadata:
  namespace: line-manager
  name: mysql
  labels:
    app: mysql
spec:
  selector:
    app: mysql
  type: ClusterIP
  clusterIP: None
  ports:
    - port: 3306
---
kind: PersistentVolume
apiVersion: v1
metadata:
  namespace: line-manager
  name: mysql-pv
  labels:
    app: mysql
spec:
  capacity:
    storage: 20Gi
  accessModes:
  - ReadWriteOnce
  storageClassName: standard
  hostPath:
    path: "/tmp/mysql"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: line-manager
  name: mysql-pvc
  labels:
    app: mysql
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: standard
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: line-manager
  name: mysql
  labels:
    app: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:5.7.17
        env:
        - name: MYSQL_DATABASE
          value: lineManage_db
        - name: MYSQL_USER
          value: root
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-pass
              key: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-pv
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-pv
        persistentVolumeClaim:
          claimName: mysql-pvc

webpacker.yaml

apiVersion: v1
kind: Service
metadata:
  namespace: line-manager
  name: webpacker
  labels:
    app: webpacker
spec:
  selector:
    app: webpacker
  type: ClusterIP
  clusterIP: None
  ports:
  - port: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: line-manager
  name: webpacker
spec:
  selector:
    matchLabels:
      app: webpacker
  template:
    metadata:
      labels:
        app: webpacker
    spec:
      containers:
      - name: webpacker
        image: khjoo19/fullout-line:v1
        imagePullPolicy: Always
        command: ["bundle", "exec", "bin/webpack-dev-server"]
        ports:
        - containerPort: 8080

rails.yaml

apiVersion: v1
kind: Service
metadata:
  namespace: line-manager
  name: web
  labels:
    app: web
spec:
  selector:
    app: web
  type: NodePort
  ports:
  - port: 3000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: line-manager
  name: web
spec:
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web
        image: khjoo19/fullout-line:v1
        imagePullPolicy: Always
        command: ["bundle", "exec", "rails", "s"]
        env:
        - name: MYSQL_DATABASE
          value: lineManage_db
        - name: MYSQL_USER
          value: root
        - name: MYSQL_HOST
          value: mysql
        - name: MYSQL_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-pass
              key: password
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-pass
              key: password
        ports:
        - containerPort: 3000
        readinessProbe:
          httpGet:
            path: /health
            port: 3000
          initialDelaySeconds: 60
          periodSeconds: 30
          timeoutSeconds: 60
        volumeMounts:
        - name: mysql-pv
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-pv
        persistentVolumeClaim:
          claimName: mysql-pvc

kubectl describe po -n

Name:               web-6d*****7d-khljl
Namespace:          line-manager
Priority:           0
PriorityClassName:  <none>
Node:               gke-core-pool-2-5*****26-v5wp/10.138.0.31
Start Time:         Fri, 17 Jan 2020 11:03:41 +0900
Labels:             app=web
                    pod-template-hash=6*****7d
Annotations:        <none>
Status:             Running
IP:                 10.44.1.252
Controlled By:      ReplicaSet/web-6*****d
Containers:
  web:
    Container ID:  docker://9eb5146d686856a*********************8465b358a04
    Image:         khjoo19/fullout-line:v1
    Image ID:      docker-pullable://khjoo19/[email protected]:ee6147ac************************8ab675ce3571
    Port:          3000/TCP
    Host Port:     0/TCP
    Command:
      bundle
      exec
      rails
      s
    State:          Running
      Started:      Fri, 17 Jan 2020 11:04:19 +0900
    Ready:          False
    Restart Count:  0
    Readiness:      http-get http://:3000/health delay=60s timeout=60s period=30s #success=1 #failure=3
    Environment:
      MYSQL_DATABASE:       lineManage_db
      MYSQL_USER:           root
      MYSQL_HOST:           mysql
      MYSQL_PASSWORD:       <set to the key 'password' in secret 'mysql-pass'>  Optional: false
      MYSQL_ROOT_PASSWORD:  <set to the key 'password' in secret 'mysql-pass'>  Optional: false
    Mounts:
      /var/lib/mysql from mysql-pv (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-lzccw (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  mysql-pv:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mysql-pvc
    ReadOnly:   false
  default-token-lzccw:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-l****
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                  From                                    Message
  ----     ------     ----                 ----                                    -------
  Normal   Scheduled  6m19s                default-scheduler                       Successfully assigned line-manager/web-6dfd6dcc7d-khljl to gke-core-pool-2-5e0c4e26-v5wp
  Normal   Pulling    6m17s                kubelet, gke-core-pool-2-5e0c4e26-v5wp  pulling image "khjoo19/fullout-line:v1"
  Normal   Pulled     5m43s                kubelet, gke-core-pool-2-5e0c4e26-v5wp  Successfully pulled image "khjoo19/fullout-line:v1"
  Normal   Created    5m41s                kubelet, gke-core-pool-2-5e0c4e26-v5wp  Created container
  Normal   Started    5m41s                kubelet, gke-core-pool-2-5e0c4e26-v5wp  Started container
  Warning  Unhealthy  24s (x9 over 4m24s)  kubelet, gke-core-pool-2-5e0c4e26-v5wp  Readiness probe failed: Get http://10.44.1.252:3000/health: dial tcp 10.44.1.252:3000: connect: connection refused

Source: StackOverflow