Kubernetes Pod – ssh time out inside docker container

  docker, kubernetes, kubernetes-helm, networking, ssh

local machine
helm chart
amazon emr

inside Kubernetes Pods, Container can not ssh into remote server(AWS EMR for this case)

while, it is possible for git-sync container to ssh into github, static container(for this case, airflow scheduler) can’t ssh into Amazon EMR server with appropriate key.

for example, outside minikube, my local mac terminal is able to connect EMR with no problem(every network is inside VPN starting 172.x.x.x)

one more strange thing is, same Kubernetes env on Amazon EKS(production), there is no ssh connection problem on same container(airflow scheduler pod – airflow container). I am literally able to ssh on a container docker bash shell command.

I suspect port problem, but on EKS, there was no ssh problem with same helm chart. Only local minikube fails.

I share my logs, I will appreciate any answers. Thanks.

-my container route table

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface         UG        0 0          0 eth0     U         0 0          0 eth0
  • netstat
[email protected]:/opt/airflow/dags$ netstat -anp |grep 22
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0      TIME_WAIT   -                   
tcp        0      0        TIME_WAIT   -                   
unix  2      [ ACC ]     STREAM     LISTENING     152204   69/airflow schedule  /tmp/pymp-6zqm69hn/listener-uqwa_n0n
unix  3      [ ]         STREAM     CONNECTED     152208   61/airflow schedule  /tmp/pymp-gieffrz0/listener-s8z5z1_k
  • debugging
[email protected]:/opt/airflow/dags$ ssh -v -i emr-pa.pem [email protected]
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1d  10 Sep 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to ip-xxx-xxx-xxx-xxx.ap-northeast-2.compute.internal [xxx.xxx.xxx.xxx] port 22.
debug1: connect to address xxx.xxx.xxx.xxx port 22: Connection timed out
ssh: connect to host ip-xxx-xxx-xxx-xxx.ap-northeast-2.compute.internal port 22: Connection timed out

Source: Docker Questions