inside Kubernetes Pods, Container can not ssh into remote server(AWS EMR for this case)
while, it is possible for git-sync container to ssh into github, static container(for this case, airflow scheduler) can’t ssh into Amazon EMR server with appropriate key.
for example, outside minikube, my local mac terminal is able to connect EMR with no problem(every network is inside VPN starting 172.x.x.x)
one more strange thing is, same Kubernetes env on Amazon EKS(production), there is no ssh connection problem on same container(airflow scheduler pod – airflow container). I am literally able to ssh on a container docker bash shell command.
I suspect port problem, but on EKS, there was no ssh problem with same helm chart. Only local minikube fails.
I share my logs, I will appreciate any answers. Thanks.
-my container route table
Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 172.17.0.1 0.0.0.0 UG 0 0 0 eth0 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
[email protected]:/opt/airflow/dags$ netstat -anp |grep 22 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp 0 0 172.17.0.2:49228 10.109.98.113:5432 TIME_WAIT - tcp 0 0 172.17.0.2:36148 188.8.131.52:22 TIME_WAIT - unix 2 [ ACC ] STREAM LISTENING 152204 69/airflow schedule /tmp/pymp-6zqm69hn/listener-uqwa_n0n unix 3 [ ] STREAM CONNECTED 152208 61/airflow schedule /tmp/pymp-gieffrz0/listener-s8z5z1_k
[email protected]:/opt/airflow/dags$ ssh -v -i emr-pa.pem [email protected] OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1d 10 Sep 2019 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: Applying options for * debug1: Connecting to ip-xxx-xxx-xxx-xxx.ap-northeast-2.compute.internal [xxx.xxx.xxx.xxx] port 22. debug1: connect to address xxx.xxx.xxx.xxx port 22: Connection timed out ssh: connect to host ip-xxx-xxx-xxx-xxx.ap-northeast-2.compute.internal port 22: Connection timed out
Source: Docker Questions