Hi everyone,

I have an Elixir app that sends out HTTP requests using HTTPoison. Usually it works fine, but I noticed it gives timeout error occasionally when a Docker container running in the same machine. The number of containers does not matter. I could narrow down the situation to only 1 container running and still got the timeout error. The container runs PostgreSQL. I got it running by just docker run -d postgres.

I suspected that because if I stop the Docker container, the Elixir app works well again. Then if I run the PostgreSQL container again, after some random time (e.g., next working day when I am back to keyboard, or next 5 minutes etc), the Elixir app gives timeout again.

The strange thing is that curl to the same website works well all the time, regardless of the Docker container is running or not.

I think it has something to do with the iptables. Here is iptables-save output with Docker container running:

$ sudo iptables-save 
# Generated by iptables-save v1.8.9 (nf_tables) on Thu Nov 28 11:54:54 2024
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [1:576]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 5432 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Thu Nov 28 11:54:54 2024
# Generated by iptables-save v1.8.9 (nf_tables) on Thu Nov 28 11:54:54 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 5432 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 5432 -j DNAT --to-destination 172.17.0.2:5432
COMMIT
# Completed on Thu Nov 28 11:54:54 2024
# Warning: iptables-legacy tables present, use iptables-legacy-save to see them

Here is the output after that Docker container stops:

$ sudo iptables-save 
# Generated by iptables-save v1.8.9 (nf_tables) on Thu Nov 28 11:55:10 2024
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [1:576]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Thu Nov 28 11:55:10 2024
# Generated by iptables-save v1.8.9 (nf_tables) on Thu Nov 28 11:55:10 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Thu Nov 28 11:55:10 2024
# Warning: iptables-legacy tables present, use iptables-legacy-save to see them

I read some articles in the Internet about Docker messing with host machine network. But I don’t know where to debug in the case of Elixir app. Did anyone experience something similar?

Thanks a lot!