Http request from within Elixir (BEAM) timeout when a Docker container is running

Questions / Help
, ,
1

Hi everyone,

I have an Elixir app that sends out HTTP requests using HTTPoison. Usually it works fine, but I noticed it gives timeout error occasionally when a Docker container running in the same machine. The number of containers does not matter. I could narrow down the situation to only 1 container running and still got the timeout error. The container runs PostgreSQL. I got it running by just docker run -d postgres.

I suspected that because if I stop the Docker container, the Elixir app works well again. Then if I run the PostgreSQL container again, after some random time (e.g., next working day when I am back to keyboard, or next 5 minutes etc), the Elixir app gives timeout again.

The strange thing is that curl to the same website works well all the time, regardless of the Docker container is running or not.

I think it has something to do with the iptables. Here is iptables-save output with Docker container running:

$ sudo iptables-save 
# Generated by iptables-save v1.8.9 (nf_tables) on Thu Nov 28 11:54:54 2024
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [1:576]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 5432 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Thu Nov 28 11:54:54 2024
# Generated by iptables-save v1.8.9 (nf_tables) on Thu Nov 28 11:54:54 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 5432 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 5432 -j DNAT --to-destination 172.17.0.2:5432
COMMIT
# Completed on Thu Nov 28 11:54:54 2024
# Warning: iptables-legacy tables present, use iptables-legacy-save to see them

Here is the output after that Docker container stops:

$ sudo iptables-save 
# Generated by iptables-save v1.8.9 (nf_tables) on Thu Nov 28 11:55:10 2024
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [1:576]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Thu Nov 28 11:55:10 2024
# Generated by iptables-save v1.8.9 (nf_tables) on Thu Nov 28 11:55:10 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Thu Nov 28 11:55:10 2024
# Warning: iptables-legacy tables present, use iptables-legacy-save to see them

I read some articles in the Internet about Docker messing with host machine network. But I don’t know where to debug in the case of Elixir app. Did anyone experience something similar?

Thanks a lot!

1 Like
2

Are you running the application in docker too?

1 Like
3

No @D4no0, the app runs outside of the Docker container.

1 Like
4

Yeah, IDK if you will have luck finding out many people doing such kind of hybrid deploys.

I would just containerize everything and check if that works.

4 Likes
5

Do you run any software that updates frequently and needs to reload iptables? If so you might need to configure it to add the docker rules back in afterwards (although it looks like your rules are fine - but still, worth checking if there’s any software that’s running that could interfere with it).

6

Actually I already narrowed down the situation to only 1 BEAM and 1 Docker container running. So no other services that would reload iptables.

I also thought the iptables rules are OK but somehow HTTPoison gives timeout. If I just do docker stop then start the container, things are back to normal until some random time later.

Strange for me is, curl always works regardless of docker is running or not. So I think maybe Elixir is doing some different underlying network calls?