Http request from within Elixir (BEAM) timeout when a Docker container is running

tduccuong

Hi everyone,

I have an Elixir app that sends out HTTP requests using HTTPoison. Usually it works fine, but I noticed it gives timeout error occasionally when a Docker container running in the same machine. The number of containers does not matter. I could narrow down the situation to only 1 container running and still got the timeout error. The container runs PostgreSQL. I got it running by just docker run -d postgres.

I suspected that because if I stop the Docker container, the Elixir app works well again. Then if I run the PostgreSQL container again, after some random time (e.g., next working day when I am back to keyboard, or next 5 minutes etc), the Elixir app gives timeout again.

The strange thing is that curl to the same website works well all the time, regardless of the Docker container is running or not.

I think it has something to do with the iptables. Here is iptables-save output with Docker container running:

$ sudo iptables-save 
# Generated by iptables-save v1.8.9 (nf_tables) on Thu Nov 28 11:54:54 2024
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [1:576]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 5432 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Thu Nov 28 11:54:54 2024
# Generated by iptables-save v1.8.9 (nf_tables) on Thu Nov 28 11:54:54 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 5432 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 5432 -j DNAT --to-destination 172.17.0.2:5432
COMMIT
# Completed on Thu Nov 28 11:54:54 2024
# Warning: iptables-legacy tables present, use iptables-legacy-save to see them

Here is the output after that Docker container stops:

$ sudo iptables-save 
# Generated by iptables-save v1.8.9 (nf_tables) on Thu Nov 28 11:55:10 2024
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [1:576]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Thu Nov 28 11:55:10 2024
# Generated by iptables-save v1.8.9 (nf_tables) on Thu Nov 28 11:55:10 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Thu Nov 28 11:55:10 2024
# Warning: iptables-legacy tables present, use iptables-legacy-save to see them

I read some articles in the Internet about Docker messing with host machine network. But I don’t know where to debug in the case of Elixir app. Did anyone experience something similar?

Thanks a lot!

6 comments

#docker #httpoison #networking

1 415 6

2024-12-10 00:34:46 UTC

Showing Posts 1 to 6

Show Best Posts
Show All (oldest first)
Show All (newest first)

D4no0

Are you running the application in docker too?

Post #1

tduccuong

No @D4no0, the app runs outside of the Docker container.

Post #2

D4no0

Yeah, IDK if you will have luck finding out many people doing such kind of hybrid deploys.

I would just containerize everything and check if that works.

Post #3

AstonJ

Do you run any software that updates frequently and needs to reload iptables? If so you might need to configure it to add the docker rules back in afterwards (although it looks like your rules are fine - but still, worth checking if there’s any software that’s running that could interfere with it).

Post #4

tduccuong

Actually I already narrowed down the situation to only 1 BEAM and 1 Docker container running. So no other services that would reload iptables.

I also thought the iptables rules are OK but somehow HTTPoison gives timeout. If I just do docker stop then start the container, things are back to normal until some random time later.

Strange for me is, curl always works regardless of docker is running or not. So I think maybe Elixir is doing some different underlying network calls?

Post #5

AstonJ

Another thing you could try next time it happens is disable the firewall temporarily - then check again. If it still does not work then it’s probably safe to say it’s nothing to do with the firewall, if it begins working then I would be of the opinion I was in my previous post, that something might be reloading iptables.

The fact that curl still works suggests it could be something else though (and in which case you may need to post more details about how everything is set-up/deployed etc)