tduccuong

tduccuong

Hi everyone,

I have an Elixir app that sends out HTTP requests using HTTPoison. Usually it works fine, but I noticed it gives timeout error occasionally when a Docker container running in the same machine. The number of containers does not matter. I could narrow down the situation to only 1 container running and still got the timeout error. The container runs PostgreSQL. I got it running by just docker run -d postgres.

I suspected that because if I stop the Docker container, the Elixir app works well again. Then if I run the PostgreSQL container again, after some random time (e.g., next working day when I am back to keyboard, or next 5 minutes etc), the Elixir app gives timeout again.

The strange thing is that curl to the same website works well all the time, regardless of the Docker container is running or not.

I think it has something to do with the iptables. Here is iptables-save output with Docker container running:

$ sudo iptables-save 
# Generated by iptables-save v1.8.9 (nf_tables) on Thu Nov 28 11:54:54 2024
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [1:576]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 5432 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Thu Nov 28 11:54:54 2024
# Generated by iptables-save v1.8.9 (nf_tables) on Thu Nov 28 11:54:54 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 5432 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 5432 -j DNAT --to-destination 172.17.0.2:5432
COMMIT
# Completed on Thu Nov 28 11:54:54 2024
# Warning: iptables-legacy tables present, use iptables-legacy-save to see them

Here is the output after that Docker container stops:

$ sudo iptables-save 
# Generated by iptables-save v1.8.9 (nf_tables) on Thu Nov 28 11:55:10 2024
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [1:576]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Thu Nov 28 11:55:10 2024
# Generated by iptables-save v1.8.9 (nf_tables) on Thu Nov 28 11:55:10 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Thu Nov 28 11:55:10 2024
# Warning: iptables-legacy tables present, use iptables-legacy-save to see them

I read some articles in the Internet about Docker messing with host machine network. But I don’t know where to debug in the case of Elixir app. Did anyone experience something similar?

Thanks a lot!

Showing Posts 1 to 6

D4no0

D4no0

Are you running the application in docker too?

tduccuong

tduccuong OP

No @D4no0, the app runs outside of the Docker container.

D4no0

D4no0

Yeah, IDK if you will have luck finding out many people doing such kind of hybrid deploys.

I would just containerize everything and check if that works.

AstonJ

AstonJ

Do you run any software that updates frequently and needs to reload iptables? If so you might need to configure it to add the docker rules back in afterwards (although it looks like your rules are fine - but still, worth checking if there’s any software that’s running that could interfere with it).

tduccuong

tduccuong OP

Actually I already narrowed down the situation to only 1 BEAM and 1 Docker container running. So no other services that would reload iptables.

I also thought the iptables rules are OK but somehow HTTPoison gives timeout. If I just do docker stop then start the container, things are back to normal until some random time later.

Strange for me is, curl always works regardless of docker is running or not. So I think maybe Elixir is doing some different underlying network calls?

AstonJ

AstonJ

Another thing you could try next time it happens is disable the firewall temporarily - then check again. If it still does not work then it’s probably safe to say it’s nothing to do with the firewall, if it begins working then I would be of the opinion I was in my previous post, that something might be reloading iptables.

The fact that curl still works suggests it could be something else though :confused: (and in which case you may need to post more details about how everything is set-up/deployed etc)

— All posts loaded —

Where Next? Top

Trending in Questions Top

Hellowman
Hello everyone! I am new to coding with Elixir/Erlang. And i am looking for some utilities for fuzzing testing like AFL++. Does anyone ...
New
yarrichar
I’m very new to Elixir, so sorry if this is a silly question: I am wondering what happens to the mailbox of a process when the machine i...
New
pragdave
How do I tell Hologram not to escape HTML when substituting into templates? {Textile.render(@product.long_description)} escapes the HTM...
New
Fl4m3Ph03n1x
Background When trying to execute mix release on a Windows 11 machine for a Phoenix project I get the following error: * assembling mark...
New
mandar1045
I load tested a plain Phoenix chat app backed by Postgres with 100 million messages in it, everything on default settings, to see what br...
New
rlipscombe
I’d like to create a file with 600 permissions (rw-------), but there’s a race condition if I create it and then change the permissions. ...
New
saleyn
Is there a way to import record definitions in the iex shell for pretty printing (similar to the rr command in the erl shell)? This woul...
New

Other Trending Topics Top

dominicletz
Hey there, I haven’t seen a list of useful Elixir specific skills for the popular AI tools yet so I’m starting one here with my first. ...
New
raa-ziq
Hi everyone :waving_hand: I wanted to share something I’ve been building — NowBlind, a random chat app designed to help people discover ...
New
mudasobwa
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
OndrejValenta
(I just needed to vent somewhere and LinkedIn is full of hope, or hype, I’m not sure which exactly) AI dream has many faces, but general...
New
kip
Localize is the next generation localisation library for Elixir. Think of it as ex_cldr version 3.0. The first version will be released ...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews