Trouble with connecting to private app on Fly.io

I deployed an app that is only accessible internally within the organization. When I try to connect to http://private-app.internal:8080, I receive the following error:

# This request is made from another Fly app within the organization
> Req.post!("http://private-app.internal:8080/messages/new")
** (Mint.TransportError) non-existing domain
    (req 0.3.0) lib/req.ex:641: Req.request!/1

The private-app is listening to IPv4 and IPv6 as per suggested in [3] & [4].

Can someone please shed some light on this problem?

References:

  1. Having issues connecting 2 golang servers over .internal network - Questions / Help - Fly.io
  2. Private application only accessible internally - Questions / Help - Fly.io
  3. Fly io Newbie: Making internal requests between apps - #5 by ignoramous - Build debugging - Fly.io
  4. Redis app refuses private connections - #5 by rugwiro - Fly.io

This sounds like an DNS issue with your application if it can’t resolve that domain correctly. If you had the ability to run curl on app, you could determine whether this is a Elixir issue or (most likely) a networking issue. I assume you’re talking about this specifically? Private Networking · Fly Docs

I was able to dig and query the DNS as shown here. So far, everything looks correct to me.

After hours of searching and testing, I was finally able to solve the problem.

For those who come across the same problem,

  1. Make sure your app is bind to :: instead of 0.0.0.0. In my case, I had to specify :: in Dockerfile for my python app like so:
FROM python:3.10-slim-bullseye

ENV PYTHONUNBUFFERED True
ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . ./

RUN pip install --no-cache-dir -r requirements.txt

CMD ["uvicorn", "main:app", "--host", "::", "--port", "8080"]
  1. If you’re using http clients like Req or Finch, make sure you explicitly enable inet6 because Fly uses IPv6 for private apps [1].
> req = Req.new(url: "http://private-app.internal:8080", connect_options: [transport_opts: [inet6: true]])

[2][3]

  1. Make sure to connect over plaintext http [4].

And boom! Problem solved!

References

  1. 6PN addressing clarification - Fly.io
  2. Connect to IPv6 addresses · Issue #554 · benoitc/hackney · GitHub
  3. IPv6 Documentation · Issue #163 · sneako/finch · GitHub
  4. Trouble connecting to private app - #4 by ignoramous - elixir - Fly.io?