Our application connects to two different databases through 2 different repos. After changing the password for one of them, our application would not respond to HTTP requests anymore. Digging in, we identified the problem, changed the password, and brought the application back online.
I am still puzzled about what exactly happened. I understand certain characters need to be escaped, so the error was on our side. I would have expected the app to throw errors if it can’t connect to the DB, which didn’t happen. I would expect the repo process to fail and the code trying to access the DB to fail as well. Instead, the whole execution was blocked and nothing happened. I was able to reproduce this locally running mix phx.server, where after compilation, nothing happens. The endpoint isn’t available, all the children from the Supervisor do not start and the app is just idling.
I tried to create a minimal example to reproduce but wasn’t able to. If I create a new phoenix app, add a second repo, and add our malformed URL, it throws errors as expected. So now I’m wondering if we are doing something wrong and how to debug further. Any ideas?
if indeed there was no error it looks like a bug. I would recommend you generate a minimal example to reproduce, (i.e. mix phx.new …) and then set the DB url to similar what you had and upload it to GitHub as public repo and open an issue on Ecto repository on Github to investigate because you’re right, it should not hang, it should crash and error.
Do you have any background processes that access that DB in your full application? Those will try to start, crash, and restart over and over until they reach their Supervisor’s limit - then the Supervisor will exit!
I tried that and am not able to reproduce it. It might be because the bug is in a combination of config and libraries or because it’s not a bug in a library but in our code.
We removed all children except for the repos and the endpoint. The endpoint didn’t start, so it seems like the repo itself is blocking. I am not sure how to debug further.