Bamboo + Mailgun: fails in production with response Forbidden

Hi all,

I’m having an extremely unpleasant time with Bamboo/Mailgun in production. I’ve tested the MailgunAdapter for Bamboo from my local machine and everything works well, however in production I’m getting this error:

 ** (Bamboo.ApiError) There was a problem sending the email through the Mailgun API.
2021-07-19T18:12:56.834196+00:00 app[web.1]: 
2021-07-19T18:39:52.510503+00:00 app[web.1]: Here is the response:
2021-07-19T18:39:52.510503+00:00 app[web.1]: 
2021-07-19T18:39:52.510504+00:00 app[web.1]: "Forbidden"
2021-07-19T18:39:52.510504+00:00 app[web.1]: 
2021-07-19T18:39:52.510505+00:00 app[web.1]: Here are the params we sent:
2021-07-19T18:39:52.510505+00:00 app[web.1]: 
2021-07-19T18:39:52.510507+00:00 app[web.1]: %{"from" => "no-reply@my-mail-domain", "html" => "<html>...</html>", "subject" => "Please confirm your email", "to" => "my_email@address.com"}
2021-07-19T18:39:52.510507+00:00 app[web.1]: 
2021-07-19T18:39:52.510510+00:00 app[web.1]: (bamboo 2.1.0) lib/bamboo/strategies/task_supervisor_strategy.ex:25: anonymous fn/3 in Bamboo.TaskSupervisorStrategy.deliver_later/3
2021-07-19T18:39:52.510510+00:00 app[web.1]: (elixir 1.11.4) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
2021-07-19T18:39:52.510511+00:00 app[web.1]: (stdlib 3.14) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
2021-07-19T18:39:52.510512+00:00 app[web.1]: Function: #Function<0.6361412/0 in Bamboo.TaskSupervisorStrategy.deliver_later/3>
2021-07-19T18:39:52.510512+00:00 app[web.1]: Args: []

...

Note, the parameters look good, I can see the correct domain there at least and I’ve also logged into the server to check the API key value, it also looks good. The error seems to suggest that either the domain or the API key are wrong though (at least that’s the first thing that came to my mind).

Here is my config file for Bamboo. I’ve triple checked that my domain and the API key are configured correctly locally and on the server (Heroku). I’ve reached out to Mailgun support for help, let’s see if they can see any more information about the error. If anyone has experienced this or has any ideas about what could be going wrong, please do let me know because I’m getting desperate.

EDIT: I’ve updated the error description

How did you check the key on the server?

I would guess that this a runtime env problem. System.get_env() will be evaluated at compile time if you put on prod.exs. If you don’t have those variables during compilation they will be nil.

If you want your environment variables during runtime you should create a runtime.exs file and put those email vars there.

Good question. I checked it by logging into the server and running System.get_env(). To be honest I doubt that they would be nil during compilation because this would have cause a lot more problems, for example I’m also reading my DB URL via System.get_env() in the same file and my DB connectivity works well. Additionally, I can see that my mail domain (also read from env) was resolved correctly by looking at the parameters that were sent: %{"from" => "no-reply@<correct-domain>"...

I think you can try getting the envs with
Application.get_env(:sleep_rescue, SleepRescue.Mail.Mailer). If everything looks correct you can test the same config in dev.exs locally and retry the email request.

Yep, running in in local dev.exs works as expected

Are you perhaps using EU zone? Quote from here node.js - Mailgun - 401 forbidden - Stack Overflow

I had this problem when my domain was in an EU zone. When you’re using an EU zone, you have to specify it in the config - this isn’t clearly explained by Mailgun.
So it would be something like this:

var mailgun = require("mailgun-js")({
  apiKey: API_KEY,
  domain: DOMAIN,
  host: "api.eu.mailgun.net",
});

Yep, I’m using the EU zone in configs and that’s how I’ve set it up on the Mailgun side. Note that I’ve successfully tested it locally

Are you using eu zone for your heroku server?

1 Like

Good question and for a second I thought that it could be the solution. I’ve checked my Heroku region and it’s set to EU and I’ve also tested the URL https://api.mailgun.net/v3 just in case, but that didn’t help.

I haven’t used Mailgun but I did a really fast peek at their docs here Introduction — Mailgun API documentation
that you need to decide when creating a domain is it for US or EU. Are you sure you created your domain for EU?

Base URL

All API calls referenced in our documentation start with a base URL. Mailgun allows the ability to send and > receive email in either our US region or our EU region. Be sure to use the appropriate base URL based on which region you’ve created your domain in.

It’s also important to note that Mailgun uses URI versioning for our API endpoints, and some endpoints may have different versions than others. Please reference the version stated in the URL for each endpoint.

For domains created in our US region the base URL is:

https://api.mailgun.net/

For domains created in our EU region the base URL is:

https://api.eu.mailgun.net/

Your Mailgun account may contain multiple sending domains. To avoid passing the domain name as a query > parameter, most API URLs must include the name of the domain you’re interested in:

https://api.mailgun.net/v3/mydomain.com

Yes, like I said in my previous post:

Good question and for a second I thought that it could be the solution. I’ve checked my Heroku region and it’s set to EU and I’ve also tested the URL https://api.mailgun.net/v3 just in case, but that didn’t help.

I’m not talking about Heroku’s region. I’m talking about that reading Mailgun docs it seems that when you add domain to Mailgun you need to select is that domain for US or EU.

Hey all,

@phcurado @wanton7 @derek-zhou @sfusato first, thank you all for trying to help. I’m a lone developer trying to get my side project running and it’s extremely useful to have someone to talk to about the problems that I run into along the way.

I reached out to Mailgun’s support team and I’ve finally fixed the problem. It turns out that I whitelisted my own IP which meant that it was rejecting all other IPs. After removing my IP from the white-list it finally works!

Thank you all again!

5 Likes