Unable to send an email via my SMTP server via SSL/TLS. But my local email client works well

[silverdr]

@fmn

to me this looks like a protocol version mismatch (for example, SSLv3 vs TLS1.2), or an attempt to use wrong protocol altogether (HTTPS on HTTP port or TLS on STARTTLS etc.)

Erm… no, the application worked for a long time correctly and there were zero changes to the configuration or any relevant parts of the application code between the time it worked and stopped working. The only change was security updates to mail server. I can’t be sure if this started the trouble because the problem was not immediately noticed, only about two days after said server upgrade. Still, I suspect the security update dropped support for pre 1.3 TLS versions and gen_smtp docs say:

" tls_options used in ssl:connect , More info at Erlang -- ssl . Defaults to [{versions , ['tlsv1', 'tlsv1.1', 'tlsv1.2']}]"

At least that’s my suspicion, which combined with the TLSv1.3 bug with OTP > 24 seems to make good sense

[fmn]

@silverdr

Erm… no, the application worked for a long time correctly and there were zero changes

sure, just suggesting an approach which helped me many many times in the past in similar situations.
reg. “nothing changed, it worked in the past” - well, obviously something changed, and you don’t know what. using Wireshark way i suggested can help you to figure it out.

[silverdr]

I only referred to the part of incorrect port/protocol/… which would typically mean misconfiguration but configuration was and is apparently correct and hasn’t been changed in any way. When troubleshooting I also dumped the SSL negotiation into mail server logs, which while not so sophisticated as Wireshark showed enough to point back to the client. In any case thank you for your input too, and I apologise if my response sounded inappropriate. Wasn’t intended so

[fmn]

no no! nothing wrong with your reply!
i still think quite explicit hints about origin of your issues lie in TLS handshake.

[silverdr]

Yes. The mail server log dumps showed that the client (the elixir application → Swoosh → gen_smtp → OTP26) didn’t react properly during 1.3 negotiation/handshake. It sent only a few bytes instead of a supposedly expected cert response. FWIW I wouldn’t know how to plug Wireshark between remote docker container and the server where the network is a docker “bridge” network (and replicating the setup locally would be a rather sizeable effort) so my only quick idea was to enable debug logging on the mail server in order to get more detailed SSL negotiation communication dumps. While this was still a lot of hexdump bytes, the behaviour looked somewhat similar to the symptoms described in the bug report and follow-up discussion on the OTP Github issue @al2o3cr linked to. All in all thanks to all who responded - it’s a devops nightmare to to get key functionality of a prod env application suddenly severed with no apparent fault on your side and no solution in sight after many hours of troubleshooting. Luckily OTP 24 works all fine now

[silverdr]

FWIW - 26.2.5 images with Elixir 1.16.2 are available now for over a week. My problem was indeed caused by that OTP 25/26 bug but… for “posterity” it was not enough to just replace OTP with 26.2.5!

Once I did built the “runner” image with 26.2.5 it started throwing another error, namely a

- {:bad_cert, :max_path_length_reached}

one, which of course was never a problem before Luckily this one was much easier to find a solution to. Going to Erlang -- ssl and doing search for “path” revealed

https://www.erlang.org/doc/man/ssl#type-allowed_cert_chain_length

which talks about “depth” so searching for “depth” lead me to an option in common_option, which given a nice, round number of 64 cured the setup. Thus my current (working) prod config – if somebody ever needs hints is:

	config :my_app, MyApp.Mailer,
		adapter: Swoosh.Adapters.SMTP,
		relay: System.get_env("SMTP_RELAY", "mail.myapp.com"),
		hostname: System.get_env("SMTP_HOSTNAME", "mail.mysmtpserver.com"),
		username: System.get_env("SMTP_USERNAME", "no-reply@myapp.com"),
		password: System.get_env("SMTP_PASSWORD"),
		auth: :always,
		ssl: false,
		tls: :always,
		tls_options: [
			versions: [:"tlsv1.3"],
			cacerts: :public_key.cacerts_get(),
			server_name_indication: 'mail.mysmtpserver.com', # what your certificate is issued for
			depth: 64, # important or stuff may crash with - {:bad_cert, :max_path_length_reached}
		],
		port: 587,
		retries: 2

hostnames and application name are “dummies” of course.

to @al2o3cr the umpteenth time

[ricw]

Not exactly the use case here, but in case you want to use swoosh + smtp adapter + gmail to simply send some emails, here is a configuration that worked for me:

email_password =
  System.get_env("GMAIL_PASSWORD") ||
  raise "environment variable GMAIL_PASSWORD is missing."

config :aidebase, Aidebase.Mailer,
  adapter: Swoosh.Adapters.SMTP,
  relay: "smtp.gmail.com",
  username: "email@googleappsdomain/gmail.com",
  password: email_password,
  auth: :always,
  ssl: false,
  tls: :always,
  tls_options: [
    versions: [:"tlsv1.3"],
    cacerts: :public_key.cacerts_get(),
    verify: :verify_none,
    depth: 64
  ],
  port: 587,
  retries: 2

Took me 2 days of tracking down missing debug statements, understanding better what’s going (not that i do now) and trying to configure swoosh’s SMTP adapter for gmail. You’d think this is a standard use case, but there really isn’t much info on this. The documentation is really lacking for this…

Anycase, this is the configuration that worked for me in the end. Hopefully it saves someone else this pain.

Here are some other errors i encountered on the way (to help people googling this):

• {:retries_exceeded, {:temporary_failure, ~c"1.2.3.9", :tls_failed}}
• {:error, {:tls_alert, {:unexpected_message, ~c"TLS client: In state hello at tls_record.erl:561 generated CLIENT ALERT: Fatal - Unexpected Message\n {unsupported_record_type,50}"}}}
• {:error, {:options, :incompatible, [verify: :verify_peer, cacerts: :undefined]}}}

[ruslandoga]

@ricw

Note: I’m not suggesting changing anything that already works but rather posting it for the next person having troubles with gen_smtp

There is now another SMTP adapter that might have been easier to set up:

config :aidebase, Aidebase.Mailer,
  adapter: Swoosh.Adapters.Mua,
  relay: "smtp.gmail.com",
  port: 587,
  auth: [username: System.fetch_env!("GMAIL_USERNAME"), password: System.fetch_env!("GMAIL_PASSWORD")]

I’ve been using it with Gmail for a self-hosted Plausible instance: plausible-mua-gmail.md · GitHub

Note that by default this configuration would use:

• cacerts: :public_key.cacerts_get()
• verify: :verify_peer
• versions: [:"tlsv1.3", :"tlsv1.2"]

[Lestrn]

THANK YOU!!! Cannot imagine how much time i`ve spent looking for exactly this!

[marschro]

And again… after searching for a solution, as I had this issue… @LostKobrakai has the solution

THANK YOU SO MUCH - took me hours again before finding your post and get it solved…
I would give you more than one if possible…

Maybe just one not… single quotes now have to be ~c’whatever’ now… otherwise not compiling…

[marschro]

By the way…

… if anyone comes here with that issue on

• FreeBSD (FreeBSD 14.2-RELEASE) with
• Erlang/OTP 26 [erts-14.2.5.5]

or in a FreeBSD jail with above specs…

There is currently a bug, preventing erlang from reading the root OS certificates.

Error looks like that:

** (MatchError) no match of right hand side value: :undefined
    (public_key 1.15.1.4) pubkey_os_cacerts.erl:39: :pubkey_os_cacerts.get/0```

Workaround is:
> cat /etc/ssl/certs/*.0 > /etc/ssl/cert.pem

The Bug is filed here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=284533