Unable to send an email via my SMTP server via SSL/TLS. But my local email client works well

I have my own SMTP server. It’s behind SSL/TLS - port 465

In my phoenix I use Swoosh to send emails. Since recently I’ve been facing this error:

delivery error:
{:retries_exceeded, {:network_failure, ~c"mail.my_mail_server.com", {:error, {:options, :incompatible, [verify: :verify_peer, cacerts: :undefined]}}}}

My config:

adapter: Swoosh.Adapters.SMTP,
    relay: host,
    username: user,
    password: password,
    port: port,

    ssl: true,
    tls: :always,

    # allowed_tls_versions: [:tlsv1, :"tlsv1.1", :"tlsv1.2", :"tlsv1.3"],
    auth: :always,
    retries: 5,
    no_mx_lookups: true,


    ssl: [
    # ssl_opts: [
    # ssl_options: [

        verify: :verify_none,
      # verify: :verify_peer,

      # cacerts: :public_key.cacerts_get(),
      # versions: [:"tlsv1.2"],
      # versions: [:"tlsv1.3"],
      #   customize_hostname_check: [
      #     match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
      #   ]
    ],

    tls_options: [
        # verify: :verify_peer,
        verify: :verify_none,

        # cacerts: :certifi.cacerts(),
        # cacerts: :public_key.cacerts_get(),
        #   server_name_indication: ~c"#{host}",
    ]

An interesting thing is that all the options that have to do with tls and ssl will be ignored.

Meaning, the error will always contain

<...> { verify: :verify_peer, cacerts: :undefined}...

There’s never been an error with verify: :verify_none and cacerts: <something_else>, even though I’ve set it up.

Why? I’ve set different values in the config. Why will they remain verify: :verify_peer, cacerts: :undefined ?

And it’s unclear wether I should used ssl_opts, ssl or ssl_options – I’ve tried dozens of combinations. The same goes for the tls_options.

What’s the matter?


The emails I’ll send from an email-client from my local computer via the same email server get sent with no issue, and under the same settings: port 465, SSL/TLS, same relay.

P.S.

OTP 26.

I’m aware of this - Erlang/OTP 26 Highlights - Erlang/OTP

But, as I’ve mentioned, it’ll ignore my verify: <...> variables in the first place.

I had an issue similar to yours, and got it working with the following configuration:

config :messenger, Messenger.Mailer,
    sockopts: [
      cacerts: :public_key.cacerts_get(),
      customize_hostname_check: [
        match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
      ],
    ],
    adapter: Swoosh.Adapters.SMTP,
    relay: "smtp-relay.gmail.com",
    username: "no-reply@colecao.moda",
    tls: :never,
    ssl: true,
    auth: :always,
    retries: 2,
    no_mx_lookups: false,
    sockopts: [
      versions: [:"tlsv1.2", :"tlsv1.3"],
      verify: :verify_peer,
      depth: 99,
      server_name_indication: 'smtp-relay.gmail.com'
    ]

I am assuming you are using STARTTLS on port 465 with a self signed certificate
.
You have 2 ssl keys in the config, and it should be set to false. for details. see the official swoosh doc:

https://hexdocs.pm/swoosh/Swoosh.Adapters.SMTP.html#module-note

Yours is different, isn’t it? You use tls: :never

Why sockopts 2 times?

Where 2 ssl keys? What set to false? Why?

https://hexdocs.pm/swoosh/Swoosh.Adapters.SMTP.html#module-note

With STARTTLS you should omit the ssl configuration or set it to false.

In my case it’s not STARTTLS - it’s SSL/TLS

The :tls setting is for STARTTLS. If you’re not using starttls you should disable it. You can find a bit more context about that here: SSL connection options · Issue #298 · gen-smtp/gen_smtp · GitHub

1 Like
tls: :never,

Still

[notice] TLS :client: In state :wait_cert_cr at ssl_handshake.erl:2140 generated CLIENT ALERT: Fatal - Handshake Failure
 - {:bad_cert, :max_path_length_reached}

You have one line as ssl: true, and one as ssl: [ ...

I was also struggling with this issue for a good while until I used almost the exact configuration mentioned here by @LostKobrakai SSL connection options · Issue #298 · gen-smtp/gen_smtp · GitHub

Here is my config which now (finally) works (using this smtp server)

  config :backend, Backend.Mailer,
    adapter: Swoosh.Adapters.SMTP,
    relay: System.get_env("EMAIL_SMTP_HOST"),
    username: System.get_env("EMAIL_SMTP_USER"),
    password: System.get_env("EMAIL_SMTP_PASSWORD"),
    port: String.to_integer(System.get_env("EMAIL_SMTP_PORT") || "465"),
    ssl: true,
    tls: :never,
    auth: :always,
    retries: 2,
    no_mx_lookups: false,
    sockopts: [
      versions: [:"tlsv1.2", :"tlsv1.3"],
      verify: :verify_peer,
      cacerts: :public_key.cacerts_get(),
      depth: 3,
      customize_hostname_check: [
        match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
      ],
      server_name_indication: 'mail.privateemail.com'
    ]

It was pretty tough to figure this out as someone very new to Elixir, especially since I couldn’t find this sockopts key documented anywhere in Swoosh or the gen_smtp Readme and I still don’t really understand why this is necessary, but maybe it helps someone.
What was also really tripping me up for a bit was the fact that server_name_indication apparently needs to be a charlist (single quoted) and I was trying to set it as a string with System.get_env("EMAIL_SMTP_HOST").

Perhaps someone who understands this better than me could add a note about it to the Swoosh SMTP adapter docs, since without these settings it doesn’t seem to be possible to get it to work?

3 Likes

I might be able to give some insight around the why here: Erlangs SSL handling is a lot more explicit (and hence complicated) than in many other places. Given nowadays emails usually use SSL over starttls you’re running into that explicitness. That’s most of the reason for the sockopts. More info can be found here: Erlang standard library: ssl | EEF Security WG

server_name_indication is additional required because by default (no_mx_lookup: false) gen_smtp doesn’t connect to the smtp server using the provided hostname, but rather it looks up the IP address for the hostname using the MX dns entry and uses that IP to connect to the server. Usually SSL certificates are supplied for a given hostname though, not an IP address. Therefore one needs to supply the hostname as SNI value, so the certificate can be validated against that hostname instead.

Disabling MX lookup does remove the need for the SNI value.

3 Likes

@LostKobrakai Unfortunately, using cacerts: :public_key.cacerts_get() inside of sockopts does not work when deploying to Fly.io. When I try to deploy with this configuration I get the following:

ERROR! Config provider Config.Reader failed with:
  ** (MatchError) no match of right hand side value: {:error, :enoent}
      pubkey_os_cacerts.erl:38: :pubkey_os_cacerts.get/0
      /app/releases/0.1.0/runtime.exs:65: (file)
      (elixir 1.14.5) src/elixir.erl:312: anonymous fn/4 in :elixir.eval_external_handler/1

It seems that this is expected behavior when no CA cert is found on the OS. However, my understanding is that Fly.io does not place the generated certificates inside its created containers. So, how would one fix this when deploying to Fly.io?

The OS is the container you’re running. You can choose to install certificates into it, given you control how the container is built. Fly doesn’t play a part in that.

1 Like

Apparently, all I had to do was install ca-certificates into the base Debian image in my Dockerfile.

So, during the RUN apt-get update -y stage, just add ca-certificates as another package to be installed.

Thank you for pointing me in the right direction.

For reference, here is a related thread on Fly.io.

2 Likes

… and I’ve got a TLS (STARTTLS) server running on “submission” port (587), which means ssl: false and began receiving errors:

{:error, {:send, {:network_failure, ~c"xx.xx.xx.xx", {:error, :closed}}}}

where “xx.xx.xx.xx” it the correct, expected IP address, and in server logs:

May  1 20:06:49 postfix/submission/smtpd[43779]: SSL_accept error from unknown[172.17.0.3]: -1
May  1 20:06:49 postfix/submission/smtpd[43779]: warning: TLS library problem: error:0A00010B:SSL routines::wrong version number:../ssl/record/ssl3_record.c:354:
May  1 20:06:49 postfix/submission/smtpd[43779]: lost connection after STARTTLS from unknown[172.17.0.3]

I suspect this started after recent server packages update, which might have hard-deprecated some earlier TLS protocols. I can use the server from every mail client I always used. Also from the container running Elixir application in question I can use openssl (of course with correct hostname rather than example.com):

openssl s_client -starttls smtp -connect example.com:587

and I get session correctly initiated:

[…]
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
[…]

and can manually interact with smptd using SMTP commands. This seems to exclude any configuration errors outside of the application itself. The relevant application config always looked like:

	config :my_app, MyApp.Mailer,
		adapter: Swoosh.Adapters.SMTP,
		relay: System.get_env("SMTP_RELAY", "mail.myapp.net"),
		hostname: System.get_env("SMTP_HOSTNAME", "server.myapp.net"),
		username: System.get_env("SMTP_USERNAME", "no-reply@myapp.net"),
		password: System.get_env("SMTP_PASSWORD"),
		# auth: :always,
		# ssl: false,
		# tls: :always,
		port: 587,
		retries: 2

and worked fine all the time, until it stopped.

I tried:

		allowed_tls_versions: [:"tlsv1.3"],
		tls_options: [
			versions: [:"tlsv1.3"]
		],

and even the

		sockopts: [
			versions: [:"tlsv1.3"]
		]

the last crashing with :badarg.

Any ideas what I might still try?

What version of OTP are you on? Have you upgraded recently?

Asking because there’s a fix that just landed for TLSv1.3 in OTP 25 + 26:

3 Likes

It’s 26! Gosh… this might be it. I didn’t upgrade OTP recently but I did upgrade mail server packages (security updates), which might have removed support for some earlier TLS protocols, which in turn might coincide with OTP 26 having problems with TLSv1.3… What’s the minimal OTP for Elixir 16?

Update: found it here:

https://hexdocs.pm/elixir/main/compatibility-and-deprecations.html#between-elixir-and-erlang-otp

So may have to try 24…

I encountered a similar error.
Same configuration works in a different application, but in a new one I’m trying to set up, things kept failing with :badarg or :tls_error with different configurations.

What helped was upgrading swoosh to its latest version.
Hope this helps.

Accorging to mix.lock:

:gen_smtp, "1.2.0"
:swoosh, "1.16.5"

In fact swoosh was at

:swoosh, "1.15.2"

when the problem started but upgrading deps was one of the first things I tried

May  1 20:06:49 postfix/submission/smtpd[43779]: warning: TLS library problem: error:0A00010B:SSL routines::wrong version number:../ssl/record/ssl3_record.c:354:

to me this looks like a protocol version mismatch (for example, SSLv3 vs TLS1.2), or an attempt to use wrong protocol altogether (HTTPS on HTTP port or TLS on STARTTLS etc.)

that kind of issues are fairly easy to detect if you will dump the traffic and interpret it. both things can be done using Wireshark.

Wireshark filters are quite easy to use: you can see what cipher suites and protocol versions both parties are supporting / offering during initial handshake.

I found this one:

https://hub.docker.com/layers/hexpm/elixir/1.16.2-erlang-24.3.4.17-debian-bookworm-20240423-slim/images/sha256-8ebf7c8b043e1f34d95bf8fa03c82124dc61f8a18a34c88e4edbfed3b64a335a

built application image based on it and… :boom: problem gone! @al2o3cr I can’t express how grateful I am for pointing this OTP bug out! :bowing_man::bowing_man:

1 Like