I had an issue similar to yours, and got it working with the following configuration:

config :messenger, Messenger.Mailer,
    sockopts: [
      cacerts: :public_key.cacerts_get(),
      customize_hostname_check: [
        match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
      ],
    ],
    adapter: Swoosh.Adapters.SMTP,
    relay: "smtp-relay.gmail.com",
    username: "no-reply@colecao.moda",
    tls: :never,
    ssl: true,
    auth: :always,
    retries: 2,
    no_mx_lookups: false,
    sockopts: [
      versions: [:"tlsv1.2", :"tlsv1.3"],
      verify: :verify_peer,
      depth: 99,
      server_name_indication: 'smtp-relay.gmail.com'
    ]

I am assuming you are using STARTTLS on port 465 with a self signed certificate
.
You have 2 ssl keys in the config, and it should be set to false. for details. see the official swoosh doc:

Yours is different, isn’t it? You use tls: :never

Why sockopts 2 times?

Where 2 ssl keys? What set to false? Why?

https://hexdocs.pm/swoosh/Swoosh.Adapters.SMTP.html#module-note

With STARTTLS you should omit the ssl configuration or set it to false.

In my case it’s not STARTTLS - it’s SSL/TLS

The :tls setting is for STARTTLS. If you’re not using starttls you should disable it. You can find a bit more context about that here: SSL connection options · Issue #298 · gen-smtp/gen_smtp · GitHub

tls: :never,

Still

[notice] TLS :client: In state :wait_cert_cr at ssl_handshake.erl:2140 generated CLIENT ALERT: Fatal - Handshake Failure
 - {:bad_cert, :max_path_length_reached}

You have one line as ssl: true, and one as ssl: [ ...

I was also struggling with this issue for a good while until I used almost the exact configuration mentioned here by @LostKobrakai SSL connection options · Issue #298 · gen-smtp/gen_smtp · GitHub

Here is my config which now (finally) works (using this smtp server)

  config :backend, Backend.Mailer,
    adapter: Swoosh.Adapters.SMTP,
    relay: System.get_env("EMAIL_SMTP_HOST"),
    username: System.get_env("EMAIL_SMTP_USER"),
    password: System.get_env("EMAIL_SMTP_PASSWORD"),
    port: String.to_integer(System.get_env("EMAIL_SMTP_PORT") || "465"),
    ssl: true,
    tls: :never,
    auth: :always,
    retries: 2,
    no_mx_lookups: false,
    sockopts: [
      versions: [:"tlsv1.2", :"tlsv1.3"],
      verify: :verify_peer,
      cacerts: :public_key.cacerts_get(),
      depth: 3,
      customize_hostname_check: [
        match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
      ],
      server_name_indication: 'mail.privateemail.com'
    ]

It was pretty tough to figure this out as someone very new to Elixir, especially since I couldn’t find this sockopts key documented anywhere in Swoosh or the gen_smtp Readme and I still don’t really understand why this is necessary, but maybe it helps someone.
What was also really tripping me up for a bit was the fact that server_name_indication apparently needs to be a charlist (single quoted) and I was trying to set it as a string with System.get_env("EMAIL_SMTP_HOST").

Perhaps someone who understands this better than me could add a note about it to the Swoosh SMTP adapter docs, since without these settings it doesn’t seem to be possible to get it to work?

I might be able to give some insight around the why here: Erlangs SSL handling is a lot more explicit (and hence complicated) than in many other places. Given nowadays emails usually use SSL over starttls you’re running into that explicitness. That’s most of the reason for the sockopts. More info can be found here: Erlang standard library: ssl | EEF Security WG

server_name_indication is additional required because by default (no_mx_lookup: false) gen_smtp doesn’t connect to the smtp server using the provided hostname, but rather it looks up the IP address for the hostname using the MX dns entry and uses that IP to connect to the server. Usually SSL certificates are supplied for a given hostname though, not an IP address. Therefore one needs to supply the hostname as SNI value, so the certificate can be validated against that hostname instead.

Disabling MX lookup does remove the need for the SNI value.

@LostKobrakai Unfortunately, using cacerts: :public_key.cacerts_get() inside of sockopts does not work when deploying to Fly.io. When I try to deploy with this configuration I get the following:

ERROR! Config provider Config.Reader failed with:
  ** (MatchError) no match of right hand side value: {:error, :enoent}
      pubkey_os_cacerts.erl:38: :pubkey_os_cacerts.get/0
      /app/releases/0.1.0/runtime.exs:65: (file)
      (elixir 1.14.5) src/elixir.erl:312: anonymous fn/4 in :elixir.eval_external_handler/1

It seems that this is expected behavior when no CA cert is found on the OS. However, my understanding is that Fly.io does not place the generated certificates inside its created containers. So, how would one fix this when deploying to Fly.io?