Issue of SERVER ALERT: Fatal - No application protocol, SERVER ALERT: Fatal - No application protocol, etc

enkr1 · December 7, 2022, 4:06am

Hi community,

I had a server running a while ago and it stopped running suddenly. I observed the logs and suspected somebody is trying to brute force my site by accessing different common WordPress (or something else’s) routes:

===== ALIVE Mon Dec  5 18:59:26 UTC 2022

===== Mon Dec  5 19:09:52 UTC 2022
19:09:52.249 [info] Plug.SSL is redirecting GET /.env to https://mysite.com with status 301
19:09:52.611 [info] Plug.SSL is redirecting GET /conf/.env to https://mysite.com with status 301
19:09:52.960 [info] Plug.SSL is redirecting GET /wp-content/.env to https://mysite.com with status 301
19:09:53.331 [info] Plug.SSL is redirecting GET /wp-admin/.env to https://mysite.com with status 301
19:09:53.682 [info] Plug.SSL is redirecting GET /library/.env to https://mysite.com with status 301
19:09:54.056 [info] Plug.SSL is redirecting GET /new/.env to https://mysite.com with status 301
19:09:54.424 [info] Plug.SSL is redirecting GET /vendor/.env to https://mysite.com with status 301
19:09:54.777 [info] Plug.SSL is redirecting GET /old/.env to https://mysite.com with status 301
19:09:55.160 [info] Plug.SSL is redirecting GET /local/.env to https://mysite.com with status 301
19:09:55.540 [info] Plug.SSL is redirecting GET /api/.env to https://mysite.com with status 301
19:09:55.874 [info] Plug.SSL is redirecting GET /blog/.env to https://mysite.com with status 301
19:09:56.242 [info] Plug.SSL is redirecting GET /crm/.env to https://mysite.com with status 301
19:09:56.609 [info] Plug.SSL is redirecting GET /admin/.env to https://mysite.com with status 301
19:09:56.962 [info] Plug.SSL is redirecting GET /laravel/.env to https://mysite.com with status 301
19:09:57.311 [info] Plug.SSL is redirecting GET /app/.env to https://mysite.com with status 301
19:09:57.648 [info] Plug.SSL is redirecting GET /app/config/.env to https://mysite.com with status 301
19:09:58.003 [info] Plug.SSL is redirecting GET /apps/.env to https://mysite.com with status 301
19:09:58.381 [info] Plug.SSL is redirecting GET /audio/.env to https://mysite.com with status 301
19:09:58.728 [info] Plug.SSL is redirecting GET /cgi-bin/.env to https://mysite.com with status 301
19:09:59.079 [info] Plug.SSL is redirecting GET /backend/.env to https://mysite.com with status 301
19:09:59.455 [info] Plug.SSL is redirecting GET /src/.env to https://mysite.com with status 301
19:09:59.829 [info] Plug.SSL is redirecting GET /base/.env to https://mysite.com with status 301
19:10:00.186 [info] Plug.SSL is redirecting GET /core/.env to https://mysite.com with status 301
19:10:00.551 [info] Plug.SSL is redirecting GET /vendor/laravel/.env to https://mysite.com with status 301
19:10:00.891 [info] Plug.SSL is redirecting GET /storage/.env to https://mysite.com with status 301
19:10:01.234 [info] Plug.SSL is redirecting GET /protected/.env to https://mysite.com with status 301
19:10:01.562 [info] Plug.SSL is redirecting GET /newsite/.env to https://mysite.com with status 301
19:10:01.904 [info] Plug.SSL is redirecting GET /www/.env to https://mysite.com with status 301
19:10:02.263 [info] Plug.SSL is redirecting GET /sites/all/libraries/mailchimp/.env to https://mysite.com with status 301
19:10:02.607 [info] Plug.SSL is redirecting GET /database/.env to https://mysite.com with status 301
19:10:02.952 [info] Plug.SSL is redirecting GET /public/.env to https://mysite.com with status 301
19:10:03.352 [info] Plug.SSL is redirecting GET /ec2-18-142-114-116.ap-southeast-1.compute.amazonaws.com/.env to https://mysite.com with status 301
19:10:03.701 [info] Plug.SSL is redirecting POST / to https://mysite.com with status 307
19:10:04.047 [info] Plug.SSL is redirecting GET /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php to https://mysite.com with status 301
19:11:01.689 [notice] TLS :server: In state :hello at tls_handshake.erl:346 generated SERVER ALERT: Fatal - Insufficient Security
 - :no_suitable_ciphers
19:11:30.675 [info] Plug.SSL is redirecting GET /.env to https://mysite.com with status 301
19:11:31.519 [info] Plug.SSL is redirecting POST / to https://mysite.com with status 307
19:14:56.662 [info] Plug.SSL is redirecting GET / to https://mysite.com with status 301
19:14:56.725 [info] Plug.SSL is redirecting GET / to https://mysite.com with status 301
19:14:58.158 request_id=Fy37nzyNa_Qcb7QAASix [info] GET /
19:14:58.159 request_id=Fy37nzyNa_Qcb7QAASix [info] Sent 200 in 952µs
19:14:59.746 request_id=Fy37n5svCSUZSNUAASkB [info] GET /
19:14:59.746 request_id=Fy37n5svCSUZSNUAASkB [info] Sent 200 in 496µs

===== ALIVE Mon Dec  5 19:29:59 UTC 2022

===== Mon Dec  5 19:42:04 UTC 2022
19:42:04.508 [info] CONNECTED TO Phoenix.LiveView.Socket in 36µs
  Transport: :websocket
  Serializer: Phoenix.Socket.V2.JSONSerializer
  Parameters: %{"_csrf_token" => "LD8vAQUbQgAJJCkbXSBYLQx7W1s6QW58oyBDKoqo1WJphU7Uf3bowvCQ", "_mounts" => "11", "_timezone" => "Asia/Singapore", "_track_static" => %{"0" => "https://mysite.com/assets/app-6676b9997926b9d99094855080ac6f52.css?vsn=d", "1" => "https://mysite.com/assets/app-f7aa12f5be2e70bd89dc872c8be91fc8.js?vsn=d"}, "vsn" => "2.0.0"}
19:45:43.028 [info] Plug.SSL is redirecting GET /.env to https://mysite.com with status 301
19:45:43.550 [info] Plug.SSL is redirecting POST / to https://mysite.com with status 307
19:45:44.615 request_id=Fy39TSXxcXD4KfIAASmR [info] GET /.env
19:45:44.615 request_id=Fy39TSXxcXD4KfIAASmR [info] Sent 404 in 293µs
19:45:45.754 request_id=Fy39TWnTBiF_6BMAASnB [info] POST /
19:45:45.754 request_id=Fy39TWnTBiF_6BMAASnB [info] Sent 404 in 312µs

===== Mon Dec  5 19:56:59 UTC 2022
19:56:59.333 [info] Plug.SSL is redirecting POST / to https://mysite.com with status 307
19:56:59.814 [info] Plug.SSL is redirecting GET /.env to https://mysite.com with status 301
20:01:54.022 [info] Plug.SSL is redirecting GET / to https://mysite.com with status 301
20:01:54.752 request_id=Fy3-Lwaf8LFiPlIAASnx [info] GET /
20:01:54.752 request_id=Fy3-Lwaf8LFiPlIAASnx [info] Sent 200 in 506µs
20:01:55.574 [notice] TLS :server: In state :hello at tls_handshake.erl:364 generated SERVER ALERT: Fatal - Protocol Version

20:01:55.900 [notice] TLS :server: In state :hello at tls_handshake.erl:364 generated SERVER ALERT: Fatal - Protocol Version

20:01:56.238 [notice] TLS :server: In state :hello at tls_handshake.erl:364 generated SERVER ALERT: Fatal - Protocol Version

20:01:56.571 [notice] TLS :server: In state :hello at tls_record.erl:564 generated SERVER ALERT: Fatal - Unexpected Message
 - {:unsupported_record_type, 128}
20:01:58.052 [notice] TLS :server: In state :hello at tls_handshake.erl:346 generated SERVER ALERT: Fatal - Insufficient Security
 - :no_suitable_ciphers
20:01:58.881 [notice] TLS :server: In state :hello at tls_handshake.erl:346 generated SERVER ALERT: Fatal - Insufficient Security
 - :no_suitable_ciphers

I am here to ask for help on why a GET & POST would trigger such errors and why would it break the whole site (stopped completely).

Thank you so much in advance!

Best wishes,
Jing Hui PANG

milangupta · December 7, 2022, 5:13am

For what it is worth …

Locking down the server is critical … the number of bots out there scanning and attempting brute-force attacks is astounding !!
There were a couple of key things I did to get my application to stay up :

front end firewall (only 443 exposed)
ufw on ubuntu (again only 443 exposed)
proper ssl cert installation - a good site to verify install - SSL Server Test (Powered by Qualys SSL Labs)
review your router.ex logic to ensure you are locked down (incl. serving static assets etc.)
use eg. paraxial.io to get the ability to review/block traffic via plugs (even though it is a bit too late, still will give you a control).
update to latest and greatest on ciphers / tls etc. My config if useful -


config :yourapp, yourappWeb.Endpoint,
  url: [host: "yoursite", port: 443],
  https: [
   port: 443,
   cipher_suite: :strong,
   otp_app: :meta,
   keyfile: "*** certificate files ***",
   certfile: "*** certificate files ***",
   cacertfile: "*** certificate files ***",
   honor_cipher_order: true,
   ciphers: [
    'TLS_AES_128_GCM_SHA256',
    'TLS_AES_256_GCM_SHA384',
    'TLS_CHACHA20_POLY1305_SHA256',
    'ECDHE-ECDSA-AES128-GCM-SHA256',
    'ECDHE-RSA-AES128-GCM-SHA256',
    'ECDHE-ECDSA-AES256-GCM-SHA384',
    'ECDHE-RSA-AES256-GCM-SHA384',
    'ECDHE-ECDSA-CHACHA20-POLY1305',
    'ECDHE-RSA-CHACHA20-POLY1305',
    'DHE-RSA-AES128-GCM-SHA256',
    'DHE-RSA-AES256-GCM-SHA384'
    ],
   eccs: [
    :x25519,
    :secp256r1,
    :secp384r1
    ],
   secure_renegotiate: true,
   reuse_sessions: true,
   versions: [:"tlsv1.3", :"tlsv1.2"]
  ],
  transport_options: [socket_opts: [:inet6]],
  force_ssl: [hsts: true],
  #force_ssl: [rewrite_on: [:x_forwarded_proto]],
  cache_static_manifest: "priv/static/cache_manifest.json",
  server: true,```

enkr1 · December 8, 2022, 3:36am

Hi @milangupta,

Thank you for the detailed information!

I have deployed the new configuration on my prod.exs and I can still observe something like this in my logs:

===== Thu Dec  8 03:03:55 UTC 2022
03:03:55.967 [notice] TLS :server: In state :hello at ssl_handshake.erl:3435 generated SERVER ALERT: Fatal - No application protocol

03:03:56.405 [notice] TLS :server: In state :hello at ssl_handshake.erl:3435 generated SERVER ALERT: Fatal - No application protocol

03:03:56.830 [notice] TLS :server: In state :hello at tls_handshake.erl:364 generated SERVER ALERT: Fatal - Protocol Version

milangupta · December 8, 2022, 3:55am

This is basically saying something is trying to connect to your ssl port using a protocol that isn’t configured. This isn’t a bad thing (in my humble opinion) as you should, ideally, be restricting/supporting only the latest TLS/Cipher suites.

Here is a good article I found … https://www.baeldung.com/java-ssl-handshake-failures

enkr1 · December 9, 2022, 3:50am

Hi @milangupta, I would like to clarify something. My situation is that after the attack, my whole site just completely stopped and unusable. And it happened again this morning, and the same thing, the last few lines are some kind of brute-forcing of the different common vulnerable paths.

I am currently trying this config after some research:

# prod.exs
config :my_app, MyWeb.Endpoint,
  url: [host: "www.my_app.com", port: 443],
  cache_static_manifest: "...", 
  server: true,
  force_ssl: [hsts: true],
  http: [
    port: 4000,
    transport_options: [socket_opts: [:inet6]]
  ],
  https: [
    port: 4040,
    cipher_suite: :strong,
    keyfile: "/var/app/ssl/cert.key",
    certfile: "/var/app/ssl/chain.crt",
    transport_options: [socket_opts: [:inet6]],
    honor_cipher_order: true,
    ciphers: [
      'TLS_AES_128_GCM_SHA256',
      'TLS_AES_256_GCM_SHA384',
      'TLS_CHACHA20_POLY1305_SHA256',
      'ECDHE-ECDSA-AES128-GCM-SHA256',
      'ECDHE-RSA-AES128-GCM-SHA256',
      'ECDHE-ECDSA-AES256-GCM-SHA384',
      'ECDHE-RSA-AES256-GCM-SHA384',
      'ECDHE-ECDSA-CHACHA20-POLY1305',
      'ECDHE-RSA-CHACHA20-POLY1305',
      'DHE-RSA-AES128-GCM-SHA256',
      'DHE-RSA-AES256-GCM-SHA384'
    ],
    eccs: [
      :x25519,
      :secp256r1,
      :secp384r1
    ],
    # avoid certain types of man-in-the-middle attacks
    secure_renegotiate: true,
    # for improved handshake performance of recurring connections
    reuse_sessions: true,
    versions: [:"tlsv1.3", :"tlsv1.2"]
  ],
  tls: [
    cipher_suite: [
      :TLS_AES_256_GCM_SHA384,
      :TLS_CHACHA20_POLY1305_SHA256
    ]
  ]

BrightEyesDavid · December 9, 2022, 4:22am

If you don’t already have a reverse proxy/webserver between the internet and your app, you could try running Caddy, or Cloudflare, or both, in the hope that, when configured appropriately, they would prevent whatever request is causing your app trouble from getting to the app.

Caddy is very easy to set up and automatically handles SSL certificates.

milangupta · December 9, 2022, 6:59am

Thanks for sharing. I’ll try this too. I wish there were a better “master” reference and one tested way of configuring this.

One question - why are you using both http on port 4000 and https on port 4040 ? I am assuming you have something else redirecting the normal http/https traffic from port 80 & 443 to these two ports … are you sure the traffic is not due to some redirection loop problem and is actually from external IPs ?

Just feels strange as a ddos attack should be quite a significant volume for (am assuming) cowboy to go down … you must be doing something interesting to attract that kind of attention

enkr1 · December 12, 2022, 6:54am

Hi @milangupta,

Yes, I have applied some redirection for this project.

I’d tried to figure out by doing some testing to spam my routes, what I observe is that when you spam by using HTTPoison.get/1, the log will be showing:

06:47:16.964 request_id=Fy_44iYyP...AAeAgR [info] GET /
06:47:16.966 request_id=Fy_44iYyP...AAAAgR [info] Sent 302 in 1ms
06:47:17.009 request_id=Fy_44ijU4...e8AAgh [info] GET /home
06:47:17.012 request_id=Fy_44ijU4...8AAAgh [info] Sent 200 in 3ms

but the log of the attacker’s routes looks like this:

06:46:04.419 [info] Plug.SSL is redirecting GET /.env to https://mysite.com with status 301
06:46:04.928 [info] Plug.SSL is redirecting POST / to https://mysite.com with status 307

How can I achieve the same thing in order to know if my cipher suits are working properly?

The reason is that the server was down again, and the last log was:

04:57:31.493 [info] Plug.SSL is redirecting GET / to https://mysite.com with status 301

enkr1 · December 13, 2022, 2:36am

Hi community,

It happened again … This time there was no error shown in the log, the last few lines are normal GETs of the routes.

benwilson512 · December 13, 2022, 2:48am

Hey @enkr1 do your logs indicate an exit reason? I wonder if you’re running out of memory when getting flooded.

enkr1 · December 13, 2022, 2:52am

Not at all …

These are the last few lines:
Logs from myapp_releases/1670825122/tmp/log/erlang.log.2:

09:18:31.520 [info] CONNECTED TO Phoenix.LiveView.Socket in 58µs
  Transport: :websocket
  Serializer: Phoenix.Socket.V2.JSONSerializer
  Parameters: %{"_csrf_token" => "U0ASUWFaHAdWBSEbOBFrGVMJRgweVSFX81ui05-K2_qbtFRJbq1NI0c6", "_mounts" => "0", "_timezone" => "Asia/Singapore", "_track_static" => %{"0" => "https://mysite.com/assets/app-b29e4b6b8d477e4a74d9bfa4b579c05a.css?vsn=d", "1" => "https://mysite.com/assets/app-f7aa12f5be2e70bd89dc872c8be91fc8.js?vsn=d"}, "vsn" => "2.0.0"}
09:18:34.888 request_id=FzABI8ScRm75MLQAACuR [info] GET /rooms
09:18:34.901 request_id=FzABI8ScRm75MLQAACuR [info] Sent 200 in 12ms
09:18:34.994 [info] CONNECTED TO Phoenix.LiveView.Socket in 61µs
  Transport: :websocket
  Serializer: Phoenix.Socket.V2.JSONSerializer
  Parameters: %{"_csrf_token" => "LCJKbhoJdBMuKz8AeyZgC2MCO3U0CiQgGS-VKfE_Jqoy7qYXRzL7cofA", "_mounts" => "0", "_timezone" => "Asia/Singapore", "_track_static" => %{"0" => "https://mysite.com/assets/app-b29e4b6b8d477e4a74d9bfa4b579c05a.css?vsn=d", "1" => "https://mysite.com/assets/app-f7aa12f5be2e70bd89dc872c8be91fc8.js?vsn=d"}, "vsn" => "2.0.0"}
09:18:45.031 request_id=FzABJiE0uOmH1LsAAAzS [info] GET /home
09:18:45.041 request_id=FzABJiE0uOmH1LsAAAzS [info] Sent 200 in 9ms
09:18:56.106 request_id=FzABKLVFEanLLBUAACvB [info] GET /home
09:18:56.121 request_id=FzABKLVFEanLLBUAACvB [info] Sent 200 in 15ms
09:19:01.697 request_id=FzABKgKFG4sg7EoAACvR [info] GET /rooms
09:19:01.723 request_id=FzABKgKFG4sg7EoAACvR [info] Sent 200 in 26ms
09:19:01.873 [info] CONNECTED TO Phoenix.LiveView.Socket in 58µs
  Transport: :websocket
  Serializer: Phoenix.Socket.V2.JSONSerializer
  Parameters: %{"_csrf_token" => "A0cKUQgbUiEjLyozCy8NB1A-EAEuFSsAh6miYtcmGuzJGx4TaFgCypia", "_mounts" => "0", "_timezone" => "Asia/Singapore", "_track_static" => %{"0" => "https://mysite.com/assets/app-b29e4b6b8d477e4a74d9bfa4b579c05a.css?vsn=d", "1" => "https://mysite.com/assets/app-f7aa12f5be2e70bd89dc872c8be91fc8.js?vsn=d"}, "vsn" => "2.0.0"}
09:19:19.331 [info] Plug.SSL is redirecting GET / to https://mysite.com with status 301
09:19:20.670 request_id=FzABLm1x7DnvlIAAAA0C [info] GET /
09:19:20.675 request_id=FzABLm1x7DnvlIAAAA0C [info] Sent 200 in 4ms
09:22:29.040 request_id=FzABWkkoT6L3aBEAACxR [info] GET /rooms
09:22:29.087 request_id=FzABWkkoT6L3aBEAACxR [info] Sent 200 in 47ms
09:22:29.250 [info] CONNECTED TO Phoenix.LiveView.Socket in 62µs
  Transport: :websocket
  Serializer: Phoenix.Socket.V2.JSONSerializer
  Parameters: %{"_csrf_token" => "HBRfbj9eQjkwDmEPFDNxPkg7PnofKicKwe8Vn1suTT1vXdHmyCI8HOek", "_mounts" => "0", "_timezone" => "Asia/Singapore", "_track_static" => %{"0" => "https://mysite.com/assets/app-b29e4b6b8d477e4a74d9bfa4b579c05a.css?vsn=d", "1" => "https://mysite.com/assets/app-f7aa12f5be2e70bd89dc872c8be91fc8.js?vsn=d"}, "vsn" => "2.0.0"}

Lemme research how to check my server’s memory

benwilson512 · December 13, 2022, 2:53am

How are you running your app? Are you using releases? Are you doing it in docker or under systemd?

milangupta · December 17, 2022, 8:28am

There are still too many variables here and the scope of where the actual issue is, is very broad. I haven’t seen crashes like you are seeing (am running the latest elixir/phoenix stack on ubuntu).

My instincts are pointing me to the redirects … eg. 301 is a permanent redirect … but what is https://mysite.com ? This looks like a “default” that hasn’t been configured. You will need to go deep into SSL & DNS (as well as the Phoenix stack, i.e. configuration of your endpoint and router) to get to the bottom.

My recommendation is to keep simplifying your config until you stop seeing the issue. If you get really lucky, you will be able to recreate the crash on demand by injecting a http request to your endpoint. That would exponentially increase your chances of tracing/catching this gremlin.

enkr1 · December 19, 2022, 8:49am

Hi @milangupta,

Sorry for the misleading mysite.com, it was just a placeholder for my website domain.

Thank you for the advice!