AlreadySentError only when deployed at Gigalixir

Phoenix Forum Questions / Help
, ,
1

I have successfully managed to deploy my application using Elixir 1.10, Phoenix 1.5.3 and Elm 0.19 at Gigalixir. Locally, everything works fine, and it seems to work on Gigalixir too if you don’t look at the logs. In the log every request trigger a AlreadySentError, and I can’t figure out why.

I don’t get the errors if I run dev locally, or if I try to run prod with Distillery locally. I have tried to remove all plugs from the pipelines router.ex, and put halt() on the responses, but it hasn’t worked. I also don’t have any plug :action, as removing them has been suggested in similar posts. I don’t know what to try next, except maybe try to start a project from scratch and add plugs, configs and packages incrementally, but that would be time consuming. Does anybody have any tips I can look at?

The error:

##### 2020-07-01 21:41:55Z  [error] #PID<0.3204.0> running Moni.Web.Endpoint (connection #PID<0.3203.0>, stream id 1) terminated
web.1  | Server: <<url>> (http)
web.1  | Request: GET /
web.1  | ** (exit) an exception was raised:
web.1  |     ** (Plug.Conn.AlreadySentError) the response was already sent
web.1  |         (plug 1.10.3) lib/plug/conn.ex:769: Plug.Conn.put_resp_header/3
web.1  |         (moni 1.0.0) lib/moni/web/endpoint.ex:1: Moni.Web.Endpoint.plug_builder_call/2
web.1  |         (moni 1.0.0) lib/plug/debugger.ex:132: Moni.Web.Endpoint."call (overridable 3)"/2
web.1  |         (moni 1.0.0) lib/moni/web/endpoint.ex:1: Moni.Web.Endpoint.call/2
web.1  |         (phoenix 1.4.17) lib/phoenix/endpoint/cowboy2_handler.ex:42: Phoenix.Endpoint.Cowboy2Handler.init/4
web.1  |         (cowboy 2.8.0) /tmp/build/deps/cowboy/src/cowboy_handler.erl:37: :cowboy_handler.execute/2
web.1  |         (cowboy 2.8.0) /tmp/build/deps/cowboy/src/cowboy_stream_h.erl:300: :cowboy_stream_h.execute/3
web.1  |         (cowboy 2.8.0) /tmp/build/deps/cowboy/src/cowboy_stream_h.erl:291: :cowboy_stream_h.request_process/3

My dependencies:

{:phoenix, "~> 1.5.3"},
      {:phoenix_pubsub, "~> 2.0"},
      {:phoenix_ecto, "~> 4.1.0"},
      {:postgrex, "~> 0.15.4"},
      {:phoenix_html, "~> 2.13"},
      {:phoenix_live_reload, "~> 1.2.1", only: [:dev, :e2e, :personal]},
      {:gettext, "~> 0.17.4"},
      {:cowboy, "~> 2.0"},
      {:plug_cowboy, "~> 2.3"},
      {:plug, "~> 1.9"},
      {:csv, "~> 2.3.1"},
      {:timex, "~> 3.6.1"},
      {:decimal, "~> 1.6"},
      {:number, "~> 1.0.1"},
      {:arc_ecto, "~> 0.11.3"},
      {:phoenix_slime, "~> 0.12.0"},
      {:dialyxir, "~> 1.0.0", only: [:dev], runtime: false},
      {:scrivener_ecto, "~> 2.4.0"},
      {:ex_machina, "~> 2.4.0"},
      {:excoveralls, "~> 0.10.6", only: :test},
      {:scribe, "~> 0.8"},
      {:bamboo, "~> 1.4"},
      {:xlsxir, github: "jsonkenl/xlsxir"},
      {:timber, "~> 3.1.2"},
      {:timber_ecto, "~> 2.0"},
      {:inquisitor, "~> 0.5.0"},
      {:inquisitor_jsonapi, "~> 0.1.0"},
      {:ex_doc, "~> 0.19.0", only: :dev, runtime: false, override: true},
      {:distillery, "~> 2.1", warn_missing: false},
      {:bcrypt_elixir, "~> 2.2"},
      {:comeonin, "~> 5.3.1"},
      {:db_connection, "~> 2.1"},
      {:ecto, "~> 3.4.4"},
      {:ecto_enum, "~>1.4.0"},
      {:ecto_sql, "~> 3.4.4"},
      {:corsica, "~> 1.1.3"},
      {:httpoison, "~> 1.6"},
      {:sentry, "~> 7.2.4"},
      {:sobelow, "~> 0.10.1", only: :dev},
      {:pot, "~> 0.10.2"},
      {:pow, "~> 1.0.19"},
      {:mock, "~> 0.3.0", only: :test},
      {:hammer, "~> 6.0"},
      {:accent, "~> 1.0"},
      {:jason, "~> 1.0"},
      {:elixlsx, "~> 0.4.2"},
      {:toml, "~> 0.6.1"},
      {:briefly, "~> 0.3"},
      {:hackney, "~> 1.16"}

My config.exs

use Mix.Config

# General application configuration
config :moni,
  ecto_repos: [Moni.Repo]

# Configures the endpoint
config :moni, Moni.Web.Endpoint,
  url: [host: "localhost"],
  render_errors: [view: Moni.Web.ErrorView, accepts: ~w(html json)],
  pubsub: [name: Moni.Web.PubSub, adapter: Phoenix.PubSub.PG2]

config :moni, env: Mix.env()

# config :phoenix, :logger, false
# Our Logger application configuration (affects all logging)
config :logger,
  backends: [
    :console
  ],
  compile_time_purge_matching: [
    [level_lower_than: :info]
  ]

# Our Console Backend-specific configuration
config :logger, :console,
  format: {Moni.LogFormatter, :format},
  metadata: [:request_id, :crash_reason, :user_id],
  level: :debug

# Configures for number (money-amounts)
config :number,
  currency: [
    unit: "",
    precision: 2,
    # To separate thousands
    delimiter: " ",
    separator: ","
  ]

# For using slime in templates
config :phoenix, :template_engines,
  slim: PhoenixSlime.Engine,
  slime: PhoenixSlime.Engine

config :phoenix_slime, :use_slim_extension, true
config :phoenix, :filter_parameters, ["password", "secret"]
config :phoenix, :json_library, Jason

# Configure Sentry
config :sentry,
  dsn: dsn,
  environment_name: :prod,
  enable_source_code_context: true,
  root_source_code_path: File.cwd!(),
  tags: %{
    env: "production"
  },
  included_environments: [:prod, :dev]

config :moni, :pow,
  user: Moni.Auth.Schemas.User,
  repo: Moni.Repo,
  extensions: [PowResetPassword, PowEmailConfirmation, PowPersistentSession],
  controller_callbacks: Moni.Web.PowControllerCallbacks,
  mailer_backend: Moni.PowMailer,
  web_mailer_module: Moni.Web,
  credentials_cache_store:
    {Pow.Store.CredentialsCache, ttl: :timer.minutes(10), namespace: "credentials"}

# cache_store_backend: Pow.Store.Backend.MnesiaCache

config :hammer,
  backend: {Hammer.Backend.ETS, [expiry_ms: 60_000 * 60 * 4, cleanup_interval_ms: 60_000 * 10]}

# Import environment specific configs
import_config("#{Mix.env()}.exs")

Prod.exs

use Mix.Config

# Configure the database
config :moni, Moni.Repo,
  adapter: Ecto.Adapters.Postgres,
  url: "#{DATABASE_URL}",
  template: "template0",
  pool_size: 2,
  log: false,
  ssl: true,
  load_from_system_env: true

# Configure Logger
config :logger,
  backends: [:console, Timber.LoggerBackends.HTTP],
  level: :debug

# Configure Timber
config :timber,
  api_key: System.get_env("TIMBER_PROD_API_KEY") || raise("TIMBER_PROD_API_KEY doesn't exist"),
  source_id:
    System.get_env("TIMBER_PROD_SOURCE_ID") || raise("TIMBER_PROD_SOURCE_ID doesn't exist")

# Distillery configurations
config :moni, Moni.Web.Endpoint,
  load_from_system_env: true,
  http: [:inet6, port: System.get_env("PORT") || 4000],
  force_ssl: [
    hsts: true,
    rewrite_on: [:x_forwarded_proto],
    exclude: [],
    host: System.get_env("HOST")
  ],
  url: [
    # scheme: "https",
    port: 443,
    host: System.get_env("HOST")
  ],
  cache_static_manifest: "priv/static/cache_manifest.json",
  server: true,
  root: ".",
  secret_key_base: System.get_env("SECRET_KEY_BASE") || raise("SECRET_KEY_BASE doesn't exist"),
  version: Application.spec(:moni, :vsn),
  debug_errors: true

# Configure mailgun
config :moni, Moni.Mailer,
  adapter: Bamboo.MailgunAdapter,
  api_key: System.get_env("MAILGUN_API_KEY") || raise("MAILGUN_API_KEY doesn't exist"),
  domain: System.get_env("MAILGUN_DOMAIN") || raise("MAILGUN_DOMAIN doesn't exist"),
  base_uri: "https://api.eu.mailgun.net/v3"

config :moni, Moni.PowMailer,
  adapter: Bamboo.MailgunAdapter,
  api_key: System.get_env("MAILGUN_API_KEY") || raise("MAILGUN_API_KEY doesn't exist"),
  domain: System.get_env("MAILGUN_DOMAIN") || raise("MAILGUN_DOMAIN doesn't exist"),
  base_uri: "https://api.eu.mailgun.net/v3"

# Configure Tink
config :moni,
  tink_client_id: System.get_env("TINK_CLIENT_ID") || raise("TINK_CLIENT_ID doesn't exist"),
  tink_client_secret:
    System.get_env("TINK_CLIENT_SECRET") || raise("TINK_CLIENT_SECRET doesn't exist")

config :moni,
  server_url: System.get_env("SERVER_URL")

config :moni,
  from_email: System.get_env("MAILGUN_FROM_EMAIL") || raise("MAILGUN_FROM_EMAIL doesn't exists")

The pipelines in router.ex

pipeline :browser do
    plug(:accepts, ["html"])
    plug(:fetch_session)
    plug(:fetch_flash)
    plug(:protect_from_forgery)
    plug(:put_secure_browser_headers)
    plug(Plug.CSRFProtection)
  end

  pipeline :api do
    plug(:accepts, ["json"])

    plug(Plug.Parsers,
      parsers: [:urlencoded, :multipart, :json],
      pass: ["*/*"],
      json_decoder: Jason
    )
    plug(:fetch_session)
    plug(:put_secure_browser_headers)
    plug(Moni.Web.Plug.CSPHeader)
    plug(Moni.Web.APIAuthPlug, otp_app: :moni)
  end

  pipeline :convert_case do
    plug(:accepts, ["json"])

    plug(Accent.Plug.Request)
    plug(Accent.Plug.Response, json_codec: Jason)
  end

  pipeline :login_required do
    plug(:accepts, ["json"])

    plug(Pow.Plug.RequireAuthenticated, error_handler: Moni.Web.APIAuthErrorHandler)
  end

  pipeline :admin_required do
    plug(Pow.Plug.RequireAuthenticated, error_handler: Moni.Web.APIAuthErrorHandler)

    plug(Moni.Web.Plug.EnsureAdmin)
  end
2

:wave:

Could you please post the full error with its stacktrace?

3

I’m sorry, posted it now. Thanks for pointing it out.

4

I managed to find the error. I had

 plug(Plug.SSL,
   rewrite_on: [:x_forwarded_proto]
 )

in my endpoint.ex. Not sure why, but after removing it, I haven’t gotten the error.

2 Likes
5

The stack trace is missing some lines (BEAM optimizations?), but given what you observed with Plug.SSL this line seems like the culprit:

But that’s not Plug.SSL’s fault - something before it in the Plug sequence must be sending the response but not halting.