How to do Plug pipelines?

Fl4m3Ph03n1x · November 19, 2018, 5:14pm

Backgroung

I have an app that is a webserver with cowboy and uses Plugs. Since this app was inherited, using Phoenix is out of the question unless we remake the entire thing, which is not happening.

My objective is to instead of having everything inside one huge file, to have several plugs and connect them via pipelines.

Code

Let’s assume I have a main router Plug, that looks like this:

defmodule MyApp.Web.Router do
  use Plug.Router

  plug(:match)

  forward "/check", to: MyApp.Route.Check
  forward "/dispatch", to: MyApp.Plug.Dispatch      
end

So here I have 2 things. A Route for the endpoint /check, which looks like this:

defmodule MyApp.Route.Check do
  use Plug.Router

  plug(:dispatch)

  get "/", do: send_resp(conn, 200, "ok")
end

And a Plug pipeline for /dispatch that looks like this:

defmodule MyApp.Plug.Dispatch do
  use Plug.Builder

  plug(Plug.Parsers, parsers: [:urlencoded])   #parses parameters
  plug(MyApp.Plug.Metrics)                     # exposes /metrics path
  plug(Cipher.ValidatePlug)                    #typical URL validation
  plug(MyApp.Route.Dispatch)                   #forwars to dispatch Route 
end

This pipeline parses the parameters, notifies a metrics service, validates the request and the sends it to the proper Router, which looks like this:

defmodule MyApp.Route.Dispatch do
  use Plug.Router

  plug(:dispatch)

  get "/", do: send_resp(conn, 200, "Info dispatched")
end

Problem

The problem here is that nothing works. Quite literally if I launch the application and try to access even the dummest endpoint ( /check ) the code blows with errors:

17:44:03.330 [error] #PID<0.402.0> running MyApp.Web.Router (connection #PID<0.401.0>, stream id 1) terminated
Server: localhost:4003 (http)
Request: GET /check
** (exit) an exception was raised:
    ** (Plug.Conn.NotSentError) a response was neither set nor sent from the connection
        (plug_cowboy) lib/plug/cowboy/handler.ex:37: Plug.Cowboy.Handler.maybe_send/2
        (plug_cowboy) lib/plug/cowboy/handler.ex:13: Plug.Cowboy.Handler.init/2
        (...)

I have now spent the entirety of my day reading documentation and this was as far as I got. The app is super simple, it is pretty much the hello world for plugs:

But with MyApp.Web.Router instead of the one they use.

What am I doing wrong?

idi527 · November 19, 2018, 7:29pm

Have you tried adding plug :dispatch to your first router plug?

defmodule MyApp.Web.Router do
  use Plug.Router

  plug(:match)
  plug(:dispatch)

  forward "/check", to: MyApp.Route.Check
  forward "/dispatch", to: MyApp.Plug.Dispatch      
end

You might also need adding plug :match to the other two router plugs.

Fl4m3Ph03n1x · November 20, 2018, 8:34am

I don’t actually want that.
You see, I want to match and then parse the requests before dispatching them:

https://hexdocs.pm/plug/Plug.Router.html#module-parameter-parsing

Also, repeating the dispatch inside the plugs wont work.

I did try your suggestions just in case, but the results were the same.

idi527 · November 20, 2018, 9:46am

I couldn’t reproduce your problem …

These tests pass just fine with the approach I outlined above.

I don’t actually want that.

You do. Otherwise no code is executed when a route is matched.

You see, I want to match and then parse the requests before dispatching them:

I see. You still can totally do that while using :dispatch in a “parent” router. The “check” route doesn’t do any parsing. The code in the repo I posted above does just that.

Also, repeating the dispatch inside the plugs wont work.

You probably can’t use :dispatch multiple times in a single router plug (I haven’t checked), but you can (must, actually) do it in different router plugs (as the repo above shows).

Fl4m3Ph03n1x · November 20, 2018, 11:10am

Since you had trouble replicating my issue, I went ahead and published a MWE:

https://github.com/Fl4m3Ph03n1x/plug-pipeline-problem

Let me know if you can fix with your solution, as I was unable to.

idi527 · November 20, 2018, 11:37am

You’d need to add a failing test, otherwise I wouldn’t know what to fix … Also, it looks quite the same as the repo I’ve posted above … Have you tried looking into it?

Fl4m3Ph03n1x · November 20, 2018, 12:19pm

There is no need for a failing test. Just try to launch the projects iex -S mix as stated in the README. Then go to localhost:8080/check and see it blow up.

The README instructions are fairly simple, adding a test would simply add more noise and detract from the real issue. I will worry with design once I get how this works properly.

And yes, I did check your code and as I stated, didn’t quite work.

hlx · November 20, 2018, 12:26pm

router.ex

defmodule Example.Router do
  use Plug.Router

  plug(:match)
  plug(:dispatch)

  forward "/check", to: Example.Route.Check
  forward "/dispatch", to: Example.Plug.Dispatch
end

check.ex

defmodule Example.Route.Check do
  use Plug.Builder

  def init(opts), do: opts

  def call(conn, _opts) do
    send_resp(conn, 200, "ok")
  end
end

route.ex

defmodule Example.Route.Dispatch do
  use Plug.Builder

  def init(opts), do: opts

  def call(conn, _opts) do
    send_resp(conn, 200, "Info dispatched")
  end
end

idi527 · November 20, 2018, 12:33pm

What didn’t work? mix test passes just fine and all your requirements about /check not parsing the request body are satisfied. I’m afraid I can’t suggest anything else.

There is no need for a failing test. Just try to launch the projects iex -S mix as stated in the README. Then go to localhost:8080/check and see it blow up.

The README instructions are fairly simple, adding a test would simply add more noise and detract from the real issue. I will worry with design once I get how this works properly.

I’d rather only have to run mix test to see it “blow up.” But yeah, to each their own.

hlx · November 20, 2018, 12:40pm

or

router.ex

defmodule Example.Router do
  use Plug.Router

  plug(:match)
  plug(:dispatch)

  forward("/check", to: Example.Route.Check)
  forward("/dispatch", to: Example.Plug.Dispatch)

  get("/", do: send_resp(conn, 200, "router"))
end

check.ex

defmodule Example.Route.Check do
  use Plug.Router

  plug(:match)
  plug(:dispatch)

  get("/", do: send_resp(conn, 200, "ok"))
end

Fl4m3Ph03n1x · November 20, 2018, 3:18pm

@idi527 I have now made the sample work using the ideas behind your code. I am still not happy, as in I have questions that need answering… but that’s for another post !

@hlx
Your first solution goes in a completely different direction what I am aiming for. I want a Plug that works as a pipeline that then redirects requests to a specific router. Your approach would be fine if I wanted a Plug is custom Logic that changes the request and then passes the information along, but that is not what I am aiming for in here.

Your second solution resembles that of @idi527 in that you are mixing the pipe setup with all the routes. Ideally I want to have a file where I have all the routes for a given endpoint, say /dispatch and I want to separate the pipe logic from it.

That is why I (and other people) have separate files for pipelines and paths. This is also the reasoning behind the Dispatch Plug - it is meant to represent a pipeline and only a pipeline.

Still, I thank you both for the time you have given to my issue.
I still have questions about Plug, but that’s a matter for another post.

hlx · November 20, 2018, 3:27pm

Could you take a look at https://github.com/hl/plug-pipeline-problem