AST based script to replace routes by verified routes in elixir and heex files

Phoenix Forum Chat / Discussions
, , , , ,
1

Hi everyone,
Long time reader, first time poster.

I wrote a nice little script that I think could be useful for people, so I’m sharing it here.

TLDR: this is a script that will convert your Phoenix routes to verified routes. Instructions are in the moduledoc and towards the bottom for converting .heex files.

the backstory

We switched to Phoenix 1.7 at work a few months ago, and enabled verified routes.
We incrementally added them, converted some of the old Routes to verified routes here and there, but our app started with about 3000 routes, so updating them all was not trivial.

When my colleague opened a PR to start the task, I got the itch, and I started looking around for automated tools. I found this script via this blog post: Phoenix Blog Post: Migrating to Verified Routes. The script, meant to run as a mix task, works with regular expressions, and therefore has some shortcomings, such as not working on multi-line routes or not dealing with arguments containing parenthesis.

my contribution

What we really needed was a tool that would act on our code AST, and armed with my experience working on the Exercism Elixir Analyzer, I modified the script into this. To run it:

  • place the file into my_app/lib/mix/tasks/convert_to_verified_routes.ex
  • replace MyAppWeb with the name of your app (the web version)
  • run mix convert_to_verified_routes

That’s it! But wait, there’s more…

You probably use routes in your .heex templates. Unfortunately, the script doesn’t work on those, since they are not pure Elixir code, they cannot be parsed with the standard library. You know what can parse those though? Phoenix.LiveView.HTMLFormatter can. It’s hacker time…

      source
      |> tokenize()
      |> Enum.map(&Mix.Tasks.ConvertToVerifiedRoutes.format_eex/1) # <--- Add this line
      |> to_tree([], [], {source, newlines})
      |> case do
  • run mix compile && mix deps.compile phoenix_live_view && mix format

And you’re done! You can clean up your hack with mix deps.clean phoenix_live_view && mix deps.get.

quirks and limitations

99% of our routes were converted perfectly, but of course, there are caveats, no tool is perfect, especially for something as complex as a codebase. Here is a quick run down of the limitation of the tools and the issues I ran into, in no particular order.

Your routes need to be aliased with exactly alias MyAppWeb.Router.Helpers, as: Routes

Otherwise it doesn’t work. The script only traverses the files that contain the string Routes..

The script may not catch all of your routes

If you use a Routes function which receives the action as a variable instead of an atom, the script won’t be able to replace it, because the path isn’t knowable at compile time.

The script renames alias MyAppWeb.Router.Helpers, as: Routes into use Elixir.MyAppWeb.VerifiedRoutes

The Elixir. part is quirky, but a search and replace fixes it so easily that I didn’t bother fixing it in code. A real issue is that if it can’t replace all of the routes in the file, you will need to reintroduce the Routes alias by hand. You might also need to add use MyAppWeb.VerifiedRoutes in other modules that you use, since the script only traverses the files that contain the string Routes..

You might get some unused variables warnings

We had a bunch of functions that took conn or MyAppWeb.Endpoint as a parameter in order to build a route. Since they are not needed in verified routes, that had to be cleaned up.

Some routes might exist only in the test environment

We have some of these, for integration tests. The fix is easy though, run MIX_ENV=test mix convert_to_verified_routes.

It will not work for arbitrary endpoints

The router and endpoint are fixed in verified routes, if your Routes function uses other ones, you might need to use path/2, path/3, url/2, or url/3.

.eex files are not covered

We converted the few of those we have left by hand. There might a formatter plugin out there that you can hack.

Some changes in .heex files might not compile ast first

Formatting blocks in .heex files is tricky, take for example

    <%= if url == Routes.some_path(@conn, :index) do %>
    # or
    <%= form_for @conn, Routes.some_path(@conn, :index), args, fn f -> %>

When running the tokenizer, Phoenix.LiveView.HTMLFormatter will isolate those lines and will not attempt to format them, since they are not valid Elixir code without an end. I wasn’t willing to let those go, so I made the script add " nil end" to the tokens, format them, and remove " nil end" afterwards. It worked for the if block, but not for the form_for block, that ended up formatted like

    <%= form_for(@conn, Routes.some_path(@conn, :index), args, fn f -> end nil) %>

which will break your compilation (because of the unmatched <% end %> later on). The reason for the parenthesis appearing in this case is that if is one of the functions/macros covered by the default formatter’s :locals_without_parens option, but form_for/3 and form_for/4 are not, so we needed to add them manually in the script’s format_string/1 function. Our app didn’t need to add other functions, yours might.

Query parameters behave slightly differently for routes and verified routes

Best to show you an example:

iex(1)> params =  %{:a => "atom", "a" => "string"}
iex(2)> Routes.log_in_path(MyAppWeb.Endpoint, :new, "en", params)
"/en/log_in/new?a=atom"
iex(3)>  ~p"/en/log_in/new?#{params}" # ~p doesn't work in iex, but if it did it would show this
"/en/log_in/new?a=atom&a=string"

This is definitely a pathological example (is it a bug? I’m not sure), but it did happen to us. Thank god for tests.

Another example:

iex(1)> params =  %{"locale" => "en", "a" => "b"}
iex(2)> Routes.log_in_path(MyAppWeb.Endpoint, :new, "en", params)
"/en/log_in/new?a=b"
iex(3)> ~p"/en/log_in/new?#{params}"  # ~p doesn't work in iex, but if it did it would show this
"/en/log_in/new?locale=en&a=b"

The reason for this interesting behavior is that the route is defined in router.ex as get("/:locale/log_in", LogInController, :new). The Routes function recognizes the locale parameter as a path parameter and therefore doesn’t include it in the query parameters, but ~p does. Again, not sure if this is a bug, and it is a weird edge case, but it did happen. Thank god for tests.

It will break if you have two routes with the same name and a different number of path parameters

In our app we have (for valid reasons, don’t judge us) a few routes like that. Something like:

get("/log_in", LogInController, :new)
get("/:locale/log_in", LogInController, :new)

In this case, the script would pick up on the first one and replace Routes.log_in_path(MyAppWeb.Endpoint, :new, "en") by ~p/log_in?en=. Unfortunate. I don’t think I would recommend structuring your routes that way if you can help it.

Fallback paths will break your verified routes validation

Part of the strength of verified routes is that they are verified at compile time.

This is not strictly related to the script, but working on it made us realize something. Our paths are localized, but we support paths without a locale, thanks to a :localized plug which adds a default locale. However, to make sure that we run that plug on all paths, we need a fallback route which will accept any path (and render a 404 if the request actually reaches it):

scope "/:locale", MyAppWeb do
    pipe_through([:localized])

    ...
    get("/*path", LocaleController, :not_found)
end

This fallback path ends up breaking the verified routes validation, as anything goes. Our response to that was to add a CI job that removes the fallback line and then runs mix compile --warnings-as-errors, which is better than nothing.

I might have missed something?

I stopped working on the script when it was able to convert all that was convertible in our app. There are most likely some edge cases out there that I didn’t address.

That’s all I got

I had great fun working on this, and I hope it can be useful for some people.
Any and all feedback is appreciated, have a good one.

9 Likes