Reality check your library idea

If you’re at the opcode level, I think you can intercept the apply opcode and substitute it with a function call.

Edit: hmm. looking over my notes, maybe not; apply is a really nasty opcode because it’s variadic in its structure. There may still be a way to surgically add guards right in front of the apply opcode…

Here’s how you would do it:

apply arity <registers: [...] Module fun>

becomes:

mov integer: arity, x: (arity + 2)
swap x:0, x: arity 
swap x:1, x: (arity + 1)
swap x: 2, x: (arity + 2)
call_ext 2 {extfunc, "Elixir.Validation", whitelist, 3} <Validation.whitelist/3> # takes "Module, fun, arity", possibly changes identity of module, returning it to x: 0 register
swap x:0, x: arity
swap x:1, x: (arity + 1)
swap x:2, x: (arity + 2)
apply arity

But yes, you would probably want to run tenant code in a separate BEAM instance, but it might not be horrible to have it cotenant on the same kvm instance or on the same metal, connected via plain old erlang distribution.

1 Like

I am not sure if Erlang Distribution is safe enough. It was mostly designed for connecting to trusted nodes.

1 Like

Instead of a raw simple apply there are tons of functions that accept MFA and use it somehow inside (mostly by calling it) and some of them are parts of Elixir/Erlang. Not sure how you would be able to deal with that, lot’s a lot’s of people told me OTP is not a good platform to develop a sandbox.

1 Like

If you decompile the modules, they are all apply opcodes under the hood.

Yet, you will need to dig into the all called functions. And AFIK not all are apply opcode as spawn/3 is BIF. It also will make logging less useful, as you will not be able to use report_cb metadata value for formatting reports.

Data fetching and caching

Occasionally, we need to fetch some configuration data in our applications, for example, client tokens, a list of ids, etc. Most of the time we want to fetch them again in some duration of time to keep it up to date.

I’m thinking about writing a small library to ease such a task. It could be used as:

defmodule MyApp.EnabledUserIds do
  use DataFetcher, name: :enabled_user_ids, ttl: :timer.seconds(6_000)
  
  # you need only implement this callback:

  @impl true
  def fetch(_),
   # returns any term, in this case, a list of user ids
   do: request_enabled_user_ids_from_some_service()

  def enabled_user_ids,
    # `data!` is a macro where you can get the data fetched
    do: data!()  

I used to use cachex for such works, but I don’t feel like the code I set it up because I need to handle the different returning values from {:ok, cached_data} and {:commit, new_data}. Furthermore, if I have high volume of traffic right when the cache expires, all the requests will hit downstream service immediately.

So my idea is to have a process to fetch, hold, then respond to queries with the data. Then it spawns a new process to fetch it again and only kills itself when the new process is successful. Therefore we have no downtime.

I’ve already finished the proof of concept and it works. This is a small idea for a common task so I’m not sure if I am reinventing anything or if anyone is interested in it.

Please give me feedback, thank you, fellows! :smile:

Under high load, there are at least two problems with using a process to hold and respond to queries with data:

  1. That single process can easily become a bottleneck
  2. Data is copied on message passing

You can avoid problem 1 by looking up data in the caller process from an ets table. If your data changes very rarely, (ideally never) you can replace ets with persistent_term which would also avoid problem 2.

4 Likes

Thanks @wojtekmach this makes my TIL! I’ll change the storage from process memory into ets.

I’ve been playing with Macro’s after reading Metaprogramming Elixir by @chrismccord.

As a result :phx_alt_routes was born, albeit in a private repository. First I want to validate the concept. It already works with dead views and live views. Simply by

  • creating one config module (adapter)
  • replacing alias MyWebApp.Router.Helpers with use MyWebApp.AltRoutes.Router.Helpers, MyWebApp.Router.Helpers in my_web_app.ex
  • adding Phx.AltRoutes.MountHelper to the :live_view set.
  • adding Phx.AltRoutes.Plug to the connection plug set

Phx.AltRoutes.Router

Provides a set of macros for generating alternative Phoenix routes with configurable assignments.

When a Gettext module is defined in the configuration, the alt_scope/2 macro uses it to build the alternative routes using the Gettext module. This allows routes to be translated along with other Gettext messages.

To summarize the process.

  • The alt_scope/2 macro splits the path of every nested route.
  • The segments are extracted into routes.po files using mix gettext.extract.
  • The individual segments are translated
  • The translations are used to create additional alternative routes with path {prefix}/{translated_segment1}/{translated_segment2}/ and helper {prefix}_{original_helper}
  • For every alternative route the configured bindings are added to assigns and session.
  • A few helper assignments are also added, which can be used by Phx.AltRoutes.Router.Helpers

Example

Given this configuration:

defmodule MyAppWeb.Phx.AltRoutes do
  use Phx.AltRoutes,
    routes: %{
      "dutch" => %{
        path_prefix: "nl",
        bindings: %{locale: "nl_NL"},
      },
      "english" => %{
        path_prefix: "en",
        bindings: %{ocale: "en_GB"},
      },
    },
    gettext_module: MyAppWeb.Gettext

To generate alternative routes for routes, those routes are nested in a alt_scope.

defmodule MyAppWeb.Router do
  use Phoenix.Router
  alt_scope MyAppWeb.Phx.AltRoutes do
    get "/users/:id/edit", UserController, :edit
    get "/users/show/:id", UserController, :show
  end
end

This will generate routes.po files including the entries for users, edit and show. You can translate the segments. The macro expand the routes during compile time with alternative variants. The example above is equal to:

defmodule MyAppWeb.Router do
  use Phoenix.Router
  scope "/" do
    get "/users/:id/edit", UserController, :edit
    get "/users/show/:id", UserController, :show

  scope "/en" do
    get "/users/:id/edit", UserController, :edit, assigns: %{locale: "en_GB", helper_prefix: "en"}, session: %{"locale": "en_GB", helper_prefix: "en"}, as: :en_user_edit_path
    get "/users/show/:id", UserController, :show, assigns: %{locale: "en_GB", helper_prefix: "en"}, session: %{"locale": "en_GB", helper_prefix: "en"} , as: :en_user_show_path
  end

  scope "/nl" do
    get "/gebruikers/:id/bewerken", UserController, :edit, assigns: %{locale: "nl_NL", helper_prefix: "nl"}, session: %{"locale": "nl_NL", helper_prefix: "nl"}, as: :nl_user_edit_path
    get "/gebruikers/tonen/:id", UserController, :show, :edit, assigns: %{locale: "nl_NL", helper_prefix: "nl"}, session: %{"locale": "nl_NL", helper_prefix: "nl"}, as: :nl_user_show_path
  end
end

The routes can be nested, which makes it easy to create /region/sub-region/ routes.

Phx.AltRoutes.Router.Helpers

Provides a macro that wraps Phoenix.Router.Helpers functions in order to generates the correct alternative route. It uses the helper_prefix assignment added to the session by Phx.AltRoutes.Router.

Thinking of it…the assignments can probably be fetched using only the helper_prefix key…

4 Likes

I was working on refactoring Slick Inbox and I looked at my admin tool and didn’t like what I saw. It’s built with LIveView, but I find that I am repeating a lot of the same things (search, pagination etc) on every admin page that I have. They’re pretty simple pages, they list the entities in the DB (Newsletter, User, Webhook etc), I can edit them etc, it’s just so I don’t need to always SSH into my DB to do admin stuffs (like manually verifying user).

I looked at the repetitive stuffs, extracted them, and right now it’s looking like it could actually be generalisable for a library idea, tentatively named Backoffice, but I thought it would be good to check here first.

I actually found that it somehow end up looking quite a bit like Kaffy, which looks like a pretty great project, but my approach slightly differs in some way.

  1. Kaffy uses controllers. Backoffice uses LiveView

  2. You use Kaffy by using it in your router, the available paths are hidden from you, and you need to find it elsewhere (config.exs or a different Config module)

  3. Kaffy works by having one controller, and then it renders things for you with a single controller as seen here. Backoffice would seamlessly integrate into your router file, you still need to declare your individual Live pages, but you can use Backoffice and that basically bootstraps your Live to an admin interface.

With Kaffy:

# https://github.com/aesmail/kaffy-demo/blob/master/lib/bakery_web/router.ex#L16

# router.exs
defmodule YourApp.Router do
  use Kaffy.Routes
end

I’m not a fan of this idea since it’s not immediately obvious what pages are available (the routes are defined in config.exs which could potentially call out to a different Config module as well, but I prefer things to be colocated)

With Backoffice, it would sit right next to your current set-up.

# router.exs

scope "/admin", YourAppWeb, do
    live("/users", UserLive.Index, :index) # these are your existing pages
    live("/users/:id/edit", UserLive.Index, :edit)

    live("/newsletters", Backoffice.NewsletterLive.Index, :index)
    live("/newsletters/:id/edit", Backoffice.NewsletterLive.Index, :edit)
end

You still need to create your own LiveView module, but you can see it sits nicely in your router, and Backoffice provides you a starting template of some sorts that you can then customize. For example, at a minimum you’d need:

defmodule YourAppWeb.Backoffice.NewsletterLive.Index do
  use Backoffice.Resources,
    repo: YourAppWeb.Repo,
    resource: YourAppWeb.Newsletter
end

This is essentially a wrapper to use Phoenix.LiveView and defines most of the callbacks for you. It also handles pagination for you (right now I’m depending on Scrivener.Ecto), and search (with a raw ilike query that doesn’t deal with input sanitization for now… works for my internal usage :man_shrugging:).

It has a list of things that you can override/customize, like what Kaffy does, so that part is not too surprising.

Kaffy already seems to be pretty great, but I didn’t like the way it configures. Backoffice is basically just me extracting out my current set-up, which may or may not be a good set-up as it might differ from how other people are building LiveViews.

I have already extracted it out in my codebase for a proof of concept and it’s working fine so far, but it still does make some assumption at a few places, so there’s quite a bit of work involved to remove those assumptions too.

What do you think? Is this something that is worth putting more effort into?

7 Likes

I wanted a process to be unique in the cluster, thanks to :global name registration. I also wanted another node to start the child if the node hosting the child died.

First I tried to achieve that with Horde, but I really wanted the child to be in my supervision tree, and no “start the child on that other supervisor” Task under my tree.

Then I found this blog post that said that, indeed, Horde was not meant for that, and that there was a simpler way with :global. Plus, the author of that blog post is the author of the Horde library, which made me confident that I was on the right path.

The author proposed its own way using global, Highlander, but it seems abandoned.

So I made a similar tool called GlobalChild (+ docs), it is simple to use and works pretty well :slight_smile:

1 Like

I propose a “Inspect/Trace Editor Plugin”

So you have this code in your editor

     def foo(bar) do
       bar
       |> do
       |> some
       |> stuff
     end

and want to inspect after some.
You just have to hit a hotkey and tiny loupe-icon appears in the sidebar

     def foo(bar) do
       bar
       |> do
🔍     |> some
       |> stuff
     end

Effectively it appends a hidden |> IO.inspect to the line.
It may also offer a prompt for a label, the default being the line number.
Also coming with a clear-all / clear-all-in-file / disable-all.

Instead of inspect one could also use tracing.

pro: would be nice
con: no idea how to do that :grin:

5 Likes

I regularly do IO.inspect everywhere in my files.
I and my friend have our own ColorIO module which does colored printing.

This would be awesome.

Any leads on how to do it?

1 Like

Yes I’ve seen that. Would be nice if the plugin could use that with an easy config interface to set coloring-rules.

None.

These two modules will help you:

Update: this project is stalled and will be refactored to use new functionalities of LiveView since initial code was written. One day…one day…

Would it be possible to set a default locale for the naked scope?

  use Phx.AltRoutes,
    routes: %{
      "dutch" => %{
        path_prefix: "nl",
        bindings: %{locale: "nl_NL"},
      },
      "english" => %{
+       naked: true,
        path_prefix: "en",
        bindings: %{locale: "en_GB"},
      },
    },
    gettext_module: MyAppWeb.Gettext

Or perhaps by having the path_prefix set to the empty string for the locale owning the naked scope. Either way, with the result being that it will not generate the scope "/:locale" do block, but simply set the scope "/" do block with the default :locale instead.

When you publish the library I would be interested in taking it for a ride.

Thanks for the great suggestion, this is already part of the lib. The current config is still being tweaked, but right now looks like:

defmodule ExampleWeb.AltRoutes do
  defmodule Assigns do
    defstruct [:contact, locale: "en", language: "en"]
  end

  use Phx.AltRoutes,
    root: %{assigns: %Assigns{contact: "root@example.com"}},
    routes: %{
      "europe" => %{
        short: "eu",
        slug: "europe",
        assigns: %Assigns{language: "en", locale: "en", contact: "europe@example.com"},
        routes: %{
          "netherlands" => %{
            slug: "nl",
            assigns: %Assigns{language: "nl", locale: "nl", contact: "verkoop@example.nl"}
          },
          "belgium" => %{
            slug: "be",
            assigns: %Assigns{language: "nl", locale: "be", contact: "handel@example.be"}
          }
        }
      },
      "united-kingdom" => %{
        slug: "uk",
        assigns: %Assigns{language: "en", locale: "en", contact: "sales@example.com"}
      }
    },
    gettext_module: ExampleWeb.Gettext
end

1 Like

Started a pre-release feedback round: Phx AltRoutes: pre release feedback / discussion