Reality check your library idea

What might be interesting in that direction could be an implementation of Petri Nets in Elixir.

I saw an interesting talk at the Code BEAM STO a few years ago about them, and was under the impression that they can solve certain kinds of problems which are hard to model in a state machine.

3 Likes

Haha no prob that’s what this thread is for! I also have one that’s been humming along in prod for a year now: https://hexdocs.pm/state_server/StateServer.html

1 Like

It is in erlang, but its already there: https://hex.pm/packages/gen_pnet

2 Likes

as promised, untangled “ansible for elixir”, very much version 0.1.0… I haven’t tested it against deploying my lab or production machines in the “untangled state” yet, so, expect errors unless my tests actually do correctly recapitulate everything correctly… https://hexdocs.pm/realbook/Realbook.html

1 Like

I’ve spent the day working on an idea of mine which had been lying in my backlog for too long:

Easy schema validation for incoming requests on a per-route basis.

In past projects I’ve occasionally used JSON schemas to ensure that incoming data actually looked like I was expecting it, which is especially relevant when doing API development.

This boiled down to either assigning the path to the schema in conn.private and evaluating it in a plug:

pipeline :api do
  # ...
  plug JSONSchemaValidator, private_key: :json_schema
end

scope "/", MyAppWeb do
  post "thing", MyController, :create, private: %{json_schema: "path/to/schema.json"}
end

or defining a plug per-route in the controller:

plug JSONSchemaValidator, [schema: "path/to/schema.json"] when action == :create

While this works it feels a bit clunky, so I dabbled a bit and ended up at this (I’m using Validation as a placeholder here, don’t have a name yet):

defmodule MyAppWeb.MyController do
  use Validation,
    adapter: Validation.JSONSchema,
    resolver: Validation.JSONSchema

  use MyAppWeb, :controller

  @validate schema: "path/to/schema.json"
  def create(conn, params) do
    # ...
  end
end

This actually complies down to the plug + action version from above but I feel like it’s so much clearer. The adapter and resolver options can of course also be defined in the config. The idea here is to stay agnostic from the kind of schema and provide these as separate libraries. One for JSON schema, one for Protobuf, etc.

What do you think? And do you have any name suggestions (I’m considering Gandalf - you shall not pass)?

1 Like

Even with your explanations I’d still be very confused on what do @validate, adapter and resolver mean. Sounds too enterprise-y somehow and could literally mean anything.

I’d probably replace this:

@validate schema: "path/to/schema.json"

With this:

@request_schema "path/to/schema.json"

This immediately makes it clear that we’re validating request parameters and not anything else.


I’m also not clear on the difference between an adapter and a resolver. Can’t they be one module or even just separate functions, described as function captures? Like this:

use RequestValidator, schema: :json
use RequestValidator, schema: :protobuf
use RequestValidator, module: MyApp.JsonSchemaValidator
use RequestValidator, adapter: &MyApp.Contexts.Validator.json/1, resolver: &MyApp.Contexts.Validator.something_else/1

…etc. And then those atoms could resolve to actual modules inside your library. I think it’s very non-ergonomic to make people write your library’s module prefix several times like in your example – Validation once and then Validation.JSONSchema twice – so I’d go the extra mile to introduce a terse DSL.

1 Like

I have a terrible jsonschema validator library here: https://github.com/ityonemo/exonerate, it’s actually been running in prod for about a year with no hiccups, if you’re interested in using that code feel free to without restriction - or if you are interested in collaborating at some point I’d like to refactor it to “not be terrible”, it’s just that i don’t actively use jsonschema very much anymore since I keeping more and more things in the BEAM these days.

1 Like

I have been looking at CRDT recently, that would be nice to have CRDT datatype because we could wrote some amazing … rich text editor for Phoenix.

I know Phoenix uses CRDT with trackers, but not datatype (array, string, etc).

We can see Action Text in Rails, and many other frameworks have their custom editor.

Often I use draftjs, or ckeditor, or tinymce, but I am sure we can do better :slight_smile:

JS can do this

and I hope Elixir will too.

1 Like

I think this text editing crdt is promising: https://youtu.be/x7drE24geUw Martin had given talks at codesync in the past iirc.

1 Like

I have been following some of his videos :slight_smile:

Thanks for the feedback! I really like the aliasing, definitely something I’ll consider!

The example I chose probably wasn’t ideal but your feedback definitely helped with pushing the API in a better direction. The idea behind adapter and resolver is a distinct separation between “validation” and “schema loading” (see below for details on what that means).

In my current iteration you (could) now use them like this (name is as before a placeholder):

use ValidationLibrary.Phoenix.Controller,
  adapter: ValidationLibrary.Adapter.JSONSchema,
  resolver: {ValidationLibrary.Resolver.File, directory: "priv/schemas"}

use MyAppWeb, :controller

@validation_library schema: "show.json"
def show(conn, params) do
  # ...
end

Of course the whole adapter and resolver config can also be moved into config/:

config :validation_library,
  adapter: ValidationLibrary.Adapter.JSONSchema,
  resolver: {ValidationLibrary.Resolver.File, directory: "priv/schemas"}

Which would allow you to reduce the use to this:

use ValidationLibrary.Phoenix.Controller
use MyWebApp, :controller

# ...

There are a number of reasons I chose this API:

adapter and resolver being distinct

This allows the user to use the actual validation logic adapter with whatever kind of schema storage, be it local storage, remote storage (such as an S3 bucket) or even a full-blown schema registry.

A reasonable default for resolver would be local storage (ValidationLibrary.Resolver.File) which is probably perfectly fine for most use-cases.

The adapter can in turn focus on actually performing the validation, without having to care about where the schema came from. It gets the resolved schema, the data and then validates.

(In the earlier example I used JSONSchema as resolver because the local storage loading “just” loads the file without json parsing. I’ve now added an optional prepare/2 callback to the Adapter which transforms the schema from a plain string to an actual JSONSchema.)

@validation_library schema: "..."

The reason behind using a module attribute with the same name as the library is that you can overwrite any configuration here.

If you for example have this one endpoint which needs to accept XML to interact with some kind of legacy system, you could then do something like this:

@validation_library schema: "priv/schemas/legacy/create.xml", adapter: ValidationLibrary.Adapter.XML

But you’re certainly right that this isn’t really self-explanatory. It’s certainly feasibly to allow “aliases” of the “full” module attribute, such as the one you proposed (I would opt for namespacing though, to reduce the risk of name clashes, the final library name would be a lot shorter anyway).

@validation_library_request_schema "show.json"

Does that clarify things a bit, @dimitarvp? Any additional thoughts?

1 Like

Here is the next crazy idea from the place that is my brain: (I’m committed to finishing Selectrix, don’t worry).

LiveView over WebRTC. The idea being that you could have a server behind a firewall, connecting up to a handshake server. Then you could from your device or terminal connect to the handshake server, and it would perform STUN/TURN and connect you P2P (ideally) to your server on the other side of the firewall.

I’m thinking the best use case would be a privacy-oriented storage device. You could build out the interface using phoenix liveview, and owners could connect into their own servers easily.

4 Likes

Ok… Now for a super half-baked idea that I don’t want to work on (but if someone wants to do this and wants help I can give some help):

Serverless Elixir/Erlang

Compile a module, or a bundle of modules. Ship it to a service. Server disassembles the BEAM bytecode, identifies operations that need to be redacted (String.to_atom), or sandboxed (for example, calls to File modules, send, calls to Node module, Process module, etc), recompiles the module, and then launches it into the VM, co-tenanted with a bunch of other modules. Might be possible to do things like track reductions spent in modules, etc, if you wanted to make a hosting service out of this.

1 Like

Sorry for being showstopper, but without spawning the additional BEAM VM in the restricted environment it is not really possible to make it safe. Main problem? apply(m, f, a) where you cannot guarantee that it will not call some unsafe code.

If you’re at the opcode level, I think you can intercept the apply opcode and substitute it with a function call.

Edit: hmm. looking over my notes, maybe not; apply is a really nasty opcode because it’s variadic in its structure. There may still be a way to surgically add guards right in front of the apply opcode…

Here’s how you would do it:

apply arity <registers: [...] Module fun>

becomes:

mov integer: arity, x: (arity + 2)
swap x:0, x: arity 
swap x:1, x: (arity + 1)
swap x: 2, x: (arity + 2)
call_ext 2 {extfunc, "Elixir.Validation", whitelist, 3} <Validation.whitelist/3> # takes "Module, fun, arity", possibly changes identity of module, returning it to x: 0 register
swap x:0, x: arity
swap x:1, x: (arity + 1)
swap x:2, x: (arity + 2)
apply arity

But yes, you would probably want to run tenant code in a separate BEAM instance, but it might not be horrible to have it cotenant on the same kvm instance or on the same metal, connected via plain old erlang distribution.

I am not sure if Erlang Distribution is safe enough. It was mostly designed for connecting to trusted nodes.

Instead of a raw simple apply there are tons of functions that accept MFA and use it somehow inside (mostly by calling it) and some of them are parts of Elixir/Erlang. Not sure how you would be able to deal with that, lot’s a lot’s of people told me OTP is not a good platform to develop a sandbox.

1 Like

If you decompile the modules, they are all apply opcodes under the hood.

Yet, you will need to dig into the all called functions. And AFIK not all are apply opcode as spawn/3 is BIF. It also will make logging less useful, as you will not be able to use report_cb metadata value for formatting reports.