ExApi - a library for creating and managing apis and their implementations

Aha! I see. Thanks for the explanation. Sounds very neat, I will check it out for sure when it is released!

Sounds like a plugin system then eh? :slight_smile:

@OvermindDL1: hmm, at least it’s not intended

I’m using GenServer that in init function reads configuration and automatically initializes apis and their implementations saving them to state. I added extra possibility to register api and implementation at runtime, because you wrote that it could be useful. As you said scanning beam files is not so easy as it originally looks, so what you need to do is just load something (beam file json, or somethimg else) and fetch/generate from them api and implementations definitions. Finally you need only to register them. I have also one extra feature: when some async calls (like registration api or implementation) are executed asynchronously then ExApi dispatch event using ^Register.dispatch/2.

1 Like

I’d been meaning to make a plugin API for elixir, I was thinking of doing a registration and just letting plugins that get loaded into the system register themselves into it.

Sounds cool though. Looking forward. :slight_smile:

2 Likes

@OvermindDL1: As soon as you can compile file with my library as dependency (especially macros) then you can register them. Of course name your api how you would like. API could be named MyApp.Plugin and then you can write MyApp.Loader that simply compiles file, gets file name (as plugin identificator) and register that plugin as a MyApp.Plugin implementation.

I’m making this firstly because I don’t want to implement protocol for empty structs that does not look good - at least for me.

1 Like

/me coughs

@OvermindDL1: I saw it, but my is little different. I need to write one mix task and add some tests then I release it and you will see my idea. It could be great if you can review my code especially my work on macros, because it’s first time I’m working on them so much.

1 Like

@OvermindDL1: I’m writing tests for it now.
You said that working on *.beam files is not good way in some situations (edge-cases).
I’m using: &Code.load_file/2 to load .ex file from test/fixtures/ project directory. This file contains only import call (for calling macro to define api) + of course call that macro - nothing more.
Then I’m using my register method passing only module name. This calls only one auto-generated API function and only work on data that this method returns - no beam or other files are used here.
Is it what you want when you said about dynamically load files or should I test it in another way?

The question becomes: “Will it work if you have no ex files, no exs files, no beam files, but the code is, say, hot-loaded into the running ERTS from, say, across a network while the server had already been up for days?”

That above is a classic use-case for plugins into, say, a Forum like this one, you can load plugins on the fly or disable them again.

1 Like

@OvermindDL1: hmm, as far as I know (understand how it works):
I think that any file that is valid for Elixir compiler (again, see that or similar function) is good. Server needs to have only compiled my library as dependency (to compile macros used in such file). Nothing stops you to use that method after any time.
I don’t yet read about Erlang Run-Time System Application (ERTS), so I can say nothing about it now.

Here are the steps to make it work:

  1. Add my library as dependency
  2. Compile your project with all dependencies
  3. Run project
  4. On specified action (like form submit) call your plug-in loader mode for example:
    4.0 Download source file if needed
    4.1 Call function that compiles it
    4.2 Call my library registration method to initialize API and/or it’s implementations.

My registration method requires only module name + of course registering API before its implementations. It does not checks if module is loaded or not.

Also I currently don’t have implemented a way to restore it’s state (in case restart application), but you can add into database boolean field and simply for each plug-in that is enabled - enable it at start-up.
However I implemented ability to auto-register already compiled (in project) APIs and implementations, (by standard Elixir configuration), so in normal (not dynamically loading code) way you do not need to call register function for each your module.

If you want I can also add ability to restore state (for example save module names of APIs and implementations to two files and at start-up read that files and try to auto-register modules).
The question is: What should I do if I can restore state and have specified modules in configuration?.
I can merge them (so I will have defaults + dynamically added) or load only from that two files skipping configuration (in case some of defaults were unregistered).

We talked also about security for plug-ins and I don’t believe that this is good way to load plug-ins from untrusted source.

Note: I’m using GenServer to keep my state.
I don’t written yet code that sync any data across all nodes. I only created setup of 3 nodes that automatically talked with each other, but I only specified them invm.args` and don’t used any function to send message. Do you have an example to dispatch message for other nodes and add handler for that message?

1 Like

For what message are you sending? You could easily add a listener remotely via an RPC call or so?

@OvermindDL1: My macros simply generate modules with extra functions + info that I read when registering it giving module name to functions in my library. I’m storing all data in memory using GenServer state.

If you say that it would be helpful to sync that state between multiple nodes then give me please an example that I can modify to dispatch my message that I will create - currently I don’t send any except event (my struct) that I’m sending via &Registry.dispatch/4, because all functions to change state are called asynchronously (to handle_cast) and I’m sending this event for catching it. Maybe I can modify this part to send that event to other nodes and then update their states.

Note: I will not send any file with code to compile and load it, because it’s not a target of this library. You can optionally use it with your code loader, so you need to implement sharing code and then register it, so state could be updated in all nodes calling register function in any of them.

I just published first release candidate for ExApi. Have fun!

3 Likes

Whooo I’ve been curious as to your release. :slight_smile:
I’ll be looking it over soon. ^.^

1 Like

@OvermindDL1: It took more time than I originally expected. I took lots of time for tests, coverage, docs and specs, catching errors and edge cases. I had also serious problems with last big storm in Poland in last days. This could be released in previous week.

I also added more features than originally planned such as grouping implementations. That could be useful in case two teams (possibly working on two libraries) are working on same api. Without groups work of one team could be overwritten while registering.

Note: This release candidate does not share GenServer state across nodes. Currently I do not need this, but I could add this if you can link or write small example, so I can catch my notifications (please see notifications code at bottom of main ex_api.ex file).

Feel free to request more documentation in case something is not clear.

1 Like

ping :ping_pong: @aseigo :077:
(not sure if you have set watching this topic)

I have updated README.md file.
Unfortunately repository does not exists. This is issue a GitLab issue. You can follow @gitlabstatus at Twitter for more informations.
Update: GitLab team have problem with cache and now repository is available.

1 Like

Hey @OvermindDL1, did you looked at it?
If so can you please let me know what you think about it (something not enough clear or maybe you see something missing)?

Do you have an idea how to dispatch my events (see &ExApi.notify_event/1 private function at bottom of main lib file) across nodes? I think about next release candidate for it. :slight_smile:

I’m also planning to create a fork of ex_doc (originally don’t plan it, but after I read original source code I can see that it’s easiest way to achieve what I’m looking for). I want to add ability to define custom modules and function groups for documentation, so for example in left sidebar you could see apis link that will work as same as modules or exceptions + each custom module could define how to group functions, so instead of: types, functions and callbacks you could see for example (on api implementation page): supported features, not supported features and not implemented features.

I think about something like:

defmodule Example do
  defmacro my_generator(arg1, ...) do
    # ...
    quote do
      # ...
      defmodule MyCustomModule do
        @module_type MyModuleType # i.e. I don't want a special fork only for ex_api
        # ...
      end
      # ...
    end
    # ...
  end
end
#  and finally
defmodule MyModuleType do
  def get_grouped_functions(module) do
    # get functions and group them
  end
  def group_label(group_name), do: # returns functions group name
  def label, do: "My custom modules" # return sidebar label
end

What do you think about it?

Looked over it a couple times but not made a project with it yet. I’m keeping it up in my active list to remind me to make a test project sometime to give it a try (still fairly busy at work). :slight_smile:

Make it hookable into different things? Phoenix.PubSub would be a nice default if it exists, else just load up a named receiving process on each node to receive the events and dispatch?

I’ve not actually delved into ex_doc as of yet, though @tmbb has a lot lately so he could probably give ideas on it. :slight_smile:

Sounds like a good PR idea though. :slight_smile:

1 Like

Ok, so wait please for next release candidate. :slight_smile:

I know that it was simple question. Just don’t used Phoenix.PubSub by myself yet (only used Phoenix that uses it),. Now I will remember it. Of course I already found docs and added support to project without bigger problem. I’m going to change some things in code, so I will not push next rc today, but in next days. I will ping you for notification. :slight_smile:

I remember his awesome work on Makeup. If he or anyone else have ideas or simple questions then I’m open to all of them and if needed I can also add additional informations to documentation.