Partisan for Elixir - need help for things not covered in the documentation

Questions / Help
1

Hi All,

Anyone using partisan in production? Seems like an interesting framework but I can’t find any documentation on it. @cmeiklejohn any ideas?

Thanks,
Hafeez

2

Partisan does look to be used in production, and have a new maintainer: Handing off Maintenance of Partisan

2 Likes
3

Isn’t hex package giving everything you need? There are the links to the GitHub repository as well as documentation which includes even it’s own cheatsheet.

1 Like
4

There just seem to be many concepts as per the video https://www.youtube.com/watch?v=KrwhOkiifQ8 . They are not covered in the documents e.g.

  • how do you connect the nodes? Is there any auth like you have with disterl.

  • do they need to be on the same subnet etc

  • how do you enable the the different topologies (peer, client server, membership models)

  • how do you do tracing and replay etc.

There’s a lot of stuff not covered by the docs. Just curious if there was anything else incl a partisan elixir book.

hb

1 Like
5

Some more…

  • Does Node.spawn work the same with partisan? Does it work over partisan? I read somewhere it’s exclusively for messages. I can think of building a custom version to make this work - but again. Would be nice if the docs answered this…specifically for elixir.

thanks
hb

6

Hi @macrocreation

Good to hear you are interested in Partisan. As mentioned above I am maintaining the library and working hard on getting v5 out including the docs.

BTW notice the latest docs are in Partisan — partisan v5.0.0-beta.21.

Being an Erlang lib you can obviously call it from Elixir but we are currently working on having an Elixir API (wrapper modules for Erlang’s).

We’ve been using Partisan in production since v3.

Let’s see if I can answer all your questions.

  • how do you connect the nodes?
    Two options:
  • manually by using partisan_peer_service:join/1 passing it the result of calling partisan:node_spec/0 on the target node (or building the spec by hand.
  • using partisan_peer_discovery_agent which allows you to use a seed list of nodes or DNS lookup. Alternatively you can implement your own backed for the agent.

There is a third alternative I still need to validate works with the latest changes and is the Orchestration modules in Partisan which offer an implementation for K8s.

We are using the DNS agent within K8s with great results in production.

Is there any auth like you have with disterl.

Nope. You can use TLS though, and I will be working on enabling Cryptosign (equivalent to TLS but without certificates).

how do you enable the the different topologies (peer, client server, membership models)

You do that by setting the config parameter peer_service_manager to the module implementing the topology.

how do you do tracing and replay etc.

Some of them are enabled via config parameters (partisan_config module) and some other are defined in the Options of the API functions.

I am working on documenting those features as we speak.

7 Likes
7

In v5 we go beyond messaging

I set myself 4 goals for v5:

  1. fix all known bugs in the full mesh topology (partisan_pluggable_peer_service_manager using partisan_full_memebership_strategy)
  2. fix all known bugs in the HyParView topology (partisan_hyparview_peer_service_managerr)
  3. Align Erlang’s and Partisan’s APIs
  4. Implement as much of OTP as possible

I am one test case away from having both 1 and 2 done and I consider the implementation of 3 and 4 to be good to go (OTP implementation passes the Erlang’s Test Suites)

So some of the capabilities you are asking for are already exposed by the partisan module and the partisan_gen_* behaviours.

Mind you OTP only works for full mesh topology at the moment.

7 Likes
8

Thank you @aramallo for taking over the partisan project. It is a great project that shows how far we can take distributed Erlang and still be very performant. I look forward to the evolution of the project and hope I can find some room to contribute more than the tiny bit I have. I also hope I can actually use it in a project some day.

2 Likes
9

Thanks a lot @Ankhers! All the kudos to @cmeiklejohn for creating the library. I am still learning all its capabilities (now turning my focus on HyParView as it is where the real distinctive value of Partisan lies).

BTW I hope I improved on your remote monitor implementation :slight_smile:. Still some more work to do there, specially if we want to try and cater for HyParView.

Looking forward to your contribs!

1 Like
10

I’m sure you did lol. I don’t remember the details, but I think I implemented it fairly quickly based on a discussion somewhere because I wanted to get my hands dirty with partisan.

Is there some place outside of github that you and others involved in the project hangout? I know there used to be a gitter space or whatever it was called. Not sure if it is active now with new maintainers.

11

We only have GitHub but I can open up a channel at Bondy.io’s Slack (recently created)

12

Thanks for this great answer. This is going to sound dumb, but do you have an example elixir config we can use? Even an example project that bootstraps the minimal system would be great.

13

Not yet, I haven’t done much elixir development honestly and I was looking recently into how to wrap the Erlang API properly i.e. using defdelegate and implement the mix file.

I’ll try to complete this in the next couple of days and ping you here.

14

I’ve been following partisan for years and could never figure out if it was an experiment or usable in production. Looks like it is getting there now. :slight_smile:

1 Like
15

But why do we need Partisan?

We had GitHub - bitwalker/swarm: Easy clustering, registration, and distribution of worker processes for Erlang/Elixir
And more recently this beautiful and simple combo:

https://hexdocs.pm/libcluster/readme.html
https://hexdocs.pm/horde/getting_started.html

And this gift demonstrating the work very well :slight_smile:

So… am I missing something? What more Partisan brings under the Sun?

2 Likes
16

Hi @daviaws, there is a section for this right here https://github.com/lasp-lang/partisan#why-do-we-need-partisan

3 Likes
17

I realize I am a couple months behind but in my opinion there is no need to defdelegate. Elixir developers should have no problem using the Erlang APIs directly (and it is often encouraged!). :slight_smile:

4 Likes
18

Thanks @josevalim I’ve been told this and no longer following that approach :blush:

1 Like