Elixir BPM (Mozart) / Compile process definitions to Elixir data structures

Questions / Help
1

I’m developing an open source Elixir BPM library (application). You can find it here:

https://github.com/CharlesIrvineKC/mozart

Right now it’s somewhere between a prototype and an MVP.

Instead of defining process models using BPMN (a graphical notation for visually specifying business processes), my goal is to create a highly readable textual process definition language.

Currently, process models are defined using Elixir maps and structs. I have a number of working examples now. They are in:

https://github.com/CharlesIrvineKC/mozart/blob/main/lib/mozart/test_models.ex

They can be run by running tests:

$ mix test

However, if you take a look at the process models defined in the file “test_models.ex” you will see that they anything but “readable”.

Here is an example process model that is runnable with the process engine:

%ProcessModel{
  name: :two_service_tasks,
  tasks: [
    %Task{
      name: :add_one,
      function: fn data -> Map.put(data, :value, data.value + 1) end,
      next: :add_two
    },
    %Task{
      name: :add_two,
      function: fn data -> Map.put(data, :value, data.value + 2) end,
      next: nil
    },
  ]
}

What I would like to write instead is something like:

defprocess two_service_tasks do
  call_service add_one do value = value + 1 end
  call_service add_two do value = value + 2 end
end

This is obviously much more readable.

This brings me to my purpose in posting. The last time that I implemented a parser I used Bison and Yacc. I wrote an embedded C-like programming language that was incorporated into a CAD tool, but this was many years ago.

Does anyone have any opinion on possibly using NimbleParsec to perform the translation? Or, are there other alternatives that I should look at?

If you have any questions or comments regarding the library, feel free to raise them with an GitHub issue on the project.

Thanks

3 Likes
2

A BPM solution for Elixir would be great, thanks for taking this on.

If you make your DSL be valid Elixir code then you don’t need a parser - you can implement as a set of macros or functions. It would be a small change to what you used as an example:

defprocess two_service_tasks do
  call_service add_one, do: value = value + 1
  call_service add_two, do: value = value + 2
end

You would then need to implement defprocess and call_service as functions. Or if you have more complex compile-time requirements, as macros.

3 Likes
3

I considered that, it’s an appealing idea and I still might go that way. The issue, though, is that I’d like Mozart to be able to be used by programmers no matter the programming language. And even by no programmers, for that matter. Of course, there’s no reason I couldn’t do both.

4

@kip

I do have a doubt whether using Elixir macros is good fit. My understanding is that macros are used to translate macro code to functional code that is then directly runnable on the Erlang virtual machine. My need is different. I need to translate business process definitions to Elixir struct and map data structures that are then loaded into and executed by a business process engine. Do you think that macros can work in that scenario?

5

I have not used it, but I’ve had interest in trying out GitHub - ash-project/spark: Tooling for building DSLs in Elixir, maybe it can help you build BPM.

6

@CharlesIrvine, macros just return AST. That AST may be executable code or it may be a data structure (a data structure is also executable code - it just returns itself when evaluated).

Overall though, the example you gave can be implemented as functions. Which also return data structures (in general).

7

Interesting. I have experimented with Elixir macro some. I’m not sure that they could produce a data structure that looks like this:

%ProcessModel{
  name: :two_service_tasks,
  tasks: [
    %Task{
      name: :add_one,
      function: fn data -> Map.put(data, :value, data.value + 1) end,
      next: :add_two
    },
    %Task{
      name: :add_two,
      function: fn data -> Map.put(data, :value, data.value + 2) end,
      next: nil
    },
  ]
}

Do you think they could?

I do need to provide some more process definition examples as they can become somewhat complex. I’ll post some examples here or add some the my GitHub readme. The example that I showed was overly trivial.

8

It would be relatively straightforward to produce that kind of structure with something like:

process_model :two_service_tasks do
  task :add_one, next: :add_two, function: fn data -> Map.put(data, :value, data.value + 1 end
  task :add_two, function: fn data -> Map.put(data, :value, data.value + 2 end
end

Whether it’s useful to have that particular syntax is harder to say, but it’s certainly possible.

Other things to look to for inspiration:

  • Ecto’s query builder uses functions + macros to build Ecto.Query structs, which are then handed to the query execution machinery (and ultimately transformed in SQL, etc)
  • Nx’s macro system (defn) takes Elixir-flavored code and compiles it to run on GPUs
1 Like
9

Everything in Elixir is, ultimately, expressed as AST and therefore nearly everything can be built in AST (a notable exception would be a process - although of course the code to create a process is most definitely expressed in AST).

Taking your struct as an example, you can see the AST representation by simply doing:

quote do
  %ProcessModel{
    name: :two_service_tasks,
    tasks: [
      %Task{
        name: :add_one,
        function: fn data -> Map.put(data, :value, data.value + 1) end,
        next: :add_two
      },
      %Task{
        name: :add_two,
        function: fn data -> Map.put(data, :value, data.value + 2) end,
        next: nil
      },
    ]
  }
end

Elixir is, in some respects, very Lisp-ish. Both data and code are expressed in AST, Macros manipulate AST and return AST.

10

If you’re interested in going the NimbleParsec route instead of macros you might some inspiration from Tablex:

And Truly is another one to take a look at, although it doesn’t use NimbleParsec:

2 Likes
11

@axelson I am leaning towards using NimbleParsec for a couple of reasons, assuming it is applicable. I still don’t have a good grasp of the purpose of NimbleParsec.

Actually, I’m glad you mentioned Tablex and decision tables. Most BPM platforms offer a business rule task, i.e. a task that is completed by evaluating a decision table. For example Camunda Business Rule Task.

I’ll add rule tasks to my todo list and take a look at Tablex.

Thanks

1 Like
12

@axelson @qhwa

FYI - I just finished my first pass at integrating Tablex into Mozart (Elixir BPM platform). The integration takes the form of what’s called a “Decision Task” in BPM terms. Decision tasks are a must have for a BPM tool. So, thanks!

Also, Tablex is a super example for the use of NimbleParsec. I’ll be studying it carefully for the parser that I need to write.

Test Case: See here

  test "process with one decision task" do
    PMS.clear_then_load_process_models(TestModels.one_decision_task())
    data = %{}

    {:ok, ppid, uid} = PE.start_supervised_pe(:process_with_single_decision_task, data)
    PE.execute(ppid)
    Process.sleep(1000)

    completed_process = PS.get_completed_process(uid)
    assert completed_process.data == %{color: "green", season: "spring"}
    assert completed_process.complete == true
  end

Process Model: See here

  def one_decision_task do
    [
    %ProcessModel{
      name: :process_with_single_decision_task,
      tasks: [
        %Decision{
          name: :decision_task,
          tablex: Tablex.new("""
          F  value  || color
          1  >90    || red
          2  80..90 || orange
          3  20..79 || green
          4  <20    || blue
          """),
          next: :identity_season
        },
        %Service{
          name: :identity_season,
          function: fn data ->
            cond do
              data.color == "green" -> Map.merge(data, %{season: "spring"})
              data.color == "orange" -> Map.merge(data, %{season: "fall"})
            end
          end
        },
      ],
      initial_task: :decision_task
    }
  ]
  end
1 Like
13

That looks like a great use case and I’m glad that the was helpful to study!

14

@al2o3cr The motivation for the higher level syntax (HLS) would be to make Mozart non-programming-language specific. This would necessitate that all Elixir code be removed from the HLS, with any necessary Elixir code being generated during parsing. Also, the “:next” indicator would be implied by ordering and any necessary element in the HLS.

15

not meant as a challenge or anything, but you are pretty much inventing your own language, for sake of making it independent from other, existing languages.

1 Like
16

I assume you are referring to the as yet to be defined or implemented high level language and not the Elixir specific syntax. But I suppose your doubt could equally apply to the Elixir syntax as well.

I have been doing Business Process Management using the visual BPMN2 off and on for 7 or 8 years. I’ve never really liked BPMN2, but I’ve used it because that is what all of the BPM platforms provided. Possibly even more significant is that a lot of programmers don’t like it either.

I developed a fairly significant BPM application using Camunda several years back and the users were very happy with it. After I left the company I kept in touch with some of the developers. The guy that took over my responsibility hated BPMN2 and he tried to transfer responsibility for it to other developers. But they didn’t like it either.

When he also l left the company, the developers got together and got approval to port the application to AWS Step Functions. AWS Step Functions, at that time at least, didn’t have a visual development tool. It was purely specified using JSON.

My current Elixir syntax is a lot like this AWS syntax. Both are textual, but both take a trained eye to understand. And that, finally, is one of the reasons for Mozart - to offer a textual syntax that is easy to understand. But there are a couple of other reasons as well. None of them are merely for the sake of making it independent from existing things already out there.

One reason is that I am between jobs. Anyone know of anything?

17

no, if i understood correctly you put strong emphasis on being language independent in your endeavor. i am super gently suggesting it’s not possible, you are just inventing your own DSL (or tweaking and adopting existing) making some assumptions on how intuitive it is vs. other languages, etc.

i have love-hate relationship with DSLs. i love toying with such ideas, but in practice, they are trying to be programming languages, but almost never achieve usability, ergonomics, power parity with “real” programming languages.

this is why i prefer stuff like Pulumi over Terraform.

don’t get me wrong, i am not criticizing your effort in general, it’s wonderful you are looking into it and sharing with us. i am more picking on your argumentation or idea of being language independent :slight_smile:

do you have any profile/up to date CV i could share with a recruiter if opportunity appears?

18

The goal of language independence is based on the goal of attracting programmers from other programming languages. I view BPM as a foundational back-end technology usable by all IT departments.

Say that someone came along and developed a relational database implementation in Elixir that offered an order of magnitude better performance than postgresql. You would want it to be programming language independent or it would have trouble catching on.

I think you may be somewhat familiar with Camunda. Camunda used to be heavily oriented towards Java developers. The latest version of Camunda strives to be programming language independent. Same reason.

BTW, the process modeling language on my todo list resembles a DSL but it is very different. As you know, DSL’s implemented with Elixir macros expand to ordinary Elixir (really Erlang) code that that then gets executed on the Erlang virtual machine.

My process models, the current Elixir ones and the to-be language independent ones, are used to spawn GenServer instances, one per business process instance. Each GenServer then drives the execution of the business process to completion.

My current plan is to use NimbleParsec to parse the language independent process models into the current Elixir data structures. I think that Elixir macros are not the right fit for what is needed.

19

I have no experience with NimbleParsec, but I’ve worked with parsing expression grammers (PEG): They’re really easy and intuitive to use (you can try it online at Online Version » Peggy – Parser Generator for JavaScript ). There’s a PEG parser generator written in Elixir (see XPeg - Powerful Elixir PEG parser generator library ) but I haven’t tried it yet. If you’re looking for a textual BPM representation that’s readable by humans I can recommend PNML (see PNML — Ontologysim 29.11.2020 documentation ). PNML can then be parsed by xpeg. For interfacing with other BPM tools you can translate PNML into any standard like BPMN, EPC, etc.

20

Thanks for the pointers. I’m catching onto NimbleParsec, enough so, to see that the implementation will take a significant amount of time, with a large portion of that being traversing the learning curve. XPeg does look interesting. I’ll have to spend a little time looking into it.

I did look at PNML. I want something much more readable.

Regarding BPMN and EPC, they are the kind of things I want to avoid, i.e. any visually created and visually represented modeling language.