Should one use pipes or with?

Chat / Discussions
, , ,
1

Background

Recently I have rediscovered the with statement to program in a more rail oriented way, and I must say I am loving it thus far.

However in work, my colleagues prefer the usual pipeline oriented approach to railway oriented programming.

My objective here is to discuss the pros and cons of which one and to put my opinions on the table. I am looking forward to reading your opinions and styles on this as well so I can build a better argument to embrace with or simply part ways with it.

ROP

ROP, or railway oriented programming is not a new concept, but it has been popularized recently with the re-introduction of functional languages. At its most simple stage (the one we will be using here) it boils down to executing X functions in a pipeline, and if a piece of the pipeline fails, it simply carries the error until the end of the pipeline without executing the missing pieces of the pipeline.

You can read a little bit more about it here:

https://medium.com/@naveenkumarmuguda/railway-oriented-programming-a-powerful-functional-programming-pattern-ab454e467f31

Code

So, in Elixir there are 2 ways of applying this pattern. With with statements:

def test(x) do
    with
      {:ok, o1}    <- f1(x),
      {:ok, o2}    <- f2(o1)
    do
      f3(o2)
    end
end

defp f1(x) do
  if x > 1 do {:ok, x} else {:error, :too_small} end
end

defp f2(x) do
  if x > 5 do {:ok, x+1} else {:error, :not_valid} end
end

defp f3(x), do: x*2

And with pipelines mixed with multiple clause functions:

def test(x) do
  x  
  |> f1() 
  |> f2() 
  |> f3()
end

defp f1({:ok, x}) do
  if x > 1 do {:ok, x} else {:error, :too_small} end
end

defp f1({:error, _reason} = err), do: err

defp f2({:ok, x}) do
  if x > 5 do {:ok, x+1} else {:error, :not_valid} end
end

defp f2({:error, _reason} = err), do: err

defp f3({:ok, x}), do: x*2

defp f3({:error, _reason} = err), do: err

Opinions !!

When comparing the with version to the pipeline one, I see with has the following advantages:

  1. errors get trickled down automatically and returned without me having to manually specify it
  2. I don’t need to manually add a multiclause function to deal with the errors
  3. my function’s signatures are very clean and don’t need to always include the boilerplate {:ok, value} input signature
  4. I write less code

However, the pipeline has the advantage of making the public function test more readable. It is very clear what the flow of information is when compared to the with version. This example only has 3 functions, but in pipelines with 10 functions or more (we have those) I am not sure with would be a winner because I believe it makes the code of the public function quite harder to read. I wish there was a way to make it clearer.

What do you guys think? Are there any other issue/benefits of pipelines VS with ?

8 Likes
2

My biggest issue with not using with is that each function in the pipeline needs to be able to handle all the error cases of the previous function, which makes reordering/restructuring more difficult besides the obvious redundancy.

4 Likes
3

A valid point which is not a problem in ROP, because once any error occurs, you “exit” the pipeline right way :smiley:

4

I use with for all pipelines where the functions can fail (i.e. return :ok or :error tuples) and we want to break out of the pipeline to deal with it and |> for simpler pipelines which either can never fail or throw when they do. This reduces the use of nested and confusing case or if statements.

Some contrived examples:

def build(params) do
  %User{}
  |> cast(params, [:name, :age])
  |> apply_changes()
end

def convert_to_csv(source_filename) do
    with {:l1, {:ok, :xlsx}} <- {:l1, detect_file_type(source_filename)},
         {:l2, {:ok, csv_filename}} <- {:l2, convert_excel_to_csv(source_filename)} do
  {:ok, csv_filename}
else
  {:l1, {:error, error}} -> {:error, error}  # (such as file not found)
  {:l1, {:ok, _other}} -> {:error, :not_an_excel_file}
  {:l2, error} -> error
end

Note wrapping the tuples in the with statement with their line numbers which I found necessary to disambiguate the else clauses. I’m not sure there’s a better way of doing that without a third-party library.

9 Likes
5

I think René really illustrates well the use-cases of with and |>, and then introduces another idea about creating a token struct to track effects (kind of like Plug) when the pipeline becomes more complicated.

8 Likes
6

As an elixir pipeline cannot exit early this doesn’t change anything for the problem though. Each function of the pipeline needs to handle errors even if it’s just a “pass the error forward”.

7

@tme_317 Ahh, quite valuable ! Your experience makes sense to me and I agree with pretty much everything you said.

@dbern Is this a 3rd party tool? Well, looks interesting non the less, I look forward giving it a check!

Which is why I placed the word ‘exit’ with quotes and in italic. Your assessment is correct, in a real pipeline the error is simply passed forward - this is ROP in action.

My point is that with does this automatically while with a normal pipeline you need to do it by hand.

8

Been there, done that, nightmares persist. In many cases such “pipelining” processing will end up as a big piele of sh
 bad code. TBH even with time I come to the concept that even Plug.Conn isn’t that good idea and having to deal with process directly, like in Raxx, Cowboy, Elli, etc. would be better solution.

9

So there is no solution to big pipelines ? =(

10

I’m not following yet; do you have an example of how you organize data transformations via processes?

11

Break them down into smaller functions? (Sorry, couldn’t resist)

Since ROP was mentioned:

Type and spec - Dialyzer not detecting error - #9 by peerreynders

4 Likes
12

As others already mentioned or implied, pipe is syntax sugar, not control flow mechanism. The question is more of how to handle control flow - and it depends what you want to achieve.

Things to consider / think about

  • Which func should care about control flow? individual func, or top-level “orchestration” func?

  • What’s the required behavior when pattern match failing?

  • Wrap small functions in a larger func (e.g. send_email vs build_recipient, build_email_body, send_email) to make orchestration func focusing on control flow

  • Warp others func as needed (e.g. make it returning ok/error tuple)

1 Like
13

Processes are irrelevant for the discussion I want to have. My main objective is to discuss the pros and cons of pipelines and functions based on the examples I have give.

So, instead of having a pipeline with 10 functions, you would have a pipeline with 9 functions, one of them being another pipeline with X functions. This is one of the problems I don’t like with pipelines - it adds useless indirection while still forcing me to multi-clause for every error case.

It’s fine if my neat example has 3 functions, but when you have pipelines of pipelines of pipelines, things get weary very quickly.

As for libraries, I was actually using this one:

https://hexdocs.pm/rop/readme.html

Which is very nice, however I was met with some resistance from my team because no one is familiar with monads and people usually don’t like to use 3rd party tools when they can do the same with bare Elixir (using pipelines with multiclause functions or with statements).

So here I am, trying to figure out which one is best :smiley:

the pipe operator is a macro, just like with. According to you, both are syntax sugars. I am afraid I am missing the point of your message. Could you elaborate?

14

The pipe (Kernel.|>/2) is a macro, with (Kernel.SpecialForms.with/1) is only documented to be one. As with every “macro” in Kernel.SpecialForms, it is something understood by the compiler itself. It is one of the basic building blocks of the languages and “expands to itself”. It is treated differently than a macro from anywhere else.

1 Like
15

So, it is a special kind of macro, correct? Or is it something else completely different but it is documented in such a way for users to better understand?

16

Basically you can say, that regular macros are syntactic sugar. Even those that are created by third party libraries or yourself. SpecialForms though are actual syntax.

17

with can be a true life saver especially when you communicate with other systems, I talk about it here. Basic idea is first validate my own data, then validate with external system(s) then insert locally. Or authentication where things might go wrong at multiple places.

|> just shows a transformation of input values to output values to me. I usually don’t expect error handling to take place there but just a smooth transformation.

3 Likes
18

Can’t you raise on unexpected errors?

I personally use with when errors are expected, and pipelines with functions that raise when errors should not happen (the boundary is not always obvious):

def my_fun(url) do
  res =
    url
    |> f1()
    |> HTTPoison.get!()
    |> Jason.decode!()
    |> f2()
    |> f3()

  {:ok, res}
rescue
  %HTTPoison.Error{} ->
    {:error, :http_download_error}

  %Jason.DecodeError{} ->
    {:error, :json_parse_error}

  e ->
    {:error, e}
end
2 Likes
19

There is no one size fits all 
 best is highly context sensitive.

Without a library the with/1 pattern demonstrated by @tme_317 is probably the best starting point.

def something(args) do
  with {:step1, {:ok, result1}} <- {:step1, task1(args)},
       {:step2, {:ok, result2}} <- {:step2, task2(result1)} do
  {:ok, result2}
else
  {_, error} -> error
end

Granted it isn’t particularly pretty but it gets the job done and there is some flexibility that goes beyond what the pipe can do.

Now I suspect that this has more to do with your own frustration - “why isn’t this already a solved problem within the language itself”.

Likely because this “problem” doesn’t actually come up all that often.

Erlang introduced {:ok, result}/{:error, reason} more than likely as a poor mans Either (or Result) type.

Given how optimized pattern matching is :ok/:error tuples are a good enough solution.

Putting my C hat on, I can easily imagine an Erlang programmer cringing at the thought of wasting precious function reductions passing an error value around through function calls just to comply with ROP. The attitude would be to drop everything and return the error value promptly - even if it meant a few more lines of code here and there, as long as it benefitted the runtime budget.

The Elixir pipe operator is merely a DevX function application feature that takes the place method chaining in OO languages and is almost as useful as function composition. The pipe operator never meant to take on the :ok/:error tuple issue.

That is really the domain of with/1. But in order to make it useful beyond just plain {:ok, result}/{:error, reason} values it is also more verbose than a pipe. And finally with/1 will quit at the first sign of trouble and is capable of soaking up all sorts of sins committed by the functions that it calls.

The same argument can be made against factoring a 1000 line function into multiple smaller functions. To me those smaller functions add value as long as they are well named and often they tend to make the code more declarative.

I hate trying to figure something like this out:

self.addEventListener('activate', event => {
  console.log('Activating new service worker...');

  const cacheWhitelist = [staticCacheName];

  event.waitUntil(
    caches.keys().then(cacheNames => {
      return Promise.all(
        cacheNames.map(cacheName => {
          if (cacheWhitelist.indexOf(cacheName) === -1) {
            return caches.delete(cacheName);
          }
        })
      );
    })
  );
});

I find this much easier to reason about:

// Activate event
const cacheWhiteList = [staticCacheName]
const isObsoleteCache = name => cacheWhiteList.indexOf(name) === -1
const selectCachesToDelete = cacheNames => cacheNames.filter(isObsoleteCache)
const deleteNamedCache = name => self.caches.delete(name)
const deleteCaches = cacheNames => Promise.all(cacheNames.map(deleteNamedCache))

function activateListener (event) {
  console.log('Activating new service worker...')
  event.waitUntil(
    self.caches.keys().then(
      selectCachesToDelete
    ).then(
      deleteCaches
    )
  )
}

self.addEventListener('install', installListener)
self.addEventListener('fetch', fetchListener)
self.addEventListener('activate', activateListener)


 code for which some members in the JS community would probably lynch me for

So when you have a 10 function pipeline (or with/1) then maybe, just maybe that pipe is spanning multiple, distinct transformations that are just begging to be named for the benefit of future maintainers.

4 Likes
20

It’s not about using pipe or not. The fundamental question is how to control flow and where the logic should be placed.

  • with is good for orchestration func (handling all control flow) calling simple func (returning ok/error tuple, focusing on single job)
  • pipe operator is good for pipeline funcs that are aware of context.
    • this control flow is delegated to individual func.

Note that it’s not all or nothing. For example, you may use pipe for funcs for data transformation or small control flow.