RFC: Extending Pipe

I’m putting this here for discussion because I’m not sure how well it’ll be received. But with tap/2 & then/2 introduced in 1.12 I’m hoping there’s at least some room for discussion.

The pipe operator already supports passing into anonymous functions using the long-form syntax:

iex> :foo |> (fn x -> to_string(x) end).()
"foo"

We can also wrap named functions this way if we really want to:

iex> 1 |> (&Enum.at([1, 2, 3], &1)).()
2

Which has the same outcome as:

iex> 1 |> then(&Enum.at([1, 2, 3], &1))
2

But even these simple examples are quite verbose. And for a while I’ve been importing a macro based on an experiment by Qqwy to override the pipe/3 & unpipe/4 functions, and I’m hoping something similar can make it into upstream.

Using Qqwy’s original override, I added it to master branch as an example and to see if there were any unintended side effects. Essentially it pattern matches on calls beginning with & or fn and wraps them in the long-form syntax above. I personally use it to avoid a lot of named assigns, although tap & then solve a lot of these problems as well. The examples above become:

iex> :foo |> fn x -> to_string(x) end
"foo"
iex> 1 |> &Enum.at([1, 2, 3], &1)
2

For a more real world example, let’s do something with XmlBuilder. By extending pipe to accept captures, we can go from:

defp build_contacts(directory) do
  title = element(:title, directory.name)

  contacts =
    for contact <- directory.contacts do
      element(:DirectoryEntry, [
        element(:Name, contact.name)),
        for tel <- contact.contacts do
          element(:Telephone, %{label: tel.type}, tel.number)
        end
      ])
    end

  document([
    element(:Directory, [
      title,
      contacts
    ])
  ])
  |> generate()
end

To a more readable:

defp build_contacts(directory) do
  title = element(:title, directory.name)

  contacts =
    for contact <- directory.contacts do
      for tel <- contact.numbers, do: element(:Telephone, %{label: tel.type}, tel.number)
      |> &[element(:Name, contact.name), &1]
      |> &element(:DirectoryEntry, &1)
    end

  [title, contacts]
  |> &element(:Directory, &1)
  |> &document([&1])
  |> generate()
end

It’s true that the capture version could be achieved much the same way with then/2, which should largely be the argument against there suggested patch, but I honestly believe that & and fn improve readability and are much easier to quickly scan through rather than buried in then calls.

What are other people’s thoughts? Am I the outlier on this one?

1 Like

There are such projects already:

@Qqwy’s capture_pipe: GitHub - Qqwy/elixir-capture_pipe: A pipe-macro for Elixir that allows bare function captures

This provides API almost exactly the same as you are proposing there.

Mine magritte: GitHub - hauleth/magritte: Ceci n'est une pipe

This uses slightly different API and is meant to be used with conjunction with Erlang functions that take subject as N’th argument. However Magritte do not allow for applying ... “operator” to non-functions, so in some parts you still would need to use then/2 or other function for wrapping the values in lists, for example:

defp build_contacts(directory) do
  title = element(:title, directory.name)

  contacts =
    for contact <- directory.contacts do
      for tel <- contact.numbers, do: element(:Telephone, %{label: tel.type}, tel.number)
      |> Enum.concat([element(:Name, contact.name)], ...)
      |> element(:DirectoryEntry, ...)
    end

  [title, contacts]
  |> element(:Directory, ...)
  |> List.wrap()
  |> document()
  |> generate()
end
2 Likes

Yeah I used Qqwy’s patch as the example because that was the basis for what I’ve been using, and frankly more trustworthy than my own.

My question is, is the community & core team open and willing to include capture_pipe in Elixir proper or was the introduction of then/2 the compromise to add functionality while keeping the pipe operator clean?

It was that. I cannot find a topic on forum about it, but if I recall correctly there was one.

The main problem which prevents CapturePipe from (a) being included in Elixir proper and (b) from being used as a library in a more widespread fashion, is because of the relative precedence of & and |> (pipes are nested inside of captures), which for instance the Elixir Formatter does not handle it properly.

CapturePipe works as a macro which rewrites the AST after it has been parsed to alter the relative precedence of & and |>. The Elixir formatter runs before/without macros being expanded, so it will use the ‘as-written’ relative precedence of &, resulting in very odd indentation of the rest of the pipe.

The cleanest way to fix this would of course be in Elixir itself, but this would be a breaking change (and also there is no 100% consensus whether we want this). And as at least currently the Elixir formatter does not give us possibilities to ensure this case is handled properly in library-only code, we’re a bit stuck.

It’s not a ‘full’ dealbreaker since the problem does not happen when the capture is the last segment of the pipe. Thus CapturePipe is still very useful to e.g. pipe into a final &{:ok, &1}, which is rather common.
Still, it makes the library less idiomatic to use. And the alternative of not using the formatter is also a dealbreaker for many people.

As far as I remember there was no discussion on the forum about this whole ordeal, but there was an in-depth discussion on the Elixir-lang GitHub repository.

Tap and then were introduced in this elixir-lang-core mailing list thread and implemented as part as this issue on the Elixir-lang GitHub repository.
Indeed, this was the compromise to improve piping without needing any ‘hackery’.
Quoting @josevalim:

[…]
2. then receives an anonymous function, invokes it, and returns the result of the anonymous function. It will be how we can pipe to anonymous functions in Elixir. It is named andThen in Scala and known as then in many promise libraries across ecosystems.

I think this can improve the piping experience considerably while keeping things functional.

3 Likes

Yes.

In general, one of my main gripes about Elixir syntax is that anonymous functions are second-class citizens. Basically this:

> a_fun = fn x -> List.flatten x end
#Function<6.54118792/1 in :erl_eval.expr/5>
> a_fun([1, [2], 3])
** (CompileError) iex:17: undefined function a_fun/1
> a_fun [1, [2], 3]
** (CompileError) iex:17: undefined function a_fun/1
iex(17)> [1, [2], 3] |> a_fun
** (CompileError) iex:17: undefined function a_fun/1
 (elixir) expanding macro: Kernel.|>/2
 iex:17: (file)

This is especially evident in pipes where you have to do all kinds of weird contortions to make a call work. As shown in the post:

1 |> (&Enum.at([1, 2, 3], &1)).()

It may be too late to change though. Because if Elixir introduces the “proper”, less verbose syntax, it will need to support both the old and the new syntax for god only knows how long.

Unfortunately it is nature of LISP-2, and Elixir is in spirit exactly that.

2 Likes

I’d expect the core teams position it to be the latter, otherwise they would’ve gone farther than just adding then. I don’t think there’s a great case to be made where then is not sufficient. Yes it’s another 4 characters + a set of parentheses, but pipes do pipe in the first parameter of a function and a core helper like then is imo enough for working around that.

2 Likes

Thanks @Qqwy those are the discussions I couldn’t find :slight_smile:

Reading through the history, I’m less confident that this will ever make it into the kernel. So I’ll have to reconcile that most of my files are going to have use CapturePipe at the top of them. That package has spoilt me but I do appreciate most of Qqwy’s experiments.

@LostKobrakai
Yes it’s another 4 characters + a set of parentheses, but pipes do pipe in the first parameter of a function and a core helper like then is imo enough for working around that.

It’s less about the number of characters. I used pipe_here for a project ended up preferring capture_pipe because & made quickly reading a lot easier.


I should also point out that this topic is in no way a criticism or complaint. I just wanted to understand other people’s opinions and the whys. I’m very happy & grateful for everything :slight_smile:

3 Likes