A simple macro to allow bare captures in pipes

EDIT: CapturePipe has been released as a library!
Please give it a go and let us know if you find any mistakes.


Hi everyone!
Yesterday evening (it was already late… :sleeping:) I was reflecting a bit about common patterns in my Elixir code.
If you’re anything like me, you’ll often end up with functions that look like this:

def some_function(argument, other_argument) do
  changed_argument = 
    argument
    |> other_function(other_argument)
    |> more_work(42)

  {:ok, changed_argument}
end

This type of code is highly prevalent when working with ok/error -tuples, but also when working with GenServers or anything based on them (like e.g. Phoenix Channels or Phoenix LiveView handlers) because the callbacks of these behaviours require you to return tuples all the time as well. An example is:

def handle_event("reset-button", _params, socket) do
  new_socket = 
    socket
    |> assign(:form_values, default_form_values())
    |> assign(:error_messages, [])
    |> assign(:reset_button_active, false)

  {:noreply, new_socket}
end

This inserting into tuples (and sometimes other datastructures) is in tension with working with pipelines. It requires ‘breaking the pipeline’. Flow no longer goes left-to-right, top-to-bottom but rather back to the top (where some new variable is bound) and then continues below the pipeline.

Yesterday evening I realized that there was a tiny sprinkle of syntactical sugar that would resolve this tension:

defmodule Capturepipe do
  @doc """
  A pipe-operator that extends the normal pipe
  in one tiny way:

  It allows the syntax of having a bare `&1` capture
  to exist inside a datastructure as one of the pipe results.
  This is useful to insert the pipe's results into a datastructure
  such as a tuple.

  What this pipe-macro does, is if it encounters a bare `&1` capture,
  it wraps the whole operand in `(&(...)).()` which is the
  anonymous-function-call syntax that the Kernel pipe accepts,
  that (argubably) is much less easy on the eyes.

  So `10 |> {:ok, &1}` is turned into `10 |> (&({:ok, &1})).()`


  To use this operator in one of your modules, you need to add the following to it:
  
      import Capturepipe
      import Kernel, except: [|>: 2]
  

  ## Examples

  Still works as normal:

  iex> [1,2,3] |> Enum.map(fn x -> x + 1 end)
  [2,3,4]

  Insert the result of an operation into a tuple

  iex> 42 |> {:ok, &1}
  {:ok, 42}

  It also works multiple times in a row

  iex> 20 |> {:ok, &1} |> [&1, 2, 3]
  [{:ok, 20}, 2, 3]
  """
  defmacro prev |> next do
    # Make sure the pipes are expanded left-to-right (top-to-bottom)
    # to allow consecutive applications of the capturepipe to work
    prev = Macro.expand(prev, __CALLER__)

    # Perform change only if we encounter a `&1` that is not wrapped in a `&(...)`
    {_, visible?} = Macro.postwalk(next, false, &capture_visible?/2)
    if visible? do
      quote do
        Kernel.|>(unquote(prev), (&(unquote(next))).())
    end
    else
      quote do
        Kernel.|>(unquote(prev), unquote(next))
      end
    end
  end

  @doc false
  def capture_visible?(ast = {:&, _, [1]}, _bool), do: {ast, true}
  def capture_visible?(ast = {:&, _, _}, _bool), do: {ast, false}
  def capture_visible?(ast, bool), do: {ast, bool}
end

Also available in this Gist.

This allows us to write above example snippet as

def handle_event("reset-button", _params, socket) do
  socket
  |> assign(:form_values, default_form_values())
  |> assign(:error_messages, [])
  |> assign(:reset_button_active, false)
  |> {:noreply, &1}
end

Now I know that opinions on enhancing the pipe-operator in general are divided.
Personally I think that this tiny bit of sugar is easier to understand (especially for people seeing the code for the first time!) than e.g. defining manual ok(...) or noreply(...) wrapping functions and I think it can be an improvement on the ‘breaking of the pipeline’ that is currently required.

That said, I am not (yet) releasing this snippet as a library, because:

  • even though it is a tiny bit of sugar and macro-code, it might still be somewhat brittle.
  • I’d rather start a bit of discussion about this syntax to hear what other people think about this sleep-deprived idea I had yesterday-evening late before committing to it :grin: .

Your input and feedback is greatly appreciated!

13 Likes

It’s a cool idea!

A simpler but less general way to address this could be to define private functions to let you do

build_up_assigns
|> noreply()

or

build_up_assigns
|> wrap(:noreply)

The names could be improved, and I’ll leave the implementation as an exercise for the reader :slight_smile:

(EDIT: Sloppy reading on my part – you mentioned that already.)

For stuff like LiveView, one could also define e.g. a handle_event to be shared by all LiveViews, inside MyAppWeb, and have it call my_handle_event and then wrap it - though personally I think I wouldn’t go that far. There’s something to be said for sticking to the conventions, slight boilerplate be damned :slight_smile:

1 Like
  1. I love the general concept, but how about overloading one of the other pipeline-like operators that’s unused?
  2. I don’t love that the capture operator is used. Sadly, I don’t have a better suggestion.

I’ve seen so many new syntax suggestions to avoid breaking the pipe. For the most part, it has always seemed unecessary. You can always create a separate function or write it inline (albeit verbose and ugly). It’s not that I don’t want an easier way to do it inline, but nothing I’ve seen has really appealed to me.

But I really love this. If I saw it in a codebase, I would immediately understand its intent because it is just a shorter version of the existing. It enhances the flow and readability of a function without introducing a new operator.

I look forward to others poking holes in this, but my initial reaction is that this is the exact elegant syntax I’ve always wanted.

4 Likes

Another common pattern is to write like this:

def handle_event("reset-button", _params, socket) do
  {:noreply, socket
             |> assign(:form_values, default_form_values())
             |> assign(:error_messages, [])
             |> assign(:reset_button_active, false)}
end
5 Likes

I like this suggestion as well. It matches expectations created around function arguments by providing an intermediary step between writing a named function that takes arguments in the correct order and using an anymous function. Personally as I have learned elixir I have become accustomed to just writing a lot more function definitions because I am a pattern matching superfan so I’m not sure I would use it over a contextual convenience like noreply(), but I’d definitely try it out.

1 Like

That is typically how I write liveview replies too. I guess I see this as a lot more than just that use case though.

I am thinking it would be potentially even more useful for the cases where you want to pipe to the next function, but that function does not expect your piped value to be the first argument. @Qqwy Am I correct in assuming it would provide for this sort of thing?..

def do_thing(user, project) do
  user
  |> something_with_user()
  |> some_other_thing(project, &1)
end
1 Like

Thank you everyone for your responses!

Personally I find that kind of code (putting pipelines inside tuple- list- map- or struct-constructors) even harder to read than the ‘breaking the pipeline’-example of the original post. That’s definitely subjective, though.

Yes, you are correct. It was not my primary goal when building this, but indeed you can also use the ‘bare’ &1-syntax to pass the pipe result to any argument of a function as well.

Thank you! :blush: This is what I am hoping for.

And that might also address @ityonemo’s question: The reason I did not want to override another operator is because operators are not self-documenting. You do not know without context what ~> does in an Elixir-application, because it depends on what libraries are in scope. You do know what |> is (supposed to) do however, since it is built-in. I am trying to extend its semantics slightly while not breaking the ‘principle of least surprise’.

I’d love to hear from more people if they agree with @baldwindavid or if they do find it too ‘magical’.
And feel free to shoot any holes in this implementation, of course :grin:!

5 Likes

One thing to look at are ways that it might feel awkward. I tried to put it through some paces via a nonsensical example and it looks nice to my eyes in various new and existing scenarios. (* denotes new function usage)

defmodule CapturepipeTest do
  use ExUnit.Case, async: true

  import Capturepipe
  import Kernel, except: [|>: 2]

  test "capture pipe usage" do
    assert speak("Jane", "Hello") == {:ok, "...HELLO, JANE..."}
  end

  def speak(name, greeting) do
    name
    # * function that requires different order
    |> greet(greeting, &1)
    # regular function still works fine
    |> String.upcase()
    # does not disrupt adding ellipses via original method
    |> (&(&1 <> "...")).()
    # * example of adding ellipses with enhanced pipe
    # could also be ("..." <> &1) if you prefer
    |> "...#{&1}"
    # * returning a tuple
    |> {:ok, &1}
  end

  defp greet(greeting, name) do
    "#{greeting}, #{name}"
  end
end

I also wonder if it causes any confusion regarding usage of captures outside the pipe operator. I can’t think of any good examples off-hand, but perhaps a developer might think the & is somehow unnecessary for, say, Enum functions. I don’t know. It is still looking really good to me.

4 Likes

You do not know without context what ~> does in an Elixir-application, because it depends on what libraries are in scope

Yeah, but that’s the point. In elixir it’s not hard to find out, since imports are lexically scoped.

For example, if you’re looking at code that uses my net_address library, you might not know what ~i"1.1.1.1" does, but you can make a pretty good guess, and then you can confirm by verifying that there’s an import IP statement and looking up the code for that.

It’s important to use something that isn’t the normal usage of the pipeline because you need to signal to the user that “something is different” here, otherwise people will scratch their heads even if it should be obvious what’s going on.

I was thinking about what you might want to replace &1 with, and I would pick __PIPE__. I don’t like &1 because I think it’s bad to confuse readers that it might be a capture, and 1 doesn’t make sense since there’s no way for it to be 2, 3, etc. __PIPE__ will ast to an alias (atom), so it should be trappable, and there is precedence for such forms to not necessarily be elixir module names (__ENV__, __CALLER__, __STACKTRACE__)

2 Likes

and 1 doesn’t make sense since there’s no way for it to be 2, 3, etc.

I see your point there, but working in a pipe we know that we’re only working with the single first argument, so &1 makes sense to me. This might be a terrible idea, but it would also be understandable to me if I saw just & rather than &1. I still think &1 is explicit and obvious though.

Are you thinking about this from the standpoint of a language enhancement or as a library? If it is only ever to be a library then I can see introducing a new keyword. A developer would then immediately know that this is provided via a library.

I’m thinking about it with the mindset of it eventually being a language enhancement and, in that case, it seems like less mental overhead to use a subset of the already existing syntax. Getting this added to the language would obviously be a much heavier lift. My assumption is that @Qqwy at least has some visions of this being added to the language, but I might be wrong there.

1 Like

I saw just &

I don’t think this is correctly parsable, as it is hard-coded in the parser to be an arity-1 operator. I could be wrong. though. it would have to be :&

language enhancement

I don’t think it’s appropriate as a language enhancement. I’m not sure what a pipe extension should look like.

for starters, I want to to sometimes assign a value to the end of the pipe.

foo = bar
|> baz()
|> quux()

to:

bar
|> baz()
|> quux()
|> foo

This should obviously be a match operation, so:

{:ok, foo} = bar
|> baz()
|> quux()

becomes:

bar
|> baz()
|> quux()
|> {:ok, foo}

well that doesn’t feel right (and conflicts, for example, with the proposal in this thread), but probably there should be SOME operator that does this, and now it becomes a challenge to decide which one “deserves” to be on the end of |>. The best choice is to have |> do one thing and one thing well.

Anyways there are two similarly shaped operators: ~> and ~>>, so those are both options, if anything is going happen to change the language it would be to ask the elixir team to release more pipe-y operators, into the parser for people to use in their libraries, like ->> and =>>, if we want to get wild, ==>, =~>, =-> might be fun too.

it would have to be :&

Yeah, you’re probably right and I shouldn’t have even mentioned it. Moreso, just saying I like sticking with the capture syntax because it is already established. &1 still looks the best to my eyes.

for starters, I want to sometimes assign a value to the end of the pipe

Well, now we’re in the land of talking about making a thing that doesn’t exist work together with another thing that does not exist, but I guess I’ll play. :slight_smile:

If I’m understanding correctly, you want to be able to bind a variable within a pipe. This seems like a totally different language feature than the syntactic sugar of OP’s feature which is focused on simplifying how a value is passed down through a pipeline. I agree that a different operator should be used for such a thing were it to exist, but I think this would work just fine with the proposed syntax of the OP.

bar
|> {:ok, &1} # OP's feature
|= {:ok, foo} # binding a variable at the end of a pipeline (or ==>, =~>, =->, etc.)
1 Like

Indeed, this is the goal I am intending.

Maybe. I know that the Elixir core-team is very cautious in adding language features in general, and pipes especially. There have been many proposals and attempts in the past to make the pipe notation more powerful on a language level.
It is difficult to come up with a syntax that is more powerful without also becoming more terse (less readable).

Anonymous functions are in general discouraged in pipes in Elixir because by wide consensus it is considered better practice (for the readability/maintainability of the code) to give those functions a name. This is one of the two reasons that you are currently required to add .() manually to an anonymous function used in a pipe (the other being to keep behaviour the same between 10 |> (fn x -> x + 1 end).() and foo = fn x -> x + 1 end; x |> foo.()).

Capture-syntax, while considered a ‘shorthand’ for anonymous functions, is treated differently from anonymous functions by the compiler. (I do not know the reason for this. @josevalim, if you have the time, could you maybe enlighten us?)

Currently, captures are not allowed in pipes at all. Only when we transform a capture into an anonymous function by wrapping the capture inside parentheses and then also add the .() required for anonymous functions is it possible to use a capture in a pipe today.
Personally I think it would improve the readability of Elixir-code by reducing the boilerplate in many functions if capture-syntax would be allowed to exist bare inside a pipe. (as e.g. {:ok, &1} but I would also be content with for instance &{:ok, &1}).

I don’t expect the Elixir-core team to agree with me on this, however, for the afore-mentioned reasons.
The one thing that might make this proposal better (arguably, in my personal opinion) w.r.t. earlier proposals is that it tries to stay very close to what is already possible in Elixir today. This means it should not surprise programmers and prevent code written with the new syntax from becoming terse/unreadable.

In the (very likely) case where this addition does not make it into the language proper, I’d like to release it as a library.

3 Likes

I think you’re missing my point. There are good arguments for any number of ‘enhancements’ to the pipe syntax. Which is more pipeish is a very squirrelly subjective call, with no one best justification for any given one, and all of them pay the penalty of “overloading the pipe operator to do more than one conceptual thing”. The most reasonable course of action is, then to basically do nothing, and let all pipe enhancements (to include OP’s, and the one that I showed, which is actually implemented in some library or another I can’t remember) live in the world of user imported libraries.

I think you’re missing my point.

I do think I missed your point, but I get you now and can completely appreciate that sentiment.

This is a controversial topic and as @Qqwy noted, it would be very unlikely to actually get accepted into the language. As I mentioned in an earlier comment, I’ve seen a bunch of these proposals (and you’ve probably seen more) and none of them were appealing to me. This one just has me excited.

“overloading the pipe operator to do more than one conceptual thing”

I guess I just don’t see it as doing more than one conceptual thing. The purpose of the pipe operator as I understand it is to pass the result of an expression as the first parameter of another expression. This just seems like a more succinct way of doing that thing AND leveraging known and unsurprising (to me, at least) syntax.

The most reasonable course of action is, then to basically do nothing

Sure, that is the likely outcome. If the OP and I were the only people for which this is appealing, I think it is safe to say it won’t be added to the language anytime soon. :). But I presume OP has posted here to vet and see if there is any groundswell of support from developers and the core team. Barring unforeseen technical issues, this probably comes down to preference.

3 Likes

Today you can write: & foo |> bar(:value, &1) |> baz(). With your proposal, how do you know if &1 binds to the wrapping & or it is just an extension of the pipe? :slight_smile:

9 Likes

Since nested captures are disallowed I would think we should expect that &1 here binds to the wrapping &. Is it possible to detect that it is already within a capture and “hold the sugar”?

Very interesting! This was a syntactical ambiguity I had not considered.

The &{:ok, &1}-syntax (still without requiring wrapping this in (...).()) that I mentioned a couple posts back would not have the same problem:

# This will raise a compile-time error about nested captures as expected
&( foo |> &bar(:value, &1) |> baz() )
1 Like

From my example above, the following tweaks should work, correct?

def speak(name, greeting) do
  name
  # * function that requires different order
  |> &greet(greeting, &1)
  # regular function still works fine
  |> String.upcase()
  # does not disrupt adding ellipses via original method
  |> (&(&1 <> "...")).()
  # * example of adding ellipses with enhanced pipe
  |> &("..." <> &1)
  # * returning a tuple
  |> &{:ok, &1}
end

I like this just as much and it looks even more obvious. It does appear the formatter would need some work to support it though.

2 Likes