How to `get` or `remove` the first matched element in a list without traverse it?

you should be looking for a better data structure - one that supports deletions from arbitrary positions faster than O(length).

This is a misunderstanding of original task. @Changxin was just looking for solution to delete the first element he has found satisfying the predicate during traversal of the structure.

So, first of all, the traversal of the structure is already a O(length) task. O(length) to delete, doesn’t change complexity

Secondly, list is one of the fastest structures which can delete value during traversal (second one is tuple).

So the most efficient solution is:

defp do_delete_first([], _func), do: []
defp do_delete_first([head | tail], func) do
  if func.(head) do
    Process.put(:"__delete_first_found__", head)
    tail
  else
    [head | do_delete_first(tail, func)]
  end
end

def delete_first(list, func) do
  list = do_delete_first(list, func)
  {Process.delete(:"__delete_first_found__"), list}
end

Because it has nothing in the stack, it traverses until reaches the value and it uses this trick with tail recursion where return value is moved into the head of the list once the function returns. But this function looks ugly, because it uses pdict.

That’s why implementation of this function is a job for a library

Considering other arguments,

Using a function that returns a list on non-list Enumerables is possible, but it’s going to 100% have to traverse the whole structure to create all the list cells.

Some strucutres can be traversed and ordered (consider tuples and keywords). For these structures you also shouldn’t traverse the whole structure

The situation on the BEAM is more complicated than “body recursion BAD, tail recursion GOOD”.

Yeah, I know that, and I’ve read about it in “7 myths of erlang efficiency”, but I’ve described why this particular case of recursion is worse than just a tail recursion. The reason were: extra concatenations and variables on the stack

Math problem. How to code `Xn+1 = Xn + Xn * S` - #10 by al2o3cr

This is a great post, thanks!

Excellent!
Well, Pathex.pop is what I want! Thx!

There’s no reason to use pdict here. Just change the return value fo do_delete_first to handle returning both the list as well as a potentially found value.

If so, the trick with tail recursion won’t work

Interesting trick. I think you meant body recursion

Just for the sake of completion, here is the tail recursive version:

  def find_delete(list, fun) when is_function(fun, 1) do
    do_find_delete(list, fun, [])
  end

  defp do_find_delete([head | tail], fun, acc) do
    if fun.(head) do
      {head, :lists.reverse(acc, tail)}
    else
      do_find_delete(tail, fun, [head | acc])
    end
  end

  defp do_find_delete([], _fun, acc) do
    {nil, :lists.reverse(acc)}
  end

This rough benchmark gives me a slightly faster time for the tail-recursive version, but takes slightly more memory (results). Of course this would vary based on OS, list size, OTP version…

I’ve tested on bigger (10000 items) lists where 5000th element is searched and median differencies are just 1%, but memory difference is 2x

Name                     ips        average  deviation         median         99th %
tail recursive       10.40 K       96.15 μs    ±16.50%      100.23 μs      127.70 μs
process dict          9.95 K      100.47 μs    ±10.58%      101.63 μs      132.07 μs

Name              Memory usage
tail recursive       156.29 KB
process dict          78.20 KB - 0.50x memory usage -78.08594 KB

So I think that your solution is more efficient in time for shorter lists, but pdict solution is always more efficient in memory

Hmm, the strange thing is that for very short lists of 10 elements pdict is faster

Name                     ips        average  deviation         median         99th %
process dict          3.33 M      299.86 ns  ±2907.65%         247 ns         647 ns
tail recursive        2.70 M      371.06 ns  ±5666.75%         254 ns         599 ns

Name              Memory usage
process dict             240 B
tail recursive           360 B - 1.50x memory usage +120 B

I have no explanation for this

emmm, since Pathex.pop performs double lookup, thus sometimes it may worse than Enum.group_by?

Yeah, it performs double lookups. I’ve just added support of the pop operation, but I haven’t performed any benchmarks for this. I have a plan for 2.3 version to improve pop, I just need to decide how.

This is a big question based mostly on heuristics. For example, I can use pdict to store the value during deletion (but this will break the inter-process paths). I can move the value onto stack, but this will significantly increase the time of delete operation. I can have both pop and delete operations in the path, but this will increase the amount of generated code.

So right now I am just gathering information about the best solution for pop, but I can assure you it’ll be optimized in the next version.

Thank you for your work, but I think the best way to achieve performance is Elixir provides first match return high order functions, they’re may be implemented by C, than used as raw functions.