std_result - A way to standardize function returns

Hi everyone,

Published a new library: StdResult!

StdResult is a library designed to standardize function returns.
Highly inspired by Rust’s std::result, this library provides a way of simplifying the management of :ok and :error tuples by providing functions for manipulating them.

The problem:
One problem I come across quite often is the lack of consistency between certain functions. In the same module, some functions will sometimes return :ok while others will return {:ok, result} and others just result. The same goes for errors. It can quickly become complicated to manipulate these results.

That’s where StdResult comes in.

Usage:
Here is a simple example: let’s say we need to retrieve an environment variable, convert it to an integer and check that it’s positive. Our function should return {:ok, value} or {:error, reason}.

Here’s an example of what it might look like with StdResult.

import StdResult

System.fetch_env("PORT")
# This will transform `:error` into a `:error` tuple
|> normalize_result()
# If there is an error, explicit the message
|> or_result(err("PORT env required"))
# If no error, parse the string as an integer
# We could also have used `Integer.parse/1` but for simplicity's sake we won't.
|> map(&String.to_integer/1)
# Test if the number is positive
|> and_then(&(if &1 >= 0, do: ok(&1), else: err("PORT must be a positive number, got: #{&1}")))

# The result will be either:
# - `{:ok, port}`
# - `{:error, "PORT env required"}`
# - `{:error, "PORT must be a positive number, got: <value>"}`

Check out the documentation for more details on existing functions.
Any issues, suggestions or contributions are welcome.
Cheers


Links:

3 Likes

Another solution is to use your editor to inspect the function (hover, keyboard shortcut, etc) to see the spec. This doesn’t introduce cognitive load, you learn the standard library, you don’t pay a performance penalty and are more likely to write idiomatic code.

It is convention that functions/macros ending with a ! raise exceptions.

4 Likes

Not having the time for more thorough feedback but I’d make this shorter i.e. to_result.

1 Like

I don’t like your solution as the reader needs to understand your way of thinking which is not as obvious as you think and in practice your library does not gives developers enough flexibility.

Also and and or naming is already used in Elixir for working with binaries (or inside Ecto.Query, but said or_where is written based on SQL’s naming).

What you can do is to introduce 2 macros: tap_on/2 and then_on/2.

"PORT"
|> System.fetch_env()
# flexibility your package does not support i.e. early raise
# `System.fetch_env(env) || raise message` is more common in config files
|> tap_on(:error, fn :error -> raise "PORT env required" end)
|> then_on(port when is_binary(port), &String.to_integer/1)
|> then_on(port when is_integer(port) and port >= 0, &Function.identity/1)
|> then_on(port when is_integer(port), &{:error, "PORT must be a positive number, got: #{&1}"})
# or shorter:
|> then(&{:error, "PORT must be a positive number, got: #{&1}"})

Another example without early raise:

# some data …
|> then(&function_call_returning_always_tuples/1)
|> then_on({:ok, _}, {:ok, data} -> function_call_sometimes_returning_atoms(data) end)
|> then_on(:error, fn err -> {err, :some_reason} end)
|> then_on({:ok, _}, fn {:ok, data} -> some_function_call(data) end)
|> then_on({:error, _}, fn {:error, reason} -> handle_error(reason) end)
|> then_on({:ok, _}, fn {:ok, result} -> IO.inspect(result, label: "Result") end)

What we win with those 2 macros?

  1. Naming is much simpler. then naming is already known, so on with extra match pattern as first argument does not require any extra knowledge of said hex package as al it do is following common naming and patterns.
  2. There is only one macro - not a whole API - developer if needed could implement their own let’s say then_on_ok/1 macro using this one
  3. We work with one type of data at a time (no weird and long ifs inside anonymous function)
  4. That way all errors could be easily piped to external handle_error(reason) function, so all error messages would be in just one place without a need to scroll all of them to see a success pipeline steps (useful on large pipelines with many different errors)
  5. Allow to work with :error atoms which allows early raise as above
  6. Those macros allow to easily write other macros without much knowledge of metaprogramming
  7. It allows to work with other custom return data such as 3-element tuple which is quite rare, but still exists

Here is how simple is to write a custom macros:

defmacro then_on_ok(left, func_ast) do
  quote do
    then_on(unquote(left), {:ok, _data}, fn {:ok, data} -> unquote(func_ast).(data) end)
  end
end

defmacro then_on_error(left, func_ast) do
  quote do
    then_on(unquote(left), {:error, _reason}, fn {:error, reason} -> unquote(func_ast).(reason) end)
  end
end

defmacro then_on_error_atom(left, func_ast) do
  quote do
    then_on(unquote(left), :error, fn :error -> unquote(func_ast).() end)
  end
end

With such custom macros we can then rewrite our pipe:

# some data …
|> then(&function_call_returning_always_tuples/1)
|> then_on_ok(&function_call_sometimes_returning_atoms/1)
|> then_on_error_atom(fn -> {:error, :some_reason} end)
|> then_on_ok(&some_function_call/1)
|> then_on_error(&handle_error/1)
|> then_on_ok(&IO.inspect(&1, label: "Result"))

I echo some sentiments here. There is reason behind different return value types. Possibly not in some libraries and certainly within private projects, but there is certainly good reason for the choice of all return types. For example there is no point in returning a 2-tuple for the happy path if we’re not passing back any other data. ok/error tuples are used as light-weight exceptions and only when the caller can do something about it (talked about here). Ecto.Repo.get/2 returns nil so it can be used in a scenario where the record might not exist. When you use this function, you are communicating to the reader that there are scenarios where the record won’t exist, but it’s not an “error” and doesn’t need any kind of messaging. Ecto.Repo.create/1 returns an ok/error tuple because something can go wrong at the database level which usually happens because the user made an error. Finally Ecto.Repo.get!/2 raises and is used when we know for a fact the data exists and if it doesn’t, there is nothing we can do about it other than Let it Crash™ or, if that doesn’t work, stay late on a Friday to fix it.

4 Likes

TL;DR: The aim of this library is to manipulate function returns in a more simple/idomatic/“standard” way rather than to standardize the function returns themselves.

It’s all about control-flow (combine/pipe functions with multiple returns types) rather than “all library developpers should always return :ok/:error tuples”.


First of all, thank you all for your answers.
They made me realize something: the purpose of this library is very clear in my mind, but the description I gave is not.

So I’m going to try to rephrase it.


Firstly, and this is my fault, I should never have used the term “standardize function returns”. Here I’m not promoting the fact that all functions should return a :ok/:error tuple. Of course an empty?/1 function, for example, should return a boolean, of course a get!/2 function should just return the data (since very often there’s a get/2 function in the same module that already returns a :ok/:error tuple).

The main use of this library is to manipulate these results in a simpler/“standard” way.
I’ve talked a bit about this in the project README (but not enough I guess), but you can think of StdResult as a new way of doing control-flow using :ok/:error tuples.

Originally, my problem stems from the use of with where the more operations there are to perform, the more complicated it becomes to write else. I know that this is a fairly recurrent problem and that many developers have already had this problem. Several posts/libraries exist on the subject and try to solve the problem. One example is this proposal: https://elixirforum.com/t/with-statement-else-index/56914.

StdResult is a library that provides an answer to the problem in specific cases.

Perhaps an example would be more telling. Let’s take this piece of code:

def update_user(updated_user, state) do
  with true <- User.exists(updated_user.id),
       {:ok, country_code} <- Country.to_country_code(updated_user.country),
       true <- User.validate_phone_number(updated_user.phone),
       old_user <- User.get(updated_user.id) do
    new_user =
      old_user
      |> Map.merge(updated_user)
      |> Map.put(:country, country_code)

    MyDb.put(state.db_conn, new_user)
    {:ok, new_user}
  else
    _ -> IO.puts("Something bad happened")
  end
end

Here, if you need more details on the step that failed in the else block, there are 2 solutions:

Solution 1: wrap each function call with a tuple

with {:user_exists, true} <- {:user_exists, User.exists?(updated_user.id)},
     ...
else
  {:user_exists, false} -> IO.puts("The user doesn't exists")
  ...
end

The main problem with this solution is that very quickly the code becomes complicated to reread and too verbose.

Solution 2: wrap functions into another that returns a :ok / :error tuple

with :ok <- validate_user_exists(updated_user.id),
     ...
else
  {:error, msg} -> IO.puts(msg)
  ...
end

def validate_user_exists(id) do
  case User.exists?(id) do
    true -> :ok
    false -> {:error, "The user doesn't exists"}
  end
end

Everyone can have an opinion on this solution, but personally sometimes I don’t want to have to scroll or press on my shortcut to see/modify the error I’m going to get, or even have a dozen functions in a module where their only use is to convert a term into a :ok/:error tuple.

The solution I propose is as follows:

import StdResult

def update_user(updated_user, state) do
  updated_user
  |> then(fn user ->
    case User.exists(user.id) do
      true -> {:ok, user}
      false -> {:error, "The user doesn't exists"}
    end
  end)
  |> and_then(fn user ->
    case Country.to_country_code(user.country) do
      {:ok, code} -> {:ok, Map.put(user, :country, code)}
      {:error, _} = err -> err
    end
  end)
  |> and_then(fn user ->
    case User.validate_phone_number(user.phone) do
      true -> {:ok, user}
      false -> {:error, "Invalid phone number"}
    end
  end)
  |> map(&(&1.id |> User.get() |> Map.merge(&1)))
  |> inspect(&MyDb.put(state.db_conn, &1))
end

I hope that’s clearer. Let me know if it isn’t.


I know, but I couldn’t think of a better name for this macro. If you have, don’t hesitate to suggest :slight_smile:
Also, the macros ok!/1 and err!/1 raise during a pattern match, so it’s not that far from convention.

I like it! It will be on the next release, thanks !

What do you mean by “the reader needs to understand your way of thinking” ?
What more do you need when you say “in practice your library doesn’t give developers enough flexibility” ?

Why do you need to early raise ? Because you expect a value ?
Then simply use StdResult.expect/2 or StdResult.unwrap/1 :slight_smile:

env_value = System.fetch_env("PORT") |> expect("You need to define the environment variable PORT")

But basically, for your example, the equivalent of your code would look like this:

import StdResult

"PORT"
|> System.fetch_env()
|> to_result()
# `System.fetch_env(env) || raise message` is more common in config files
# So just use `inspect_err/2` :)
|> inspect_err(fn :error -> raise "PORT env required" end)
|> map(&String.to_integer/1)
|> and_then(fn port ->
  if port >= 0 do
    {:ok, port}
  else
    {:error, "PORT must be a positive number, got: #{port}"}
  end
end)

As for the rest of your message, in short then_on_ok/2 is an and_then/2 and then_on_error/2 is or_else/2.

And using functions is a choice, because I don’t like using macros when I can simply use functions.

@sodapopcan I think I’ve answered your comment in this one, but don’t hesitate if you have any questions.

I’ll update the current post and README for the next version.

Naming is very hard as we all know, I don’t claim mine as better in any way.

Here’s one more idea:

"PORT"
|> System.fetch_env()
|> ok_err()
1 Like

Ok, so let’s assume the reader knows only core Elixir API and it’s naming…

This original version requires me to see what those functions are actually doing which is against of what you wrote in quote above.

Here I have no idea what “normalization” you mean unless I read your hex package’s docs. This part without a context of the next lines could make me think that normalize_result returns a boolean and or_result is doing something like & &1 || err("PORT env required").

The new version is not better:

  1. Since to_result is not in Elixir core I would expect that every to_* functions would return a custom type i.e. struct.
  2. inspect_err have no sense for me. inspect is naming from IO, but in anonymous function the raise is called. So … when the result is error we inspect it and otherwise call anonymous function? Wait, that’s conflicting with the message in raise.

The macros I have proposed uses existing Elixir core naming properly. Both tap and then explains what the macro is doing. on is a short of match on.

See first quote above and how I can see your naming as said assuming the reader knows only Elixir core API.

Oh, I have missed the expect. Still it’s yet another function to check in documentation. Unlike tap which is already known expect is not intuitive.

expect(data, "Something went wrong")

Unless I read your docs I could think that it means:

expect data to be "Something went wrong"

Alternatively expect could mean that we expect a truthy value for data variable and return the passed string otherwise.

As a senior developer reading your examples makes sense for me, but as said you are using the existing naming to do things your way instead which for new developers or people who did not read the docs could be unclear.

The macros are for on thing i.e. for pattern matching.

Not really, as in my case those are examples of how a developer could use my macros and in your case it’s your package functions. I wrote those examples to show that said 2 macros is all the developer needs. Their naming is proper (having in mind Elixir core naming of course). Then if the repository maintainer decides to even shorter the code he could write then_on_ok and then_on_error custom macros like in my last example in that reply message.

Because I expect something (environment variable, directory structure, command arguments and so on) outside the app to exist outside the supervisor tree. The cases are:

  1. system environment variables (application config, scripts and tasks)
  2. command arguments i.e. System.argv/0 or task arguments (scripts and tasks)
  3. directory structure or some seed data in priv and so on (scripts and tasks)

Ok, so …

  1. The hex package is simple and good for it’s use case
  2. What I don’t like personally is using the Elixir core naming in a different way
  3. It works well, but it requires reading documentation - we do so instead of scrolling the the function which changes result into tuple, so that argument is bad for me
  4. Personally for me it’s too much functions to work with return values. As said I would prefer a simpler and more generic macro with a pattern matching in first argument.

Also I reminded that ok hex package. If I would use a library to deal with results my preference would be ok package.

The above ideas reminded me of the macro I implemented a few months ago: heresy/lib/heresy.ex at main · doorgan/heresy · GitHub

That lets you write:

begin do
  match true = User.xists?(updated_user.id), else: {:error, :user_not_found}
  match {:ok, country_code} = Country.to_country_code(updated_user.country)
  match true = User.validate_phone_number(updated_user.phone), else: {:error, :invalid_phone_number}
  old_user = User.get(updated_user.id)
  
  new_user =
    old_user
    |> Map.merge(updated_user)
    |> Map.put(:country, country_code)

  MyDb.put(state.db_conn, new_user)
  {:ok, new_user}
end

And later was reminded that it’s similar to the older GitHub - vic/happy_with: Avoid commas on Elixir's with special form.

Every now and then the topic resurfaces; having a normalized value structure is valuable

I think I understand a bit better, though I think it’s just your wording I find confusing. In your problem statement it reads as “every function in the same module should return the same result.” I think I misunderstood thinking this library was for ensuring such return values when it actually unifies existing ones?

I have far less problem with the latter for those who would be interested in this. I personally don’t believe this type of branching logic belongs in pipelines, though. I’m a big believer that Elixir’s very few constructs are all valuable and can themselves be used to signal what’s going on. For me, with is the tool to use when there are a series of dependent steps. When I see a pipeline, I generally just expect raw data in, raw data out. This is huge for scannability. But of course, I don’t think there is anything inherently wrong here (now that I understand better).

@dorgan Isn’t happy_with merely alternate but similar syntax for with with nothing heretic about it? :smiley:

2 Likes

More often than not I find that general purpose macros are not always welcomed because they’re “magic” or “people need to learn them” :slight_smile:

I’m with you in that I really really dislike the term “magic” when it comes to macros. They aren’t magic and I feel the difficulty people have with them is their own mental blockers thinking they are harder than they are. Macros are great and are at the heart of what makes Elixir what it is.

However, I do personally dislike the general purpose type stuff. I actually prefer happy_with’s syntax but I’ve never used it because regular with isn’t bad and I would always rather use the vanilla language. I also do get annoyed at “having to learn” (macros or functions) different ways of doing things there are already perfectly good answers to. It’s a death by a thousand cuts scenario for me. Really, I just really like Elixir how it is :slight_smile:

2 Likes

ok_tuple, wrap_in_ok ok_wrap, wrap_in_ok_tuple, to_ok_tuple.

Solution 1 is evil, bad, bad, naughty, ugly, eww, yucky.
Solution 3 is to return exception structs, as discussed in the thread you linked, in keathley’s blog post that @soadpopcan lined and this one. Using this method, you can either ignore the error, raise it or use it.

Your example is still contrived because you either don’t care which error it is or all you’re doing with the error is IO.puts. All of those User and Country functions must be in your codebase(?), so you could easily write functions that return ok/error tuples if that is what you need.

To me, it feels easier to rewrite a function in a legible way than to introduce a library that does things in an idiomatic-Rust/non-idomatic-Elixir way. You can’t call your library idiomatic Elixir code if you just wrote it and nobody is using it yet.

As @sodapopcan touched on, pipes are often data transformations, and putting a bunch of conditionals in a pipe makes the control flow difficult to see at a glance.

2 Likes

It’s not, I love with, I just wish it was better, and thankfully elixir is extensible via macros(and I kinda like Zig’s try). I’d love if we didn’t have to wait for those changes to land in the languages themselves or to be added in a library by someone with an important name.

In the context of the OP I think it’s valuable because, well, having a normalized return type is valuable but it has the same side effect as with: it makes you create functions

I have a problem personally with that because there wasn’t a case where this sequence doesn’t happen:

  1. A few lines of code exist, and they’re extracted to a private function to make with less painful or avoid credo’s cyclomatic complexity check, or the formatter butchers the code, or with would only handle sparse branching points and you need many nested cases and Credo gets mad at you, or some other “small” issue
  2. Now that the private function exist, that piece of code can be called by other code in the same module
  3. 2 or more functions that use those private functions have slightly different use cases, so they start to add opts to it to change the behavior
  4. These functions get used by even more functions ouside the module, which also have different requirements
  5. On top of the many slightly changes in required behavior, they also produce side effects, so now you need to keep track of that too

Functions have the side effect of encouraging reuse and that reuse is rarely intentional enough to not cause problems

So while, again, I value value normalization, I don’t think libraries like the OP help a lot in the long term. I have similar gripes with Ecto.Multi vs Repo.transact like in the other thread. The common suggestion of “create more functions”(and maybe pipe them) sounds good on paper but it almost always gives me problems and I don’t want all that.

1 Like

Why? Is there some kind of company policy that restricts you defining a DSL/features that are valuable to your business needs?

IMO with beats competition by far. Take a look at how scala deals with their “monoids”, in a for comprehension, it just feels like an afterthought to some feature they couldn’t implement.

The usage of course varies and there is room for abuse, this especially happens from people coming from other ecosystems, however if you use it correctly it is invaluable.

1 Like

It’s very hard to convince people to adop something or to not thoroughly question the implementation to the point of inaction if you don’t have a well known name or reputation. To be more concrete also in reply of the previous “why heresy?” question, I don’t see it mentioned as often nowadays but people were very reluctant to adop patterns outside of “the blessed way”, so heresy seemed appropriate for something that explores the opposite thing. It was nothing intentional, it just happened. (Also to be fair Jose got a lot of pushback in the proposals to improve for, change is hard)

It does, but again, it could be better. And the best way to find what’s better is to actually try other approaches and try to identify what exactly could be better. I pointed above what I personally think that is. Sometimes you are not “holding it wrong” but it could really be different.

1 Like

I agree that the “result tuple” is such a widespread convention in Elixir it’s a little puzzling there’s not something built in to help you construct those tuples from raw values. But there have been many libraries created to “help” with the problem but they don’t ever seem to gain widespread traction I believe because it is too easy to just add something to your project to assist with whatever specific flavor of that convention the maintainers want to promote.

I think to move forward on this issue we would need a proposal for a change to the standard lib (Tuple.result?) rather than another lib, which there very well might have been, but I don’t think I’ve seen any.

1 Like

It is kinda true, this is our nature as humans, even though I would recommend you to try, you never know.

Delivering the idea correctly is also very important, this is why even though the library from this post might prove useful in other ecosystems, there is a clear lack between idea and practicality in elixir ecosystem.

I think this is a terrible idea. The tuple convention {:ok, response} is a ghetto monad that counts as a pure data structure.

When you will start enforcing in the language a convention that functions must return a specific shape of data, you will start to have inconsistent behavior:

  • What if my function doesn’t abide by these rules? Example bang functions.
  • What if my return response is a more comprehensive data structure? Example: {:ok, response, partial_errors}

Instead of dealing with that proactively it is better to let the user decide, defensive programming is bad practice that needs to die out.

This however doesn’t mean that you can’t use any kind of linters, specific credo rules or dialyzer configurations that would invlolve your local codebase following some specific rules.

2 Likes

I don’t disagree, and I wouldn’t be surprised if this is how the core team feels as well, but it’s unfortunate that the strength of the convention continually creates the pressure for these various libs to help “standardize” the practice…

3 Likes