What is the idiomatic shape for extensible keyword opts?

Questions & Help Questions
, ,
1

I’m working on a small library (GitHub - fuelen/mold: A tiny, zero-dependency parsing library for external payloads · GitHub) where schemas are plain data. Type options live in the 2nd element of the tuple: {:string, min_length: 2, max_length: 50}.I’d like to officially support custom keys here, so companion libraries can extend these opts.

Here’s what the typespec for :string looks like today:

@type string_type() ::
  {:string,
   trim: boolean(),
   nilable: boolean(),
   default: default(),
   format: Regex.t(),
   min_length: non_neg_integer(),
   max_length: non_neg_integer(),
   in: Enumerable.t(),
   transform: transform(),
   validate: validate()}
  | :string

This renders well in docs. I didn’t even factor the common opts (nilable, default, transform, validate) into a shared type, because then I’d have to write:

{:string, [
  {:trim, boolean()} |
  {:format, Regex.t()} |
  {:min_length, non_neg_integer()} |
  {:max_length, non_neg_integer()} | shared_option()
]}

It renders not as nice as the inlined keyword typespec, but I’m fine with it if needed.

Technically, I can already write

{:string, min_length: 2, my_custom_opt_for_another_library: :something}

and the library swallows unknown options. But the typespec says you can’t add custom options.

Here are the options I’m considering:

Option 1. Open keyword

@type string_type() ::
        {:string,
         [
           {:trim, boolean()}
           | {:format, Regex.t()}
           | {:min_length, non_neg_integer()}
           | ...
           | {atom(), any()} # <-- custom option
         ]}
        | :string

In this case, the list of known opts reads more like a hint than a contract.
Probably, it would be interesting to have an ability to write something like

{custom_option_name :: atom(), any()} when custom_option_name not in [:trim, :format, :min_length, ...]

but it actually means I want a map with the syntax of keyword list :slight_smile:

Option 2. Explicit :ext namespace

{:string, min_length: 2, ext: [some_ext_opt: ...]}

Verbose, but core opts stay strictly typed. Everything inside :ext is keyword() and available to extensions. But feels a bit artificial, like a pattern grabbed from other programming languages where type system doesn’t allow anything else.

Option 3. Just keyword()

Give up on typing opts and simply document allowed keys in @typedoc.

I think the first option is a good mix between 2nd and 3rd.
What would you pick? Is there an idiomatic shape at all?

2

I would use maps :slight_smile:

Otherwise yeah, treat the specs as hints, and add good documentation on top of it.

3

This sounds exactly like the problem the core team had with Inspect protocol, resolved by Inspect.Optsstruct. I wouldn’t reinvent a wheel. Mold.Opts or like sounds good.

@type t() :: %Mold.Opts{
  …,
  custom_options: keyword(),
  …
}
4

Maps aren’t that nice due to lack of syntactic sugar
{:string, min_length: 1} vs {:string, %{min_length: 1}}

I like this idea the most, especially when there is a warning in docs:

Typespecs may be phased out as the set-theoretic type effort moves forward.

If new set-theoretic types could express such types – good. Otherwise, we won’t lose anything. I think it’s good to design the API around DX and Elixir’s expressiveness, not around the limitations of the current typespec tooling. If typespec can’t express it cleanly then typespec, not the API, should give way.

c’mon, that’s our job :smiley:

so, that’s, basically, option 2 from my list. Having a struct is not a necessary thing here, as public API for inspect/2 still uses a keyword list.

5

Correct me if I’m wrong, but you asked about the idiomatic solution. I honestly don’t know what would be more idiomatic than the language core. It accepts keyword() for convenience, but it immediately raises when keys are not known.

That is detectable by the typing system, unlike map().

6

Man, I’d love if Jason also supported this but the options are opaque and you can’t easily thread metadata when encoding.

7

I am not sure I understand what you do mean.

Jason supports encoding of whatever, and the only opaque thing there is custom_options, which might be explicitly asked to form some expected shape.

I have implemented Jason.Encoder for my structs gazzillion times and it’s perfectly handling metadata and propagates it all the turtles down the encoding.

8

Don’t compromise correctness to save typing three characters, especially in the AI coding era. If a map is what you need then just use that.

9

Syntactic sugar is important; Developer ergonomics is what make Elixir popular.

Having said that, my opinion is to use keywords when the user will provide often 0, at most 3 options. The OP’s pattern is clearly beyond that.

10

If you want to stick with the keyword list, I’d write it like this:

@type string_type :: :string | {:string, string_type_opt}
@type string_type_opt :: {:trim, boolean} | {:nilable, boolean} | ... | {atom, term}

However, with an open keyword list like this, there’s a collision risk if someone picks an option key that is later added to the library. To prevent that, it’s better to add a dedicated key under which custom options can be added. I’m not sure what the most common name is in the Elixir ecosystem. I’ve seen both custom_options and extra. custom_options seems clearer, and it’s used by Inspect.Opts, as mentioned above.

@type string_type :: :string | {:string, string_type_opt}
@type string_type_opt :: {:trim, boolean} | {:nilable, boolean} | ... | {:custom_options, keyword}

And with shared options:

@type string_type :: :string | {:string, string_type_opt | shared_opt}
@type string_type_opt :: {:trim, boolean} | {:nilable, boolean} | ... | {:custom_options, keyword}
@type shared_opt :: {:something, boolean} | ...
11 
@type string_type_option_default() ::
    {:trim, boolean()}
    | {:format, Regex.t()}
    | {:min_length, non_neg_integer()}

@type string_type(t) :: {:string, string_type_option_deafult() | t} | :string

@type string_type() :: {:string, string_type_option_deafult()} | :string
12

Disclaimer: I’m still learning, so not an Elixir expert.

This second option seems the best to me. Even if the type system can make sense of a big bag of atoms, having things namespaced like this makes things easier to reason about for everyone as long as it’s not nested imo.

Another option would be to allow the user to register their custom options with the lib via a macro, so that all options can be treated the same and verified at compile time by checking them against the list of registered atoms.

13

Core doesn’t actually raise on unknown keys here. Both inspect/2 and String.split/3 just ignore them:

iex> inspect(1, my_option: 1)
"1"

iex> String.split("Hello World", " ", test: 1)
["Hello", "World"]

custom_options option was only added in 1.9.0. The struct itself existed long before that. A struct is not an open map by definition, so when the need for caller-supplied extras showed up, the only place to put them was a separate custom_options field. So I wouldn’t read it as evidence that the closed/struct approach is the idiomatic one. There wasn’t really a choice given the existing shape.

Logger is in the standard library too, and it goes the other way from Inspect.Opts. It uses a single flat keyword for everything. Name collisions are possible, but they’re handled by convention and documentation, not by validation or a separate field.

Logger.info("hello", ansi_color: :green, whatever: 1)

That’s more about reading, rather than typing.
I wouldn’t say map is correct in this case and keyword isn’t. Theoretically, there might be options in the future that’ll benefit from duplicate keys :thinking:

You forgot about wrapping type to a list :slight_smile: [EDIT: I misread shared options as open options and came to this example] With shared options it makes sense to use parameterized open_keyword type:

@type open_keyword(t) :: [t | {atom(), term()}]

@type string_option :: {:trim, boolean()} | ...
@type shared_option :: {:nilable, boolean()} | ...

@type string_type :: :string | {:string, open_keyword(string_option() | shared_option())}

Actually… it looks nice :smiley:

Yeah, I know Req library uses this approach. But Mold positions itself as a lightweight, shape-first library where schemas are plain data and ergonomics matter more than catching mistyped option names – String.split/3 doesn’t check options and that’s not the end of the world.

14

Yeah, I am the author of the PR allowing custom options. AFAIR, my initial intent was to treat anything unknown as custom, but there was a rock-solid argument we should distinguish custom stuff and the expected one.

Also, inspect/2 is a Kernel helper, I was talking about the protocol implementation.