tomekowal

tomekowal

Change of behaviour in Elixir 1.18 macros: (ArgumentError) tried to unquote invalid AST

Hi!

I have a piece of code that parses some json and creates validations from them. In one place, I need to create a regular expression from string.

In Elixir 1.17 this worked

iex(1)> regex_string = "[0-9]{4}"
"[0-9]{4}"
iex(2)> quote do unquote(~r/#{regex_string}/) end |> Macro.to_string
"~r/[0-9]{4}/"

In Elixir 1.18, I get an error

iex(1)> regex_string = "[0-9]{4}"
"[0-9]{4}"
iex(2)> quote do unquote(~r/#{regex_string}/) end |> Macro.to_string
** (ArgumentError) tried to unquote invalid AST: ~r/[0-9]{4}/
Did you forget to escape term using Macro.escape/1?
    (elixir 1.18.2) src/elixir_quote.erl:542: :elixir_quote.argument_error/1
    iex:3: (file)

I thought, it might be because of interpolation, so I tried interpolating outside of quote:

iex(1)> regex_string = "[0-9]{4}"
"[0-9]{4}"
iex(2)> regex = ~r/#{regex_string}/
~r/[0-9]{4}/
iex(3)> quote do unquote(regex) end |> Macro.to_string
** (ArgumentError) tried to unquote invalid AST: ~r/[0-9]{4}/
Did you forget to escape term using Macro.escape/1?
    (elixir 1.18.2) src/elixir_quote.erl:542: :elixir_quote.argument_error/1
    iex:7: (file)

The error suggests using Macro.escape, but it doesn’t make sense to me. Unquote should already escape and indeed, I would the generated code will have AST of the code isntead of the code.

iex(1)> quote do ~r/[0-9]{4}/ end
{:sigil_r, [delimiter: "/", context: Elixir, imports: [{2, Kernel}]],
 [{:<<>>, [], ["[0-9]{4}"]}, []]}
iex(2)> regex = ~r/#{regex_string}/
~r/[0-9]{4}/
iex(3)> escaped = Macro.escape(regex)
{:%{}, [],
 [
   __struct__: Regex,
   opts: [],
   re_pattern: {:{}, [],
    [
      :re_pattern,
      0,
      0,
      0,
      <<69, 82, 67, 80, 109, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 255, 255, 255,
        255, 255, 255, 255, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 64, 0, ...>>
    ]},
   re_version: {"8.44 2020-02-12", :little},
   source: "[0-9]{4}"
 ]}
iex(3)> quote do unquote(escaped) end |> Macro.to_string
"%{\n  __struct__: Regex,\n  opts: [],\n  re_pattern:\n    {:re_pattern, 0, 0, 0,\n     \"ERCPm\\0\\0\\0\\0\\0\\0\\0\\x01\\0\\0\\0\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0@\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\x83\\0)n\\0\\0\\0\\0\\0\\0\\xFF\\x03\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0m\\0\\x04\\0\\x04x\\0)\\0\"},\n  re_version: {\"8.44 2020-02-12\", :little},\n  source: \"[0-9]{4}\"\n}"

I am starting to wonder if Elixir 1.18 became more strict or is it a bug in Elixir?

Marked As Solved

Eiji

Eiji

In that case I guess you can compile the regular expression in compile time, then inspect it to get a string with sigil and finally convert a string to the AST.

Here is a complete script I have prepared:

Mix.install([:ecto])

defmodule MyLib.Schema do
  @doc """
  Generates the changeset function.

  The `name` is a changeset function name. Default to `:changeset`.
  The `properties` is an `Elixir` map with `atom` keys.
  The `func` is an optional funtion that could be used by the developer to add other validations. 
  """
  defmacro changeset(name \\ :changeset, properties, func \\ nil) do
    quote bind_quoted: [func: func, module: __MODULE__, name: name, properties: properties] do
      pipe =
        properties
        |> module.from_properties(__MODULE__)
        |> module.pipe_func(func)

      # pipe
      # |> Macro.to_string()
      # |> Code.format_string!()
      # =>
      # struct
      # |> cast(params, [:four_digit_code])
      # |> validate_format(:four_digit_code, ~r/[0-9]{4}/)

      struct = Macro.var(:struct, __MODULE__)
      params = Macro.var(:params, __MODULE__)

      def unquote(name)(unquote(struct), unquote(params)) do
        unquote(pipe)
      end
    end
  end

  @doc false
  def from_properties(properties, module) do
    fields = Map.keys(properties)
    struct = Macro.var(:struct, module)
    params = Macro.var(:params, module)

    cast =
      quote do
        cast(unquote(params), unquote(fields))
      end

    pipe = ast_pipe(struct, cast)
    Enum.reduce(properties, pipe, &from_field_properties/2)
  end

  @supported_validators ~w[pattern]a

  defp from_field_properties({field, properties}, acc) do
    properties
    |> Map.take(@supported_validators)
    |> Enum.reduce(acc, fn {key, value}, acc ->
      ast_pipe(acc, validator(field, key, value))
    end)
  end

  defp validator(field, :pattern, value) do
    quote do
      validate_format(unquote(field), unquote(quoted_regex_sigil(value)))
    end
  end

  defp quoted_regex_sigil(source) do
    source
    # creates regular expression from source
    |> Regex.compile!()
    # inspect returns a string with a sigil
    |> inspect()
    # escaped AST form
    |> Code.string_to_quoted!()
  end

  def pipe_func(left, nil), do: left

  def pipe_func(left, func) do
    ast_pipe(
      left,
      quote do
        then(unquote(func))
      end
    )
  end

  defp ast_pipe(left, right) do
    quote do
      unquote(left) |> unquote(right)
    end
  end
end

defmodule MyApp.Schema do
  use Ecto.Schema

  import Ecto.Changeset
  import MyLib.Schema

  embedded_schema do
    field(:four_digit_code, :string)
  end

  changeset(%{four_digit_code: %{pattern: "[0-9]{4}"}})
end

defmodule Example do
  def sample do
    MyApp.Schema.changeset(%MyApp.Schema{}, %{four_digit_code: "0007"})
    # => %Ecto.Changeset{valid?: true}
  end
end

You should be able to easily adapt the example code to your needs.

Also Liked

tomekowal

tomekowal

Thank you!

This part is exactly what I was missing!

  defp quoted_regex_sigil(source) do
    source
    # creates regular expression from source
    |> Regex.compile!()
    # inspect returns a string with a sigil
    |> inspect()
    # escaped AST form
    |> Code.string_to_quoted!()
  end

I didn’t realise, Code has string_to_quoted!. That will ensure correct representation in the final generated code.

Brilliant answer!

Where Next?

Popular in Questions Top

hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a &gt; b) do {:ok, "a"} end if (a &lt; b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

Other popular topics Top

hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a &gt; b) do {:ok, "a"} end if (a &lt; b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
274 41989 114
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
AngeloChecked
What learn first? Rust or Elixir Hi Elixir community! I’m here because i want learn a new language. I’m a junior developer and mainly i ...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New

We're in Beta

About us Mission Statement