tomekowal
Change of behaviour in Elixir 1.18 macros: (ArgumentError) tried to unquote invalid AST
Hi!
I have a piece of code that parses some json and creates validations from them. In one place, I need to create a regular expression from string.
In Elixir 1.17 this worked
iex(1)> regex_string = "[0-9]{4}"
"[0-9]{4}"
iex(2)> quote do unquote(~r/#{regex_string}/) end |> Macro.to_string
"~r/[0-9]{4}/"
In Elixir 1.18, I get an error
iex(1)> regex_string = "[0-9]{4}"
"[0-9]{4}"
iex(2)> quote do unquote(~r/#{regex_string}/) end |> Macro.to_string
** (ArgumentError) tried to unquote invalid AST: ~r/[0-9]{4}/
Did you forget to escape term using Macro.escape/1?
(elixir 1.18.2) src/elixir_quote.erl:542: :elixir_quote.argument_error/1
iex:3: (file)
I thought, it might be because of interpolation, so I tried interpolating outside of quote:
iex(1)> regex_string = "[0-9]{4}"
"[0-9]{4}"
iex(2)> regex = ~r/#{regex_string}/
~r/[0-9]{4}/
iex(3)> quote do unquote(regex) end |> Macro.to_string
** (ArgumentError) tried to unquote invalid AST: ~r/[0-9]{4}/
Did you forget to escape term using Macro.escape/1?
(elixir 1.18.2) src/elixir_quote.erl:542: :elixir_quote.argument_error/1
iex:7: (file)
The error suggests using Macro.escape, but it doesn’t make sense to me. Unquote should already escape and indeed, I would the generated code will have AST of the code isntead of the code.
iex(1)> quote do ~r/[0-9]{4}/ end
{:sigil_r, [delimiter: "/", context: Elixir, imports: [{2, Kernel}]],
[{:<<>>, [], ["[0-9]{4}"]}, []]}
iex(2)> regex = ~r/#{regex_string}/
~r/[0-9]{4}/
iex(3)> escaped = Macro.escape(regex)
{:%{}, [],
[
__struct__: Regex,
opts: [],
re_pattern: {:{}, [],
[
:re_pattern,
0,
0,
0,
<<69, 82, 67, 80, 109, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 255, 255, 255,
255, 255, 255, 255, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 64, 0, ...>>
]},
re_version: {"8.44 2020-02-12", :little},
source: "[0-9]{4}"
]}
iex(3)> quote do unquote(escaped) end |> Macro.to_string
"%{\n __struct__: Regex,\n opts: [],\n re_pattern:\n {:re_pattern, 0, 0, 0,\n \"ERCPm\\0\\0\\0\\0\\0\\0\\0\\x01\\0\\0\\0\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0@\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\x83\\0)n\\0\\0\\0\\0\\0\\0\\xFF\\x03\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0m\\0\\x04\\0\\x04x\\0)\\0\"},\n re_version: {\"8.44 2020-02-12\", :little},\n source: \"[0-9]{4}\"\n}"
I am starting to wonder if Elixir 1.18 became more strict or is it a bug in Elixir?
Marked As Solved
Eiji
In that case I guess you can compile the regular expression in compile time, then inspect it to get a string with sigil and finally convert a string to the AST.
Here is a complete script I have prepared:
Mix.install([:ecto])
defmodule MyLib.Schema do
@doc """
Generates the changeset function.
The `name` is a changeset function name. Default to `:changeset`.
The `properties` is an `Elixir` map with `atom` keys.
The `func` is an optional funtion that could be used by the developer to add other validations.
"""
defmacro changeset(name \\ :changeset, properties, func \\ nil) do
quote bind_quoted: [func: func, module: __MODULE__, name: name, properties: properties] do
pipe =
properties
|> module.from_properties(__MODULE__)
|> module.pipe_func(func)
# pipe
# |> Macro.to_string()
# |> Code.format_string!()
# =>
# struct
# |> cast(params, [:four_digit_code])
# |> validate_format(:four_digit_code, ~r/[0-9]{4}/)
struct = Macro.var(:struct, __MODULE__)
params = Macro.var(:params, __MODULE__)
def unquote(name)(unquote(struct), unquote(params)) do
unquote(pipe)
end
end
end
@doc false
def from_properties(properties, module) do
fields = Map.keys(properties)
struct = Macro.var(:struct, module)
params = Macro.var(:params, module)
cast =
quote do
cast(unquote(params), unquote(fields))
end
pipe = ast_pipe(struct, cast)
Enum.reduce(properties, pipe, &from_field_properties/2)
end
@supported_validators ~w[pattern]a
defp from_field_properties({field, properties}, acc) do
properties
|> Map.take(@supported_validators)
|> Enum.reduce(acc, fn {key, value}, acc ->
ast_pipe(acc, validator(field, key, value))
end)
end
defp validator(field, :pattern, value) do
quote do
validate_format(unquote(field), unquote(quoted_regex_sigil(value)))
end
end
defp quoted_regex_sigil(source) do
source
# creates regular expression from source
|> Regex.compile!()
# inspect returns a string with a sigil
|> inspect()
# escaped AST form
|> Code.string_to_quoted!()
end
def pipe_func(left, nil), do: left
def pipe_func(left, func) do
ast_pipe(
left,
quote do
then(unquote(func))
end
)
end
defp ast_pipe(left, right) do
quote do
unquote(left) |> unquote(right)
end
end
end
defmodule MyApp.Schema do
use Ecto.Schema
import Ecto.Changeset
import MyLib.Schema
embedded_schema do
field(:four_digit_code, :string)
end
changeset(%{four_digit_code: %{pattern: "[0-9]{4}"}})
end
defmodule Example do
def sample do
MyApp.Schema.changeset(%MyApp.Schema{}, %{four_digit_code: "0007"})
# => %Ecto.Changeset{valid?: true}
end
end
You should be able to easily adapt the example code to your needs.
Also Liked
tomekowal
Thank you!
This part is exactly what I was missing!
defp quoted_regex_sigil(source) do
source
# creates regular expression from source
|> Regex.compile!()
# inspect returns a string with a sigil
|> inspect()
# escaped AST form
|> Code.string_to_quoted!()
end
I didn’t realise, Code has string_to_quoted!. That will ensure correct representation in the final generated code.
Brilliant answer!
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance










