How do I pass variables to Protox's __using__ macro?

scott2e

Apologies, this question has been asked in numerous variations before, but it seems I haven’t found the answer I’m looking for.

I’ve found myself in a situation where I need to support multiple versions of protobuf messages. This was due to some poor decisions by the vendor in changing protobuf files for different versions of software they’re running. i.e. they’ve kept the same protobuf files between different software versions but changed some data types on existing fields in messages. I need to be able to interface with both versions of their software..

So I need to generate both sets of protobuf files but with different namespaces

I did start doing this (which works) but it’s very tedious:

defmodule MyModule do
  use Protox, files: ["./proto/v1/foo.proto"], namespace: V1
  use Protox, files: ["./proto/v2/foo.proto"], namespace: V2
  ...
  use Protox, files: ["./proto/vN/foo.proto"], namespace: VN
end

I’ve attempted this (does not work):

defmodule MyModule do
  MyOtherModule.get_files_and_namespaces()
  |> Enum.each(fn {files, namespace} ->
    use Protox, files: files, namespace: namespace
  end)
end

This responds with the usual:

variable "files" does not exist and is being expanded to "files()", please use parentheses to remove the ambiguity or change the variable name

I’ve attempted to use unquote/1

defmodule MyModule do
  MyOtherModule.get_files_and_namespaces()
  |> Enum.each(fn {files, namespace} ->
    use Protox, unquote(files: files, namespace: namespace)
  end)
end

This responds with:

== Compilation error in file lib/mymodule.ex ==
** (CompileError) nofile:4: unquote called outside quote
    (elixir 1.12.3) lib/code.ex:1036: Code.eval_quoted/3
    (protox 1.7.0) expanding macro: Protox.__using__/1
    lib/mymodule.ex:4: MyModule (module)
    (elixir 1.12.3) expanding macro: Kernel.use/2
    lib/mymodule.ex:4: MyModule (module)
    (elixir 1.12.3) expanding macro: Kernel.|>/2

The __using__/1 macro from the Protox module looks like this:

 defmacro __using__(opts) do
    {opts, _} = Code.eval_quoted(opts)

    {paths, opts} = get_paths(opts)
    {files, opts} = get_files(opts)

    {:ok, file_descriptor_set} = Protox.Protoc.run(files, paths)

    %{enums: enums, messages: messages} = Protox.Parse.parse(file_descriptor_set, opts)

    quote do
      unquote(make_external_resources(files))
      unquote(Protox.Define.define(enums, messages, opts))
    end
  end

Can anyone relieve me from my macro ignorance?

Thank-you in advance

7 comments

#macros #troubleshooting #protobuf #protox

0 1072 7

2022-09-19 02:57:39 UTC

Showing Posts 1 to 7

Show Best Posts
Show All (oldest first)
Show All (newest first)

zachallaun

Hi Scott!

On my phone so I can’t give in depth code examples, but a few notes to hopefully nudge you in the right direction:

The important thing you remember with macros is that they run at compile time (a step called macro expansion). So in your attempt with Enum.each, the call to use is being expanded before the loop runs. As seen in the pasted using code, they then evaluate the passes options — at that point, files isn’t bound to anything because the function isn’t being run simultaneously.

So, the question now is how do we get those values at macro expansion time? The answer is macros! What you need is to define a macro that generates quoted expressions containing the use calls that you need, with the right arguments injected into the expressions. You might have something like this:

defmacro define_protox_versions do
  for {files, namespaces} <- get_files_and_namespaces() do
    quote do
      use Protox, files: unquote(files), namespace: unquote(namespace)
    end
  end
end

and then you can do:

defmodule MyModule do
  require MyOtherModule
  MyOtherModule.define_protox_versions()
end

So, now your macro is going to return a list of quoted expressions, injecting them into MyModule. Those expressions contain macros, so they will be expanded as well, but now they also contain the literal lists and namespaces that use Protox needs.

Hopefully this helps!

Post #1

scott2e

Oh nice, that makes total sense!
I’m in bed now (also on mobile) so I’ll try first thing tomorrow.

Thank-you for that amazing response, and on your mobile. What dedication to the community!

Post #2

dimitarvp

Not to be mistaken with the at least 20 answers I posted when on phone and they were all wrong, lol.

But yeah I was about to suggest the same as @zachallaun. Try it and let us know.

Post #3

zachallaun

Yep, this is always the fear it’s amazing how reliant I sometimes am on just being able to quickly try something out in a scratch file.

Post #4

zachallaun

One other thing I’ll mention is this: after you get this working, you may want to consider still co-locating the files/namespaces, or maybe a shortened version (I’m not sure what your theoretical get_files_and_namespaces() is doing).

define_protox_versions(
  v1: V1,
  v2: V2,
  …
  vN: VN
)

As cool as macros are, someone (even yourself sometime later!) will have to go through multiple layers of indirection to figure out which versions are being defined. I don’t begrudge someone wanting to save some keystrokes, but explicit is often better than implicit. Also might make some things easier down the road, like when you decide that V4 was a total buggy piece of crap that absolutely shouldn’t be used — now you can just delete that version from the list instead of having to add some special exclusion somewhere.

Post #5

scott2e

zachallaun:

Hi Scott!

On my phone so I can’t give in depth code examples, but a few notes to hopefully nudge you in the right direction:

The important thing you remember with macros is that they run at compile time (a step called macro expansion). So in your attempt with Enum.each, the call to use is being expanded before the loop runs. As seen in the pasted using code, they then evaluate the passes options — at that point, files isn’t bound to anything because the function isn’t being run simultaneously.

So, the question now is how do we get those values at macro expansion time? The answer is macros! What you need is to define a macro that generates quoted expressions containing the use calls that you need, with the right arguments injected into the expressions. You might have something like this:
defmacro define_protox_versions do
  for {files, namespaces} <- get_files_and_namespaces() do
    quote do
      use Protox, files: unquote(files), namespace: unquote(namespace)
    end
  end
end
and then you can do:
defmodule MyModule do
  require MyOtherModule
  MyOtherModule.define_protox_versions()
end
So, now your macro is going to return a list of quoted expressions, injecting them into MyModule. Those expressions contain macros, so they will be expanded as well, but now they also contain the literal lists and namespaces that use Protox needs.

Hopefully this helps!

OMG it works!! I was stuck on this for longer than I care to admit. Thank-you

Post #6

scott2e

zachallaun:

One other thing I’ll mention is this: after you get this working, you may want to consider still co-locating the files/namespaces, or maybe a shortened version (I’m not sure what your theoretical get_files_and_namespaces() is doing).
define_protox_versions(
  v1: V1,
  v2: V2,
  …
  vN: VN
)
As cool as macros are, someone (even yourself sometime later!) will have to go through multiple layers of indirection to figure out which versions are being defined. I don’t begrudge someone wanting to save some keystrokes, but explicit is often better than implicit. Also might make some things easier down the road, like when you decide that V4 was a total buggy piece of crap that absolutely shouldn’t be used — now you can just delete that version from the list instead of having to add some special exclusion somewhere.

The get_files_and_namespaces() I made just parses the directory hierarchy where the protobufs are stored and determines the version and namespaces based off the names of the directories. But yes, perhaps I should make it more explicit.

Thank-you for the input

Post #7

— All posts loaded —