ChatGPT generated code for eval_string alternative

Questions / Help
,
1

Was talking with ChatGPT, is the following code a valid/good replacement for Code.eval_string?

def create_function(name, args, code) do
  ast = Macro.escape(code, [:strict])
  code = Macro.to_string(ast)
  {name, Code.string_to_quoted(code)}
end

This code would run in a template engine of sorts to load function schema/definitions from a json file.

defmodule Engine do
  def start(schema_files) do
    functions =
      schema_files
      |> Enum.map(&parse_schema_file/1)
      |> Enum.map(&create_function/1)
      |> Map.new

    # Store the functions in the `functions` map
    Process.put(:functions, functions)
  end
end

ChatGPT explanation:

In this example, the Macro.escape/2 function is used to compile the code string into an AST. The :strict option is passed as the second argument to Macro.escape to ensure that the code is compiled in strict mode, which means that all variables must be declared before they are used. The Macro.to_string/1 function is then used to convert the AST back into a string of Elixir code. Finally, the Code.string_to_quoted/1 function is used to parse the code string and return a tuple containing the function name and the function itself.

This approach has the advantage of being more efficient than Code.eval_string, since the code is only compiled once and then stored as a quoted expression, rather than being compiled every time it is called. It also has the advantage of being safer than Code.eval_string, since it does not allow arbitrary code to be executed. However, it is a bit more complex to use and requires a deeper understanding of Elixir’s macro system.

2

No, this simply does not work. Code.string_to_quoted/1 does not do what it says it does. GPT confuses parsing to AST with compilation, and the Process.put bit is pure fiction.

3

As is pretty common with ChatGPT, some parts of this are outright hallucinations:

  • there is no strict option to Macro.escape
  • Macro.escape does not compile strings into ASTs; if you pass it a string it will return it unchanged
  • Code.string_to_quoted (no !) does not return an AST, it returns either {:ok, ast} or an error tuple
  • the result of Code.string_to_quoted has not been “compiled”, it has been parsed into an AST ready to be compiled

But some parts are mostly correct. There is a relationship between Code.eval_string and Code.string_to_quoted! + Code.eval_quoted because they both make use of the same underlying compiler machinery.

Code.eval_string has three steps:

  • prepare an env with env_for_eval
  • convert the supplied string to forms with :elixir.string_to_quoted!
  • evaluate the forms by calling :elixir.eval_forms

Code.string_to_quoted! does only the middle step:

while Code.eval_quoted does the first and last steps:

A minor detail: :elixir.eval_quoted does a tiny bit of argument reformatting before calling :elixir.eval_forms

As to your original question, the above analysis should make clear that Code.string_to_quoted! at boot + Code.eval_quoted invoked over and over will be faster than Code.eval_string invoked over and over since it does the string_to_quoted! part once instead of over and over (and otherwise does the same thing).

An even faster alternative would be:

  • at boot, parse all the AST once with Code.string_to_quoted!
  • at boot, take all those AST fragments and stitch them together into a module definition
  • call the functions over and over with apply, at normal speed for compiled code

IMO a bigger question is “what happens when one of these functions is configured with a fatal error, like closing a block with ned?”

  • Code.eval_string will crash with an exception when you try to run the function
  • Code.string_to_quoted! called at boot-time will crash with an exception and prevent application startup
  • Code.string_to_quoted (note no !) will return an error tuple explaining the situation

Your application’s specific needs should help you decide which of these is desirable / acceptable.

4 Likes
4

Thanks for those answers!

As a general design idea, I am thinking about how to dynamically load/run code on a general purpose engine thats easy to scale up.

  • user writes a function in phoenix frontend (using a custom SDK)
  • function code is stored in spec file as string or file is saved on shared disk somewhere
  • engine dynamically loads spec file or imports disk file and runs code

ChatGPT also suggested Code.require_file/2 or Code.require_file/3 for dynamic imports

5

Is your goal to play with ChatGPT, or is your goal to do this dynamic function thing? Because I don’t think interacting with ChatGPT is going to help very much here. It is far more competent than it has any right to be, but it’s still pretty incompetent, particularly if you want to do something off the beaten path.

And accepting and running user defined functions is definitely off the beaten path. If you want a user to write functions, I would strongly suggest that you pick say LUA and run that in GitHub - rvirding/luerl: Lua in Erlang and not try to compile actual elixir code from stuff users provide, because compiling running user code as elixir code is a security nightmare.

6

My goal is to do both and have fun :stuck_out_tongue:. I’ll read up on Luerl as that sounds cool.

For this dynamic defined function idea though, security is less of a concern if the engine runs in a container.