Is it possible to define structs at runtime

Questions / Help
, , , ,
1

I’m considering writing an integration with Salesforce. Salesforce has SObjects of various types like Case, Quote, Lead, CustomObject__c, etc. These SObjects are like database tables with their own schemas. I’d like to convert the JSON responses I get back from Salesforce into structs that represent the SObject type. For example, %Case{}, %Quote{} instead of something more generic and known at compile time like %SObject{type: "Case"}. There are hundreds of SObject types, so defining them all isn’t reasonable and wouldn’t cover custom SObjects or custom SObject fields not provided by Salesforce itself.

Is there any way to define structs at runtime to achieve something like %{SObjectName}{}?

2

What is the value of having those schemas if you are defining them at runtime?

The whole reason behind schemas is to have more compile-time guarantees, if you generate them at runtime there is no way your codebase will have any idea of those schemas.

Maybe the question was not addressed correctly and you are looking for one of the following things:

  • validation - validate the incoming data based on some runtime rules. This can be done with schemaless changesets from ecto;
  • code-generation - generate those schemas in your codebase based on a spec given by Salesforce. This case is possible however you will have to venture into metaprogramming and that is not the most welcome field for beginners.
1 Like
3

Hello and welcome!

There is not, one of the whole points of structs is to provide compile time checks, including against keys.

If you are going for dynamic runtime, maps should be largely fine. You can still pattern match. I guess the only problem you might face is accidentally adding a wrong key after initialization but you could guard against that with custom functions, I suppose (though probably not worth it).

2 Likes
4

There’s no database involved, so schemas are necessary. This is just an integration wrapper for Salesforce APIs. The value is pattern matching on the struct itself and knowing its SObjectType without looking at a field value. I can make a generic SObject struct and have a type field if that is impossible.

5

Thanks for the feedback. I suspected such, but I wanted to check.

6

It highly depends on what you are looking for, first of all ecto can be used without a database, and it is widely used for data validation. In your case, from what I understand the protocol defines the type of the structure in one of its fields.

As long as you can define those types in your codebase, you can always use ecto + polymorphic embed.

5 Likes
7

Thanks for the information. I consider this one. I hadn’t heard about polymorphic embeds before.

1 Like
8

Yes, you can create structs in runtime. Consider this code

def create_struct(name, fields) do
  defmodule name do
    defstrcut fields
  end
end

This is a dark magic, but there is no need to limit yourself, be expressive, break laws, make people be afraid of yourself and make them treat your code with fear and respect.

To match on such structs in runtime, consider something like

name = Case
create_struct(name, [x: 1, y: 2])
...
case something do
  %^name{} ->
    ...
end
3 Likes
10

@chazwatkins Oh boy, do not follow the advice above. You’re opening yourself up to atom overflow issues and as well as a host of warnings about module redefinitions you’ll have to code around.

4 Likes
15

The use case I’m looking at wouldn’t go anywhere near the atom limit as there are only, at most, one to two thousand SObject types in complex Salesforce orgs. You’re likely only using a subset of them, a few hundred at most for large organizations.

It doesn’t look like there is a solid way to convert a string to a module name without a prefix of : like %:Case{}. It’d be great for using them with Protocols, but it’s just as easy to pattern match on something like %SObject{type: "Case", fields: ...} and dispatch to the corresponding Case business logic.

It’s good to know that this is possible, though. I completely understand the point of not using it if the input is unbound, where it could overflow the atom table.

1 Like
16

The problem lies more in that you are dynamically creating modules at run time. Once those are created, they’re there. If you need to change them and your users are, for instance, using hot code reloading, they won’t get your changes. As far as atom overflow goes, sure, there may be a finite number of entities you are dealing with, but it’s now something you have to think about and secure against. In short, this is not well documented for a reason and why I didn’t mention it in the first place. If you’re ok mixing artsy code into your business logic, that’s on you. It’s not something I would deem responsible in a library you are wanting people to use, though. I personally would never use something if I knew it was doing this.

2 Likes
17

I’ve worked with Salesforce a bit. The object types are known and constant, why do you need runtime structs? You could just create your own structs for the objects you’re actually using - in my experience more like 10, rather than 2000.

18

I thought it would be nice to return the SObjectType as a struct by default, but I have determined %SObject{type: …} would be sufficient.

Doing an Ecto Adapter or Ash DataLayer with Salesforce as the backend would be cool. Then you could create Ecto.Schemas or Ash.Resources for only the SObjects you’re actually using.

19

If you wanted to go meta, I did wonder if you could build a macro that called the introspection APIs and use that to create the structs at compile time. I don’t know if you can call APIs in macros, and I assume it’s bad practice, but could be an interesting experiment.

20

For folks looking to do something similar in the future, my suggestion here would be to dump some representation (ideally a json schema) of the salesforce objects into a file, and then derive your structs from that at compile time. Then, rebuild and redeploy as necessary. If you actually need these structs, of course

I agree with @D4no0 and @sodapopcan that in this case you’re not getting any benefit from dynamic runtime struct definition, in exchange for “weirdness” later down the line.

If what you’re looking for is something syntactic to signal that you’re operating on a salesforce object of X type, or to pattern match on that in some interesting or novel way, consider a macro. To be clear, I’m not suggesting that you do this necessarily, just showing that you don’t need structs to get customized pattern matching. Elixir has effectively all the facilities you need to modify the language.

defmodule SObject do
  defstruct [:type, data: %{}]

   defmacro so({:%, _, [type, {:%{}, meta, keys]}) do
     string_keys = 
       Enum.map(keys, fn {key, value} -> 
         {to_string(key), value}
       end)

     contents_with_string_keys = {:%{}, meta, keys}

     quote do
       %SObject{type: unquote(to_string(type)), data: unquote(contents_with_string_keys)}
     end
   end
end

Then you can do this kind of thing

defmodule SomeModule do
  import  SObject

  def do_something(so(%Case{number: case_number})) do
    case_number
  end
end

That all works without ever having to turn salesforce data from strings to atoms. You could then build in a known set of fields that exist for certain builtin objects (or don’t exist) and validate that at compile time with your macro.

Whether you ultimately want to use something like this is 100% up to you, but the main point I want to get across is that, with macros, you can often approach syntax that you like, without having to sacrifice how the underlying system works.

6 Likes