I’m generating some code that currently relies on generating multiple function heads with guards and leveraging a lot of pattern matching. My test suite generates over 30,000 variations of the same function. Thats if compilation ever finishes In a practical deployment and in development the number is at least 2 orders of magnitude lower and it’s not an issue. So somewhere between ~500 functions with the same signature but different heads and ~30,000 or so compilation goes up exponentially and I’ve been letting it compile for over and hour now on my CI machine with no completion.
Whats the considered view of a practical maximum of functions / function heads before this strategy tops out?
Well I can’t answer your question but you probably answered it yourself. The thing I wanted to point out is that there still may be a way to optimize things: by removing guard clauses.
I am uncertain if it will help with the compilation phase, but it will certainly help with the execution. Compiler can better optimize pattern matching this way, and having guards in function clauses, because the resulting program no longer has to test against each and single guard expression before doing the pattern matching.
Whenever that’s an option for you - I don’t know. But I would try to look into how performance looks like once you do have only pattern matching for parameters and not guards - when the number of clauses/guards is great.
It’s also worth mentioning that for reasons I don’t fully understand myself that it seems like having a lot of local variables in a module body can lead to extraordinary-compilation times. I would limit how much logic you do inside the module body itself and instead try to hand off to a macro which then returns AST which just has your def parts in it.
Put another way, the issue here seems less likely to be the number of function heads and more likely to be the how they’re being created. Are you able to gist the module in question?
The body of the function is empty right now. I had an inkling this might be an issue, hence the research. Am curious to understand more about the sources of slower than expected compilation. I have one other situation (not nearly as extreme) where I am generating 511 functions where the function is nothing more than a data bag - but of a fairly large map (contains the definitions of about 300 currencies in locale specific language). That module takes about 3 minutes to compile.
I have taken out the last generator in the list comprehension and the guard clauses as @hubertlepicki suggested and it gets compile time down to single digit minutes rather than four hours. But I’m very curious where the time goes - its not complex code!
For efficiency you want to order the heads well so it can skip large chunks at a time. If they are truly large enough then an ETS lookup might even end up faster (I’d benchmark that to verify, never tried)…
In a practical deployment and in development the number is at least 2 orders of magnitude lower and it’s not an issue. So somewhere between ~500 functions with the same signature but different heads and ~30,000 or so compilation goes up exponentially and I’ve been letting it compile for over and hour now on my CI machine with no completion.
