Long compile times

I have a package that takes a long time to compile because it uses the ExSpirit parser, which does lots of compile time optimizations (but which at runtime is actually very fast).

The package will have lots of parser modules to parse and lex programming languages (one module per programming language), and individual omdules might take well over 15s to compile. Currently there are 2 such modules, and compile times are already a little long.

If I ever get to have, say, 50 modules (support for 50 languages). Compile times might baloon to over 750s (12mins). This is unacceptable for someone who only needs a couple of programming languages, right?

So I guess I’ll have to go the nodejs route of publishinng micropackages -.- (have a top level package, a module with the common functionality and one package per language). I’dreslly like to avoid this…

Is there any way of distributing “precompiled” packages on hex? I don’t mind having my machine running a build for 12mins but I wouldn’t want to impose that on users

In general, I’d prefer to have per language extensions instead of having to have a plentora of stuff available which I do not need at all. Also it would make it much easier to use experimental lexers as a monolithic application would do.

And last but not least, it would take the burden from you to actually keep all lexers maintained and workable with the current version, and also you wouldn’t need to bundle a new release for every language added or fixed.

I was in a situation where I needed a pygments parser for a study related project that had been merged, but wasn’t in the latest released version. Due to restrictions made by my university, I wasn’t allowed to use “unreleased” software as well. So I had to leave those bits unhighlighted at the end, which wasn’t a problem at all, but didn’t look as good as I wanted it.

1 Like

Well, I guess it’s not too hard to do. I can have a core package with a user-facing module an a core module. Lexers would require the core module, and the user interface would call lexers by module name. It’s more packages but as you say, they can be updated independently.

Well you could break up the rules into many reusable module parts? That would help a lot for now…

But otherwise I’d think a base package, that contains maybe nothing, then little packages that depend on ‘it’ and just register their own language, so a package per language would be best?

I’m really leaning toward building a DSEL via something expression template’y…

1 Like

Yes, I’m leaning towards this solution. I might make each lexer a behavior or something, and the user would call it by module name. That way I wouldn’t even need to register anything. Currently the lexers are almost a behavior anyway…

That doesn’t help with compile times for production, only for dev. In production, compile times will still be long.

Eh not as bad though, considering I can entirely skip unused functionality then it could potentially be a lot faster, however it might change the API (not too much, maybe not at all).

As long as I still have seq, alt and repeat, lexeme, char and tag I don’t really care.

Lol, yep.

Then go wild :smile:

And regarding the “registration” thing. Do you think registring languages would be important? Or just allowing the user to pass module names would be enough? A central registry would require some overhead like setting an option at the :makeup app’s config. Do you have any concrete use case for “registration”?

I’d probably just define a namespace for languages to take, like Markup.Parsers so Elixir would be, for example, Markup.Parsers.Elixir, and the user can just pass in :Elixir, then you can make the module atom (to existing of course) and use it directly. That is what I do in things like safescript and such.

Yes, that’s just what I had in mind.

1 Like