Working around circular library dependency

[tmbb]

Yes, ex_spirit depends on makeup to generate the docs. This means makeup needs to be specified as a :dev dependency, yes. Other than that ex_spirit has no dependencies at runtime. But yes, you’re right. I had misunderstood you.

[OvermindDL1]

Ooo, that’s an idea, be able to specify a dependency as a run-time only dependency, nothing compile-time needed (nor available at compile-time), that way they can be compiled without depending on each other but all get included at runtime regardless.

[tmbb]

It looks like it doesn’t: https://github.com/elixir-lang/ex_doc/blob/master/lib/ex_doc/cli.ex#L105

And I wouldn’t be comfortable adding a new command line option just to document the only package that can’t be highlighted with makeup… It seems like a very big change for little gain.

[ericmj]

@OvermindDL1

Ah, ugh, that is definitely not standard in the native compiling world… o.O

I feel like you often have this attitude. If you have ideas for how to improve things we are always open PRs or suggestions on how to improve things.

@OvermindDL1

Well in that case, is there any way to define to load a dependency but not actually ‘depend’ on it as a dependency? That would be a perfect use-case for ex_doc and things like this for example. ^.^

@OvermindDL1

Ooo, that’s an idea, be able to specify a dependency as a run-time only dependency, nothing compile-time needed (nor available at compile-time), that way they can be compiled without depending on each other but all get included at runtime regardless.

But there is a compile time dependency, changing the dependency declaration doesn’t change this fact. Not sure what you mean by loading a dependency but not depending it.

@tmbb

It looks like it doesn’t: ex_doc/lib/ex_doc/cli.ex at main · elixir-lang/ex_doc · GitHub

And I wouldn’t be comfortable adding a new command line option just to document the only package that can’t be highlighted with makeup… It seems like a very big change for little gain.

Your own PR for this addition hasn’t even been merged yet so I don’t think flags exist yet Support for custom markdown processors by tmbb · Pull Request #787 · elixir-lang/ex_doc · GitHub. You can add the flag to your PR, it doesn’t look like a big change.

[tmbb]

@ericmj

Your own PR for this addition hasn’t even been merged yet so I don’t think flags exist yet https://github.com/elixir-lang/ex_doc/pull/787

I think you’re confusing some things that really don’t belong together:

A couple months ago, I submitted a PR to allow for extra content (extra JS and CSS) to be included in the HTML files. I didn’t add any option to choose the markdown processor. You were supposed to choose the markdown processor in ex_doc’s config, which was already possible.

Some days later, I submitted a PR to allow for a custom markdown processor, given in the :docs option of the mix.exs file, which was rejected, because it did some sketchy things and @josevalim couldn’t see a good case for that

At this point, it was already possible to use makeup with ex_doc. The protocol_ex package, by @OvermindDL1 uses it, for example: ProtocolEx — protocol_ex v0.4.4

Then, a couple of days ago, José reverted his position and decided that configuring the markdown processor from the mix.exs file was not such a bad idea, and merged the changes himself. Then, after discussing it with José, I submitted a new PR that further simplifies the use of custom markdown processors. This is the one waiting to be merged. It adds absolutely no functionality, only simplifies the use of what’s already there.

This PR doesn’t touch anything related to the command line, and I don’t think it’s appropriate for it to do so. That’s an orthogonal concern. The goal of this last PR was to simplify the way you’d use ex_doc from Mix.

[tmbb]

@OvermindDL1

Ooo, that’s an idea, be able to specify a dependency as a run-time only dependency, nothing compile-time needed (nor available at compile-time), that way they can be compiled without depending on each other but all get included at runtime regardless.

I don’t get it too… makeup still depends on ex_spirit at compile-time. How would this help?

[ericmj]

@tmbb

I think you’re confusing some things that really don’t belong together:

Thanks for clarifying. Adding this flag to the command line would be a pretty small change though.

[tmbb]

On the surface it would be a small change indeed… But the new flag would have to be the name of a module (lay ExDocMakeup). This would have to be converted into an atom at runtime, which is icky, but not too bad; this is not something that can freeze the BEAM. But then, the module would have to be fetched from somewhere, right? Since the module doesn’t come included in ex_doc, I don’t think this is a viable option. Please correct me if I’m wrong, of course, but I think this is a fundamental limitation.

I can of course write another script that calls ex_doc with the module as the markdown processor, but since I’m writing the dummy package myself, I see no reason for adding a new flag to the original ex_doc package.

[josevalim]

@OvermindDL1

Ah so the top-level project is force compiled last regardless of if it would fit further down in the dependency tree? Funky… I’m guessing that is a ruby’ism (I’m more used to distinct modules that are calculated file by file instead of project by project in programming languages, like ocaml for example).

Can we stop saying that everything is a rubyism? Few things are and they mostly belong to the syntax. Ruby is not even a compiled language, so this doesn’t apply to Ruby at all.

EDIT: to clarify, this is not a poke at Ruby. Pinning all behaviour (desired or undesired) on Ruby is not fair to Ruby nor Elixir. Regardless of the inspiration, features added to the language were evaluated in the context of Elixir and the Erlang VM. And if Ruby does something unexpected and we decided to do it too, it is on us, not Ruby.

[ericmj]

@tmbb

I can of course write another script that calls ex_doc with the module as the markdown processor, but since I’m writing the dummy package myself, I see no reason for adding a new flag to the original ex_doc package.

Yeah, you are right. Since dependencies need to be added it wouldn’t really work as a command-line flag without manually going into the ex_doc project and adding the dependency so I think your solution using a script is better.

[devonestes]

They are a Benchee extension, but what Devin wants to use is the HTML formatter, which depends on JSON to get some data into the JavaScript generated by the HTML formatter to make the pretty graphs and such. Sorry, I should have been more clear about the specifics there!

[OvermindDL1]

@ericmj

I feel like you often have this attitude. If you have ideas for how to improve things we are always open PRs or suggestions on how to improve things.

Sorry! I don’t mean anything by it! ^.^;
Just there is a LOT in the Elixir-world that is not documented, both in ‘how’ it works but also ‘why’, so a lot is entirely confusing when it is hit… ^.^;

@ericmj

Not sure what you mean by loading a dependency but not depending it.

As in having it be downloaded and loaded when the program is finally run, but not needing to be compiled before anything else, thus a run-time circular dependency that is not a compile-time circular dependency becomes a non-issue.

Basically, ex_spirit does not depend on makeup in any way, it does not even depend on ex_doc in any way, those are entirely used just for publishing.

@tmbb

I don’t get it too… makeup still depends on ex_spirit at compile-time. How would this help?

But not the other way around, ex_spirit does not depend on makeup at compile-time. Basically I’m just curious if there is an option that could be added to the mix dependencies to say that something is a run-time only dependency on something else without having a compile-time dependency.

@josevalim

Can we stop saying that everything is a rubyism? Few things are and they mostly belong to the syntax. Ruby is not even a compiled language, so this doesn’t apply to Ruby at all.

I figured it had something to do with how it loaded modules. ^.^;
Python is not unsimilar in that regard either. (On a side note, I find python rather quite nice to embed in a program as a scripting language, even nicer than lua’s api :-))

[ericmj]

@OvermindDL1

As in having it be downloaded and loaded when the program is finally run, but not needing to be compiled before anything else, thus a run-time circular dependency that is not a compile-time circular dependency becomes a non-issue.

It wouldn’t help in this case. Which dependency would be compiled last to solve this issue? Remember that dependencies compile before the project. Changing so that some dependencies compile after the project would be a major change Mix, introducing a lot of complexity for an uncommon edge case that you can solve in other ways.

[OvermindDL1]

@ericmj

It wouldn’t help in this case. Which dependency would be compiled last to solve this issue? Remember that dependencies compile before the project. Changing so that some dependencies compile after the project would be a major change Mix, introducing a lot of complexity for an uncommon edge case that you can solve in other ways.

Indeed, I looked and mix does not build a dependency graph and uses it straight out but makes a lot of assumptions, so it would be a non-trivial fix. ^.^;

[ericmj]

@OvermindDL1

Indeed, I looked and mix does not build a dependency graph and uses it straight out but makes a lot of assumptions, so it would be a non-trivial fix. ^.^;

Mix does build a sorted dependency graph but only for the dependencies, the top-level project is not seen as a dependency. The assumption Mix makes is that dependencies are actually dependencies, which is a fair assumption. What you are looking for is a special-case where this is not the case.

Since all the examples that have been shown so far have workarounds I don’t think it’s worth it to introduce the extra complexity to Mix until more examples where this is needed are shown.