Spinning this off in a new thread because I think it’s interesting and merits its own discussion. Agree with @hauleth that the basis should be spinning up a c-node inside of julia, but it’s going to need a bit more than that. A few questions:
How to manage Julia versions to run on
Julia has a --worker node setting, should we use that?
How do you load up the correct dependencies? Also, julia annoyingly uses global dependencies, more like python than elixir, which has project-level dependencies.
Is there something really high performance that one could do like memmapping memory segments and passing data across that way? How do you detect if that capability exists? (windows? Linux? MacOS? etc)
Separately, would anybody be interested in doing a collab on this for SpawnFest 2020?
OK, I’m slightly biased because I came to Elixir directly from Julia because I didn’t think Julia had a “deploying real user-facing things” story despite being 1.0… maybe I’m the only one, ha.
Julia matches extremely well with the academic mindset and the mindset of machine learning professionals (even though some of them don’t know it yet), which is why it’s in the beginning stages of blowing up in that field. It also meshes really well with elixir in terms of its programming paradigms (functional, escape hatches for statefulness, capitalized modules, secretly a lisp). I think stronger julia <-> elixir interop is a thing that could be really amazing, and is currently an unsolved problem. I also know that there have been lots of rumblings in the elixir community about wanting this…
I only started learning Julia a few days ago. So far, I think it is good at what it is supposed to be good at, not much for other things. For one the core api documentation is no as easy to navigate as elixir’s.
I wholeheartedly agree better interop between elixir and julia would be beneficial to both communites.
I would love to see really strong Julia interop and for Julia to adopt Elixir’s expectations/standards for high-quality documentation and tooling. Those two languages compliment each other really well. But I also agree that a lot of the user base is usually happy to just hack something together in Python and call it a day. An Elixir/Julia alternative could possibly occupy a niche for applications that need to be durable and therefore more thoughtfully engineered.
Why should we care? Really. This is all on the “Julia developer” side, not on “Elixir developer” side.
If such integration would use UNIX sockets for communication instead of C-Node (fairly easy to do with socket module) then one could pass mmaped FD to the second process. However IIRC this is still not available on Windows (even with introduction of UNIX socket on Windows).
Why should we care? Really. This is all on the “Julia developer” side, not on “Elixir developer” side.
I think if you’re going to want to write an app that you deploy somewhere “in prod”, you are not going to want to struggle with, say virtualenvs. I’ve discussed this with the Julia founders, and honestly, the current Julia deploy strategy for solving this is “just containerize it”. Since Elixir is the more opinionated system, and acting as the ‘orchestrator’ anyways, I think it’s reasonable to take the position that in a mixed language deployment, (that will necessarily have to share container, for example), Elixir should be the “responsible parent” and manage those sorts of things. I think julia has an escape hatch where you can set the deps path, so forcing julia deps into the elixir project tree is “the right thing”.
Agreed. Just don’t make it overly complicated and hidden. Also if people want to stick to the global setting of the Julia environment they should be able to opt out of this.
That is the place where we disagree. Each of the applications should have their own containers and do not share stuff if not needed. I need to check, but I think that sharing FDs via UNIX socket could work between containers.
I do not think that we should patronise such applications. From the viewpoint of the Elixir developer this may seem like a good idea, but from the Julia developer, who is accustomed to their own tooling, this may not seem like something worth their time.
IIRC there is such possibility.
I wouldn’t say so. I think that there should be library for the two projects to communicate with each other, but making Julia part of Elixir deployment is not worth the hassle in most cases. Julia (like R) is designed to be simple and easy for scientists, and they want to be able to just Pkgs.install("foo"); using Foo; and call it a day. Managing that from within Elixir would be IMHO hell of unneeded journey (as you would for example need C and Fortran compilers in your machine, that would need to be managed by Elixir as well).
I think that it is possible, by hook or by crook to share FDs through containers via unix socket. Do CNodes work between containers? Also, I think you would want your Elixir app to supervise the Julia processes and julia can certainly go rogue and hog its own thread management to the point where it wouldn’t be able to service a kill request over the cnode channel. If they’re in the same container, the elixir vm has the “out” of being able to slaughter julia at will.
In any case it would be easier to not bother with the dependencies so maybe I’m overthinking it, but I think that the target audience for a julia + elixir combined is not a julia developer that is a data scientist or academic researcher, it’s an elixir developer that wants to deploy something that a data scientist or academic researcher has written, or a data scientist or academic researcher that actually cares about sane deployment. So those problems are going to have to be solved.
These do. C-Nodes are treated the same way the regular “Erlang nodes”. So these even works over network (additional advantage - you can run your Erlang application on simpler machine, and run Julia code on GPU-backed beast).
Absolutely not, this is why we have system supervisors like systemd. Erlang isn’t the best solution out there for managing external processes, especially when you run it as unprivileged user. This also solves problem of “unresponsive node” you mention.
No if Julia would for example hog RAM. From within Elixir you also cannot run application in cgroup or extensively limit it capabilities (as running Erlang VM as root is generally poor idea).
Agreed on the no reinventing the wheel. A julia project within an Elixir project, within the same git repo, would be nice though. Sort if like how a JavaScript project lives in a phoenix project. The julia sub project can still be autonomous. I would love to use exunit to drive regression tests for julia.
Nothing prevents you from having both codebases in one repo. That doesn’t change there to be honest. The difference will be how you launch and manage components of the application.
Joining this discussion late on but wanted to say that we are using both Elixir and Julia in production in a kubernetes cluster communicating back and forth over HTTP api’s. By adopting a functional approach in Julia with no state management we are able to have acceptable performance.
We use Julia’s package compiler to create shared object files that contain the compiled version of our Julia code, its dependencies, and the Julia runtime itself. Then in Erlang we start a new operating system process (erlang:open_port/2) and run our Julia code. Our Julia code is listening for requests on its standard input and sends the responses to its standard output. (This part was inspired by https://github.com/hdima/erlport.) We plan to modify our code to use sockets instead of standard IO.
Thank you both very much for your valuable insight. I wonder if @dokie would benefit from something a bit less burdensome than all-out http parsing.
@hcs42 I’m surprised and impressed that statically compiled Julia works well enough to do that, static compilation has come a long way from the 1.0 days I guess!
I just quit my job and am evaluating what to do next. I think it won’t be an elixir-julia bridge, but that would be second on my list.
