Ive already taken a quick look at the code and I would like to put native support for vector processing in Erlang (initially floating point vectors).
When I say Native I mean in the BEAM not using a shared library NIF.
Once that is done I’m going to figure out what it would take to do a sympathetic implementation of vectors in Elixir.
If there are any businesses that would appreciate that then I am looking for a sponsor for
this work.
I am proposing this for X86_64 since that’s probably what gets used the most. I am prepared to do the same project for any other hardware if there is good enough reason.
contact: marcel@aexa.uk
30 years in kernel design incl. OpenVMS Kernel, OSF/1, USL Single-System-Image Amadeus
Author of deqdip on hex.pm
Personally I think that it would be both hard and pointless. There are few reasons for that:
Erlang wasn’t designed for number crunching, so almost no one is doing that, there is no point. Erlang provides enough FFI facilities (which are almost astonishingly easy to use) that it is better to use other languages for that. Not only via NIF, but also as ports or C-Nodes.
Erlang operates on arbitrary big numbers, so it would require artificial limitations in such type which would be incompatible with other parts of the system.
We should wait till JIT implementation lands before even thinking about such features. Maybe it will be there already.
Hard=>
It’s only Software. Nothing is impossible when it comes to software. (Hardware is a different story)
Incompatible=>
Everything is relative - as incompatible as different versions of IBMs many operating systems? It’s
not going to be a perfect dovetail fit but that’s just another engineering trade-off.
Pointless=>
I think it’s a bad show that something as terrible as Javascript only ever had 1 datatype 1995-2018
floatingpoint. While Erlang/Elixir/Phoenix a much more capable system does the least possible for
FP with a “double” implementation. Not much better than what Javascript offers.
BigInt was only added to Javascript after 2018.
Wait until its done in the Jit=>
For reasons mentioned above I’m 100% sure someone else will be doing it. That doesn’t help
pay my rent.
Erlang/Elixir/Phoenix is a great system but it has to grow to stay relevant…
Case in Point:
All these IoT apps are going to be using A.I.
A.I. requires F16 vectors
If your going to be in the business of serving webapps much beyond 2020 then your system
better have a good A.I. story…
Last time I checked, detecting whether given set of instructions will reach halt state is not possible. For sure it looks like software problem.
Incompatible with user expectations that (assuming new syntax for vectors #(a, b, c, d)) this code can fail in runtime:
foo(X) when is_integer(X) ->
#(X, X, X, X).
As X can be greater than allowed vectored integer, as Erlang allows arbitrary length of integers (for example it can be 2048-bit integer and AFAIK there is no architecture that allows such integers, especially in the vectored operations).
Erlang has 2 numeric types - integers of arbitrary length and IEEE754-like floating points (not fully compliant as Erlang do not support NaNs and infinities).
We have different thoughts about it. First of all, each technology has it’s niche where it was meant to be used, for example C for systems programming, R for statisticians, Matlab for engineering, Julia for scientific computations, Go for Google networking stack, Java for Sun applications, ,NET for MS applications, etc. Going outside your niche rarely works, especially as you were strictly designed with completely different use case (immutability isn’t the best thing out there for number crunching).
I disagree, but ok, let’s assume you are right.
If you say so, I am not an expert there.
I disagree even more, but in such cases it is way better to use other languages like Julia or Python to do number crunching and connect to BEAM application via one of many possible ways to do FFI there (already mentioned NIF, ports, and C-Nodes, just pick your poison). For few reasons:
There is already a lot of libraries for such use cases in other languages, porting all of that to Erlang-native implementation would be enormous job that no-one would have time for.
New algorithms, ideas, and libraries will be always implemented in mentioned above languages, no-one will come and say “well Erlang seems like great networking language which also supports F16, so I will write my paper in it, despite the fact that all my colleagues use different language, so I will need to implement everything from the ground up”.
Finding people proficient with these languages, and libraries for them, will be much easier. And knowledgeable people are much more important than any issues with FFI, especially as this FFI is simple and straightforward in Erlang (which I said earlier).
So if you want/need AI in you web application (99% of applications do not need it and even if they market it as containing “AI” these are often simple algorithms that I do not count as AI, for example simulated annealing, k-means or Bayes filter), then there is plenty of solutions that do not need BEAM changes. What is more, BEAM changes would be an issue there, as it means more places where it can break and more maintenance burden for the maintainers.
IMHO not worth it, and if you really want to do it, then make it as a commercially available NIF library. It will be much simpler, easier to maintain, and equivalently fast.
I am really curious as to why do you believe that? On what is such a belief based? Are you observing certain software market trends? Or did you continuously work with customers which requires AI + webapps (no clue what that means though)?
For example Microsoft have one language C# doing just about anything and everything.
I’m C# programmer by profession and this is not completely true. If you need to do some really low latency stuff or things that need memory to be returned to OS as fast as possible then .NET is not good for those cases because of garbage collector pauses and because memory releasing is controlled by it.
I also don’t see why machine learning can’t be implemented as separate service or NIFs. As an example Discord is created mainly with Elixir but it uses some Rust NIFs and also some separate services are created with Rust only.
I develop business software and as of now our company hasn’t really needed machine learning at all to be competitive and I don’t see that changing. We have done some ML.NET proof of concepts but customers haven’t been really interested in those. Also haven’t heard anything about machine learning taking web development by storm in programming circles I’m in.
This forum is for Elixir language and other related stuff too but it’s not for BEAM development. BEAM is developed by separate entity. So you can’t really propose any BEAM features through this forum. I’m also quite sure not many businesses want to use a customly modified BEAM. If you want to propose these features to BEAM you should look into GitHub - erlang/otp: Erlang/OTP how to do it.
I hope you get money to pay your rent but from personal experience it’s lot harder to find someone to pay programming work you are interested in than just finding paying programming work.
