Rant / Vent Re: most popular languages

Chat / Discussions
1

Could I just vent a little? I have something I want to say but I don’t know where else to say it.

I’m really frustrated by this issue of language popularity. I tried asking the people over at [Boutique Supercomputer Company] about hardware for parallel computing, specifically with Elixir in mind, and the response was pretty much just, “there exist some languages which are built for parallel computing, but they’re not very popular”.

This is so frustrating to me. The popularity of a language has almost nothing to do with the merit of the language or its capability.

The reason companies pick languages is often based on foolish criteria. Personal preference, group-think, follow-the-leader / herd mentality, and appeal to authority, are all bigger factors in language selection over more logical criteria like developer productivity, scalability, performance at scale, maintainability, cost of ownership, etc.

And then to compound the situation further, the languages that new programmers choose to focus on is MOST influenced by what the big companies are using, because developers have to chase jobs. Maybe schools play a role here too, but I actually don’t think it matters that much what somebody’s first language is.

I feel like we need someone to show off what Elixir can do for parallel processing tasks and put it up on YouTube. Anybody willing to suggest some project ideas? I just pulled up a quick search result, and all the ideas are way out of my league:

Notable applications for parallel processing […] include computational astrophysics, geoprocessing (or seismic surveying), climate modeling, agriculture estimates, financial risk management, video color correction, computational fluid dynamics, medical imaging and drug discovery.
12 Parallel Processing Examples to Know | Built In

What do you think? Do you think Elixir would destroy the competition if applied to these problems?

2 Likes
2

Just find an API from where you can ingest realtime data at high scale or an huge public dataset that you can import and process in parallel.

Some years ago Twitter allowed to subscribe to a stream of a subset of the realtime tweets, but no idea if is still possible.

We have now some courses around using Elixir for trading platforms, and I think it’s a good example of showing off its potential for concurrent, parallel and distributed computing.

1 Like
3

Do you have a link? I’ve been interested in this topic and I have a time series analysis library I’m writing.

1 Like
4

Crypto Currency Trading Bot

You can check my video notes, including the code from the first course video here:

Screenshot from 2021-04-10 08-34-32

Crypto Currency Dashaboard

Your Time Series Analysis Library

Do you have a link?

4 Likes
5

elixir is unfortunately too slow to be used in the areas you listed, as a rule of thumb you should do computationally intensive stuff in nifs.
e.g. rust, c/c++, ocaml, java, scala would blow elixir out of the water, but elixir is far easier to use when writing concurrent code.

6

Random side story about the Twitter streaming API:
I actually used that twitter stream with extwitter during the 2016 US election day, collecting tweets with slogans for each campaign or phrases I would consider definitively for one side. It ended up collecting about 60,000 tweets in total and roughly matched the election results for percentage of votes Hilary vs trump. I thought it was definitely broken at about 2pm because it was showing trump as winning. Then later it flipped to Hillary but he ended up winning.

I did sentiment analysis after using some API I can’t remember, and the emotions it came up with for trump tweets were a plurality of fear, anger, and excitement, while Hillary supporter tweets were a plurality of hopeful, excited, sad.

1 Like
7

My history is from some 5/6 years ago. I was just listening to every tweet that was tweeted, not looking for a tag, just literally * in order to populate the MemSQL databse that I was trying out by following on of their tutorials. I remember that I ingested millions of tweets pretty quickly(minutes, but no idea how many now) with a Python script with 10 or 20 threads(don’t recall anymore how many).

I don’t think Twitter allows this anymore. I think that now they are very picky with who they allow to listen for *.

8

Those things aren’t usually very exciting and presentable.

My biggest successes with Elixir’s transparent parallel computing were:

  • Ingesting a ton of order / line item data from DB and making financial reports;
  • Scraping data, transforming it, aggregating and storing it in a data warehouse (for later analysis);
  • Doing the said analysis (too long to list the details here but basically spawn CPU_Threads number of Elixir processes and do work like that).
  • Parallel scraping of 100+ separate domains (although to be fair modern kernels are extremely good at hyper-parallelizing I/O so any semi-decent language will do here);
  • Receive logs in a fire-hose manner (many open connections from clients) and aggregate on the next stage (through heavy use of Flow);
  • …and I know I am forgetting at least half of what I did that utilizes BEAM’s transparent parallelization to the maximum.

As I said, not very exciting stuff but when you have internal divisions struggling with what you perceive to be trivially fixable and then you actually fix it, you can feel like a king for 5 minutes after. :slight_smile: And you will deliver a direct business value; nobody wants to wait for a financial report for 4 hours.

1 Like
9

Yes and no:

YES

  • Because Elixir in my eyes wins with its extremely tiny LOC (lines-of-code) metric. You can literally do Task.async_stream and feed it a stream of files, network sockets, async DB requests and what-have-you and it will just mostly work. It feels extremely empowering and you are often like “OMG how can this be so easy here and why did I never find it before?!”
  • Because the runtime guarantees of the BEAM are unmatched. If something crashes it will be restarted with its initial state. That’s not ideal for all scenarios and that’s why Erlang preaches idempotent tasks be put in processes.

NO

Because other languages are gradually catching up. For example, if in Rust you make a pre-warmed thread pool (meaning it asynchronously starts all threads that will be needed immediately after the program starts up and before the thread pool is used in your code) with N threads for CPU-intensive work, where N == CPU cores/threads, and another pool with N*5 threads for I/O intensive work, and then appropriately allocate tasks to them as you go (while heavily using Rust’s async constructs – which can be a challenge at times) then you’ll end up with 95% of the BEAM’s parallel abilities for something like 30x - 100x the speed.

The only exception would be the fault tolerance but this can be worked around: f.ex. if a task fails you’ll just put it back to a centralized task queue and then restart the loop (all worker threads in a thread pool simply pull one task at a time from that queue).

…Buuuuuut, Rust is much more verbose than Elixir. That can and has been a huge turnoff for me at times.

And finally, OCaml 5.0 is due at the end of 2021 (although I’d bet they will miss that timeline so it will likely arrive 1 year from now) and it will have muuuuuuuuch more finer-grained control over the parallel execution of your code compared to practically everything out there, Erlang/Elixir and Rust included. And OCaml, like Elixir, is extremely terse, only it’s strongly and statically typed, compiles to native machine code / single executable binary, and its compiler is lightning fast.

These language creators there are scary, dude: they are both perfection-driven academics and people working for big financial corporations. They both know their theory and can make it happen.

Now don’t get me wrong. I love Elixir to tears. It’s one of the very best language innovations on the planet. The said planet isn’t sitting still however; many people have took notice of Erlang/BEAM’s concurrency/parallelism approach and are working on both (a) mimicking it and (b) improving it.

What I am doing here is only shining a spotlight on a small part of the bigger picture out there. Draw your own conclusions and never ever become a language fanboy. Critical and tradeoff-driven thinking should dominate all your analysis.

7 Likes
10

I haven’t really seen any other language that are relevant today that has lightweight actor model so tied to platform itself and I think that a huge deal coming from C#. I think it makes all the difference when the platform itself has this support and not some 3rd party library. Sure for CPU intensive work BEAM isn’t that fast but example Discord has been written in Elixir and Rust and surely they would have rewritten lot of their code with Rust if it was feasible to do so, to get more out of their hardware.

3 Likes
11

Yep, agreed. Elixir is still unbeaten as an orchestrator and a fault-tolerant core of your program. Discord is the perfect case study of “why isn’t everything rewritten in Rust yet?” indeed.

3 Likes
12

The reason companies pick languages is often based on foolish criteria. Personal preference, group-think, follow-the-leader / herd mentality, and appeal to authority, are all bigger factors in language selection over more logical criteria like developer productivity, scalability, performance at scale, maintainability, cost of ownership, etc.

as a counterpoint, many of your logical criteria come with popularity. more popular languages will have wider access to expertise, libraries, tools and other resources that can act as a multiplier for developer productivity, maintainability and cost of ownership

2 Likes
13

That does happen part of the time, granted. The other times people just happily copy-paste from StackOverflow and, paradoxically, make their project harder to maintain even if all the benefits of popular tech you enumerated are there for the taking.

Dig deep enough, it’s all about the people underneath, and rarely about the technology itself.

1 Like
14

WIP: GitHub - rm-rf-etc/time_series_sliding_window: Rolling window for time-series analysis in Elixir, implemented in Rust NIF

My use case is stock analysis, but one could use it for any time series analysis. For financial data, it’s not limited to just OHLCV. Next thing I want to do is write an ATR indicator for this library.

2 Likes