Elixir ecosystem is a dream come true and future proof, except...for...,

Chat / Discussions
41

My main point was not to start a discussion around what database type is better but exactly about the fact that Elixir should provide us the choice at least between the “classical” RDBMS and “new” approaches like Graph DBs.

I realise the vast majority of people using Elixir at the moment are very happy and true believers in RDBMS but I expect they to understand that that belief is probably preventing others for using Elixir and should be addressed seriously by the community, especially now that all the other key parts of the ecosystem are almost finished in a very good way.
So, addressing this topic could be the next big step.

PS: I also learned that I need to be very careful next time in how to frame the question to not mislead the conversation into a discussion about the intrinsic value of different databases. I wrongly assumed the majority had the same picture about the dynamics of DBs, which obviously is not true :slight_smile:

1 Like
42

I agree on that point: having nice tools as Ecto for a large number of database engine would be great.
But it’s a lot of work, and hard work too and I think that for almost every language, the number of available tool depends on… the community. Because it is time and work and everybody has a limited amount of each.
Then sometimes, the best thing to do when you don’t find what you need is to build it yourself or at least try to build it yourself and to share it with the community. And if you’re lucky, some people will help you to build the thing.
Elixir is not so popular that there is a lot of mighty brilliant individuals achieving all the things just for the fun then we have to build things ourselves. There’s a lot of people in this forum who’ve contributed in one way or another to Elixir ecosystem (thank you guys :)). Why don’t you bring your love of alternative dbs and your love of Elixir into a new library and share it with us? It can (will?) be hard but you’ll learn a lot but after that Elixir will appeal more people like you :).

@marcalchca I’m the author of ecto_neo4j and I strongly discouraged you from using because I will stop maintaining it really soon. It’s very unsatisfying to work on it as developer (imagine putting squares in triangles…) and imo not satisfying as an end user as it rely on relational-concept.
Then, you maybe should use seraph which way more graph friendly and heavily inspired by Ecto (an rely on it for some parts). I didn’t make a annoucement yet because I want to add the query api first, I hope it will be available next week.

7 Likes
43

Ok. Anyway, as you can read from my comments, I fully agree with extending Phoenix as you suggest. At this moment I’m starting to study neo4j in more depth. I read in 1 of the comments that ECT would somehow support it already.
Thx

44

Ok. Thanks for warning me and informing me about the alternative.
As for contributing … I have to balance very well the time I can spend too and if I’m the only one interested it makes no sense, but maybe all the interested people have not yet adopted Elixir and are not in this forum. So if I could realize something and increase the size of the Elixir community thanks to it, that would motivate me. Still I must find a paid project of which that would be a spin off. I still got 5 kids and my wife to support and haven’t won the lottery so far :grin:.

But I keep your advice in the back of my head.

And please let me know when the alternative graph library is live. :hugs:

Thanks

Marc

45

No. I mean, I like them, but here I am fighting with fear and misunderstanding about them.

Sorry, but with that experience and the fact that you still confused relation with foreign keys/joins, I find it hard to believe.

Me too. I. just find graph DB or document DB terrible choice at the beginning of the project. Relational DB is always a better choice unless you are 100% sure that you will need the “NoSQL” (and I would say that in the beginning you never can say with 100% certainty that it will be the case). While SQL has it disadvantages it is also the most flexible solution out there. Especially with PostgreSQL which can be used as:

  • “Classical” relational DB
  • Document DB (via JSONB or XML store)
  • KV store (either as a simple CREATE TABLE foo(key STRING PRIMARY KEY, value JSONB) or hstore)
  • Full-text search engine (via tsquery)
  • Queues (NOTIFY)
  • GIS (PostGIS)
  • TSDB (hll, TimescaleDB)

This also gives you another advantage in the beginning of the project - less moving parts that you need to configure.

TL;DR: Yeah I like RDBMs, but my posts there are mostly fighting with fear and misunderstandings about them.

9 Likes
46

I agree with you that document databases are not a solution for a full application. I had a very painful experience with MongoDb a few years ago. However graphdb line Neo4j is of a totally different kind since objects can be typed. Just have a non-biased look at it.

As for mixing up relation with foreign keys/joins … maybe I wasn’t clear in the way I formulated it. I wanted to say that the basic traditional relational theory only offers us a limited toolbox: zero/one-to-one/zero and one-to-many (and reverse of course) relations between entities. The many-to-many relations (i.e. 85 - 95% of cases in real world) can only be achieved by composing 2 one-to-many relations. It is not a “primitive”.

Either this is artificial as in some cases you’re forced to have that intermediate entity where you only use it for joining the other ones. And worse, if you do need the middle entity (most common case), you quickly get into a myriad of other related entities each with their intermediate entities just to express a moderate part of reality.

So for something conceptually simple, you end up with quite some complexity. Both for maintaining the inevitable data to feed the links between all these entities but also to query them. If on top of that you need a hierarchy of several levels deep (not to mention a graph!) then complexity explodes. It often makes people denormalize their databases which they’ll pay down the road. Been there. Done that. It’s not that RDBMSs cannot do it. It’s the pain it costs to do so that I try to flee.

If you compare the expressiveness of Neo4j to solve the same problem it is dazzling. And the related queries … my God. To think we have been struggling ask those years with inner and outer joins!

This was about classical RDBMS. If you add layers of graph support or other tools, then it may be less bad. But to me that feels like a maskerade of complexity under the hood. I think sooner or later you pay the price for such hybrid constructions. So I prefer to examine the pure version.

And don’t worry, if ever I discover it has limitations that make me return to RDBMS, I’ll humbly admit so. But at this point in time I strongly doubt it.

So I hope you see that I haven’t spent 30 years working with and teaching about RDBMS … not even grasping the basics of it!!! I know the covid19 lockdown can make people very stressed, so I forgive you your inappropriate judgement of my professional capabilities. I’d never judge someone I don’t know based on 1 post. So with you I hope we soon will be free again to walk where we want. It will probably make us a lot more gentle towards each other.

Kind regards
Marc

1 Like
47

Yes, but that isn’t what “relation” in “relational database management system” mean. Relation in sense of RDBMS is what we call “table”. That is real basis of the relational algebra on which RDBMS are built.

This is common misconception that is repeated over and over again.

So the discussion goes and goes over the same misconception.

If this is discussion about query languages (because storage in graph DBs and relational DBs will be similar) then it is mostly irrelevant as you can design language that will use graph-like queries and translate them to the relational algebra as you can describe the graph DB as a set of 2 relations:

Vertices ( id, data )
Edges ( from, to, data )

You still are. As I have showed above there is a way to use relational algebra to store and query graph-like data. What graph DB is doing, is that it just “hides” that all joins from the user of the query language. It also may automatically create indices to make queries faster, but not much more. It is like saying that

If you compare the expressiveness of Java to solve the same problem it is dazzling (as C). […] To think we have been struggling all those years with memory management!

Yes, Java is more expressive than the C, but that is not the point. C is more flexible and more powerful, even if sometimes require a little bit more code to write.

There is not much, because as I said, you can pretty simply translate relational algebra to graph query language.

You probably do not. It is just that RDBMS are simpler. Do you know how all these “NoSQL” DBs happened? I will give you quick tour:

  1. We use SQL.
  2. Yay, we are successful and we are growing! :champagne:
  3. Oh snap, we are growing out of our current DB. We should scale (horizontally/vertically).
  4. Oh snap, we cannot scale our current DB anymore. Write DB accustomed to our needs which is graph/KV/columnar/document/etc. DB.
  5. New NoSQL is born :partying_face:

It is not just one post, you are again and again telling that “relation is connection between two tables”, which is not what R in RDBMS is for. JOINs aren’t relations, joins are operation on two relations (known as tables) that generates new relation.

There is no “one to one”, “one to many”, nor “many to many” relation in RDBMS if we are speaking in terms of relational algebra, because these aren’t relations.

2 Likes
48

A graph database is not really new. It’s a network-model database with improved query language. Relational databases supplanted network databases decades ago because of their greater generality and flexibility. If you object to adding graph query functions to RDBMS, then you might as well try to find a CODASYL database and use that.

I think Neptune actually uses PostgreSQL as its storage engine. Amazon doesn’t admit that, but when you dig deep into some of the config & replication capabilities & limitations, it EXACTLY matches PostgreSQL. (And I am not the only person to notice this…)

1 Like
49

You insist on explaining in a wrong way what I’m saying in my post. It becomes a semantic discussion iin the end. See further about the"table/relation-obsession".

I know a relation in the theory concerns a table. But when we build an ERD you will admit that the common parlance is about “relations” between entities. And those are key in expressing real world needs.

A relation or table in the purist sense of the definition, is very limited since it only has 2 dimensions.

E.g. for each conditional data you’re forced to define another table holding a 1-0 link (I avoid using the word relation just for you, but you should get used to the fact that in the real world “overloaded” definitions are inevitable, so it was not a misconception of mine, but a deliberate use of the word relation in a slightly different context).

E.g. ternary relationships are unnatural to RDBMS although one can build them.

You put the world upside down! You seem to imply that graphdb under de hood are just RDBMS iin which they mask the “tables”. Clearly you have never taken any effort to study them seriously. It simply is not true.

I’m astonished. Didn’t know RDBMS fundamentalists existed. I met one.

I’m always open to learn and despite your annoying and disrespectful way of treating me, I’m still tempted to look up some if the things you shared. Thay is because I’m really open minded.

But if your goal is to impress people and make them shut up, then you met the wrong guy. I believe in freedom of speech.

Cheers

50

That is simply not correct, it is an Entity RELATIONSHIP Diagram. Words get misused in casual speech, this is true. But it’s not unreasonable to expect more precision from someone who’s been teaching a subject for 30 years–especially when engaging in a technical debate.

  1. He didn’t state (nor imply) that all graph databases are implemented that way. He pointed out how one could be implemented that way.

  2. As per my previous post, in at least one prominent case that is actually how it is implemented.

Finally, his tone is neutral, while yours is hostile and disrespectful.

4 Likes
51

I like wine.
I love Champagne. I think the future is Champagne.
I don’t drink as much beer as I used to drink when I was younger.
Please let’s not go into discussions about if red wine should go with red meat and beer with…more beer.
If I invite my friends for a barbecue I’ll have wine, beer and champagne because I just love it.
3 options. More than that they need to bring them to the barbecue.
Elixir has only wine.
Doesn’t matter if it’s what tradition and professional taste says it is what should go with meat.
It only has wine.
I love Champagne.
My friends love beer. Younger generation prefer beer more and more instead of the perfect red wine. Or my favourite Champagne.
I like to have my friends with me enjoying the barbecue. I have wine, beer and champagne. I want my friends to keep coming and be happy.
If I was Elixir I would have my barbecue drinking wine, not my loved Champagne and no one of my younger friends that like beer would come.

Is this clear enough?

PS: yes, I have to spend some extra money and time buying 2 extra type of drinks I won’t drink but it’s really worth it.

52

Please let’s not get too hot on this topic.

I don’t think anyone is trying to annoy, or disrespect others.

There have been people integrating Neo4j…

But as mentionned by @domvas, it’s hard to fit squares in triangles, and Ecto works well for relational concepts.

It would be nice to see how other frameworks do.

5 Likes
53

Erlang is not complexer than Elixir. It is Elixir that is adding complexity with macros, DSLs and a more sophisticated syntax. Both languages are great, but it is wrong to make your statement, especially when Erlang has a very simple syntax - much simpler than most programming languages. It is just a bit unusual when you come from Ruby/Java/C…

1 Like
54

In addition, learning Prolog syntax explains a lot about Erlang syntax :slight_smile:

1 Like
55

This playing with words continues (“relation” … “relationship” - this is a forum, not a university exam - my point was quite clear after I reexplained it with examples: entities relate to each other, how else would you expect me to state it?) and … based on that I should accept someone questioning (2×) my complete career ?

Wouldn’t you read this as disrespectful?

But now my reaction is hostile? Really?

I didn’t question anyone’s competences, did I?

As my first real experience with this forum it’s quite demotivating (not to mention I don’t feel welcome)! To think that in my first post I explicitly said I didn’t want to hurt anyone’s feelings and I suggested Neo4j only on top of RDBMS. I thought: let’s walk over eggs as I’m new here, but that clearly didn’t suffice.

I didn’t expect this subject to be so touchy.

For peace sake, I won’t react anymore regarding this matter (I’ll for now search other sources to pursue my quest).

I hope one day we can have a grown up conversation on database technology, but today there’s some short-circuit preventing us from doing so (covid19?). It’s a pity. I’ll reread my posts to see where I could have said something to contribute to this.

So for now, let’s just leave it to that, ok?

Let’s go for a nice walk and take a deep breath. But please note that I was just interested in this matter. I’m very positive regarding Elixir (and LiveView).

1 Like
56

Let me reinforce that the Elixir community is one of the best I’ve ever experienced.
In fact, it is one of the main forces for me to have stayed with Elixir although it was tough in the beginning coming from Object Oriented languages and focusing on other databases types.
At the end of the day you always find colleagues that care about you problem and core team will always find a way to help you and make things just work.
That’s why, it’s important to stand for this issue about databases, not into an argument about “mine is bigger than yours” but the barbecue style: " I want my friends to feel welcome because even if they have different taste than me they are very important for me".
Thank you for you participation and for the way you could bring elevation to this dialogue. :slight_smile:

57

You make me feel welcome! Thanks very much.

Maybe I was a bit too fierce about defending my ideas (I seem to have a more Latin temper even though I live in Northern Europe ;-)). I don’t know.

I was very happy about finding this Elixir community to share things with.

Before that I’ve never had something like that. Only professional co-workers.
In fact this is my very first experience with an open source community.

I used to be always very suspicious about them in the past … so you can imagine the thoughts I had tonight.

Sorry if I said something (probably my 2nd last post) where I was too harsh, maybe.

Probably the side-effects of Covid-19 bit me too :-).

I’m definitely intending to try working on Elixir and use LiveView for front-end work.

I’ve been a professional developer a very long time ago, but I tasted different software engineering technologies last 10 years toying with many of them but with no full-time involvement (I had other roles but still they allowed me to get a good view on all the constraints developers are facing today, so I’m prepared now (I think ;-))).

I really consider starting up again as a full-stack developer on my own (at least at first).

If it works out well, I might drop the rest and focus on it for a living.

So being excluded from a community like this, is the last thing I was hoping for.

And if after a while I’m able to contribute I would be glad to do so.

This gives you a little bit of context about my intentions with Elixir.

Anyway thanks
Best regards

1 Like
58

What the younger generation prefers has zero bearing on what is objectively a good technology and what isn’t. So what’s “old” and “new” has zero weight in a technical discussion.

(Example: I like and learn Rust not because I think C++ is “old” but because in my eyes it’s a lot more trouble to work reliably with. Rust being “new” is accidental; I simply find it better in the space of strong and statically typed natively compiled languages.)

I am not a scientist but I’ve read and experienced enough to see that the RDBMS model is good enough for a lot of apps. Is it inherently good for time series data? Not very much. Is it good for making an append-only journal? Hell no, the RDBMS nature of your DB will only get in the way.

But that’s not the point. The point to me is more along the lines of “people create non-RDBMS storage plugins in an RDBMS database as a way to brainstorm and see if a concept sticks and can serve practical business needs out there”.

I’d rather experiment with a new storage paradigm inside my current DB even if I full well understand that’s fitting a square peg into a round hole. I don’t care if it will perform [much] worse. I want to see if with that new storage mechanism I can write code that will be shorter, clearer to upgrade and maintain, and easier to onboard new team members in. If the goals I have for this new tech aren’t achieved then I didn’t lose much; I’ll likely just do git rm -r lib/myapp/experimental_storage/ or something, and revert a DB migration. Done.

But if I did invest in a new DB and went all the way, only the discover that it doesn’t work as well as I hoped, then that would be a huge double investment: once to migrate the current code to it, and one more time to undo all of that and get back to the “old” SQL way of doing storage.

As for your seeming disappointment that Elixir only puts the RDBMS paradigm front-and-center:

  • Elixir has plenty of DB adapters (but not many of them integrate with Ecto; that’s true).
  • You are free to start an effort to create a non-RDBMS storage library and indeed, if you have some luck, people will come on board and help you out. (I for example am currently writing an Elixir sqlite adapter and I am nearly done with the first stable working release.)
  • When working with Phoenix, you can actually skip the Ecto parts with its generators and just write your own context / model modules. As long as you properly insert / update / delete / read pieces of data then Phoenix – or any other Elixir app really – will not care.

I do personally disagree that Ecto being RDBMS-centric and practical “discourages” the Elixir community to try new storage models. I don’t think so. What’s more likely is that a lot of people discovered they don’t need those new storage models, and/or simply never had the time or political capital to invest in them in their paid work.

9 Likes
59

A former employer of mine will probably be doing that in 12-24 months. A massive undertaking, just because a decision maker read a blog post and some marketing materials and went for something new based on zero technical analysis. (And I do mean 0!)

2 Likes
60

But right or wrong, Elixir would benefit a lot of having options.
I understand many options is difficult to mantain.
But options means at least 2. Right now you can only go “Relational” if you want to use the “official” Ecto that is very, very god, for…Relational DBs.

PS: if you had at least 2 options you would avoid debates about the best one and you would get more people on board. Then you could debate while using Elixir ecosystem. Now we are debating when some people are outside the ecosystem.