"Do you really need a database?"

Well, it really depends on what you are doing. There’s no one size fits all solution here. That said, I reckon most web apps do use, and do need, at least one database.

With regard to phoenix, there are certainly some new ‘best practices’ in 1.3 but they don’t concern whether you should use a db or not.

Well, most of this thoughts about not needing a database assume a basic thing: You never shut down your application.

If this is a fact and you can guarantee that it will never shutdown, then leave DBs completely.

But since you can’t guarantee zero downtime, you have to persist your data somehow.

To persist your data and getting it back into your live system, you have a couple of options:

• Put them into a databse of any kind (SQL or not doesn’t matter) and reload it when necessary
• Log changes of the system into a file. On restart load that file and “replay”
• Combine both ways.
• probably others

One of the most important things to remember to understand that mantra is, that it means “you don’t need the database as primary source of truth, but it may be a valid backup strategy”.

To follow on others : Have an app that write in multiple place so there is always one that is never shut down.

Of course it is not the most practical thing, and in general you would not do that like that.

But the use case i see the most out there for database (and i include redis and co into it) is to make it handle your distribution and concurrency problems.

As the saying goes (I forget who originally said it):

“The database is not the truth: it’s a cache of a subset of the truth.”

It’s very valid to use a classic SQL database sparingly in a BEAM application, using it only for persisting in case of emergency reload of “start state”, because we have so many more choices between “no database” and “the database is everything” to work with.

Lots of other languages lack the abstractions for creating data store layers that can work both as the primary source of truth and also persist things to other sources, but they’re so cheap in terms of developer time because of OTP that it’s almost wasteful not to use them and treat everything as if it’s Ruby.

I guess what I want to say, in short, is the question should probably be “Do you really need to read/write to the database all the time, or can you just use it as a bucket where you pull starting state from?”.

Thanks for the feedback guys.

I guess what I want to say, in short, is the question should probably be “Do you really need to read/write to the database all the time, or can you just use it as a bucket where you pull starting state from?”.

What would be a “good” strategy for dumping stuff into that bucket?

In the Joe Armstrong example of if he were to rewrite Twitter today, he would have each user and each tweet be a process. How often would have the app dump tweets into the database at a set interval?

The Phoenix app…SORRY Elixir app .. that I started on was a Home brewing app. I had users, recipies that consisted of malt profiles, yeasts, hops and so on. All of these things can be represented as prosesses and communicate via messaging. A batch of a given recipe would have a brewdate and so forth. Everything is fine (hopefully), until the server crashes.

So is the only question how vulnerable the app is to data loss?

That seems to me like the fundamental question. With regards to “everything should be a process”, as in every tweet and so on I guess I don’t really feel all that convinced everything needs to be a process.

In terms of setting time limits on when things should be cached to the DB and so on I’d say you have to make that judgment call on a case by case basis. There are things that you wouldn’t mind losing a minute of in case of emergency and things where you need absolute guarantees that they’ve been written to reliable storage immediately. Different data calls for different parameters.

Lately I’ve toyed with the idea of representing tournaments spawned on-demand as gen_statems and having these be restored from DB data on startup, meaning I’d probably have to store the last finished matches in the tournament, etc.. In the end, it’s about exploring what data you need and if it’s a feasible idea at all, just like any other development. This involves a lot of decisions that aren’t necessarily 100% right/wrong. As an example maybe you’d say that any started matches that are disrupted by a node failure would have to be replayed from the start, as live game data isn’t saved.

All in all, decide what your limit of data loss is and follow the breadcrumbs to your solution.

But a general pattern is to cache the state to a DB at some interval of time? Or is it more common to persist it at a given user or system event?

I’m asking because I’m having a hard time seeing a “natural” point where I would cache the state.

I mean, if every process caches to DB I see little point in doing this. Rewriting the state of the whole app every few minutes seems like an expensive task. So would it be more natural to let supervisors cache whatever hasen’t been cached since the last cache.

There’s a lot to take in here, so sorry if the questions are not very well formulated.

Great discussion!

Where did he say that? Any links?

One thing I’d worry about (if things weren’t in the DB) is running out of memory - guess not as big an issue when the app is something like stack overflow (who I believe keep all data in memory) but when you get to the size of a big social network it might be a different matter?

What does that “DB” word do there? ;-).

I’ve had a couple of times in the past where we went full-out TDD and YAGNI, and ended up with no database. Just some code dumping data out to disk in whatever the native format was (in this world, :erlang.term_to_binary is your friend). As someone remarked, a database is especially useful if you need to coordinate multiple instances of your app, so in modern systems it often serves more as a distribution tool than a persistence tool, if you want that categorization. To that end, it is often the simplest (most readily available, best documented) tool for the job.

AFAICT Pat Helland, e.g. http://queue.acm.org/detail.cfm?id=2884038. You’re welcome

(and yes, yes yes - databases throw so much information away while condensing the stream of events that created them down to a mere snapshot).