Sanjibukai

Sanjibukai

How do you handle databases backup when (some) important data is involved?

Hello everybody,

It’s not really a Phoenix problem (and might even be a devops concern) but I’m asking in the context of a Phoenix App.
Until now I never really thought about database redundancy or backup (besides what it’s already done in the file system level by the VPS provider in case of a disk failure).

In my current project, payments will be involved and I’m wondering how I can make the payments data redundant so that if something bad happen I can start fresh with at least the data of who made payments. (In fact this particular scenario is already covered by the payment provider itself e.g. stripe which provide the history of the transactions).

I don’t really know what are the best practices for this kind of use-case and wanted to know what are the best practices?

  • Is it simply managed at the OS/FS level for disk failure handling? But what if it’s the VPS itself which is broken?
  • Do people use two (or more) databases at once (for each write/update/delete requests) which will run ideally on different provider?
  • Does another solution might be to still use only one DB and perform regular dump? (it’s an easy task when doing it manually from time to time but I saw that quickly becoming complex if it’s needed to do it more often and ideally by transferring the dumped data into another location)
  • What about an “event store” or some kind of dedicated DB/queue/pubsub system which will only store the payments events (with the associated data like email of the payer, etc.)?
  • Maybe people are using DB systems which have built-in support for write replicas? (IIRC mongodb have this kind of feature)

I’m really interested to hear how you guys are handling that or if you have some resources explaining what are the best practices.

Thank you!

Most Liked

shanesveller

shanesveller

Please don’t ever make this your “plan A”. That is not a DR strategy, that’s what you do under duress to save a capsizing business because you have no other choice. Logs are not a system of record, as many many things have to go just right for them to even get collected and retained.

hauleth

hauleth

VPS probably has some kind of RAID setup for preventing disk failures, you do not need to worry much there. But remember - RAID is not backup.

There is concept of hot/warm/cold-replica that replicates the data from the master and sometimes is used for reads as well.

You can use dumps or if your DB supports it, you can also use streaming of write-ahead-log (if your DB supports this). The latter is a little bit more complex to set up, but will provide you with point-of-time recovery (so you can recover state of the DB at any point you like).

It is log that will contain events as they happen in your application. That is it.

Multi-master replication is quite hard to get right, but most of the DBs have support for such (either built in or 3rd party). But if all you want is redundancy, then master-replica is simpler and easier approach.

danielzfranklin

danielzfranklin

Consider using an external service like Datadog or Papertrail or Logly for monitoring and log management. It’s useful to get alerts about the status of your server anyway.

Then, if you log everything needed to recover (like user x bought plan y) you can download the logs and write a little script to restore the db if you need to.

Where Next?

Popular in Discussions Top

andre1sk
A big advantage to Elixir is all the distributed goodness but for many applications running on multiple nodes having integrated Etcd, Zoo...
New
jeramyRR
This is an interesting article to read. Elixir’s performance, like usual, is excellent. However, it seems like the high CPU usage is co...
New
MarioFlach
Hello, I want to share a project I’ve been working on for a while: https://github.com/almightycouch/gitgud Background Some time ago I ...
New
axelson
Decided against including more info in the title, but the gist is that Plataformatec sponsored projects will continue with the assets bei...
New
AlexMcConnell
The reason that Rails is as popular as it is is because it’s very easy for relatively inexperienced developers to get a lot of work done....
588 19568 166
New
Crowdhailer
I’ve been hearing much about the new formatter and it’s something I have been keen to try. I find examples buy far the most illuminating...
248 19204 150
New
crabonature
I’m still quite new to Elixir. As I understand we got in Elixir “multi guards” as convention to simplify one large guard with or’s?: de...
New
shishini
I think this twitter post and youtube video didn’t get as much attention as I hoped I am still new to Elixir, so can’t really judge ...
New
pdgonzalez872
If this has been asked here before, please point me to where it was asked as I didn’t find it when I searched the forum. Maybe a mailing ...
New
paulanthonywilson
I like Umbrella projects and pretty much always use them for personal Elixir stuff, especially Nerves things. But I don’t think this is ...
New

Other popular topics Top

Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a > b) do {:ok, "a"} end if (a < b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
gausby
I asked this very same question on twitter and got some interesting feedback, but I thought it would be a good question to ask here as we...
1207 39297 209
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
marick
I had some trouble figuring out how to make many-to-many associations work. Once I got it working, I wrote a blog post. Because I’m a nov...
New
dogweather
I wrote this comment on r/haskell, and it’s not popular there. :wink: But I think I’m on to something… Haskell reminds me of Java, and e...
New

We're in Beta

About us Mission Statement