So for my resources, and assuming I can’t yet guarantee 100% uptime, is it either Postgres or SQLite? Is there a way of perhaps working with ETS and saving its state to disk from Ash? There is no MongoDB support just yet, right?
EDIT: A little context, in case it helps: My goal here is to a) simplify my systems as much as I can. If I could just do away with the database and all its accompanying headaches, I would be a happy man. And b) engineer my systems to be performant from the start. The fewer roundtrips to external services, the better. Again, if I could just work in-memory, for most or even all parts of my apps, it would probably be ideal.
I get that it’s not always realistic or feasible though, of course. I’m only trying to get the lay of the land for now, so to speak.
So you want to build a system that should be persistent and scale also in the future but at the same time you don’t want to use the very software that was invented to do exactly this?
You can spin up a postgres instance somewhere for almost free and then connect to it using a query string. It can’t get any easier than this.
If you are not happy with the supplier of your DB as a service, you simply dump it, spin up a new copy somewhere else, load the dump and bob’s your aunty.
I would focus my efforts on trying to solve business problems and not re-invent things like performant persistent storage.
Ultimately, using PostgreSQL is that simplification. We have in memory options, and they are useful when you need them but the truth is (and this is true for pretty much any app, framework, language) that choosing an in memory store when a database would suffice is riddled with caveats, new challenges, and solving solved problems.
In the case of Ash, AshPostgres is the only data layer that supports every capability of Ash. AshSqlite doesnt support aggregates or atomic updates, but if you must eliminate a service, then I’d suggest using Sqlite and mounting it on a persistent volume.
There are very few universals in software engineering but probably the closest I’ve come to is “just use Postgres if at all possible”.
I guess I’m just a little wary about it. I have had bad experiences with minor version updates breaking the very format used to save the data so after a routine system upgrade I got backends refusing to connect and crashing. Happened at least once a year when I would use it. Later I got to work with MongoDB and that wasn’t a problem anymore. I guess I’ll just have to use a specific containerized version and leave it at that.
Interesting, I’ve not had that problem. In fact I’ve done Postgres major version upgrades in place across multiple major versions (and every major version since 9) and never had a problem Maybe you have bad luck
To be fair, I’ve never been an expert at PostgreSQL. Maybe I did something wrong. I remember distinctly the error message saying something like “This is PostgreSQL v x.y.z. Your database was created by version x.y-1.z, so we can’t read it and you’ll have to migrate it manually. Go to this website to learn how” More or less… The 2nd time it happened I just ran away and been avoiding it ever since lol
True, I’ve migrated once a 2TB production database from PG9 to PG14 using replication and it just worked out of the box, we were all shocked on how we did the migration in less than a week with a few minutes of downtime.
See, that’s exactly the kind of problem I am trying to avoid. It just seems so unnecessary to waste a whole week doing a maintenance chore when it should just run flawlessly IMHO, like it had been doing for the past weeks or months. Maybe I’m too used to MongoDB, but I would die a little inside if I had to pause my one-man-startup for a whole week every once in a while just because of a tool upgrade.
Please don’t try to twist my words, I never said anything was wrong with the running setup that was working for well over 7 years under some really heavy load. We migrated to the new version because we wanted some of the nicer shiny things for optimization and new features. We also changed caching configuration and tuned some other things for the new instance. It also took a week, because we did the migration with downtime under 5 minutes, that was a hard requirement.
If mongo works for you then nobody said not to use it, but I can say that I’ve used postgres for all the startups I’ve worked on and I would never consider using mongo ever, I am not a fan of their paid plan and the limitations around it, also the businesses I worked in, data integrity and database level validations were a requirement.
Sorry, I wasn’t trying to twist your words. I just saw that migration as a problem, but I see it wasn’t, only an opportunity you took to improve your system.
Thank you all for your replies. I learned a lot today
“This is PostgreSQL v x.y.z. Your database was created by version x.y-1.z, so we can’t read it and you’ll have to migrate it manually
is because you ran postgresql as a container and tried to bump the version in the docker-compose version. (Of course I’m just guessing here but this I have seen this exact error when people (might have been me… I woudn’t recall…) tried to bump the version in the compose file )
It is generally a very bad idea to run postgres (or any DB in a container). For dev and test we do this but production systems should not have the DB in a container.
Nope. Native, installed from the distro’s package manager (I believe it was pacman at the time). The PM got wind of new packages, I updated, and the rest is history.
But how do you not run into this problem if you do not containerize your DB? Every time there’s an update there’s the risk of a breaking change, isn’t it? Other than freezing the whole system or risking partial updates I don’t see how
Postgres provides an extra utility for upgrading called pg_upgrade:
pg_upgrade (formerly called pg_migrator) allows data stored in PostgreSQL data files to be upgraded to a later PostgreSQL major version without the data dump/restore typically required for major version upgrades, e.g., from 12.14 to 13.10 or from 14.9 to 15.5. It is not required for minor version upgrades, e.g., from 12.7 to 12.8 or from 14.1 to 14.5.
All of our customers run postgresql and I have never heard of any issue with a postgresql update (that wasn’t caused by the meatbag between the monitor and the seat).
Also nobody just casually upgrades the DB version. I mean it should work but your sys admins should at least have the common sense to not just go “hurr durr lets upgrade postgresql while I’m at it. What could go wrong?”
Well in my defense it was my personal computer, which I use for prototyping and I like to keep up-to-date. I know, I know, it was the meatbag’s fault. I’ll see what I can do in the future so it doesn’t affect me again, at least without me knowing in advance.
Postgres releases require the use of pg_upgrade for major versions, while minor versions will migrate seamlessly. Every distro handles this differently, but Arch specifically expects you to do the upgrade yourself. It’s a minor nuisance, but major Postgres releases only happen once a year.
Edit: Forget to mention, but on Debian, I think most people use the official apt repo the Postgres team provides, which namespaces by version number. So, you never get caught off-guard with a version change while installing updates. Or they provide the option of using one that automatically upgrades and migrates for you, always keeping it the latest version without hassle. That’s what do personally.
Maybe you have bad luck 
)
