Hi, I hope it wouldn’t be off topic.

I was wondering for a long period of time to create another web search. For now, I’m vastly disappointed with my search experience especially when I want to search something unrelated to StackOverflow (like a cooking recipe or some discussions). My boiling point has reached the final destination and I’ve started to look into how to build a solution.

My goal is to build web search only with high-quality websites (like websites who not asks for push notifications, doesn’t have popups or obtrusive ads and not using dark UI patterns).

I’m considering to use Sphinx to power search but also I need to store cached pages in order to generate previews or to reindex it later.

The hard part now is to decide what to use to store pages with metadata. For now, best candidates are Cassandra, CouchDB and RiakKV. Despite all solutions is schema-free (except Cassandra) I think strict schema will be better for my data structure.

My general requirements are to create a solution which can be managed with spare resources either time or money because it’s a sort of hobby project.

Cassandra looks like a good fit to store TB of cached webpages with Spinx on top of it but I don’t think JVM based tools is a good fit for my requirements.

So, I’m looking forward for any ideas for a tool to store TBs of cached web pages which can be horizontally scaled on demand.

Thank you