A second opinion on importing multiple large XML Files

Before we can answer that: what’s your goal with that project? Why Elastic docs? Why events?

It’s essentially a price comparison website. Standard use would be like any regular eCommerce store and you can browse products and search (hence elastic) etc. The main difference is that the product page itself shows the price at multiple retailers, you click the one you want to purchase from and we affiliate link you off direct the product on their website.

Each retailer provides a product feed and I need to import them, filter unwanted products, match up to existing MPNs and save the prices at each retailer.

Having thought about it some more there’s an option to avoid events and queues and just use something like flow inside each Oban worker/task? But in short, I just need a reliable and relatively quick (in terms of completing the task) way to import from multiple feeds.

The more I think about it, Flow may well be a better way to tackle this initially.

At a glance this does seem like overkill. Mixing technologies is a large source of complexity, and you can accomplish all of this with fewer tools. Given my natural biases, I would stick to Oban and PostgreSQL:

1. Define a periodic job that runs every four hours. The job fetches the xml itself and spawns sub-jobs to fetch in parallel.
2. Sub jobs process each feed either as a whole or in chunks—1000 items won’t take long to process.
3. Write documents directly into Postgres and add some indexes to speed up full text search.

This implementation only requires one database and one-two Oban workers, depending how you define them. Then all of the effort is in your feed handling and processing, not in gluing multiple systems together.

Down the line, if it is working well, you could make ingestion into a batch and layer notification callbacks on it. For now, start simple!

You’re right and the solution I threw together this morning pretty much mirrors this.

I have an Oban worker that runs every few hours and iterates through the retailers, creating a new worker for each which fetches the XML, streams through it and saves it into the database.

You read about these things and try to use them, often forgetting that it’s really not necessary (especially at this sort of MVP stage)

The huge advantage of your (and @sorentwo’s) approach is that you can extend it in the future by reading the Postgres records and import them in a proper searching backend like Elastic. But you don’t have to do it immediately and can get away with having a half-baked search through Postgres’ full-text searching abilities for a time.