andrewnclark
A second opinion on importing multiple large XML Files
I’m fairly new to Elixir/Phoenix and just wanted some second opinions on my approach; as a side-project being developed by myself in a language non of my colleagues are familiar with I’m missing the ability to bounce ideas around. I’m hoping you can help!
The problem
Periodically (say every 4 hours), I want to import ten (will eventually be more) Google Shopping feeds in XML, process each of the products, filter some out and then store the data into Postgres and create an Elasticsearch document for each.
Solution
- A scheduled Oban task to get the XML files from URLs and then create an SQS event for each of the products in each feed (ten feeds x 1000+ products each)
- Broadway with SQS Consumer to ingest the tens of thousands of products, filter then, save to database and create Elasticsearch documents.
Question(s)
Do you think this is a suitable solution? Overkill? Am I missing something much easier that I’m not aware of?
Marked As Solved
sorentwo
At a glance this does seem like overkill. Mixing technologies is a large source of complexity, and you can accomplish all of this with fewer tools. Given my natural biases, I would stick to Oban and PostgreSQL:
- Define a periodic job that runs every four hours. The job fetches the xml itself and spawns sub-jobs to fetch in parallel.
- Sub jobs process each feed either as a whole or in chunks—1000 items won’t take long to process.
- Write documents directly into Postgres and add some indexes to speed up full text search.
This implementation only requires one database and one-two Oban workers, depending how you define them. Then all of the effort is in your feed handling and processing, not in gluing multiple systems together.
Down the line, if it is working well, you could make ingestion into a batch and layer notification callbacks on it. For now, start simple!
Also Liked
dimitarvp
The huge advantage of your (and @sorentwo’s) approach is that you can extend it in the future by reading the Postgres records and import them in a proper searching backend like Elastic. But you don’t have to do it immediately and can get away with having a half-baked search through Postgres’ full-text searching abilities for a time.
andrewnclark
You’re right and the solution I threw together this morning pretty much mirrors this.
I have an Oban worker that runs every few hours and iterates through the retailers, creating a new worker for each which fetches the XML, streams through it and saves it into the database.
You read about these things and try to use them, often forgetting that it’s really not necessary (especially at this sort of MVP stage)
Popular in Questions
Other popular topics
Latest Oban Threads
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"