andrewnclark
A second opinion on importing multiple large XML Files
I’m fairly new to Elixir/Phoenix and just wanted some second opinions on my approach; as a side-project being developed by myself in a language non of my colleagues are familiar with I’m missing the ability to bounce ideas around. I’m hoping you can help!
The problem
Periodically (say every 4 hours), I want to import ten (will eventually be more) Google Shopping feeds in XML, process each of the products, filter some out and then store the data into Postgres and create an Elasticsearch document for each.
Solution
- A scheduled Oban task to get the XML files from URLs and then create an SQS event for each of the products in each feed (ten feeds x 1000+ products each)
- Broadway with SQS Consumer to ingest the tens of thousands of products, filter then, save to database and create Elasticsearch documents.
Question(s)
Do you think this is a suitable solution? Overkill? Am I missing something much easier that I’m not aware of?
Trending in Questions
Other Trending Topics
Latest Oban Threads
Latest on Elixir Forum
Sponsor Spotlight
Courses that'll move you from confusion to "Aha, now I get it!"
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #performance
- #security
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Marked As Solved
sorentwo
At a glance this does seem like overkill. Mixing technologies is a large source of complexity, and you can accomplish all of this with fewer tools. Given my natural biases, I would stick to Oban and PostgreSQL:
This implementation only requires one database and one-two Oban workers, depending how you define them. Then all of the effort is in your feed handling and processing, not in gluing multiple systems together.
Down the line, if it is working well, you could make ingestion into a batch and layer notification callbacks on it. For now, start simple!
Also Liked
dimitarvp
The huge advantage of your (and @sorentwo’s) approach is that you can extend it in the future by reading the Postgres records and import them in a proper searching backend like Elastic. But you don’t have to do it immediately and can get away with having a half-baked search through Postgres’ full-text searching abilities for a time.
andrewnclark
You’re right and the solution I threw together this morning pretty much mirrors this.
I have an Oban worker that runs every few hours and iterates through the retailers, creating a new worker for each which fetches the XML, streams through it and saves it into the database.
You read about these things and try to use them, often forgetting that it’s really not necessary (especially at this sort of MVP stage)