I am planning on building my first phoenix app in a couple years. I want to decouple the project into several applications that work together.
App1 is a web scraper that pulls info from one site
App2 is a phoenix app as the front end to allow users to use the web scraper info
Should I have a third application that uses Ecto to persist data from App1 and App2? Is Ecto still popular compared to things like graphql, or ets tables.
just a few ideas …
decoupling components has the advantage that you have a defined interface (eg json) between those and later you can replace the persistence layer…
Onthe other hand then you could have network latency issues, that slows down your app and you cannot use DB-transactions as you would have when you would use ecto wit an RDBMS directly in the apps.
For your first application, I’d strongly suggest building it as a single application with two or more libraries. In other words, decouple at the code level, not at the deployment artefact level. The beauty of Elixir is you can sanely ship a monolith that is internally broken up into individual services and save yourself a whole bunch of deployment headaches. This is the default for a phoenix application - there is a myapp_web library and a my_app library generated. The former manages the front end interactions (web serving, routing etc) and the latter manages business logic and persistence. Commonly the business logic / persistence library would also include background workers (e.g. to do the scraping).
Don’t overthink the background web scraping at first. In Elixir, thanks to OTP, it’s really easy to spin up background workers with no additional infrastructure. You can add timers etc to make them periodic, and you can add supervision to make them restart if they crash for whatever reason.
Ecto is definitely the go-to for bridging between Elixir and typical RDBMS (Postgres is the database server of choice for the community, and probably the best supported option in Ecto). The Ash Framework offers a more declarative approach built on top of Ecto.
ETS is useful where you only need temporary storage - it gets wiped when the instance restarts. There are other options for storage native to the platform (ie no need for Ecto and Postgres), but you would need to explain your use case in a little more detail to get useful advice.
Graphql is more about interop than storage, and is well supported through absinthe.
I have already built this app once as a full stand alone app 2 years ago using a single phoenix application. The web scraper was a GenServer that used wallaby to scrape the page (it is very JS heavy). I want to rebuild it, but this time I made an API for the scraper by reverse engineering the site.
I watched Dave’s coding gnome course recently and I did enjoy how he broke up his hangman app, but he mentioned how he never uses ecto. It made me wonder what other people use because I haven’t kept as up to date lately.
What I am thinking so far is packaging my API into it’s own app. The data it is scraping does not need to be stored to any sort of database because my other app makes actions based on availability of what is being scraped at that very time. In my original app the web scraper would look to see if an item was listed and update a GenServer with the state of how many items are currently available. Each item is completely unique and will never be seen again and will have no importance later on in the future.
Right now I am leaning towards making a Phoenix with ecto to store data like users, and purchases, and a second app that is the new API. I might use an ETS table in the phoenix app to store the state from the scraper.
If you’re willing to listen I can DM you the actual project details if you’re willing to give some advice. I have built many tiny fun phoenix projects over the years and was previously a Rails developer on a small team a few years ago, but I moved to a very remote location and haven’t met with other devs like I used to so I am not getting the answers to stuff like I did in Rails.
© Copyright Elixir Forum | Terms | Privacy & Cookies
