Happy new year!
I’m trying to create a migration system that takes ids from another websites and convert them to my app ids
There are 3 websites available to import the data, the quantity of ids from every website is ~200.000 which are divided in 4 different categories of migration (EG: Movies category ids, series category ids etc…)
The migration data (website id to my app id) is stored in the database, but some users can have really large lists of content to migrate so fetch from the database every single migration value isn’t a good option, so I decided to use Elixir to handle the problem, my main idea looks like this:
- Create an Agent to maintain the state
- On init the Agent query the database and get all the migration records and save into memory
- User give a list of ids and the Agent return the app ids
- If a given website id isn’t on the agent, query the database to check if it’s on it, if is return the value and update the Agent state (The database can be updated with new migration ids over the time)
But here is where I need some help to design the app,
- If I use a single Agent to maintain the state can lead to bottlenecks if many processes request that data
- The point 4 can lead to performance problems, since a user can give a huge list of non existing ids will query the database a large amount of times just to ensure the id is existing or not
- Since the database migration ids is updated regularly, the Agent don’t have any way to know that a new migration id is added and that’s why the point 4 is on the list
- I’m not sure about this, but the use of memory can be really high since I need to save ~800.000 ids (the 3 websites and my app ids) always persistent in memory
So that are the problems that I identified (feel free to tell me if there is more), this is the first serious project that I’m doing with Elixir and I would like to have some feedback and help to design this migration system in a Elixir way, what are the approaches to solve this problem ^^
I also have a question regarding to the project in general… I have in mind to add this system to my phoenix app, so every instance of my app running on a server will have this migration system with it’s own state, is this a good approach? Or is better to run this service on a separate machine and the phoenix app communicating with it? (I want to point out that this migration process is computational expensive since I need to parse some import files and in some cases scrape content)
I hope can help me to find the best option to implement this ^^
Cheers!
PD: Feel free to tell me (or edit the post) if there is any grammatical error