Hello guys! (This is my firrst post in this forum, been just creeping everyone’s post)
So I have been playing with elixir for a bit now(also erlang recently very briefly), and I this idea that came up. I would like to introduce elixir into my company by building a sort of small server/server that can monitor my company’s interfaces. So for example we have multiple step for a single process “Create a vendor on one of our platform” -> “Save to another database that sits on another server” -> “check info against some other database or platform that we got”. Of course each step does more than what I am mentioned, but i would just like to have a representation of whats important to us.
Now here comes my questions. I was thinking I could create a server/client that would receive each task completion via a REST API, and display a sort of animation of the task next step or route with a client. I am thinking of using GenServer(or GenEvent) and DTS/ETS to accomplish this.
GenServer(GenEvent) : has the manager that handles all the income task completion , instead of having a controller expose all the logic.
DTS: to sort of manage all the task data and hold the next step that will proceed to and if it has been accomplish or not, and save the info in a file(sort of a log), and i was thinking of have a sort of job that would deleting old file base or certain that frame that I select. But my worry is how would I hold user account if I would like to implement it this way. is ok? or safe? I dont know have the answer to these questions.
I am not too familiar with these methods, but since I would just like to just show updates of each task, and keep the records for max a month or so, I’m thinking that avoiding using a database could be possible. So the idea would be that every time a data comes in I would update the client via channel (or maybe Server Sent Event?).
Anyone with experience could advise me on how I could proceed with this or what are the pros and cons ? I’m not familiar with elixir an erlang, but with all the concurrency and OTP ability that it has I feel like this could be the right language to build this in, and it seems like it could be a fun project.
It is pretty slow here on weekends, should get more responses come tomorrow.
Welcome!
Big first question is how can those other systems be monitored? Is it some polling that has to be done? Can they broadcast events to some existing infrastructure that you can hook in to? Regardless of how it is done it can be done on the EVM, just need to know how to be able to describe better.
GenServer is fine if you want the system to perform any processing on the EVM side. Unsure what you mean by animation, are you wanting to display a webpage of the overall system monitoring? If so then that would best be done by ‘channels’ in Phoenix, designed just for that kind of purpose.
Depends on the work it needs to be done. A GenServer(Gen*) is not the basic separation of data in the functional world, but rather a module is, so you may not need a GenServer depending on how you get the data (like via an event), though you may need to with others (like polling).
For user accounts and such you can use DETS or Mnesia fine (Mnesia over DETS though in 90% of cases), however for generic data storage especially of logging data that can potentially exceed 2gigs over time, definitely use a real database like PostgreSQL or something (and Ecto works natively with it so you’d have a very nice interface in to it).
If you are just holding known state and not needing to store anything past restarts then a simple web interface with a fairly trivial no-state login would be quite easy to do, but again that all builds up based on how you get the information from what you are monitoring. And yes a Channel is one of the most efficient ways to do the client-updates of a webpage.
Perfect language for it, just definitely need to figure out the interfaces better of what and how you are monitoring so you know how to start processing that information.
Well for an example we have an interface(Boomi) that is in the middle of two platforms. First platform is where the end user creates for example the vendors, then sends an outbound to our interface which process the data and creates the info to another platform(for e.g with got NiFi but i don’t think that will be one of the systems) that we got. Within the interface/platforms I can make http requests. The idea would be that the interface would push some data to notify my EVM server/client to notify that a task that has been executed, and would that info that I receive would let me know what is the current condition or next step that the task will need to go through. Wither it has been created successfully on one of the platform or not.
The thing is that we have multiple platforms which a data might need to go through, and I would like to build a (yes with some html/javascript) specific monitoring system. So what I was thinking is that my server/client would be receiving chunks of data just containing small information, and then update the webpage based on the incoming info(so same as a server event). So do I really need to keep a constant channel open? If I’m not losing anything then that’s fine.But, the idea is that I would have to define the step of each task that I want to monitor and hold it somewhere, so that when the data comes it knows what task its reflecting. So I could be monitoring different type of tasks for the same systems(so multiple channel?).Does that makes sense?
e.g:
Task 1 : create vendor(platform 1) → (interface) process data → create data to another platform(platform 2)
Task 2 : platform(2) prepare to sync data to platform(1) → interface receives data check if platform(1) ready for receive → platform(1) save new data
Same systems but different task that i could display on a web page.(So each type of task would have is own page. Maybe?)
Well i would really like to use Mnesia or DETS because it would help me understand them better and give me a reason to touch them . PostgreSQL I’m already familiar and used a lot in Rails. But because I need to define the structure of my task and maybe I would also like to have user accounts(which is not really needed now) I was thinking that I might definitely need a real database. The thing is that I dont want to have to manage a database later on if this project fits perfectly. haha.
Well right now just to find out where something went wrong they gotta backtrace and they can only hold 2 weeks old info. So I was being nice when I said that I would hold like a month past of log info. If later on I feel that I’d rather use a database I will switch it to a database. But, if its clearly pushing it using DETS or Mnesia then ill just go with a database now.
@OvermindDL1 Thanks a lot for your detailed reply. Replying back to you took me a least 5 -7 mins to write. I appreciate you taking the time to reply with such detail!
Ahh so it will be mostly a Push interface, that makes it wonderfully simple.
In that case you’d want to just set up a proper path in your router for them to communicate too, and if it is just status information and you do not care about history then I’d keep the current state of each in mnesia while also broadcasting it out via the Phoenix.PubSub system. A channel can grab the ‘current’ state from mnesia and send that to the client while also subscribing to updates (subscribe to the PubSub topic before grabbing the current state just to make sure no race condition) that the events will be sent over. Assuming some simple kind of event where the msg is just something like a map with keys, one for the server and the other for its state, then very simple, can add more with ease as well. Just have the channels listen to those PubSub messages and re-route them over the channel as necessary.
A persistent channel is always better, and very cheap in Phoenix unlike most web frameworks, so yes, keep it open.
But yep, with the style I mentioned above it would be easy. But I’d definitely say that you’d want to make a unified view of your messages, I’d encapsulate it in a struct, maybe something like:
defmodule TaskState do
@derive [Poison.Encoder] # This makes it so you can auto-serialize it to json
defstruct [:server_id, :state, :comment]
end
And I’d have each just be a binary/string for ease of use since it is mostly for being consumed by clients. I’d maybe add some more parts for things like how long it has been running or so if you want to display that information too. Then just store this even in mnesia with a key of the server_id or so along with also broadcasting it out on the PubSub then have the channels listen for it and re-push it out along their channel (at this step you could easily filter so someone only sees the server_id’s they are interested in too).
You could easily have each server_id have its own page with a listing of its own tasks, if you go more fine grained like that then your mnesia key would be a tuple of, say, server_id and task_id for example.
Mnesia is fine as long as your data stays below 2 gigs, which if you are just storing the current state and little or no history as well as users, that is all fine, definitely use Mnesia though, Mnesia is a wrapper around ETS/DETS that provides transaction support and better look-up abilities. You rarely want to use DETS straight (it forces a disk sync on near every operation), although using ETS straight has many uses (like a global in-memory map).
As long as the total data stays less than 2 gigs it is fine. Regardless of what you choose, mnesia or postgresql, always put all functions that access them in other modules that you call so you can swap it out when you need. If Ecto had an mnesia layer (someone needs to make one) that would be great for abstracting it out, but for now just a module is fine.
Heh, sorry, I have a bad habit of making long posts. ^.^
. PostgreSQL I’m already familiar and used a lot in Rails. But because I need to define the structure of my task and maybe I would also like to have user accounts(which is not really needed now) I was thinking that I might definitely need a real database. The thing is that I dont want to have to manage a database later on if this project fits perfectly. 
haha.
