General Architectural Advice For A Ported Application

Questions / Help
, , , ,
1

I’m looking for some general, high-level architectural advice and opinions because I want to port a system over to Elixir which is currently written in Node (data processing server) and the meteor JavaScript framework (web interface). Sorry that this is such a long post!

I have a number of devices which send data to a server via TCP. We also have data coming in over HTTP which comes from things like an SMS service, webhooks from MailGun, etc. Incoming data is stored in a database after it has been validated and processed. At the moment this is all done on servers running Node.

There is also a web application which allows users to log in and view data as well as allowing administrative users to set up and manage devices, etc. This is currently a Meteor-based system.

We also have a basic API so that an iOS app can interact with the system and the API is part of the Meteor application (it was 50/50 as to whether it should be part of the Node system but credential authentication was easier if it was part of the Meteor application).

In Elixir I think that I will need the following but this is where I need some advice before I head too far down the wrong path:

  • A database application which is an Ecto ‘layer’. It will contain schemas and data management functions and essentially be an API for everything else to use to centralise database access.
  • An Absinthe GraphQL application which will be used to allow external systems to interact with the system. For example, the iOS application will use this as and we could also allow third parties to use it to retrieve data for reporting purposes, etc.
  • An incoming data application which will use Cowboy to handle HTTP data for things like the incoming SMS data and webhooks.
  • Incoming data applications which will use Ranch to create TCP servers for incoming device data. Rather than having a single server as I did i the Node application it would make life easier to have several of these, one per device type so that I can simplify parsing data from different device types rather than having to determine the type of incoming data first and then parsing it accordingly.
  • A Phoenix application which will be the web application for viewing data, managing devices, etc.

I’m sure that some of these could be rationalised and merged (a single Phoenix application could, I think, encapsulate Ecto, Absinthe and the Cowboy functionality) but I want to be able to scale functional areas separately. In the current system, mainly for redundancy, we have two servers running the Node data processing code which sit behind a load balancer and we have two servers running the Meteor application, again sitting behind a load balancer. We have separated the data processing from the web application because the volume of devices reporting in will always grow much faster than the number of operators using the web application so we want to be able to add more data processing servers without having the overhead of having to add more web application servers.

At the moment my thoughts are to have incoming data processing handled on a server which runs an application which supervises an instance of the Cowboy application and instances of the Ranch applications The Cowboy application and the Ranch applications would all use the Ecto application for database access.

The web application server would be a Phoenix application which use the Ecto application for database access.

The API application server would be an Absinthe application which also uses the Ecto application for database access. I’d like to separate this from the Phoenix application, again for scalability but also so that if the Phoenix server(s) went down there would still be access via the iOS app (and vice versa). Am I right in thinking that Absinthe doesn’t need Phoenix and can use Plug instead?

Does that all sound reasonable or is there a better way to structure things?

Can I separate my Ecto application from Phoenix and Absinthe in the way I’m thinking I can?

I really just want to be as sure as I can that I’m not starting off down the wrong path.

2

Phoenix already separates your web layer (MyAppWeb) from your application layer (MyApp).

You can choose to create an umbrella app or just go with a simple phoenix setup and just expand it:

lib/
- my_app # contains business logic (Ecto, etc)
- my_app_web # contains phoenix web
- my_app_api # contains your GraphQL layer
- my_app_ingress # contains your incoming device data
( ... )

At Novistore we’re using the simple approach and all our containers contain the same code base with all apps started on different ports. This allows us to have one code base with which we can use to build different images.

edit: Absinthe does not need Phoenix, see: https://hex.pm/packages/absinthe_plug
edit2: Maybe you don’t need a web layer, you can always expand the GraphQL layer and build an JS app for the frontend.

3

Great, thank you and there are definite advantages to having the same code base across servers which are configured differently.

I’m going to develop the web layer last because I want to see what Phoenix LiveView offers but using GraphQL and a JS app is definitely a possibility.

Thanks again.

4

To help others who might find it useful, my searching skills were obviously lacking because I’ve just come across another forum post about this sort of thing which gives some great advice and opinions.

1 Like