TomGrozev
I am trying to setup elixir in a microservice architecture and have each of them (i.e. nodes) communicating efficiently. For context, there is a product microservice and a phoenix frontend microservice (there are more but these are relevant). I have got them connected through some custom module that is a dependency on both. This code basically spawns a task locally to spawn a task on another service to do whatever (i.e. create a product), this works great using a call similar to Connector.call(ProductMicroService.Products, :create, [some_product]). The downfall of this is when it comes to using Ecto schemas in Phoenix since it will try to call the schema module functions (e.g. struct) locally and not in the products microservice where the schema actually exists.
Now I have a couple possible solutions but none of them seem to appetizing and all seem very ‘hacky’. Some ideas I had are:
- Create copies of the schema modules on the frontend microservice
- Create fake schema modules on the frontend microservice that has all the functions but redirects calls the the real schema module
- Override the relevant functions in Phoenix (there are a lot)
- Some alternative?
For the more synchronous calls I believe the direct node communication is best and for asynchronous things a message queue (RabbitMQ, etc.) but is there a better way that I am missing? Anyway this is done, Phoenix still needs to be able to access these methods on the schema. It doesn’t seem to be an issue with anything other than phoenix.
Where Next?
Trending in Questions
Other Trending Topics
Latest Phoenix Threads
Latest on Elixir Forum
Sponsor Spotlight
Producing high quality Elixir screencasts since 2017.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #code-sync
- #discussion
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #ai
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #blog-post
- #elixirconf-us
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #elixirconf-eu
- #processes
- #security
- #api
- #forms
- #performance
Our Sponsors
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync. Next: ElixirConf EU 2027
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
**Building the future of software with AI, trust, and human connection.**
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 8- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
olivermt
Could you elaborate a bit on why you want it this way? It sounds like you are adding a lot of complexity for gains I don’t see.
To use built in node communication in a somewhat easy manner you need codebases that are either the same or very aware of eachother, which sort of brings you back to a monorepo just split across more servers.
Why not use a http layer for the synchronous parts? Build a authority of sorts on the web node and pass it along with each request to keep the microservice acl etc.
Laetitia
In microservices architecture, the microservices share “raw data”, i.e the interfaces can expose maps, not struct. Each microservice must have its “view” of the model. For instance, a product in microservice A may have some different attributes from a product in microservice B. The output of microservice A will be a map, and microservice B will receive this map and convert it to its struct or schema.
Phoenix offers a backend-template application, so you don’t need microservices and you can use the schema along all your application but if you really need microservices, it’s a good way to use Erlang node communication and to apply decoupling between them.
al2o3cr
Option 4: follow the first rule of distributed objects and don’t divide the system up this way.
AFAIK there aren’t any common patterns that would recommend turning context or view functions into remote procedure calls. Code in the “frontend” should interact with the “backend” via a clearly-defined API.
One other thing worth thinking about: the remote
Task.Supervisoris single-threaded, so unless you add a lot of complexity that can be a bottleneck at high concurrency.mpope
I have seen this solved before in a Java microservice cluster and it might work well for your case. Services that communicate had a shared library of DTO class definitions. They weren’t the entire schema, but it was more of a request level object. The same could work for your schema definitions for task spawning. Two services could share a hex package (or even a git submodule) that has the shared definitions. That might not be a very ‘erlangy’ way of solving things, however it seemed clean enough in the Java environment. The downside was that both services needed a redeploy for schema changes, and the receiving serviced needed code around to handle the previous ‘version’ of message format that could be cut away after a full deploy.
TomGrozev
The idea is that each service will likely be split across multiple servers with some extra instances of some whereas others may only have a few instances. Having the services split allows for teams to work on them individually and operate on their own and merely interface with other services. Using a HTTP layer is no different than using node communication, just the node communication is a bit more of a robust thing that will not require as much extra development effort as an entire HTTP API.
I understand what you are saying, so basically each instance has no knowledge of the source or, in a way, meaning of data, hence why structs will not work.
That is quite helpful thank you.
That is a good point, either will need multiple task supervisors or use an alternative for spawning tasks.
Yes, this is a significant downside as it doesn’t isolate the services entirely.
Exadra37
I would recommend you to read the @PragDave Programming 1.6 Elixir book or his video course to see how he suggests to tackle what you are trying to do. Basically you will work with what the Nerves team coined of Poncho projects in terms of folder structure, but with a very nice and clean separation as you want, were teams can work independently and code can live in separated repos, but are deployed in the same server until the day it is necessary to scale it to multiple servers.
al2o3cr
The phrasing is a little flippant, but the message is serious. Imposing a network boundary where there was only a module boundary before has significant costs:
The biggest hazard in building a distributed system is that you get all of those headaches automatically, but you only get the good stuff (independent scaling, independent deploy, etc) with the right implementation.
For instance, an Elixir novice who confuses
GenServerwith “a server” and makes each piece of their application a singleGenServerwill produce a single-threaded performance tarpit.The “rule” is a reminder to carefully consider the consequences of your decisions - what are you getting in exchange for all the additional complexity?
TomGrozev
I read this a long time ago, will give it a re-read with this project in mind, thanks.
This is a good point, but the idea of separating services is that this communication between them should be minimal. I see the error I made in having a separate service for the frontend, it does not seperate them correctly as it must know the context behind all the data, which breaks the rule.