I used to deal with redis, celery, mq for long run process. What the benefit of using redis or mq in elixir
3 Likes
First and foremost, Redis, RabbitMQ, etc. are made to interface with many different languages.
Elixir(Erlang)'s built-in distributed data stores (such as ETS) and message-queues (such as the new GenStage) do not typically have a gateway to the outside world. As long as everything you build is on the BEAM (Made with Erlang, Elixir or other languages on the Erlang Virtual Machine), this is not a problem, and it will really simplify your stack and increase maintainability and portability (You can easily create an āall-inclusiveā distribution of your app).
The main reason to use Redis, RabbitMQ et al. is to interface with environments that cannot communicate in this way. The advantage is interoptability with other languages. Another important reason why Redis is still used in Elixir projects, is that Heroku, which is a place where many people deploy their applications, does not (as of now) allow Elixir applications to talk to each other in the BEAM-style.
Interestingly, both RabbitMQ and Riak, a common alternative to Redis, are built using Erlang.
9 Likes
It really comes down to the requirements of your system.
Durability
If your machine(s) is unplugged and the disks are destroyed, is it OK to lose all the jobs that were in the queue and currently running? If not, you need to use something like redis/rabbit/sqs.
Distribution
Do you need to be able to split the load across multiple machines? If so, then it is much easier to use a system like redis/rabbit/sqs. You can do this yourself, but Iād recommend against it because itās not a trivial problem to solve yourself. (Iād love to be proved wrong on this.)
If you are building something super small and it just needs to be good enough, then please do use elixirās facilities. However, if the above two issues matter then you should look into adding a real message queue to the system.
Cheers,
Ben
2 Likes
@nerdyworm I find your arguments a little flawed; here is why:
Both Persistence (what you call durability) and Distribution can be managed without problems from within the BEAM. In fact, the BEAM was built exactly for this purpose: To have multiple systems communicate and divide the work, and to keep the service running even in case of a hardware failure.
This has been used by Ericcson to create a phone-handling server with 99.999999999% (nine nines) uptime. It has also been used by WhatsApp to create a messaging application with one billion users that is maintained by just 50 developers.
3 Likes
@Qqwy Like I said it does depend on your systemās requirements and use cases.
However, Iād love to see a real example of a roll your own message queue that can deal with machine failures, net splits, and data loss. The only projects that I can find that handle these things well are rabbit, sqs, disque, etc.
Please, please, please. Iām begging you. If this is actually trivial to implement as a production-grade system in erlang/elixir I want to see it.
I really hope to one day change my opinion on this matter.
Thanks for the response!
Cheers,
Ben
P.S. I skipped the benefits of using such systems. You donāt need to deal with implementing any of the stuff that Iām saying is nontrivalā¦ you get that for āfreeā 
That is all Iām trying to convey.
@nerdyworm Thank you for your reply!
Yes, youāre very right. It depends on your projectās requirements. Of course there still are things that you cannot do (or can only do badly) using Elixir/Erlang. A good example of this would for instance be math-heavy calculations on large chunks of (big) data: While Elixir is very good (and arguably better than the alternatives) at distributing and paralellizing work, performing math is slower than in low-level (imperative) languages that can re-use the same memory location.
Another problem Elixir faces right now, is that there are many outside resources that only expose bindings to other languages. To interface with these resources, either a new interface needs to be made, or you have to work with an intermediary program in another language, and interface with that.
As for the subject of message queues, RabbitMQ itself is actually built using Erlang, and will (as far as I know ā I am not an expert and havenāt used RabbitMQ in production) happily work alongside as well as inside the BEAM.
And in the cases you only need to connect nodes that all run Elixir/Erlang, you can just use the built-in OTP tools to do the job: Message passing in Elixir/Erlang works transparent between nodes; it works exactly the same regardless of if two processes are on your local computer or one of them is someplace else. You can completely configure what should happen in case the connection with another node is lost; failover, takeover, etc. As OTP is part of the standard library of Elixir/Erlang, I would consider it ābuilt inā, rather than ātrivial to buildā.
The best source for more information about these built-in tools in OTP is probably the Learn You Some Erlang for Greater Good book which can be read for free online, and the Elixir in Action book, which is a very nice and comprehensive guide, specific to Elixir (but cannot be read for free).
Relying on external systems for things you donāt want to build yourself is great! However, relying on too much work made by others :
- makes it very hard to reason about what is going on; The main problem being that these tools are often written in their own (domain specific) language or at least have some very specific settings that are hard to grasp if you donāt know exactly what they do.
- It becomes harder to deploy/relocate your application, because you have to move, install and configure many separate parts, each with their own caveats.
- You need to place trust for the correctness of your application in third parties. In some cases this becomes a problem.
Sola dosis facit venenum. Using external dependencies isnāt ābadā, but you can definitely overdo it. (I have seen too much applications that haveā¦) Often, adding a dependency is a lot easier than removing one.
Thank you very much for your reply!
~W-M/Qqwy
3 Likes
If all you need redis for is for pubsub messages, Phoenix already has a fantastic pubsub system built in. That is what many people use redis for Iāve noticed. Just pubsub to send notifications, emails, etcā¦
2 Likes
Question what will happen if speed of incoming messages is >> speed of how fast you can process these messages?
How big buffer you can have in BEAM? Can you persist queue on disk (for example you kill machine where queue is)?
Question if for some reason you need to reprocess messages again. Let assume you have some bug in code. You applied fix in code and you want to reprocess some messages again.
For example in Kafka you can configure to store messages for x time after they are deleted.
2 Likes
I wish this nine nines figure werenāt quoted so much - itās not representative of actual usage in the field.
See e.g.: http://stackoverflow.com/a/26447543
1 Like
I actually ran into this exact same question. For my use case we need to crawl URLās in a controlled fashion, not as fast as the machine can handle.
So using exq, redis and GenServer allowed us to generated different queues, control concurrency and schedule work. (GenServer is only used to schedule the initial āburstā - exq queues up all the other url hits after).
1 Like
Itās a bit irrelevant you can build system in Erlang that will fail every n seconds on the other hand if you had to design a system with ānine ninesā I would guess itās not a bad starting point
There are a few things to consider that havenāt been mentioned:
- You have more options for deployment if your cache is deployed separately (like Redis). In other wordsāyou donāt need to do hot upgrades, which while neat and all, take extra effort.
- Things like Redis and RabbitMQ have a ton of work put into them. Theyāre highly optimized, patched for security issues, etc.
- Many people have worked with Redis/RabbitMQ, but only your team has worked with your custom solution. This can actually reduce the overhead of learning a new system. There are also plenty of docs/examples for them.
- Some of the bigger hosts offer easily deployed and managed versions of Redis, which allows for easier deployment and scaling. Amazon has Elasticache and Rackspace has Object Rocket, for 2 examples.
Maybe itās because Iām still new to the Erlang world, but the idea of tying any sort of in-memory persisted data to the thing Iām deploying multiple times a day seems much worse than tying it to the thing that I upgrade once a quarter or so.
1 Like
Let me stress that using tools that exist inside the Erlang Virtual Machine does not at all mean that youāre limited to storing everything in memory. Nor does it mean that you need to (re)build everything from the ground up.
There are also security concerns to take into consideration. Sure you can build your system as a large cluster, but that has some serious implications on security. If your nodes are able to communicate directly and execute code on each other, meaning single point of failure gives attacker full access to the system.
Having something like RabbitMQ, Riak, redis in betweens (depending on the context) allows you to separate the nodes, set up various security precautions that you canāt do otherwise.
Of course you can do it all yourself, but why when there are good and proven existing solutions to the problem?
Thanks @Qqwy for touching on this. Can you elaborate a bit more on the suitability of Elixir for math/analytics-oriented applications?
Iām just learning Elixir and considering using Elixir/Phoenix for the backend of a Mint-like web app. The app will principally track budget/actual entries, perform Monte Carlo / scenario analysis and produce datasets for front-end rendered charts.
Iām excited to learn and apply Elixir to this task; but would like to qualify the suitability of Elixir/Phoenix to this type of app.
Thoughts?
Wrote a blog post semi-covering it. http://www.brightball.com/articles/elixir-ets-and-mnesia-vs-redis
2 Likes
