I am an absolute Elixir beginner migrating from php, java, c# world. I want to build a super fast api server for Forex ( ie realtime currency trading exchange) with Elixir and (Redis, Cassandra or SqlServer, MySQL ?) Database. I would evaluate the system under 10million API calls and transactions per second. Elixir / Elang as i have read in some places offers the highest concurrecy and i assume it would make the best programming language for this experiment. I want to minimize the server overhead (CPU, RAM) at any cost. Must I used Phoenix or what am I to lose without it. Thank you.
NB. Any relevant resource to study as concerns such a project is appreciated!
Of course there is nothing requiring you to use Phoenix You have several options in the community, such as dropping down to plain Plug (which phoenix is built on), or using another Elixir webserver/library combo like Ace/Raxx, or even using Cowboy (an Erlang web server) from Elixir. Using Phoenix will not impose measurable overhead performance wise, and what you may lose by forgoing it depends on what exactly you’re building– which we have light details on. I will say Phoenix is great for API servers and it’s probably your most mature option. The SQLServer integration will be handled the same way regardless of your choice web wise. Maybe prototype out a couple options and see what fits best?
I guess you’d need to clarify what you mean by “super fast”, “forex api server”, and how you are planning to use sqlserver in this setup.
Since phoenix is mostly geared towards web dev, it’s TCP-based. For “super fast” you might want to go with UDP. Although I’m not sure what “forex api server” here means, do you plan to make API requests to some other exchanges, and aggregate the results? And you might want to pick an in-memory database instead of a disk-based one.
I know Erlang has been used for matching engine, so if you do the same then the most difficult problem is solved. As for supporting stuff like API, not sure Erlang/Elixir is a good choice, if you truly want to minimize CPU/RAM use you should be looking at C/C++.
Thanks for contributing! I will use Redis in place of sqlserver. “Forex api server” means I want to build a backend for currency exchange and trading where a frontend using angular or anything will feed from. “Super fast” bec it is expected to support millions of concurrent transactions. I am new to Elixir but I believe elixir is the favorite option.
Thanks for contributing. Since Phoenix has no performance overhead, then I will go with it. I ve updated the questions just to add details on what I am about to build. Appreciation!!
“Super fast” and supporting millions of transactions is not the same thing. One is speed, the other is concurrency. What I meant when I asked you for further clarifications was, I wanted you to specify some thresholds like <100us for how long a transaction in your opinion should take.
For financial data I’d worry about consistency first and foremost, not the speed. I’m not quite familiar with redis, but if it’s single threaded like voltdb, then it could be a good choice.
If you’re wondering whether to use Phoenix or not, I highly recommend you read the preface and opening chapter of Programming Phoenix - these chapters are free to download and honestly does one of the best jobs to sell a piece of technology I have ever seen! It’ll make the hairs on the back of you neck stand up
Havent been active for a while on this forum, but I do check the Forum Summaries regularly. Maybe it’s not my place to say, but I just wanted to say that the OP sounds a little “junior”.
Couple of things I noted:
If you had real requirements from a Forex trader you would have given more info on the performance aspect. And not just “millions” cause that says absolutely nothing.
It seems you take advice way to easy without further research (Yes, the answer was given by Chris himself, but still you would want to have measurable results if this was a real case for a real Forex related company).
A project of this size would have lots of money dedicated to building this and so buying a book must be absolutely no problem.
So conclusion that I can make is that you are building this in your free time, perhaps with some (online) friends. If that’s the case, I would start thinking about how to build this in your preferred language because the overhead of learning Elixir (Erlang) ecosystems besides thinking about the architecture of a Forex application might be to much :). This is no discouragement, just giving you my two cents. Build a POC first and keep on improving that based on requirements.
Thank you @Ilyes512 for contributing. I am an absolute beginner in Erlang/Elixir ecosystem.
Your assumptions are correct. I lead a group of hobbyists and we choose this project
because we read Elixir is optimized for apps that require fault tolerance and high concurrency.
We would love to buy recommended Advanced Elixir books but that would be when we
have exhausted free resources like https://elixirschool.com. We choose this scale of work
to broaden our knowledge to the core and to experiment with Elixir to its limit.
We have learnt that we have to use Phoenix (it’s probably for the best)!
We are investigating the most optimal database i.e. transaction safe RDBMS like PostgreSQL
or Scalable NoSQL like Mongo assuming we target 2million transactions per sec etc.
The Role of a Message Broker like Redis or RabbitQL
Accessing through WebSocket with UDP(native to Phoenix it seems) or REST API (TCP based)
As you see, it’s still a long way and already we have learned a lot like getting to know this forum.
When we are done, we can create Elixir Meetups, Elixir Youtube Channels and Medium Posts.
Just want to add a few more thoughts since I’m a trader myself and have previously thought about creating something like this. Most retail traders are trading CFD, and this is illegal in the US. While some may think CFDs are a scam, they are easier to implement from a technical perspective, because after all the broker house is not actually matching your order with someone else’s. Also it’s now possible to trade CFDs quoted in bitcoin…
Suppose you want to make a real exchange that does matching, apart from auditing, reporting, legality, how on earth are you going establish yourself as an international exchange? If you have the resources already I don’t understand why you asking in this forum.
I suggest you narrow down the scope of your problem, define what exactly your API is going to fulfill, and pick your technology last.
Apologies if I sound too critical but I hope to save you some time going down the wrong rabbit hole.
Thank you Sir @xlphs for your advice. The API is to serve any mobile, desktop, or web Forex application. Any exchange can just build on it without much work.
That’s ambitious, I like it. I would say build a bullet proof matching engine first, then implement all the different order types, trailing stops of course, and fancier ones like OCO. Data storage is probably the last thing I would worry about. And margin orders too, depends on how the actual exchange implements margin this may vary significantly.
Welcome to the community, @4dcoder! We’re glad to have you on-board
I skimmed through the previous replies and I can see that most of the tips and advice I would have liked to share with you have already been said, so I won’t repeat those things.
There is one additional thing I would like to mention:
I can see that you’re mentioning databases such as Redis in combination with performance-related topics, which suggests to me that maybe you’re considering using Redis as a cache of sorts.
If you’re working with ephemeral data (data which does not need to be persisted, e.g. a cache), then you might have better options with Elixir. For exmaple, you could store some state in memory by using processes or ETS tables, and persist to a database on some interval if necessary.
The book Adopting Elixir has a very good chapter about persistence strategies and dealing with ephemeral data, which you will find on pages 107-108 and 116-117.
Also, I highly recommend reading Programming Elixir, which is a very thorough and well-organised book. It’s not free, but it might save you quite a bit of time and effort to make up for the cost.
You have several options in the community, such as dropping down to plain Plug (which phoenix is built on), or using another Elixir webserver/library combo like Ace/Raxx, or even using Cowboy (an Erlang web server) from Elixir. Using Phoenix will not impose measurable overhead performance wise, and what you may lose by forgoing it depends on what exactly you’re building– which we have light details on. I will say Phoenix is great for API servers and it’s probably your most mature option. The SQLServer integration will be handled the same way regardless of your choice web wise. Maybe prototype out a couple options and see what fits best?
