Thanks for contributing! I will use Redis in place of sqlserver. “Forex api server” means I want to build a backend for currency exchange and trading where a frontend using angular or anything will feed from. “Super fast” bec it is expected to support millions of concurrent transactions. I am new to Elixir but I believe elixir is the favorite option.
Thanks for contributing. Since Phoenix has no performance overhead, then I will go with it. I ve updated the questions just to add details on what I am about to build. Appreciation!!
“Super fast” and supporting millions of transactions is not the same thing. One is speed, the other is concurrency. What I meant when I asked you for further clarifications was, I wanted you to specify some thresholds like <100us for how long a transaction in your opinion should take.
For financial data I’d worry about consistency first and foremost, not the speed. I’m not quite familiar with redis, but if it’s single threaded like voltdb, then it could be a good choice.
I recommend to read article: The Road to 2 Million Websocket Connections in Phoenix and many other tips, tutorials etc. shared by community.
If you’re wondering whether to use Phoenix or not, I highly recommend you read the preface and opening chapter of Programming Phoenix - these chapters are free to download and honestly does one of the best jobs to sell a piece of technology I have ever seen! It’ll make the hairs on the back of you neck stand up
I checked out the book but it wan’t free.
Sorry, only the Preface and Intro are free - but you can get 35% off the ebook with your forum discount
Havent been active for a while on this forum, but I do check the Forum Summaries regularly. Maybe it’s not my place to say, but I just wanted to say that the OP sounds a little “junior”.
Couple of things I noted:
- If you had real requirements from a Forex trader you would have given more info on the performance aspect. And not just “millions” cause that says absolutely nothing.
- It seems you take advice way to easy without further research (Yes, the answer was given by Chris himself, but still you would want to have measurable results if this was a real case for a real Forex related company).
- A project of this size would have lots of money dedicated to building this and so buying a book must be absolutely no problem.
So conclusion that I can make is that you are building this in your free time, perhaps with some (online) friends. If that’s the case, I would start thinking about how to build this in your preferred language because the overhead of learning Elixir (Erlang) ecosystems besides thinking about the architecture of a Forex application might be to much :). This is no discouragement, just giving you my two cents. Build a POC first and keep on improving that based on requirements.
Thank you @Ilyes512 for contributing. I am an absolute beginner in Erlang/Elixir ecosystem.
Your assumptions are correct. I lead a group of hobbyists and we choose this project
because we read Elixir is optimized for apps that require fault tolerance and high concurrency.
We would love to buy recommended Advanced Elixir books but that would be when we
have exhausted free resources like https://elixirschool.com. We choose this scale of work
to broaden our knowledge to the core and to experiment with Elixir to its limit.
- We have learnt that we have to use Phoenix (it’s probably for the best)!
- We are investigating the most optimal database i.e. transaction safe RDBMS like PostgreSQL
or Scalable NoSQL like Mongo assuming we target 2million transactions per sec etc.
- The Role of a Message Broker like Redis or RabbitQL
- Accessing through WebSocket with UDP(native to Phoenix it seems) or REST API (TCP based)
As you see, it’s still a long way and already we have learned a lot like getting to know this forum.
When we are done, we can create Elixir Meetups, Elixir Youtube Channels and Medium Posts.
- Thanks for your contribution and remain blessed!
websockets don’t work over udp …
Just want to add a few more thoughts since I’m a trader myself and have previously thought about creating something like this. Most retail traders are trading CFD, and this is illegal in the US. While some may think CFDs are a scam, they are easier to implement from a technical perspective, because after all the broker house is not actually matching your order with someone else’s. Also it’s now possible to trade CFDs quoted in bitcoin…
Suppose you want to make a real exchange that does matching, apart from auditing, reporting, legality, how on earth are you going establish yourself as an international exchange? If you have the resources already I don’t understand why you asking in this forum.
I suggest you narrow down the scope of your problem, define what exactly your API is going to fulfill, and pick your technology last.
Apologies if I sound too critical but I hope to save you some time going down the wrong rabbit hole.
Thank you sir @idi527 correction taken. I will read more on that.
Thank you Sir @xlphs for your advice. The API is to serve any mobile, desktop, or web Forex application. Any exchange can just build on it without much work.
That’s ambitious, I like it. I would say build a bullet proof matching engine first, then implement all the different order types, trailing stops of course, and fancier ones like OCO. Data storage is probably the last thing I would worry about. And margin orders too, depends on how the actual exchange implements margin this may vary significantly.
Welcome to the community, @4dcoder! We’re glad to have you on-board
I skimmed through the previous replies and I can see that most of the tips and advice I would have liked to share with you have already been said, so I won’t repeat those things.
There is one additional thing I would like to mention:
I can see that you’re mentioning databases such as Redis in combination with performance-related topics, which suggests to me that maybe you’re considering using Redis as a cache of sorts.
If you’re working with ephemeral data (data which does not need to be persisted, e.g. a cache), then you might have better options with Elixir. For exmaple, you could store some state in memory by using processes or ETS tables, and persist to a database on some interval if necessary.
The book Adopting Elixir has a very good chapter about persistence strategies and dealing with ephemeral data, which you will find on pages 107-108 and 116-117.
Also, I highly recommend reading Programming Elixir, which is a very thorough and well-organised book. It’s not free, but it might save you quite a bit of time and effort to make up for the cost.
Since you know what I want, I strongly believe that those books recommended will put me on the right path. I will spend time digesting them and when I am done, I believe I won’t be too naive. I love this community!
Any book that covers UDP with Phoenix too would be great.
Thanks for your contributions
You’re very welcome! Please feel free to post any questions or thoughts you might have to the community and don’t worry about being “too naive” and such things—the vast majority of us would be happy to help. We’re all learners with varying degrees of experience
FWIW I’ve had good luck implementing APIs with Cowboy or Plug in commercial projects. The consumers of the APIs were either single page apps (React, Angular) or third-party backends. In those cases we didn’t need all the extra features that come along with Phoenix so we opted for the lighter approach.
Since you’re new to Elixir, I recommend picking 2 or 3 libraries/frameworks to evaluate a small part of your app in and getting hands-on experience with each one. Not only will that give you a better idea of each library, you’ll gain more experience with the Elixir language and ecosystem as well.
If you need to interface with exchanges via UDP or provide a UDP connection to your clients, check out the documentation for gen_udp.
The new keynote from Elixirconf 2018 might give you another reason on why to use Phoenix.