Secure files transfers servers in Elixir?

Our system with hundreds of thousands of requests per seconds and the fact we are looking into alternatives to phoenix templating because of that.

I don’t know if the OP will test his system the same way we use ours, it may also be because we are not using Phoenix at the best of its potential, but this is my experience.

Still, I find what I say not to be polemic - I still recommended Phoenix to the OP and even directed him to some documentation.

It’s up to the OP to check both options and decide for himself. If Phoenix was borderline unusable with high traffic (like some community packeges in hex) then I wouldn’t recommend it in the first place.

Can you share what app/site that is? I’d love to see :smiley:

I can’t :frowning:

We do backend and for all purposes we are invisible to people. We deal mainly with data from providers and we use Phoenix Views as an easier means to modify XML, HTML, JS and JSON files (a decision was made that using Phoenix would be easier than dealing with AST trees). We are now updating Phoenix from 1.2 but it is proving to be a battle we want to avoid.

Not only that, sharing company code would get me fired. Hence the reason why I always have MWE in my forum threads and not real code.

I know this is not the answer one would expect. It even undermines my argument (I am asking readers to trust my word without real proof).

To add to that, I don’t even know if the OP will use Phoenix the same way we did. These reasons are exactly why I still recommended OP to try it out in my first post.

I think Phoenix is awesome, it’s not working for us and I am sorry I can’t provide additional context.

1 Like

If your goal is file transfers only, then it seems like Erlang should contain mechanisms to handle that already. I believe Erlang implements an sftp server (http://erlang.org/doc/man/ssh_sftp.html). I also see this article:

https://ninenines.eu/articles/ranch-ftp/

Which talks about building an ftp server on top of Ranch.

Accessing those things from Elixir to help with distributed coordination shouldn’t be too hard.

5 Likes

Honestly, if you find a faster option for templating at 100k req / sec I think we’d all like to know. :slight_smile:

It might even be worth contacting the folks over at Discord for some suggestions. Their engineering team has done a lot to push the limits of Elixir, open sourced and blogged about too.

2 Likes

We are currently looking into this Engine:

https://elixirforum.com/t/fast-eex-iolist-option-for-eex-engine/16145/7

Looks promising thus far, I hope it can serve you well.

This is not anything new however, some folks at the community already gave it a shot, but I am not sure why this was not implemented (at the best of my knowledge, nothing was done with it).

That’s pushing the limits indeed. But IMO at this scale you’ll be much better off having 2-3 servers and putting a load balancer in front of them. Trying to squeeze every last drop of performance per watt is not using the dynamic languages like Erlang and Elixir to good use. They can crumble under such pressure.

If you really really want that server to remain singular then maybe it’s time to look into Rust or Go or OCaml.

2 Likes

At that scale on a single server then you are looking at pure socket performance, so C/C++ or Rust with specific libraries, or Nim (I’m still heavily impressed at its significantly loaded socket performance). I’d not recommend Go or OCaml, neither perform as well as C/C++/Rust/Nim when finely tuned (though they do well when not everything is finely tuned). You will be looking at memory pools, no-alloc structures, etc… etc… Honestly just adding more servers as better as redundency is always better for such things, at which point Elixir is just fine.

Still, hundreds of thousands per second is crazy high, I’m exceptionally curious what this is. ^.^

3 Likes

Well a while back I wanted to transfer some files so …

:slight_smile:

6 Likes

Well you have sockets, and something to compute an MD5 checksum - which is actually all you need.

Forget all the generic this and that and make a simple socket client and server (homework - find out the smallest socket client and server code)

Making a client/server over a socket is virtually the first thing I do in any language when learning - very instructive

Repeat in C, TCL, JS, Java, Ruby, Python, Perl, C#

Seriously, the round trip client -> server -> client with a check that you transfer all data and loose nothing is an essential programming technique - learn to do this first THEN add JSON/XML (whatever) on top.

Once you can get two programs in different languages communicating raw bytes over a socket then you can start having fun - Use the low level socket libraries and NOT fancy frameworks - this is the slow way to program - BUT in the long term the best way - understanding is the key to programming libraries which hide what you are really doing should only be used once you understand what’s really happening

Cheers

20 Likes

Possibly even DPDK…

1 Like

I will follow your “slow” way in Elixir to learn it with the help of your article “Why I often implement things from scratch”.
Thanks for your advice ! :wink:

24 posts were split to a new topic: Secure Files Transfers Servers in various languages Discussion

Very exciting debate on languages comparisons… Even if it drives me a bit far from my original concern ! :roll_eyes::grinning:
Nevertheless, to fix some boundaries about my initial request:

  1. The fact that I posted my request on Elixirforum and nowhere else is a bit related to the fact that my benchmarks focus on Elixir only as it is what I have to evaluate: So Rust, Go, OCaml or others (surely) fine and powerfull languages are, for the moment at least, out of my scope,
  2. As I mentionned in the original post, my tinies “files exchange” servers will be located on several computers/VM. This is indeed one of the main reasons that leaded me to test the Elixir/Erlang/OTP system as its distributed aspect seems to be praised everywhere,
  3. I’m only querying practical and clear advices to lead an “Elixir n00b” like me on the jungle of Elixir concepts (GenServers, GenStages, Supervisors, Tasks, Agents and the like) in order to achieve my goals while trying as much as possible to avoid the waste of time generated by exploring some unefficient dead ends (NB: this experimentation is done by me in addition to my regular job, partly outside of my working hours). Just like did Joe Armstrong or Scott Thompson in this thread, for instance…:+1:
  4. I also hope that the advices given could help some others beginners, facing problems similar to mine…
2 Likes

Don’t forget that Elixir is a fantastic ‘glue’ language between languages as well, it’s Ports and Nif’s are not to be discounted. :slight_smile:

Well the first decision is what protocol to use, like do you want to use a standard protocol like HTTPS or SFTP, or do you want some custom protocol inside an SSL connection? If you just want something secure and fast then the beam has a built-in SFTP module, the server documentation of which is at: Erlang -- ssh_sftpd :slight_smile:

If using https then cowboy/plug/phoenix (depending on what extra functionality and features you want, and what kind of clients will be pulling it) will work well.

Though more information and details can be given if this can be filled out. :slight_smile:

  1. What client(s) do you want pulling from it (something prebuilt and if so what, or will you make it yourself)?
  2. What protocol(s) do you want to use (something pre-defined, or making something custom)?
  3. What do you want the BEAM server to do that something like an SFTP server itself cannot do?
  4. Etc… anything else that might be relevant. :slight_smile:
1 Like

Fisrt, thanks for your help.

Will make it myself as the client will be (only) the server: BTW each server should be able to send and receive files and these files should only be transfered from a server (node ?) to another server (no others clients).

I’m not really firmly decided on that for the moment. My specs are that the files should be crypted during the transfer (so sftp would fit) but also that integrity should be verified at the end of each transfer (I don’t think that sftp achieves that) by some checksum or others means .
A plus could be that theses servers should also resume partials transfers but this is optionnal…

The integrity thing mentionned above ?

I’d like this server to be the most efficient possible (big files transfers and simultaneous transfers of numerous small files) as we will do a benchmark and compare the Elixir one to some others same servers made in C++ by another team.
(I actually try to promote Elixir inside my company ! :wink: )

Surprised no one’s mentioned BitTorrent yet. By definition, a way to efficiently distribute files across peers. Found this slide deck about the details: http://www.erlang-factory.com/static/upload/media/1474809327462227martingausbyimplementingbittorrentinelixireuc2016.pdf

1 Like

Perhaps because implementing a whole P2P protocol such as BitTorrent seems to be, as stated in the slides, a bit of, an “hard challenge” (at least for me) to achieve the goals that, at first glance, didn’t seem so complicated to me (but I’m certainly wrong indeed): securely transfering files between (only) two servers/nodes with integrity checks ?
Anyway, thanks for your advice.

If it’s just two servers, yea, BitTorrent is overkill.

2 Likes

If it’s purely a BEAM mesh network then you can transfer a file as easily as sending a message, you can even access a file on a remote node from another node directly. :slight_smile:

Well there is a check-file extension in the sftp protocol for that but it’s an optional extension so I don’t know if the system supports it. Might be good to emulate it by just having each end expose a md5/sha1/whatever prefixed file of the same name that when accessed will hash the file and return the hash as that would be pretty simple to set up.

You can pack a CRC/hash along with it, just send something like a tuple or a map with fields for the hash, filename, and content. :slight_smile:

I/O is what the beam does very well, especially with sendfile calls too if you don’t mind setting up transfer-specific sockets between the servers, but even the distribution network is fine for that until it gets too loaded down and messages get delayed. :slight_smile:

1 Like