Recommendations for large eCommerce venture?

Questions / Help
,
1

I need the communities assistance to make some rather difficult choices.

I have been given the assignment to research the best tools for a very large eCommerce venture.

My number 1. principal is to keep costs way way down. I other words I cannot go and get an off-the-shelf solution; no Oracle, no IBM, Amazon, none of the usual suspects that we would have to pay $$millions for licenses.

So it looks like we are going to have to build it from scratch. So the principles in play now are. Robustness, ease of maintenance, scalability, low cost, no fads, it’s got to last 20 to 30 years, lightening fast web interface, millions of concurrent users, no faults, no down time, minimal moving parts (not too many technology dependencies)

My search brought me to Go, Scala and Elixir and Erlang, COBOL (Yes!, COBOL) Adabase and Natural.

We have to make some choices. So far I am leaning towards

1. Phoenix (web platform) + Elixir (Erlang) (Development language) + NuoDB (scalable, elastic SQL DB)

Ubuntu seems the logical OS.

At another time in my life I would have chosen IBM AIX, because it scales and scales on IBM hardware with DB2, but this is not that time., and it would have cost $$Millions.

Folks of the community.

Am I heading in the right direction. What is your opinion and advise.

Your thoughts and advice would be much appreciated.

Thank you
SammiS

3 Likes
2

I know you mention no off the shelf solutions, but everything you mentioned is way bigger than you’d likely need initially. I’ll point out https://www.shopify.com/pricing and https://www.bigcommerce.com/essentials/pricing/ as two things that don’t cost nearly as much and are likely cheaper than paying a team to build something equivalent.

6 Likes
3

Developing an E-Commerce platform from scratch will be a massive amount of time, effort and money, saying this, I’ll let you know what I think:

I don’t like Go, just my personal opinion, but I think a big code base will be a pain to maintain. I may change my opinion when Go 2 came out, seems like it will improve a lot, but it’s not here yet.

I’ve never tried Scala, it may work for you and your company, just want to add that you may want to consider Kotlin too, it’s built by and for the enterprise.

COBOL? is that a thing? I don’t know, never tried it.

Elixir and Erlang are the only sane choices from the languages you mentioned. I think there are great stories of success from big enterprises where concurrency is a must, like WhatsApp, Discourse, Pinterest, and so on.

Also from all options you mentioned, I think Elixir is the only one that has a well-established web framework, and that is an advantage to speed up development. Go has a lot of frameworks available, none of them like Phoenix, complete solutions like Phoenix doesn’t have much value in the Go community, in fact some people think that is much better to build your own stack with a toolkit to have more flexibility, that means you’ll need to figure out the libraries required for the project if you follow that path. Kotlin has support for Spring a well-known framework in the enterprise’s world. I don’t know about Scala or COBOL.

4 Likes
4

Honestly for something maintainable over a huge period of time I’d opt for something stronger typed than elixir or go or so. Scala isn’t bad for that, better than kotlin, but for working with money and the detail you need with that then I’d not opt for it either. Something like COBOL actually isn’t bad for this, not that I’d start any new project in it, Idris would be perfect except it is more academic then ‘useful’, so I’d probably opt for Rust, it’s still very strongly typed (though not dependently typed, it is a mock-refined typing though, so better than normal HM type systems) and is very reliable and seems to have the staying power to stay those decades (it really is the first language I’ve seen that I think can replace C/C++).

Elixir, however, would be great for scalability, however you’d want a most excessive test suite, dialyzer types everywhere, and a lot of enforcement. Combining Elixir and Rust would be a fantastic mix though and they work together exceptionally well, so Elixir would save you time for the bulk coding and Rust would give you the numerical safety and speed.

Still, I’d opt for something pre-made, this kind of task is anything but trivial.

4 Likes
5

I know nothing about Scala, (I tend to stay away from the Java-related ecosystem, only Kotlin took my interest), so yeah, it may be a good fit for this.

Totally agree with everything you said about Rust, it’s such a great language gaining a lot of attraction. However I’m still unsure of how it’s supposed to replace the well-established C / C++ ecosystem, of course, it has a refreshing modern syntax, and support for hot topics like WebAssembly that increases the interest and hype for the language. However I don’t know what happened to Mozilla’s engineers working on Rust, after reading the reasons of Steve Klabnik and Nick Cameron on why they left Mozilla (both did -and still do- a remarkable job with Rust and its ecosystem), I’m very skeptical about what the future of this project is, this kind of things are really bad for its reputation.

As a game development aficionado, I didn’t see anything relevant in the ecosystem, on the other hand C++ has Godot and Unreal, two mature and stable game engines.

1 Like
6

There are actually a lot of building game engine libraries and full frameworks building up in Rust. There only aren’t any yet because a game engine is huge and Rust is still relatively new. There are rust bindings for both Godot and Unreal though. ^.^
Also see:

3 Likes
7

Here’s also a tutorial on how to get started with web development in Rust: https://zupzup.org/rust-webapp/

It uses actix-web which is the fastest web framework available today. actix also uses an actor model.

2 Likes
8

As a person who used to contribute to Solidus, I HIGHLY recommend it. It’s very well looked after and lots of major ecommerce companies use it. It uses Ruby instead of Elixir though.

9

Wooo, never heard about the bindings of Unreal Engine, I may take a look then, since my knowledge of C++ is very limited. There’s a great talk of what Blizzard thinks about adopting Rust:

(It seems like some people at Blizzard uses Erlang)

Also I heard that Chucklefish Games has had great success using Rust, this talk is by Catherine West, a very funny one!

3 Likes
10

From reading the requirements what jumps out at me if that it is required to last for 30 years, so I would think carefully about what technology is going to last that long.

The Java ecosystem has enough momentum for that, is currently very mature, and is not difficult to hire for, so I would be seriously Kotlin, Scala or Java.

I’ve not come across NuoDB before. It seems to be a niche proprietary database. Given proprietary databases have a habit of disappearing due to funding problems and niche databases have a problem of disappearing due to contributor shortages do you think it’ll be around in 10+ years? You may end up having to maintain an unfamiliar database by yourself, which is a scary idea. I would likely use Postgres or MySQL/MariaDB, both will still be around and supported for that long and both have been proven to usable at scale.

5 Likes
11

There’s shopify? Unless there is something I’m missing here or you want more control. It’s super easy to set up and running and you can move to build something custom when you need to or do that while shopify is running.

Cobol? From what I’ve read the majority of where Cobol is used is banking and it’s legacy stuff.

Adabase is Ada? Ada is an interesting language for system
 good luck finding support though.

I did Scala. The learning curve was high and having several programmers with different level of skill set was interesting. Scala have a unified type system and I inherited a code base where my coworker typecast everything up to any type and basically use it as a dynamic language
 Examples of high learning curve is the amount of pattern arises from it’s feature such as implicit class, cake pattern, etc
 And akka, their actor system isn’t as good at Erlang’s BEAM VM. This is what I appreciate about Elixir and Erlang how small the language is and how terse it is.

Yeah I would go with Java. It’ll stay for a long time with the amount of code base and libraries invested in it.

I love elixir but it still not a sure thing at least to me to be banking for 30 years.

3 Likes
12

For what it’s worth, I write Scala for a hedge fund as my job. I find it’s actually really good precisely for modelling all sorts of complex financial situations with lots of numbers all meaning diffferent things flying around, thanks to its strong type system and easy introduction of “newtypes”, lightweight (or no-weight in the case of AnyVal) case classes whose only purpose is to tag e.g. an integer or BigDecimal with a domain-specific type. You can then define your business logic in terms of operations on these higher-level objects more specific to your actual domain.

1 Like
13

Well, Erlang has been around for 30 years already.

But in Erlang/Elixir you’ll end up writing almost everything from scratch. There were efforts and ideas about it but let’s be real: without real funding and somebody’s livelihood depending on coding that, nothing will happen. We all have jobs to do and families to support. And free time to enjoy.

Outside of that, either Solidus (Ruby isn’t a very good idea for the scale you’re looking for) or Java – although scale might be a problem there as well.

I’d naively pick Elixir and work my way through the difficulties, asking first if I’ll get guaranteed 3 years of funding first. Erlang/Elixir are technologically best equipped for the scale and longevity you’re looking for. But again, it’ll be a custom-tailored and hand-crafted solution.

2 Likes
14

Hey Sammis,

before giving any technical advise, or seeking technical advise, I would always recommend to lay out the requirements as precisely as possible.

You state that

Let me put it this way: maybe. But this is most certainly not your real concern right now. Every choice you can make will inavitably have its downsides.

So the real questions are:

  1. Why is money a problem? If you’re going to do business for 20-30 years you need to make millions yourself with this enterprise. So if money is a problem now, developing from scratch is maybe not a smart option.
    (That beeing said: economically it makes sense to keep costs down. Just remember that licensing costs will almost always be lower than development costs if you can get the same system. This is true for almost any part of the world)
  2. What are you up to? Are you a startup? Are you a development department or agency for a company that wants to expand or change its business? Is this a public government sector type of thing, where paying people is not a problem because they are “there anyway”?
    2b) How many people will be working on the project.

Pointing to any particular technology is not doing any good, without answering at least question 2). Because it massively depends on what to optimize for.

That beeing said: if you are sure (and I mean absolutely, rock solid, super dooper, “I have seen the contract” sure) that this project will be very long running you need to optimize for one thing: changeability / refactoring. Don’t hardwire stuff, keep everything as separated as possible. Create clean APIs that stabilize fast and don’t change. Keep the scope of every part of the system as small and focused as humanly possible.

This advice is, I think, universal for large systems and does not depend on any particular technology. Just like: try to introduce as little side effects as possible and only in well defined and understood places.

So, if you can, please give us a little more info about your project :slight_smile:

(Not that I don’t enjoy a good technology discussion but I think this isn’t helping OP as much :wink:)

7 Likes
15

I like what you saying. Very pragmatic. Thanks

16

Those are still HM level types though, meaning actually dealing with the monetary values has to be done in runtime code. As a basic comparison take Idris, where you can enforce how the money values change in the type system itself (not just the money type, but rather how it changes), or a not near as powerful Rust where the refined typing can ensure that money is not mishandled at runtime (that is of course if you use rust’s typing system for that), neither of which Scala’s types handle.

Still though, I entirely state that use something off-the-shelf that is well tested and well used.

1 Like
17

mmmrrr

Thank you for your insight.

This is really great advice I am receiving.

Much appreciated.

18

dimitarvp

Thank you for your words of wisdom.

19

If I were you, I would start with language and database I know the best. If you are learning language now then you will be lost for some time doing nothing about your business nor building your sales skills.

  1. Define what is your MVP,
  2. set KPIs
  3. do MVP it (implement) and then release it and see what happens.
  4. Do some cheap marketing, e.g. put some $ on FB adds and wait for result. Check what audience is best for you and then add more $ just for such audience. Open FB page, Instagram or what ever is popular thees days.

At some point, you will see that you are spending 10 grades more on marketing than on licenses and infrastructure. But that marketing money is returned in revenue probably. If you do this carefully you will quickly build some value. Then go with this value to some investors and present it and give 15-30% of your company so you get funds, BUT prepare new plan how you will spend that money to create bigger value.

As you see above, only one point is about coding.

If you are still concern about what platform and tools to use, I can say from my 20 years experience that every company I worked for or consult, had smaller or bigger platform change at later point, did some “rewrite” in same or new language or add new service using new platform. What so ever, people that should work for your company (developers) will always try to find new DB engine or language to replace current. Luckily the change happens rarely :slight_smile: otherwise it would slow down business development.

2 Likes
20

(This is getting offtopic so perhaps a thread-split is in order.)

I do quite a bit of Rust on the side, and I love it, but I actually don’t see how its type system would be superior for the types of business logic problems that we solve. In fact, the couple of feature of Rust that I occasionally miss in Scala are the Result type (but mostly just because it’s consistent across libraries) and move semantics—I’d love to be able to express that a function uses up a value and it can no longer be accessed—but even then that mostly comes up when I’m interfacing with some mutable outside Java thing. Neither of these are really a property of the type system. I can do ADTs/enums (sealed trait), I can do traits (“typeclasses” aka implicits), and even some more heavyweight functional stuff with scalaz (though only a few projects use that).

I’m not sure what you’re picturing exactly when you say

Rust where the refined typing can ensure that money is not mishandled at runtime

—do you just mean that I won’t e.g. add 100GBP to 100USD? That I won’t accidentally mix up balances on two different accounts? Because all of those things we can ensure with Scala, with e.g. overloaded operators. Yes, the checks happen at runtime, but I really don’t think they could happen statically, because all of this data is only seen at runtime, and we cannot make static assumptions about our data sources, because they are sometimes unreliable. I’m no expert with Idris or other languages with real dependent types, so perhaps I am missing something there, but no matter what the compiler / type checker must work with static information only, and I’m not sure whether we could capture any more static information about our domain using dependent types, than we already do.

If what you mean by the fact that Scala doesn’t “guarantee” this things is “if you tried, you could write code which would compile and do the wrong thing”, e.g. by explicitly extracting 100 from the object representing 100 USD, and another 100 from the object representing 100 GBP, and adding them together, and packing it back into a 200 USD object, then sure, I agree. However, we have the luxury of a reasonably small team of extremely competent people and always doing code review, and we enforce disciplined code which does things the “right way”, to allow all of the necessary checks to occur.

I do have complaints about Scala, but they’re pretty much all about the JVM itself, or the ecosystem (Akka is extremely painful after having programmed Elixir), and not the type-system.