Nomad

sashaafm · May 17, 2016, 1:33pm

Hello everyone, this is a project I’ve been working for my Master’s Thesis for the last few months (since December roughly). It’s an SDK for the cloud that also has deployment capabilites. The project can be seen here Nomad at GitHub

Create cloud portable Elixir and Phoenix apps. Write once, use everywhere!

What is the Nomad Project?

Nomad is an Open-Source API for Elixir that enables developer to use the most popular cloud providers interchangeabily at will without having to change any of their code. Nomad provides generic API’s for cloud services (mainly Virtual Machines, Storage, SQL and JSON datastores). These API’s offer the most useful features and functionalities that are common across the same kind of service between cloud providers. They are meant to be simple and user friendly. This enables applications to be portable and allows the migration of applications from one provider to the next.

Moreso Nomad offers Mix Tasks for automatic management of cloud resources for your application. Need a SQL database in the Cloud for your application? Create it and automatically configure your app to use the one you just created. Want to host your Phoenix app in the Cloud? Launch a Virtual Machine through Nomad Mix Tasks and let it automatically deploy a production release of your app in said VM.

This project is my subject for my Master’s Thesis but once it’s delivered it will be a pure Open Source application.

Disclaimer: Nomad is under heavy development and is prone to changes

What are the use cases for Nomad?

The main goal of the Nomad project is to defeat Cloud Vendor Lock-in in the Elixir ecosystem. With the rise in popularity of cloud services Elixir users should not be restricted to one provider and should be able to make the most out of Cloud Computing. Some of the identified use cases are the following (but there are many more):

You’re using S3 storage for a Phoenix application but a few months later Google has better pricing? Just let Nomad automatically copy your whole S3 bucket to a Google Storage bucket and relaunch the app with Google configurations.
Recently got a new userbase in a remote region? Launch a new Virtual Machine with your application in the closest available region.
Need a powerful SQL database but you’re not really knowledgeable in Cloud related affairs? Just use Nomad’s user friendly API and it will get done in a jiffy!

How is this achieved?

Nomad takes a lot of inspiration from Ecto and uses adapters in a similar fashion. First and foremost the needed API functions are identified based on what the cloud providers’ APIs have to offer and on what are the most recurring use cases and used functionalities. From this a general Elixir Behaviour is defined. These Behaviours must be implemented by each provider’s adapter.

Each adapter will implement the APIs for each cloud service (Virtual Machines, SQL, Storage and Datastore). Upon starting up an application a simple config in the desired environment config is needed - mainly cloud account credentials and a key to identify the chosen provider.

Finally through macros and metaprogramming, the adapters logic will be generated for the Nomad main modules (VirtualMachines, SQL, Storage and Datastore modules).

Roadmap

SDK

[x] Storage API
[x] SQL API
[ ] Virtual Machines API
[ ] Datastore API
[ ] Friendly Mode and Full Control Mode (return parsed response vs. complete HTTP response)

Mix Tasks/Deployment Tool

[x] Database Listing
[x] Database Creation
[x] Database Deletion
[x] Database Reboot
[ ] Automatic Deployment
[ ] Virtual Machine Creation
[ ] Virtual Machine Reboot
[ ] Virtual Machine Deletion
[ ] Storage Creation
[ ] Storage Deletion

sashaafm · May 19, 2016, 9:41am

I’ve updated the README with better descriptions and with links for tutorials.

The tutorials can be found here.

greyhwndz · May 23, 2016, 2:05pm

Hmm. This looks interesting @sashaafm

sashaafm · May 23, 2016, 3:03pm

@greyhwndz Hi! I’ve been working really hard on it. It’s my baby right now ahah. Could you please give me some insight on why you find it interesting? I’ve been looking for feedback from other developers.

Right now I’m doing the API for virtual machines. It’s been a bit more difficult than the other APIs but it’s doable. After that I’ll work on one for datastorage (JSON).

Ultimately my goal for my demo at university will be to apply it to a web app I’m working on in my internship and show how I can migrate from AWS to Google and back without code changing. Even for people not interested in migrating this library will be useful for regular cloud interaction since it leverages the most relevant use cases from the cloud services.

I’m also thinking about building Mix Tasks like for automatic deploying prod releases on cloud VM’s created on the fly and other things like creating a database in the cloud for usage with Ecto, etc… (I’ve actually built some of these but they still need to be better tested).

It’s been quite interesting. Specially since 6 months ago I barely knew how the Cloud worked. I’ve been learning a ton.

sashaafm · June 1, 2016, 5:19pm

Nomad has just been updated with the Virtual Machines API. You can now interact with Google Compute Engine and Amazon EC2 interchangeably!

Qqwy · June 3, 2016, 9:55pm

This looks so amazing.

Once my application becomes deploy-ready I’ll definitely try it out.

What a wonderful idea!

sashaafm · June 3, 2016, 11:09pm

Thank you @Qqwy! Please feel free to give me any suggestions, feedback or opinions. I’m really looking forward to making this the best tool possible.

I’ve got a ton of ideas for it and things to build with it. I just need time tell me if you need any helping setting it up or using it!

sashaafm · June 14, 2016, 9:55am

Automatic Mix Tasks for virtual machines have now been added to master. You can now create, delete, restart and list virtual machines through a mix task.

Full control mode will appear sometime this week or the next one. Full control will be bang versions of the functions where the results are not parsed and the HTTP reply (HTTPoison) is returned fully . So instead of something like create_storage returning :ok the HTTPoison.HTTPResponse type will be returned.

sashaafm · June 15, 2016, 2:23pm

Full control mode has been added to all APIs (Storage, SQL and Virtual Machines). Full control mode are bang versions of the API functions which return the complete HTTP reply (with HTTPoison) instead of the specification defined return type.

list_virtual_machines :: [binary] | binary => list_virtual_machines! :: HTTPoison.Response.t

This of course breaks the abstraction layer since the HTTP replies from different cloud providers will differ between each other. However, I’ve found some use cases where this may be relevant to have and having more options doesn’t hurt. Mind that using the non-bang versions is still the default way to don’t break cloud compatibility.

With this Nomad has been bumped to version 0.7.0.

Next steps are:

Mix Task for automatic generation of applications’ prod releases and uploading to a given VM (also a variant of this Mix Task where the VM is created on the fly)
Storage migration (copying all files from one Storage service to another in order to be feasible and easier to change between clouds)

cmelgarejo · July 21, 2016, 3:35pm

Awesome! Great tool you’re making @sashaafm ! Looking forward to use it on my elixir pet project

sashaafm · July 23, 2016, 12:25pm

@cmelgarejo thank you! I’ve been finishing my Msc. Thesis (which Nomad is part of) and from August I’ll start implementing a lot more features.

Do you mind sharing how you’re gonna be using Nomad (what kind of application/main use case for Nomad)? If you need any help setting up or using Nomad please contact me anytime here, GitHub, Slack or Twitter

Crowdhailer · October 10, 2016, 3:33pm

I’ve started looking at nomad. I was wondering do you have any plans to integrate it with distilery/exrm etc. So that you can spin up a compute instance with a release ready to go on it.

sashaafm · October 10, 2016, 6:15pm

Hey again @Crowdhailer I haven’t thought about integrating into distillery or exrm directly. However, I plan on automating a task similar to what you described:

Spin up a compute instance
Generate a production release of the app
Transfer the release to the instance
Automatically set it up and have it running

Please any feedback you may have feel free to tell me. The version I have on GitHub is not the most up to date, because I’m working on a new version privately to first send it for a conference and then I’ll merge into the public repo.