Okay so I am using Heroku for a lot of Elixir/Phoenix stuff. We also use it for Ruby, it is the default platform for both solutions at the moment in my company.
The reason behind using Heroku is that deployment is simple, and we do not have to care much about maintenance. We are developers, we don't want to install/configure security updates and fiddle around with firewall.
The deployment configuration with ansible/capistrano/ec2 etc - we also do it for some clients. Usually at certain time of life of project, we choose to migrate from Heroku because it either gets too expensive, or limiting (e.g. no filesystem access). Then we choose EC2, or other services and do the deployment architecture set up, which is quite a lot of work to be fair.
Heroku allows us to skip all the work we'd have to do with say capistrano or ansible, and focus on building application. We don't have to care to write special code to compile assets, run migrations etc - this is happening automaticaly with connection to our CI server.
So, I am having a look at your solution, and it would sort out some problems we have indeed. Heroku shuts down/restarts dynos as they please, the storage is being cleared out etc, and we can'd do clustering at all (unless on Enterprise solution I think).
But if we do have to mess around with distillery/exrm, write special code to run migrations etc, this is actually more work than we would want to do. Ideally, our CI server would deliver code with
git push, and your service would take care of building a release, including assets, doing an update to the system and running migrations. I don't care much about hot code swapping either.
So, if I think about ideal platform for hosting Elixir/Phoenix, this is Heroku with dynos that we can cluster/name/assign ip, plus persistent storage we can use.