I was running into the same issues when I started out with phoenix. I ended up making https://github.com/jpiepkow/accesspass to fit a lot of the concerns of a full auth system. I have quite a bit on the roadmap for what I want to accomplish with it down the road but it is very much usable right now and I believe quite unique in its solution of authentication and security.
Check AccessPass - a opinionated authorization library for more information.
Please do put any questions or concerns here/in the link above or on the github because I really do want to continue to improve it to something that many people cause use.
