How do you create hex packages while still supporting in umbrella dependencies?

This is the perfect use case for :path dependencies. You can publish rtmp_session to hex and use it from there, then if you want to test some changes locally you temporarily change the dependency to fetch it from a local :path. If you need both to be :path dependencies you can declare the transitive dependency with an override to make sure it it wont be fetched from hex.