The remark that it implements special process by hand made me interested, so I had a look at the code. Here are some things I spotted, I hope you don’t mind.

• you don’t handle the regular naming conventions - it would be nice to be able to pass the name option like with all the Elixir’s stdlib behaviours. When you do this, there are some race conditions to consider, so I would advise looking at using the :gen module that powers all the Erlang behaviours (it’s not documented publicly, but I saw many projects using it, I’m not sure how bad of an idea that is - but it definitely handles a lot of common scenarios).
• whenever you call user module callbacks, they should be wrapped in a try and errors should be handled appropriately (e.g. you want to call the terminate callback in that case before dying).
• it’s nice to use :sys.handle_debug for any messages received and sent out (or any “major” events that happen in a process) - this is really useful when things go wrong.
• in the receive loop, you probably want to look for the {:EXIT, parent, reason} message in case the user starts trapping exits in the process. The special process has to exit with the same reason as the parent in case that happens.
• it looks like you don’t implement the system_code_change/4 callback
• there’s an additional format_status/2 callback you can implement to control what information is returned on :sys.get_status and :sys.get_state used, for example, by observer.

WebSockex 0.4.1 Released!

Enhancements

• Allow :via and :global tuples for named registration.
  • This includes handling for cast/2 and send_frame/2.
• Add access to response headers during handle_connect/2 via Conn.resp_headers.
• Add Conn.parse_url/1 to handle url to URI conversion.
• Allow Conn.new/2 to use a url string instead of a URI struct.
• Automatically add a “/” path to a pathless url.
  • The HTTP request will break without a valid path!
• Add child_spec definitions for Elixir 1.5+
  • Or any version that exports Supervisor.child_spec/2
• Some documentation tweaks

Bug Fixes

• No longer invoke handle_disconnect if there is reason to exit from invoking a callback. (e.g. an exception was raised)
• Properly handle unexpected SSL socket termination.
  • This seems pretty important, but I don’t know…
• Return a descriptive error when trying to use send_frame/2 in a callback.

Take a look at the v0.4.0...v0.4.1 diff or the release on hex.pm for more information.

I absolutely appreciate and encourage trying different things, and even overcomplicating the code for the purpose of learning and gaining a deeper understanding. That being said, if learning is the only motivation for using your own special process here, and assuming that this library is meant to be used by others and contributed to, I think it might be wise to switch to GenServer. Doing this would make it easier to solve (and in some cases automatically solve) issues mentioned by @michalmuskala.

It might also simplify some future additions. For example, I think that calls should be supported as well, and this would be pretty simple if the process was a GenServer. Making the interface of your behaviour more similar to GenServer by supporting timeouts and hibernate options in return tuples, as well as stop option would also be nice. Again, this should be fairly trivial if your behaviour was powered by GenServer.

I have similar sentiments about the current state of WS clients. About a year ago, I looked into a couple of them, and wasn’t really completely pleased with any. In the end, I settled for this branch. I like that your project starts with a clean slate, and has good test coverage. That seems very promising!