Hi guys,
New to the forum here and somewhat new to elixir as well. I’ve been going through tutorials and doing the Dave Thomas’ course on elixir, but I’ve been looking to build something in the mean time. I had a project that I wrote in javascript which output sounds through the speakers and I was thinking of porting that to elixir for funsies.
However, I couldn’t find anything related to audio output through google or any package on hex.pm, is there anything already? Does anyone have an idea?
For reference: I was using this package in javascript (https://www.npmjs.com/package/speaker)
Hmm, not at all the usual work area for BEAM projects, but I could see someone making a NIF or Port or so being just fine for audio processing. Perhaps ‘you’ should make such a library?
Lol, that’s exactly what I was researching
Looking at rustler right now to get me started (really good packages for audio in rust), maybe I’ll hack something together.
Another set of packages to watch is https://www.membraneframework.org/ although perhaps that’s more focused on streaming than what you’re looking for (it’s also still in beta)
Update: made a small nif project that uses rodio under the hood. The rust part is able to play the audio, however, now I’m looking on how to stream the file from elixir down to rust, but having some difficultly regarding the types (what are the corresponding rust types for a file stream in elixir?) - need to read up on that.
Essentially I want to make it so that I’ll be able to use File.Stream from elixir and pass it down to rust and then be able to handle pause/play/stop etc.
I would just stream chunks of binary to the NIF. You can also have the NIF do it’s own file handling as well (pass it a name). Files on the BEAM directly are actually just Ports (PortDrivers specifically) and they send messages of file data to the process, so it’s no different to you sending binary to the NIF to fill it’s internal buffer.
Yup, the types for that binary in rust got me really confused since I couldn’t figure it out. So I ended up writing the streaming part in rust as well. So from elixir you just pass a URL to the audio file and rust fetches it and plays it.
Going to implement some wrapper methods for play/pause/stop and see how it works. Also, is it a good practice to start a nif in a supervision tree?
Nif’s are just function, nothing to supervise. If however you hold a resource on the elixir side that was given by the NIF and you need to manage it, then yes for that, but you can’t supervise a NIF itself as it’s just a function call.
Got a working version, would you mind reviewing it? I’m still unsure about best practices in elixir so might be doing something blatantly wrong or not following better conventions.
https://github.com/razagill/speakers/blob/master/lib/speakers/player.ex
Yay specs, love specs, wonder if those come through the defdelegate, if not they should be added there too. ^.^
Noticing some interesting PlayerValidator calls in here, I’ll look at that file shortly, but I’m wondering why the NIF isn’t validating these and returning appropriate values (so someone can’t bypass these checks for example, by calling the NIF straight).
A curiosity, does the add_to_queue taking a URL accept a file:///some/path/to/file.ogg work? This isn’t documented.
Otherwise most calls just call to the NIF (they could indeed be defdelegates too). ^.^
https://github.com/razagill/speakers/blob/master/lib/speakers/player_validator.ex
Hmm yeah these checks should really be checked on the NIF side so the user can’t call the NIF directly to bypass these, and it would save the time from, for example, decoding the URI twice needlessly. And why not accept an empty schema/host if it’s a file:// path anyway so you can default to passing just a filesystem path like /blah/thing/here.mp3 or so?
Hmm, it’s a very basic interface, simple for simple playing. Definitely could use a higher level interface for controlling of sinks and sources and mixing though, but that can be done later. Instead of lazy allocating the device sink I’m wondering if it would still be better to handle it as a BEAM resource and let the elixir side hold on to it, so it can deallocate it whenever it’s actually ready (either by closing it via a NIF call or by letting it be GC’d). That would pollute the simple interface though, but it might making a more controlled interface more annoying…
I recommend at least make a way to deallocate the lazy allocated device sink via a NIF call (close_global_sink() or something?), and just let it re-lazy-allocate if it is needed again after that.
There definitely needs to be a way to pass in a binary and some metadata about the binary (type, like raw data, a wave, ogg, whatever) and then instance a memory source from that ran through the decoder appropriate for the type.
Instead of renaming rodio things like play to be resume, even though more accurate, I wonder if it is better to follow rodio’s naming conventions instead, so it’s docs are more relevant…
reqwest::get(audio_url) Hmm, I don’t actually know if reqwest supports file paths… File paths definitely need to be supported somehow though.
.expect("Failed to get audio stream"); Eh no, actually test if it fails and report a proper {:error, "Failed to get audio stream"} tuple back to elixir instead, ditto with the other expect calls. expect calls should never ever appear in released/production Rust code! The rustler error result type is more for program faults, or badarg’s or so, not improper logic or state.
Overall it looks like a good start. I’m getting ideas of a full rodio exposure to elixir, mixing and all. ^.^
This is really nice, thanks a bunch for the great review.
I have been following Dave Thomas’ course on Elixir and he recommends an approach to only have public methods in the root file and the rest should essentially go to modules of their own (which at least for now I think is a good approach) and then you defdelegate to the module methods.
I haven’t published it to hex yet since I wanted to iron out some stuff and make it a bit more stable-ish (not that anybody is gonna use it right away hehe).
Haven’t tried it on NIF modules, but I would add some there as well.
Specs are awesome and yup, they do come through defdelegate.
I added the validation through PlayerValidator because to be honest I just didn’t know how to do that in rust, I’m only starting to explore rust for this library. If you can point me to some examples, I can rework the validation.
And no, at the moment add_to_queue doesn’t accept file:// type paths, though I am definitely going to add that since it’s natively supported by rodio.
That was my first approach as well, but then I kinda got blocked by the elixir → rustler → rust type system and gave up on that to at least have some working version first. If you have any more ideas, I would happily try it again.
Yes, that’s actually easy since rodio provides a function to do it already.
As far as I understood, this is something that rodio does already, I haven’t really tested if it’s based on the file extension or what but you can pass different types of audio URLs and they’ll be played fine. But yeah, I need to research this more and maybe create a struct as AudioURL or something.
Had no idea how to use expect, I would have read up on error handling in rust. Can you expand a little on why they should never be in production code though?
Yeah, hopefully, I’ll keep on exposing more rodio functions
Should be a fairly simple mapping, what did you get stuck on in it?
A source can be user defined, rodio just comes with some default things. For a binary with a ‘type’ you’d instance the proper decoder for the given type (wave, ogg, etc…) and it will return a source for you. For raw data you can just feed it directly in, though it would need some other information like it’s rate and so forth.
expect hard panics, bringing down the entire process, this will include the BEAM itself, it should only be used when the system is in an entirely unrecoverable state at that time.
Well, streams created by File.stream are functions, I doubt you can simply convert them to a slice of bytes.
We already suggested in the slack to either do the Filehandling on the rust side in a dirty NIF or in a port, or to read chunks into a binary and call the NIF repeatedly with those chunks.
Oh I’d never say pass an Elixir stream over, rather a stream messages to fill it in (or a complete message of the entire thing). Reading an elixir stream over and over and over to get the full contents would be so slow in comparison, they aren’t exactly no-cost like Rust streams.
@NobbZ that’s why based on your suggestion in slack I wired up the remote streaming part in rust.
But @OvermindDL1 just to understand your comment better and wrap my head around it, why exactly can’t we pass the stream to rust?
I’m not sure I understood the filling it in using a stream messages part. If I have something like the following
IO.stream(:stdio, :line)
|> Speakers.add_to_queue
how would I pass that as a “streaming message” to rust? If I am completely off base here, please point me to some resources where I can understand this better.
