I am working with CNODE at the moment and I don’t fully understand how binary data is passed between Elixir/Erlang node and C node.
I am reading binary file on the Elixir node using File.stream!(path, [:raw, :read_ahead, :binary], @file_chunk_size) and passing chunks to the CNODE. On the CNODE side I receive message from Elixir node and extract passed chunk using following:
len = ERL_BIN_SIZE(method_param);
ptr = ERL_BIN_PTR(method_param);
Method that I need to call with received data requires short int[] type (lame encoding).
First question I have is do I receive binary data as uint8_t[] type ?
I assumed so and added following conversion between types:
for (i = 0; i < len; i += 2)
input_buffer[i/2] = bytes_buffer[i] | (short int)bytes_buffer[i+1] << 8; //little endian
where:
short int input_buffer[8192/2];
uint8_t bytes_buffer[8192];
Problem is that resulting data in input_buffer is different to what I get if I read same file straight out of C reading directly into short int array.
My C knowledge is far from great, so any help would be greatly appreciated.
In C most file reading function do give you some []int back, but only the lower 8 bit of each integer are used.
So if you can hand your read-from-C string directly into the target function, you should be able to simply do this (semi pseudo C, its been a while, but should show my point):
int[8192] foo
for (int i = 0; i < 8192; i++) {
foo[i] = (int)ptr[i]
}
If though your processing function does require to get unicode codepoints, it will get more complicated.
Thank you, I just tried your solution and it adds 00 to every byte in the buffer, so hex 52 49 46 becomes 52 00 49 00 46 00, which is not right as I think I should pack 2 bytes into single int.
As you can see from code I drop first 4 blocks of encoding into separate files. After this I do a binary comparison with original source file ready by Elixir and see if blocks are there and if they are sequential. From this analysis I can see that blocks are not adjacent in the source file.
Not sure what do you mean?
I use binary comparison (via BeyondCompare binary mode) to identify where blocks in temp files located in source file
Yep, it is in Gist above, but let me know if you need more info
Can’t you simplify this code drastically? Start with small things first… Send a simple binary like <<0x20>>, also read from a file which does contain only a single space (no line breaks) and compare those.
You said you have a function which takes an array of int, does this function work with the array you created with my function?
Can you perhaps even give a link to that functions documentation?
Not what I asked. You said when you convert the input using my method you inspect it and get this 52 00 49 00 46 00, how does it look like using the same inspection method after reading that file from C?
I’d prefer a minified project with both ends of your testing. Elixir and C. Would make it much easier to test and fiddle locally.
Sure it is this call in my gist: encoded_len = lame_encode_buffer(lame_config, input_buffer, input_buffer, len, output_buffer, len);
Method definition from lame.h is:
int CDECL lame_encode_buffer (
lame_global_flags* gfp, /* global context handle /
const short int buffer_l [], / PCM data for left channel /
const short int buffer_r [], / PCM data for right channel /
const int nsamples, / number of samples per channel /
unsigned char mp3buf, /* pointer to encoded MP3 stream /
const int mp3buf_size ); / number of valid octets in this
stream */
It looks like 52 49 46.
I’ll prepare one for you to look at tomorrow morning - getting late here in Australia
P.S. I made couple of changes to my code, to skip all byte to int changes I simply copied memory block I got from Elixir onto short int input_buffer[8192], which after dumping to file resulted in what appears to be a proper data - after writing it to file I get sequential chunks of data from my original file, my call to lame method yielded incorrect results though as result mp3 file contains corrupted data.
C file reading functions give a byte array, which you can then interpret however you wish, none of that for loop stuff you show.
A binary in Erlang (or a CNODE) is also just a byte array, interpret it directly as you wish.
for (i = 0; i < len; i += 2)
input_buffer[i/2] = bytes_buffer[i] | (short int)bytes_buffer[i+1] << 8;
This yells out to me ‘entirely wrong’. First of all, binaries in erlang are not endian aligned in any way, they are a byte stream. How you read or write to/from them can be aligned in one way or another, but the byte streams themselves are not and they would identically match the input file if not other processed first.
Definitely this, setup a git repo of a minimal reproduceable step. I’m willing to bet either you are not decoding it how it is encoded or it is not being encoded properly.
But overall just remember, a BEAM Binary is almost identical to a C byte array.
Thank you OvermindDL1,
I already realised that I get a memory block from Elixir side and I can point any type to this area that I can interpret as any type I like (byte, int, long…). So I made the code change to use memcpy from binary data received from Elixir onto short int[] array and looks like it worked. The battle however is not won yet
Problem I have at the moment is following: blocks of the binary file seem to arrive in tact and match those that I get when I read file directly using C (in this method: https://github.com/andre-dubber/CNodeDataTransfer/blob/master/cnode/src/cnodeserver.c#L217), however outcome from the call to lame_encode_buffer is different between Elixir sourced data and C sourced data (C data is encoded correctly). So obviously something is still missing there.
─➤ ./bin/cnodeserver 3456
Starting up server
Devision result: 0
Calling lame setup...
Lame setup cmplete.
Starting test encode
[1] 81602 segmentation fault ./bin/cnodeserver 3456
That is with both your included binary and trying to recompile it myself (of which I could not initially do because holy-wow-ton-of-errors that I had to fix first, starting with the Makefile, which very much assumed ‘your’ system and not standard installs, you should use cmake). And yes I have lame installed:
─➤ lame --version 1 ↵
LAME 64bits version 3.99.5 (http://lame.sf.net)
Including the development libraries and more.
On initial guess I’d say one of those many errors in the compile (I run a fairly strict compile set, warnings-as-errors and such even) is probably the cause of the discrepancy you see. Do not ignore errors or warnings, because there are a ton here… ^.^
I don’t have time at the moment to debug into it or rewrite it but that is at least a starting point? ^.^;
EDIT: Got it running, had to put the 1.wav as /tmp/1.wav instead of in the project directory. ^.^
if ( memcmp(input_buffer, bytes_buffer, len)) {
fprintf(stderr, "\n Arrays match after casting ");
} else {
fprintf(stderr, "\n Arrays DONT match after casting ");
}
return value indicates
<0 the first byte that does not match in both memory blocks has a lower value in ptr1 than in ptr2 (if evaluated as unsigned char values)
0 the contents of both memory blocks are equal
>0 the first byte that does not match in both memory blocks has a greater value in ptr1 than in ptr2 (if evaluated as unsigned char values)
I.E. your comparison is backwards, you are saying they are not equal when they ‘are’ equal. It does not return a true/false if matching or not, it returns -1 if the left is less than the right, 1 if the left is greater than the right, or 0 if they are equal, it is a ‘comparison’ function (even elixir’s comparisons work the same way), not a ‘match’ function.
/me is loving how consistent discourse is, the spoiler works in the preview, but not the post?! Wtf?!
Yeah, but a price I like to pay as a user as well as a hoster…
There are basically 3 ways to solve this:
Render everything on the client: as a user I do not want this, discourse already eats up more than enough of my ressources, especially when on phone…
Render everything on the server, even the preview: I do not want this as a server operator since it burns down my traffic as well as my CPU, also remember, ruby is still singlethreaded as a user I don’t like the idea of such highfrequent up/down loading my WIP-post.
Use a markdown renderer that behaves the same on both ends: I like this as an server operator as well as an user, lets delegate the work to keep those in sync to the programmers of discourse… Anyone who would +1 such a feature request?
Thanks for your comments and deserved criticism. Just to clarify, I haven’t done much of C coding for about 18 years, so you may consider me as noob. Another thing is that this project is at prototype stage, so I have not done any cleanup of the warnings since taking CNODE sample app and modifying to make it work they way I expect it to.
Now on the subject I’m brushing up project to get rid of the warnings and hope it will resolve my issues, I’ll post an update later on.
I have done further changes to that repo. All warnings are gone (at least on my system) and I attempted to bring C native and Elixir received block as closed as I could (both now read char[] and pass it into lame encoding. As I can see from the dumps of the binary blocks I have exact match between inputs from both C and Elixir, however for some reason that is beyond my understanding I get different outcomes between calls to lame_encode_buffer() results however consistent between runs, i.e. on the same input data I get results that are consistent between multiple runs, but different between data received from reading chunks using C fread() and chunks received from Elixir node.
I am sure there is something trivial that I miss, but I don’t see what!
Could you please have a look at the latest version of the repo and see if you can spot why above might me the case?
I cannot test it at the moment (busy at work) but from what I’m seeing I’d immediately look at making sure the blocks are the same length between both test, the number of blocks are the same between both tests, the number of chunks you process is the same between both tests, and so forth. A difference in any of those will cause a different mp3 to generate because of the way mp3 encoding works (though it’d still sound the same).
Thanks OvermindDL1, both C and Elixir sourced blocks are written into /tmp/tef_ix.wav and /tmp/elixir_ix.wav respectively. Generated files are identical, which is what confuses me, as it appears that from the same input I get 2 different results… Obviously something must be different, but I fail to see what it is.
Just FYI, I have managed to get it working (kinda) by passing accurate source file audio properties and using them to configure lame encoder. As result encoding has correct sound, but playback speed is not right yet, this is however outside of scope for my original question.
results however consistent between runs, i.e. on the same input data I get results that are consistent between multiple runs, but different between data received from reading chunks using C 
