Is there a way to create a tar file in memory in elixir?

Questions / Help
,
1

Is there a way to create a tar file in memory in elixir?

2 Likes
2

Zip files would work, I don’t know for tar… Erlang has a zip package readily available:

 :zip.create(file_list, [{:cwd, path}, :memory])

See the docs. I see there is a tar module as well.

1 Like
3

Erlang has :erl_tar, but I think the OP is looking for a way to not to store the file in disk, but in memory, which I think it is only available for extracting but nor for creating files.

4

Ye that’s what i’m looking for, I’m wondering if it’d be too hard to create my own? Do u have any material on tar formats that i can use

5

The pieces are almost all there:

  • you can create an IO descriptor for an in-memory file with the :ram option for :file.open/2
  • you can pass an IO descriptor to :erl_tar.open/2

so you might think this would work:

{:ok, fd} = :file.open("", [:read, :write, :ram])
{:ok, tar_desc} = :erl_tar.open({:file, fd}, [:write])

but this fails with an argument error:

** (FunctionClauseError) no function clause matching in :erl_tar.open1/4    
    
    The following arguments were given to :erl_tar.open1/4:
    
        # 1
        {:file, {:file_descriptor, :ram_file, #Port<0.7>}}
    
        # 2
        :write
    
        # 3
        [:raw]
    
        # 4
        []
    
    (stdlib 3.15.2) erl_tar.erl:324: :erl_tar.open1/4
2 Likes
6

I was trying the same approach with file name "", but using init/3, and I 'm not able to make it work.

7

Read the source code for erl_tar, I am sure you can find a way around, maybe even submit a patch so it can work for your needs.

  1. otp/erl_tar.erl at master · erlang/otp · GitHub
  2. otp/tar_SUITE.erl at master · erlang/otp · GitHub
8

I was thinking of writing a simple code base that converted a folder to a tar object and save it as a binary in a genserver. Right now i have a few helper functiosn that will flatten a list into it’s filename and the contents.
It doesn’t need to meet all the requirements of :erl_tar but it does need to read from a path and compress it to a tar repo. U can check

9

Thanks, i’ll look into it and see if i can submit a patch to get around this.

1 Like
10

All the best. Share here the link if you do submit a patch or made it work.

11 
{:ok, {:file_descriptor, :ram_file, port}} = File.open("somefilename.tar", [:read, :write, :ram])
pid = Port.info(port) |> Keyword.fetch!(:connected)
:erl_tar.open({:file, pid}, [:read])

This code allows u to open a file that’s read in memory but u can’t write to that file because the guards on :erl_tar.open doesn’t allow writes for pids.

open1({file, Fd}=Handle, read, [raw], Opts) ->
    case not lists:member(compressed, Opts) of
        true ->
            Reader = #reader{handle=Fd,access=read,func=fun file_op/2},
            case do_position(Reader, {cur, 0}) of
                {ok, Pos, Reader2} ->
                    {ok, Reader2#reader{pos=Pos}};
                {error, Reason} ->
                    {error, {Handle, Reason}}
            end;
        false ->
            {error, {Handle, {incompatible_option, compressed}}}
    end;
1 Like
12

Exactly, it is line 342 and 344 that needs to be edited. replace read with Access and make a guard to accept read and write.
That solves the issue.

I have ported the ‘erl_tar’ module to Elixir.
https://github.com/eksperimental/tar_open
feel free to include it in your project.

# Using TarOpen
{:ok, fd1} = File.open("", [:read, :write, :ram])
{:ok, tar_desc1} = TarOpen.open({:file, fd1}, [:write])
:erl_tar.add(tar_desc1, "xxx", 'README.md', [])
{ok, contents1} = :file.pread(fd1, 0, 8192)
File.write("foo1.tar", contents1) 

# Using Tar
{:ok, fd2} = File.open("", [:read, :write, :ram])
{:ok, tar_desc2} = Tar.open({:file, fd2}, [:write])
Tar.add(tar_desc2, "xxx", 'README.md', [])
{ok, contents2} = :file.pread(fd2, 0, 8192)
File.write("foo2.tar", contents2)
1 Like
13

FWIW, you are almost certainly going to get better performance with doing file-based operations against something like tmpfs (versus bending a ram_file to your will); depending on how many times ram_file needs to resize its internal binary it could get very expensive…

2 Likes
14

Oh wow, thanks, i’ll look into it and see if any more changes need to be made.

1 Like
15

One last question, how did you create this library so quickly? Did you use a generator or did you write it from scratch? I’m asking simply because I would like to expand the library to fully cover :erl_tar in erlang. Thanks again for your help!

1 Like
16

Hi, I use GitHub - marianoguerra/efe: Elixir Flavoured Erlang: an Erlang to Elixir Transpiler which is really super cool tool (I give credits both in the README and the top of the .ex file)

It required a bit of tweaking, but it worked.

The library (Tar) is a 100% port of :erl_tar. TarOpen is just the minimum working version to offer the patched open/2.

1 Like
17

I would simply use tmpfs as mentioned earlier, you can mount it pretty much anywhere, it will give good IO performance and won’t require patching Erlang (and then running on your own build for a while, while it gets to the released version).

2 Likes