CharlesO

CharlesO

I have read through this thread: binary-pattern-matching-when-input-is-a-stream on a related subject, but i’m not able to make any headway.

I get this error trying to load and parse a large binary directly:

iex> JQL.save "--1566364097905", "STMT.ENTRY"
eheap_alloc: Cannot allocate 18446744071692551144 bytes of memory (of type "heap").

Crash dump is being written to: erl_crash.dump...

I’m doing an ETL (extract - transform - load) for a reporting project, moving data from a proprietary platform into SQL Server for easier reporting.

The parser has worked fine until I hit this size issue.

Please how / can we use streams to handle these kind of situations?
(particularly when initial or existing parser had not been built with streaming in mind)

JQL Parser: https://gist.github.com/CharlesOkwuagwu/4c6c89d96db7876bc0d27fecd518340e (updated)

Sample large file (~850mb unzipped): https://paperlesssolutionsltd.com.ng/java/--1566364097905.7z

Thanks.

Marked As Solved

CharlesO

CharlesO OP

Found a really simple way to solve this. Just read the data-header, then parse row by row:

  def read(src, cnt \\ nil) do
    {:ok, fid} = :file.open("_cache/#{src}", [:raw, :read, :binary])
    {:h, start, n} = _p(fid, :h)

    cnt = cnt || n

    Logger.debug("Reading: #{cnt} of #{n} rows")

    Process.put(:start_pos, start)

    for i <- 1..cnt do
      {:r, start, row} = _p(fid, Process.get(:start_pos), :r)
      Process.put(:start_pos, start)
      Logger.debug(inspect({i, row}, @format))
    end

    :file.close(fid)
  end

  # READER
  defp _p(fid, :h) do
    # 000007775B4C4900000136490000000049FFFFFFFF4C490000006749000000004900000005
    {:ok, <<n::32>>} = :file.pread(fid, 33, 4)
    {:h, 37, n}
  end

  defp _p(fid, start, :r) do
    # 490000000049000000006A62 00000165 3136333737303....
    {:ok, <<l::32>>} = :file.pread(fid, start + 12, 4)
    {:ok, b} = :file.pread(fid, start + 16, l)

    {:r, start + l + 18, _split(b)}
  end

  defp _split(v) do
    b = for(<<c <- v>>, c in 32..126 || c in [252, 253, 254], into: "", do: <<c>>)
    b = :binary.replace(b, <<253>>, ";", [:global])
    b = :binary.replace(b, <<252>>, "^", [:global])
    :binary.split(b, <<254>>, [:global])
  end

Also Liked

NobbZ

NobbZ

But the error message you posted, it claims that it wants to allocate another 18446744071692551144 byte, which is 18014398507512256 kiB, 17592186042492 MiB, 17179869182 GiB, 16777215 TiB, 16383 PiB, so ~16 EiB…

As you can see @hauleth even missed by a factor of ~100…

There might be plenty of reasons why such a huge amount of memory might be allocated…

hauleth

hauleth

No surprise there as I assume that you do not have 163 PB of RAM available for your machine. It seems like you want to load a lot of data at once. Maybe try to split it into reasonable chunks?

NobbZ

NobbZ

Does the problem persist if you remove this line?

https://gist.github.com/CharlesOkwuagwu/4c6c89d96db7876bc0d27fecd518340e#file-jql-ex-L102

You are not even using the result of term_to_binary… And unless compressed, it will always require more memory than the input.

Where Next? Top

Trending in Questions Top

Blokh
Hey guys, I’ve got a huge CSV ( around 10 GB ) that needs to be processed hourly Do you guys have any suggestions what is the best prac...
New
kszambelanczyk
Hello! Could someone please give me a help/sample code, how to delete a file from s3 using waffle/waffle_ecto from Phoenix app. I creat...
New
Onor.io
I have what I’ve heard referred to as a “lookup table” in my database. This is a way of assigning codes to common values. One common lo...
New
Trolleger
What approach to take when sending live updates to “random” users Hi! I have a question, I have a little chat app, and when I create a DM...
New
RemyXRenard
I’m seeing that a list inside a Kino.DataTable will be interpreted as a charlist, even if the Kino.configure() is set to charlists: :as_l...
New
matt-savvy
Anyone here using Honeybadger? My Honeybadger account is being overwhelmed with noise from some bots. Seeing a lot of Bandit.HTTPError...
New
samoloth
Hi, I’ve just set up an application with ash_authentication. There is only magic link strategy for now, so there is no confirmation add o...
New

Other Trending Topics Top

mudasobwa
I am happy to introduce the very α version of the new programming language compiled to BEAM. Welcome Cure. It has literally three kille...
New
garrison
Hobbes is a low-level distributed database for the Elixir programming language. Hobbes provides a simple, safe, and scalable storage lay...
New
mcass19
ExRatatui lets you cook up rich terminal UIs in Elixir, powered by Rust’s ratatui via Rustler NIFs. Build interactive terminal applicatio...
New
Damirados
Hello everyone. After busy few months I am happy to announce v0.1.0 of Emerge &amp; Solve. They are GUI (Emerge) and State management (S...
New
netoum
Corex is an accessible, unstyled UI component library for Phoenix that integrates Zag.js state machines using Vanilla JavaScript and Live...
New
wintermeyer
There are three potential reasons for members of this forum to have a look at https://vutuv.de You are tired or annoyed of LinkedIn. Yo...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews