AleksandarFilipov

AleksandarFilipov

How to seek a huge text file at the speed of light (well elixir :)

Hi forum, it’s been a while,

I would like some input from you brainics on the following;
I have huge textfile (GB) where each line starts timestamp.

timestamp0 data
timestamp1 data
...

So given a timestamp as input I need to locate the closest timestamp in the file and then parse line by line. Naturally this seek operation should be quick!

Well we know that this has been done a billion times before, however it hasn’t been done by me :wink:

I should be mentioned that the huge textfile, is produced during a recording session which I control - opening for a possibility to produce a secondary meta data file during the recording. (well to be fair there is actually already a meta data file containing some json (however no indexing)).

Surely there must exist a “by the book” solution here.

Thanks!

Most Liked

dimitarvp

dimitarvp

…or you can import it into SQLite and query that. :person_shrugging:

hst337

hst337

Is the file ordered by timestamp? If it is, you can perform binary search using :file.pread

fmn

fmn

Well, perhaps I should put more emphasis Something in “Something like:” :wink:
I am sure if you will take a closer look, you will find more issues - it’s just an idea.
That being said, I don’t think what you wrote is correct. If timestamp is at the end of file, none of the processes will exit because of the condition you described.

EDIT: To clarify - in what I described, search is going forward. If timestamp you are looking for is lower than first timestamp you encounter during the search, it means this particular process can’t find it. But other can.

Where Next?

Popular in Questions Top

lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
mgjohns61585
Could someone help me? I’m making my first elixir program, number guessing game. I can’t figure out how to convert the user’s guess from ...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
jaysoifer
Is there a way to rollback a specific migration and only that one (“skipping” all the other ones)? Would mix ecto.rollback -v 200809061...
New

Other popular topics Top

sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
985 43657 311
New
chrismccord
As promised, the first release candidate of Phoenix 1.3.0 is out! This release focuses on code generators with improved project structure...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
johnnyicon
Hi all, I’ve just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I’m trying to use Postgres...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
boundedvariable
I am going through the kafka architecture. All the features what the kafka is providing are already in Erlang. I would like hear your opi...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
komlanvi
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New

We're in Beta

About us Mission Statement