AleksandarFilipov
How to seek a huge text file at the speed of light (well elixir :)
Hi forum, it’s been a while,
I would like some input from you brainics on the following;
I have huge textfile (GB) where each line starts timestamp.
timestamp0 data
timestamp1 data
...
So given a timestamp as input I need to locate the closest timestamp in the file and then parse line by line. Naturally this seek operation should be quick!
Well we know that this has been done a billion times before, however it hasn’t been done by me ![]()
I should be mentioned that the huge textfile, is produced during a recording session which I control - opening for a possibility to produce a secondary meta data file during the recording. (well to be fair there is actually already a meta data file containing some json (however no indexing)).
Surely there must exist a “by the book” solution here.
Thanks!
Most Liked
dimitarvp
hst337
Is the file ordered by timestamp? If it is, you can perform binary search using :file.pread
fmn
Well, perhaps I should put more emphasis Something in “Something like:” ![]()
I am sure if you will take a closer look, you will find more issues - it’s just an idea.
That being said, I don’t think what you wrote is correct. If timestamp is at the end of file, none of the processes will exit because of the condition you described.
EDIT: To clarify - in what I described, search is going forward. If timestamp you are looking for is lower than first timestamp you encounter during the search, it means this particular process can’t find it. But other can.
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance









