codeRaider

codeRaider

Build correctly Massive Data Structure

TL;DR
Better way and Efficient methods to build a massive data structure (like a snapshot), or this is job for another language?

In-deep
In a Redis server, we have segmented objects by parameters. The base object is institute, this have a base json object with parameters like address, phone number, code, id, etc. Then, this institute have nested keys with data (json object), like institute:<code>:teachers, institute:<code>:executive and others.
There is a builder that create this massive structure by many institutes with a list of institute codes, one by one, this generate a base structure and merge and add the nested objects.

A little example:

%{
    "institute-a" => %{
      address: "Street A 453",
      id: 12,
      status: "active",
      online: true,
      location: %{
        country: "United State",
        state: "CA",
        city: "Los Angeles"
      },
      teachers: %{
        "john-doe" => %{
          age: 35,
          asignatures: ["Art", "Music"],
          id: 999
        },
        ...
      },
      courses: %{
        "A1" => {
          students: %{
            "carol-doe" => %{
              age: 15,
              grades: %{
                "music" => "A",
                "arts" => "B+"
              },
              ....
            }
          }
        }
      }
    },
    ....
  }

The duration to build this structure was nice (5 secs), but more institutes was added to the Redis server, already we have 500 institutes.
I try many forms, but, when the data is so massive, the request time is like 15~20secs to show this request data or send as response in a channel event.
I know elixir is not friendly with this kind of process, this is temporary, but I need ideas to be more efficient and respecting the elixir way to code.

Most Liked Responses

chulkilee

chulkilee

Did you find out what part is actually taking time mostly?

If building the data is 5 sec but the whole request takes 15-20.. it means something takes 10-15s :wink:

Things to check before going further (e.g. rustler)

  • Use iolist not string for redis command (if the library supports)
  • Check Redis library/server setting which makes thing slow - especially redis response time (not your app response time)
  • Do not rebuild data again
  • Avoid copying large data across erlang nodes - maybe better to refetch from phx channel process
chulkilee

chulkilee

Okay then you have N+1 query problem - this is data fetching problem not necessarily elixir.

You may make it faster by performing N+1 queries in parallel (use Task for instance) - but in this case you loose the benefit of Redis pipelining.

Anyway you’re accessing the data not in “Redis way”. You may prefetch most values if knows the pattern in advance - e.g. using SCAN or it’s family.

Or you can use Lua script to preprocess a little bit on redis side.

As you have redis, you may use redis for caching as well - which is helpful if transformation is expensive compared to network bandwidth between erlang nodes and redis instance.

darnahsan

darnahsan

You can have a look at this it might be of some help or atleast give you idea over hwo to tackle your problem

https://github.com/discord/sorted_set_nif

Where Next?

Popular in Questions Top

aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New

Other popular topics Top

aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? Ecto.Repo — Ecto v3.14.0 has exampl...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
RisingFromAshes
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
svb
Hi! Currently I want to submit a form by pressing the Enter key. However, since my input field is of type “textarea” this is just adds a...
New

We're in Beta

About us Mission Statement