codeRaider

codeRaider

Build correctly Massive Data Structure

TL;DR
Better way and Efficient methods to build a massive data structure (like a snapshot), or this is job for another language?

In-deep
In a Redis server, we have segmented objects by parameters. The base object is institute, this have a base json object with parameters like address, phone number, code, id, etc. Then, this institute have nested keys with data (json object), like institute:<code>:teachers, institute:<code>:executive and others.
There is a builder that create this massive structure by many institutes with a list of institute codes, one by one, this generate a base structure and merge and add the nested objects.

A little example:

%{
    "institute-a" => %{
      address: "Street A 453",
      id: 12,
      status: "active",
      online: true,
      location: %{
        country: "United State",
        state: "CA",
        city: "Los Angeles"
      },
      teachers: %{
        "john-doe" => %{
          age: 35,
          asignatures: ["Art", "Music"],
          id: 999
        },
        ...
      },
      courses: %{
        "A1" => {
          students: %{
            "carol-doe" => %{
              age: 15,
              grades: %{
                "music" => "A",
                "arts" => "B+"
              },
              ....
            }
          }
        }
      }
    },
    ....
  }

The duration to build this structure was nice (5 secs), but more institutes was added to the Redis server, already we have 500 institutes.
I try many forms, but, when the data is so massive, the request time is like 15~20secs to show this request data or send as response in a channel event.
I know elixir is not friendly with this kind of process, this is temporary, but I need ideas to be more efficient and respecting the elixir way to code.

Most Liked

chulkilee

chulkilee

Did you find out what part is actually taking time mostly?

If building the data is 5 sec but the whole request takes 15-20.. it means something takes 10-15s :wink:

Things to check before going further (e.g. rustler)

  • Use iolist not string for redis command (if the library supports)
  • Check Redis library/server setting which makes thing slow - especially redis response time (not your app response time)
  • Do not rebuild data again
  • Avoid copying large data across erlang nodes - maybe better to refetch from phx channel process
chulkilee

chulkilee

Okay then you have N+1 query problem - this is data fetching problem not necessarily elixir.

You may make it faster by performing N+1 queries in parallel (use Task for instance) - but in this case you loose the benefit of Redis pipelining.

Anyway you’re accessing the data not in “Redis way”. You may prefetch most values if knows the pattern in advance - e.g. using SCAN or it’s family.

Or you can use Lua script to preprocess a little bit on redis side.

As you have redis, you may use redis for caching as well - which is helpful if transformation is expensive compared to network bandwidth between erlang nodes and redis instance.

darnahsan

darnahsan

You can have a look at this it might be of some help or atleast give you idea over hwo to tackle your problem

https://github.com/discord/sorted_set_nif

Last Post!

fsa

fsa

I know very little about this topic - but before doing any code changes or system alterations - check if the increase in the size of the data being processed is causing swaps to disks anywhere… just a thought :slight_smile:

Where Next?

Popular in Questions Top

rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New

Other popular topics Top

nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
gausby
I asked this very same question on twitter and got some interesting feedback, but I thought it would be a good question to ask here as we...
1207 40082 209
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New

We're in Beta

About us Mission Statement