Wow that presentation is full of interesting insights, thank you! And gb_trees looks like a useful structure, will read more about it (esp. once erlang documentation site starts loading for me).
@jonathanleang Interesting. I’ve never tried using these Ecto streams, I don’t think they play too well with a case where the stream should live for a long time. In our case I wrote my own module called StreamExt and implemented a couple of streaming functions in it. One based on offset/limit, and another (more efficient one) based on id comparisons.
defmodule StreamExt do
@doc """
Stream data in batches from an Ecto repo using a query.
"""
def ecto_in_batches(repo, query, batch_size \\ 1000) do
import Ecto.Query, only: [from: 1, from: 2]
Stream.unfold(0, fn
:done ->
nil
offset ->
results =
repo.all(from(_ in query, offset: ^offset, limit: ^batch_size))
if length(results) < batch_size,
do: {results, :done},
else: {results, offset + batch_size}
end)
end
def ecto_in_batches_by_id(repo, query, batch_size \\ 1000) do
import Ecto.Query, only: [from: 2]
Stream.unfold(-1, fn
:done ->
nil
last_id ->
results =
repo.all(
from(
r in query,
where: r.id > ^last_id,
limit: ^batch_size
)
)
{new_last_id, count} =
results
|> Enum.reduce({-1, 0}, fn r, {_, i} -> {elem(r, 0), i + 1} end)
if count < batch_size,
do: {results, :done},
else: {results, new_last_id}
end)
end
end
Oh you’re right! That function always expects id as the first item, and if I rearrange select fields it won’t work. Great find! However, I don’t recommend customizing the function because it’s meant to be reusable in any place where you need to stream data from database. Instead I suggest to go back to what you were doing originally, but it’s much simpler now:
I have an other question.
Currently the smaller or more negative the score is the higher the rank is, how do I rank the player higher, the higher the score is ?
Notice how in the README example the scores are all negative. I didn’t want to implement any special sorting logic, and instead left this simply up to the user, so all you need to do is put a minus in front of your scores, then map them back (minus again) when getting them for display.
Example:
def leaderboard_stream() do
import Ecto.Query, only: [from: 2]
query = from(r in Darkmoor.User, select: {r.id, r.elo, r.name})
Darkmoor.Repo
|> StreamExt.ecto(query, batch_size: 5, strategy: :id)
|> Stream.map(fn {id, elo, name} -> {{-elo, id}, name} end) # <=== minus added on this line
end
Hi there, we just launched our game and got featured on the app store, the CPU load is really high and our leaderboard controller is taking up most of the time.
is there any quick fix, our server is hosted on DigitalOcean 8 cores now, and soon there will be no bigger instance to upgrade to, we have been doubling the traffic every 1-2 days, the leaderboard seems to run much faster on MBP maybe memory bandwidth is much lower on vm.
