Help: Check My Work - Fetching Basic Presence Details With Ecto - Advice Welcome

Phoenix Forum Chat / Discussions
,
1

Greetings!

I’m still in the learning process and I’m trying to piece together a basic script to fetch user data from a database to use with Phoenix.Presence. I am starting with the example in the documentation: Phoenix.Presence — Phoenix v1.6.2 . I almost have a fully working example, but I hit an error that I can’t figure out, and I need some advice on whether I’m using “import”, “use” and “alias” to tie together my modules correctly.

Here is what I got so far:

/lib/exchat_web/channels/presence.ex:

defmodule ExchatWeb.Presence do
  alias ExchatWeb.Accounts

  use Phoenix.Presence, otp_app: :exchat,
                        pubsub_server: Exchat.PubSub

  def fetch(_topic, presences) do
    users = presences |> Map.keys() |> Accounts.get_users_map()

    for {key, %{metas: metas}} <- presences, into: %{} do
      {key, %{metas: metas, user: users[String.to_integer(key)]}}
    end
  end

end

/lib/exchat_web/accounts/accounts.ex:

defmodule ExchatWeb.Accounts do
  import Ecto.Query
  alias Exchat.Repo

  alias Accounts
  @derive Jason.Encoder
  def get_users_map(ids) do
    query =
      from u in Profiles, # use lib/exchat_web/schemas/profiles.ex
        where: u.id in ^ids,
        select: {u.id, u}

    query |> Repo.all() |> Enum.into(%{})
  end

end

/lib/exchat_web/schemas/profiles.ex:

defmodule Profiles do
  use Ecto.Schema

  schema "members" do
    field :username, :string
  end
end

Background:
Phoenix.Presence will track registered users that have integer user ids (also found in database), as well as anonymous readers that have randomly generated alphanumeric characters stored in cookies (not found in database). The anonymous readers can enter public chat rooms and read, but they won’t be able to chat. I’ll be using a simple “fetch” function to retrieve the user’s username, profile image, age, etc from the database - as recommended in the Phoenix.Presence documentation. We won’t need (nor want) to fetch anything in the database for these anonymous users. We’ll just count up those anonymous users in the presence.list using client javascript and display a count (# of viewers).

Question 1: The first thing I don’t know how to do is when “fetch” function is called, how would I be able to exclude the non-integer user ids (the anonymous users), as these won’t be found in the database, and sometimes the database query will come up empty if everyone is anonymous in a room? What would be a good way to handle this? I would need a simple code example to look at.

The Error Message: When I run the app above, I eventually end up with errors related to Jason.Encoder (I’m using an API setup for my app). I’m assuming the data coming from the database needs to be encoded to JSON, or does this happen at a different step in the “fetch” function or presence module? Or do I have to set this up in the Schema? I’m not really familiar with this:

[error] Task #PID<0.506.0> started from ExchatWeb.Presence_shard0 terminating
** (Protocol.UndefinedError) protocol Jason.Encoder not implemented for %Profiles{__meta__: #Ecto.Schema.Metadata<:loaded, "members">, id: 1, username: "John"} of type Profiles (a struct), Jason.Encoder protocol must always be explicitly implemented.

If you own the struct, you can derive the implementation specifying which fields should be encoded to JSON:

    @derive {Jason.Encoder, only: [....]}
    defstruct ...

It is also possible to encode all fields, although this should be used carefully to avoid accidentally leaking private information when new fields are added:

    @derive Jason.Encoder
    defstruct ...

Finally, if you don't own the struct you want to encode to JSON, you may use Protocol.derive/3 placed outside of any module:

    Protocol.derive(Jason.Encoder, NameOfTheStruct, only: [...])
    Protocol.derive(Jason.Encoder, NameOfTheStruct)
. This protocol is implemented for the following type(s): Ecto.Association.NotLoaded, Ecto.Schema.Metadata, DateTime, Atom, Float, Integer, Jason.Fragment, Any, Map, Date, NaiveDateTime, Time, BitString, Decimal, List
    (jason 1.2.2) lib/jason.ex:199: Jason.encode_to_iodata!/2
    (phoenix 1.5.12) lib/phoenix/socket/serializers/v2_json_serializer.ex:29: Phoenix.Socket.V2.JSONSerializer.fastlane!/1
    (phoenix 1.5.12) lib/phoenix/channel/server.ex:101: anonymous fn/5 in Phoenix.Channel.Server.dispatch/3
    (elixir 1.12.2) lib/enum.ex:2385: Enum."-reduce/3-lists^foldl/2-0-"/3
    (phoenix 1.5.12) lib/phoenix/channel/server.ex:86: Phoenix.Channel.Server.dispatch/3
    (elixir 1.12.2) lib/registry.ex:474: Registry.dispatch/4
    (phoenix_pubsub 2.0.0) lib/phoenix/pubsub.ex:286: Phoenix.PubSub.dispatch/5
    (phoenix 1.5.12) lib/phoenix/presence.ex:364: anonymous fn/4 in Phoenix.Presence.Tracker.handle_diff/2
    (stdlib 3.15.2) maps.erl:410: :maps.fold_1/3
    (phoenix 1.5.12) lib/phoenix/presence.ex:363: anonymous fn/3 in Phoenix.Presence.Tracker.handle_diff/2
    (elixir 1.12.2) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
    (stdlib 3.15.2) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Function: #Function<0.58243068/0 in Phoenix.Presence.Tracker.handle_diff/2>
    Args: []

Any other advice and tips would be greatly appreciated.

Thanks!

2

Update: By trial and error, I was able to resolve the Error problem by adding a jason.encoder line in the schema file, so that it is now:


defmodule Profiles do
  use Ecto.Schema
  @derive {Jason.Encoder, only: [:username]}

  schema "members" do
    field :username, :string
  end
end

I can now fetch the details for presence from ecto. However, I am still trying to figure out how to pass the ids to the ecto query when one id has non-integers. For example, this is the error message I am receiving:

[error] GenServer #PID<0.541.0> terminating
** (Ecto.Query.CastError) lib/exchat_web/accounts/accounts.ex:9: value `["1", "17852", "20365", "26845", "29900000", "74563", "8832692c0609e6e523c9a0d3b9e734"]` in `where` cannot be cast to type {:in, :id} in query:

from p0 in Profiles,
  where: p0.id in ^["1", "17852", "20365", "26845", "29900000", "74563", "8832692c0609e6e523c9a0d3b9e734"],
  select: {p0.id, p0}

    (elixir 1.12.2) lib/enum.ex:2385: Enum."-reduce/3-lists^foldl/2-0-"/3
    (elixir 1.12.2) lib/enum.ex:1704: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3
    (elixir 1.12.2) lib/enum.ex:2385: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ecto 3.7.1) lib/ecto/repo/queryable.ex:203: Ecto.Repo.Queryable.execute/4
    (ecto 3.7.1) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
    (exchat 0.1.0) lib/exchat_web/accounts/accounts.ex:13: ExchatWeb.Accounts.get_users_map/1
    (exchat 0.1.0) lib/exchat_web/channels/presence.ex:14: ExchatWeb.Presence.fetch/2
    (exchat 0.1.0) lib/exchat_web/channels/room_channel.ex:20: ExchatWeb.RoomChannel.handle_info/2
    (phoenix 1.5.12) lib/phoenix/channel/server.ex:343: Phoenix.Channel.Server.handle_info/2
    (stdlib 3.15.2) gen_server.erl:695: :gen_server.try_dispatch/4
Last message: :after_join
State: %Phoenix.Socket{assigns: %{status: 0, user_id: "8832692c0609e6e523c9a0d3b9e734"}, channel: ExchatWeb.RoomChannel, channel_pid: #PID<0.541.0>, endpoint: ExchatWeb.Endpoint, handler: ExchatWeb.UserSocket, id: nil, join_ref: "3", joined: true, private: %{log_handle_in: :debug, log_join: :info}, pubsub_server: Exchat.PubSub, ref: nil, serializer: Phoenix.Socket.V2.JSONSerializer, topic: "room:lobby", transport: :websocket, transport_pid: #PID<0.538.0>}

In my query, you can see that the first 6 “registered users” have integer ids. If I just query those, everything goes through fine. If the id doesn’t exist in the database, the user details result in “nil” which is prefect:

However, when the 7th user joins, who is an anonymous user that has a randomized alpha-numeric user id (example: 8832692c0609e6e523c9a0d3b9e734), I get the error message above. My goal is to set the user details to “nil” for these users.

from p0 in Profiles,
  where: p0.id in ^["1", "17852", "20365", "26845", "29900000", "74563", "8832692c0609e6e523c9a0d3b9e734"],
  select: {p0.id, p0}

How can I do this and resolve the error message?

2 Likes
3

Trying to separate users from anonymous by checking if the key might be an integer or a string is not really clean.

What if You start to use binary-id? Why not set a status in presence… like authentified/anonymous? For example, when You generate this uuid?

Are You making db calls each time presence change?

This alias is very suspicious. Even if it is right, aliasing a one word module has no benefits.

Anyway, did You try Enum.filter, combined with Integer.parse() to separate your ids?

1 Like
4

I’m actually having a hard time just trying to find the basic elixir functions to check if a key, or each item in a map is an integer or string (or even !integer or !string). I came across something in the docs that seemed promising and had a combined function with Integer.parse: Enum — Elixir v1.12.3

def get_users_map(ids) do
  ids = ["1234", "abc", "12ab"]

  IO.inspect(ids)
  Enum.flat_map(ids, fn string ->
    case Integer.parse(string) do
      # transform to integer
      {int, _rest} -> [int]
      # skip the value
      :error -> []
    end
  end)
  IO.inspect(ids)
.......
......
end

I thought that would be perfect to call just before the ecto query to the database, but that doesn’t do anything. The IO.inspect(ids) shows the same thing before and after the function.

I would also need to find the string keys in that list as well, to set the user profile data to “nil” for each of those “anonymous user” keys and combine that with the database user data and return the whole thing to the presence.list.

As for using a binary-id or something like adding “is_anonymous” variable and storing it into presence, I had thought about this, as well as just adding the username, profile image and everything else from the socket.assigns into the presence tracker. However, it seems that the phoenix.presence docs recommend against this: Phoenix.Presence — Phoenix v1.6.2

For example, if I added something like this:

    {:ok, _} = Presence.track(socket, socket.assigns.user_id, %{
      online_at: inspect(System.system_time(:second)),
      is_anonymous: 1
    })

The docs are saying this will increase the size of the metadata. They recommend keeping the metadata as small as possible, and use the “fetch” function to get user profile data from the database. I asked about this in the forum last week, and the consensus appears to be the same as well, so I’m trying to set this up the way the docs recommend with the “fetch”.

1 Like
5

Adding is_anonymous: 1 won’t supercharge your metadata. It will help You filter your ids.

1 Like
6

Why do You return list, if after You flat_map?

There is no way “abc” will be parsed to an integer.

What about testing if rest is “”? This would reject “12ab”

{int, ""} -> int
{_, _} -> nil
_ -> nil
1 Like
7

Thanks for the alias advice. I deleted that line and the script still works, so that line must have been unnecessary to get the script to run.

Even with the is_anonymous: 1, I’d still have to use the “fetch” and somehow get the integers out of that list to query them.

Also, I believe the database is being queried each time the presence changes. It seems like it’s making 4 calls each time a new user joins just “room:lobby”:

["1", "17852", "20365", "26845", "29900000", "74563"]
["1", "17852", "20365", "26845", "29900000", "74563"]
[debug] QUERY OK source="members" db=1.5ms queue=0.9ms idle=823.4ms
SELECT m0.`id`, m0.`username`, m0.`id` FROM `members` AS m0 WHERE (m0.`id` IN (?,?,?,?,?,?)) [1, 17852, 20365, 26845, 29900000, 74563]
["1", "17852", "20365", "26845", "29900000", "74563"]
["1", "17852", "20365", "26845", "29900000", "74563"]
[debug] QUERY OK source="members" db=0.7ms queue=46.9ms idle=0.0ms
SELECT m0.`id`, m0.`username`, m0.`id` FROM `members` AS m0 WHERE (m0.`id` IN (?)) [17852]
[]
[]
[debug] QUERY OK source="members" db=0.5ms queue=43.4ms idle=0.0ms
SELECT m0.`id`, m0.`username`, m0.`id` FROM `members` AS m0 WHERE (m0.`id` IN (?,?,?,?,?,?)) [1, 17852, 20365, 26845, 29900000, 74563]
[debug] QUERY OK source="members" db=0.3ms queue=3.7ms idle=0.0ms
SELECT m0.`id`, m0.`username`, m0.`id` FROM `members` AS m0 WHERE (false) []

The second call has an empty ids list and the third call is a duplicate of the first and the fourth call is another empty list. I was going to ask about that later. I’m not sure why that is happening, but it seems like a huge waste of db resources.

8

Why do You return list, if after You flat_map?

I’m actually just using the example directly from the documentation: Enum — Elixir v1.16.0 . I thought the example was a map of strings and was the same thing as the ids list I’m trying to pass into my function from the accounts.ex script in my example: ids = ["1234", "abc", "12ab"] From what the documentation describes, I assumed the the flat_map / Integer.parse functions would simply take my map of strings and turn it into a map of integers (and removes all the non-integer strings, like “abc” or “12ab” - or ultimately the anonymous user ids). That way the ecto query will run without errors when the anon user ids are removed.

9 
iex> ids = ["1234", "abc", "12ab"]
["1234", "abc", "12ab"]
iex> Enum.filter ids, fn x -> r = Integer.parse(x); is_tuple(r) && elem(r, 1) == "" end
["1234"]
10

I converted your function and tried to put it in the module and run the script, but it’s still not working:

defmodule ExchatWeb.Accounts do
  import Ecto.Query
  alias Exchat.Repo

  def get_users_map(ids) do
    IO.inspect(ids)   # ["1", "29900000", "8832692c0609e6e523c9a0d3b9e734"]

    Enum.filter(ids, fn x ->
      r = Integer.parse(x);
      is_tuple(r) && elem(r, 1) == ""
    end)
    IO.inspect(ids) # ["1", "29900000", "8832692c0609e6e523c9a0d3b9e734"]

    query =
      from u in Profiles, # use lib/exchat_web/schemas/profiles.ex
        where: u.id in ^ids,
        select: {u.id, u}

    query |> Repo.all() |> Enum.into(%{})
  end

end

Perhaps I converted your function wrong (I tried it “as is” as well), or may the map of keys being sent from presence to the fetch function to “get_users_map” is different than I’m expected, or it’s not a map? Or maybe IO.inspects(ids) is making it look like it’s a map, but it’s really not?? I don’t understand why it’s not working.

11

I have added ; because I wanted to write on one line…
You don’t need it when writing multiline.

Elixir variables are immutable, but You can rebind…

    ids = Enum.filter(ids, fn x ->
      r = Integer.parse(x)
      is_tuple(r) && elem(r, 1) == ""
    end)
    IO.inspect(ids) # ["1", "29900000"]
1 Like
12 
defmodule ExchatWeb.Accounts do
  import Ecto.Query
  alias Exchat.Repo

  def get_users_map(ids) do
    IO.inspect(ids)   # ["1", "29900000", "8832692c0609e6e523c9a0d3b9e734"]

    ids = Enum.reduce(ids, [], fn x, agg -> 
       case Integer.parse(x) do
          {i,""} -> [i|agg]
           _ -> agg
       end
    end)
    |> IO.inspect() # [29900000, 1]

    from(u in Profiles, # use lib/exchat_web/schemas/profiles.ex
        where: u.id in ^ids,
        select: {u.id, u}
    )
    |> Repo.all()
    |> Map.new()
  end
end

The reduce call filters out the non-integers and returns a list of integers, al be it in reverse order.
And as @kokolegorille mentions, you need to reassign ids, as variables in Elixir are immutable.

2 Likes
13

Yes, Enum.reduce is a way to filter AND transform at the same time…

1 Like
14

Flat map is also a way to filter and transform at the same time, which is what the original function was doing.

Enum.reduce(ids, [], fn x, agg -> 
  case Integer.parse(x) do
     {i,""} -> [i|agg]
     _ -> agg
  end
end)

Is equivalent to

Enum.flat_map(ids, fn x -> 
  case Integer.parse(x) do
     {i,""} -> [i]
     _ -> [] 
  end
end)

Empty lists will be removed during the flatten.

EDIT: actually, the reduce version will reverse the list while flat_map will preserve the original order. So if you want to transform and filter flat_map is actually preferable.

2 Likes
15

Yes, I saw the docs where the example came from… and the reason why flat_map is used.

Or List.foldr… so You don’t have to reverse the result.

1 Like
16

Thanks a lot for all of the help from everyone, I learned quite a bit more about elixir in general as well.

17

Last tiny question, if the ids list is empty, I would like to skip the ecto query entirely to save database resources. I came up with this simple logic:

  unless(Enum.empty?(ids)) do
    IO.inspect("ids list not empty - do query")
    from(u in Profiles, # use lib/exchat_web/schemas/profiles.ex
        where: u.id in ^ids,
        select: {u.id, u}
    )
    |> Repo.all()
    |> Map.new(fn {key, value} -> {Kernel.to_string(key), value} end)
  else
    IO.inspect("ids list is empty - skip query")
    Map.new(%{}) # return an empty map
  end

It works, but I was just wondering if there was a more efficient way of writing this in the Elixir mindset? How would you write this?

18

I would do it with 2 functions where you pattern match.
And %{} is already a map, so no need for Map.new/1.

1 Like
19

Thanks for the tips, I greatly appreciate it. I’ve removed the unnecessary %{} and attempted to make two functions with pattern matching.

I’m still trying to wrap my head around this new elixir type of programming, so I’m not completely sure I’m doing it right. What do you think of these functions / pattern matching?

  def get_users_map([]) do
    IO.inspect("pattern match empty query")
    Map.new()
  end

  def get_users_map(ids) do
    IO.inspect("pattern match - ids found")
    from(u in Profiles, # use lib/exchat_web/schemas/profiles.ex
      left_join: img in ProfileImage,
      on: img.id == u.mainimageid,
      where: u.id in ^ids,
      select: {u.id, %{"username" => u.username, "gender" => u.gender, "status" => u.status, "image" => img.path}}
    )
    |> Repo.all()
    |> Map.new(fn {key, value} -> {Kernel.to_string(key), value} end)
  end

What would your functions look like?

20

They would probably look the same.

1 Like