Questions & Help>Questions

keys distribution between processes

#job-processing#pooling
gpr

gpr

keys distribution between processes

Hi. I have the following task:
I have a server, which support subscription to different topics. One connection supports subscription for up to some limited number of topics, say 1000. I want to subscribe to, say, up to 10000 topics, so I use pool of up to 10 connections. Connections can fail and be restarted. The task is to distribute topics between connections and I would like to avoid manual tracking of which topic belongs to which connection. If number of subscribed topics is low, I would like to migrate topics between connections and shutdown some connections.

If I abstract from topics/connections, task sounds like distributing keys (topics) between processes with some configuration of limits, migration strategy, etc. I’m doing it manually now, but code is more complex than I wanted it to be, and I’m wondering, if there are some libraries in erlang/elixir which support such key distribution algorithms.
ANy suggestions are welcome.

1 comment
#job-processing#pooling
0 552 1

First Post!

dmitrykleymenov

dmitrykleymenov

I’m not completely sure what you mean, but if i’ve understood correctly, consistent hashing is what you need. If so, this may help.

Post #2

Where Next?

View thread on forum (has 1 response!)
job-processing
pooling
Home Questions & Help>Questions
#job-processing#pooling
0 555 1
Last post

Popular in Questions Top

sen
Questions & Help>Questions

How to set environment variables in dev.exs?
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
/phoenix
31 21927 15
New
chokchit
Questions & Help>Questions

DBConnection Error - connection not available and request was dropped
** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2733ms. You can configure how long re...
/phoenix#ecto
16 15283 10
New
_russellb
Questions & Help>Questions

Web scraping tools
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
#web-scraping
123 19532 45
New
Trevoke
Questions & Help>Questions

(EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
I have an umbrella app. Some of the apps inside depend on other apps in the umbrella, unsurprisingly. I’m writing a test for one of the...
#otp#umbrella#testing#exunit#errors
26 21602 11
New
skosch
Questions & Help>Questions

Put/update deep inside nested maps (and auto-create intermediate keys)
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
#data-structures#maps#immutability
52 20278 11
New
Fl4m3Ph03n1x
Questions & Help>Questions

Learning Elixir, frst impressions ( plz don't kill me ! )
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
#learning-elixir
371 26239 61
New
vegabook
Questions & Help>Questions

How to serve an img with Phoenix?
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
/phoenix
15 18745 5
New
RisingFromAshes
Questions & Help>Questions

How to get the current URL?
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
/phoenix
20 18753 18
New
chensan
Questions & Help>Questions

(Postgrex.Error) ERROR 42804 (datatype_mismatch): column "" cannot be cast automatically to type integer
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
#ecto
29 13524 4
New
Brian
Questions & Help>Questions

Import a module from a file into IEX
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
#iex
35 32626 16
New

Other popular topics Top

Nvim
Chat & Discussions>Discussions

Django vs Phoenix
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
/phoenix#learning-elixir
184 21696 82
New
JeremM34
Questions & Help>Questions

How can I check Phoenix version?
Hello, how can I check the Phoenix version ? Thanks !
/phoenix
35 28221 8
New
johnnyicon
Questions & Help>Questions

How do I use the Postgres JSONB / Postgrex JSON extension?
Hi all, I’ve just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I’m trying to use Postgres...
#ecto
22 17673 9
New
RisingFromAshes
Questions & Help>Questions

How to get the current URL?
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
/phoenix
20 18753 18
New
sergio_101
Questions & Help>Questions

Pattern matching against a string
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
#pattern-matching
28 27317 14
New
bsollish-terakeet
Questions & Help>Questions

Checking if an enum is empty - Credo vs Compiler
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
#compiler#credo
8 23194 6
New
nobody
Questions & Help>Questions

How to get the server ip address?
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
/phoenix
19 49320 16
New
klo
Chat & Discussions>Discussions

Concat/appending lists
Got a question about when to concat vs. prepending items to list then reversing to achieve appending. So i know lists boil down to [1 | ...
#erlang
20 15629 11
New
PeterCarter
Questions & Help>Questions

Good auth solutions for Elixir/Phoenix?
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
/phoenix#authentication
48 21542 20
New
lanycrost
Questions & Help>Questions

How To Implement if...else if...else condition
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
#how-to-question
40 52243 6
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

Coders51

We build reliable cloud platforms for business-critical systems.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

Manning

Develop your skills with books, videos, and courses.

WyewWorks

Enabling companies to succeed by building software people love.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

Manning

Develop your skills with books, videos, and courses.

WyewWorks

Enabling companies to succeed by building software people love.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement