Piping Toniq jobs

Questions / Help
1

Hello *, I’m a little confused by how Toniq works.
I can read on the documentation that I can actually use pipelines to queue jobs (https://github.com/joakimk/toniq) using the enqueue_to function.
If I try once everything works like a charm, my jobs are enqueued and run. If I try more than once it fails because the second worker receives a different data structure.
I solved with Enum.map, but honestly I’m not in love with my solution:

Enum.map([Service.Macaw.CreationWorker, Service.Gannet.CreationWorker],
         fn(worker) -> Toniq.enqueue(worker, service) end)

I would obviously prefer something like:

service
|> Toniq.enqueue_to(Service.Macaw.CreationWorker)
|> Toniq.enqueue_to(Service.Gannet.CreationWorker)

which seems much more idiomatic.

Suggestions?
ngw

2

Perhaps you should file an issue/feature request at toniqs tracker?

But I have to be honest: I think it feels odd to enqueue a single job item to multiple workers. But perhaps I am missing something here, since I do not use Toniq.

3

Well, it’s not a single job, it’s the data the job needs to perform it’s task.
There for example I’m creating something using Ecto, and then sending that to workers to notify some external services which will use the “service” in different ways.
I don’t think it’s that weird, right?

4

Well, no, your explanation makes absolutely sense and I do understand your usecase.

Have you already thought about using it like this?

Enum.map(workers, fn (worker) ->
  service
  |> Toniq.enqueue_to(worker)
end)

There is nothing in the documentation which makes me assume that Toniq.enqueue_to/2 could be used as you suggest in your original post. To me it looks like a finalizer.

You could create a simple helper behaving as you suggested for now:

def enqueue_helper(data, worker) do
  Toniq.enqueue(worker, data)
  data
end

service
|> enqueue_helper(Service.Macaw.CreationWorker)
|> enqueue_helper(Service.Gannet.CreationWorker)

This is only a quick draft and might not work until adjustments have been made.