Structuring Kafka topic consumer --> web UI

Questions / Help
1

I’m new to Elixir and FP, and I’ve been reading this forum for a few months waiting for the right project to pop up. I think I found my project — and I need advice already.

Background

Our company has around 150 partners, each with decently large interests with us, and it’s critical to contact them directly, quickly by phone for certain issues relating to their assets. Oddly, these people are bad at keeping up their profiles with us.

To make sure we always have the right contact information, we periodically run names/biz addresses through data providers like Pitney Bowes or whoever to get updated information. We get an updated PhoneNumber for the Name we provide.

Because some of our clients work across verticals/companies, some numbers returned will have multiple names attached. That is to say that BOB SMITH @ 123 MAIN ST might have num1 – but FIONA JONES @ 2244 SOUTH AVE might also be associated with num1.

This application is designed to validate the information that our providers give us.

Infrastructure

We run three providers in a waterfall. A main Query (name, address) goes to a microservice, and the microservice sends out three SubQuery requests to each of the three Providers.

All is hosted on GCP - Google PubSub logs the query and results for each request of a Provider.

A customer service agent will call ~5 clients per day. The exact agent will rotate to avoid “burn in” of the task. A mobile website will allow the agent to dial a number and select one of:

  • Who the number belongs to (multiple options available)
  • Disconnected
  • Wrong Number
  • Unknown

We will use this information to save money by reducing the number of our profile enhancement sources.

Questions

  • I want a web UI, but also have some need for ETL. Should I structure such a small app all in a Phoenix project?
  • Any libraries you would recommend?
  • (aside) Graph or Relational?
  • Umbrella project or multiple projects?
  • Is Elixir/Phoenix a choice you would make for this app?

Thanks - I’m so excited to finally get the opportunity to use this language!

2

Updates

I read my post. It’s missing some things. Also it’s not Kafka - it’s Google PubSub. Same same.

So this new service (that’s you are helping with) will:

  1. Consume the SubQuery topic
  2. ETL results to database (Postgres or Neo4j)
  3. Serve numbers to validate to customer service agents
  4. Store agent feedback (good, disconnected, wrong num) per Name+PhoneNumber combination
  5. Serve reports on the results

Maybe that will help tie it all together.

3

Hi,

Welcome to the forum

Maybe add a 2FA for the users so that you confirm there number up front and they can’t login without the correct phone number.

Something like this https://searchsecurity.techtarget.com/definition/two-factor-authentication

4

Thanks for the welcome!

The 2FA thing doesn’t solve the problem, since there isn’t a web app for them to authenticate with. This is strictly our own diligence.

I’ve simplified some of the business processes for this post - but I would be so grateful to know how you would solve the stated problem!

5

I still think that solution where you create a 2 Factor auth using the clients mobile is a good idea:

  • you will always have the correct numbers and emails and other relevant information
  • the people will agree to the terms and services
  • you can track orders easier and also create statistics with real data

Also others may have different views then me, so you should wait for others opinions and experiences.

Good luck with your project