How can I train a model to predict a master record when trying to merge Maps?

Elixir Framework Forums Nx Forum
, ,
1

Almost every day we are struggling at work merging objects based on some conditions that keep manually. I wonder how difficult this could be to train a model using elixir ML ecosystem that helps predicting the master for the others to be merged onto?

Any guide out there on how perform this trainings in Elixir? I do not have ML background

Thanks! :smiley:

2

Let me get this straight, you want an AI to merge maps in Elixir instead of BEAM?

1 Like
3

Just one question: what?

4

This sounds like a task that cannot tolerate inaccuracy. If that’s the case you will be very sad using AI for it.

1 Like
5

These are not just Maps in Elixir. This is information I get from an external system (Hubspot) and these records when hey have the same value on key “domain” then should be merged into one, we do it manually following kind of same process to determine the master to merge into, so if I provide these merges as input and train a model where the master record is tagged as such then I can predict future ones.

I got something working using Python but was wondering how to translate same thing to Elixir.

6

And don´t be that sarcastics, it´s becoming a rust-like forum otherwise, what a shame

1 Like
7

Yes, you can write NNs with Elixir using Nx, Axon and family
No, you shouldn’t use machine learning for this kind of problem

8

You have some ML working in Python that does what you want?

9

So really this is the crux of what you’re hoping to solve with ML right? Like you’re merging a few large data structures, and obviously you can ignore any unset fields, but for fields that are set and in conflict you need to decide which of those structures is going to take precedence in the merge?

So yeah, Axon is definitely worth your time there, you’ll find good docs and a nice overview linked from the README.

1 Like
10

Cynicism/sarcasm aside, this is a totally valid question. If the merging process requires recognizing certain patterns in the data itself (regardless of whether it’s being represented in a map), then training a model can be a good way to go about it. It depends on which data is being interpreted and how the decisions are made. I don’t think there is any reason why ML could not be a good fit, based on what OP shared.

1 Like
11

Thanks @linusdm , that’s exactly the point. To explain it better, I’m NOT trying to merge Elixir Maps, instead I have an endpoint that receives the master and the other records that should be merged. So, in summary:

I get the records via API, want to pass them to a trained model that helps me to predict the master one, when master is detected, call the merge endpoint ({master record}, [the other records]).

Right now my team is doing it manually based on some patterns that can be easily put into samples to train a model (I guess), so then we can start testing.

I got something in Python, sooo basic, but wanted this in Elixir to keep learning and also because I found in the pass this forum pointing me out really good documentation and samples

12

Hello :wave:

If you can collapse all that patterns in an algorithm written by yourself I think it would be the most stable/debuggable solution.

If you can’t, going for a ML solution is a very good try :slight_smile: , I would suggest to go for a non opaque model like decision tree. You will have less issues if you need to debug it and you will need less samples to train it.

I think the SciKit-Learn version for NX is Scholar, don’t know which models are available currently.

1 Like