MihailPertsev
Hi there!
I have a tough understanding what to do:
I need to categorize a product by it’s keywords.
I have a several “Topics”, for example: “New Technologies”, “Health”.
Each of those “Topics” have several “Subtopics”, for example: "New Technologies => “Internet of Things (IoT)”, “New Technologies => 3-D Printing”
There are in average about 250 categorized products in each “Subtopic”.
I went ahead and grabbed all keywords from already categorized products, and made a list for each “Subtopic” with keywords and their frequency, for example: %{“keyword”: “stereolithography”, “uniqueCount”: 8, “totalCount”: 17}
So, long story short: I have a bunch of lists with keywords that I need to somehow “measure” and “rank” them.
I guess my next step should be creating a keywords for each “Topic” from its “Subtopic”'s combine “dictionary” of all “important” keywords… I have no idea how to do it. The keywords are overlapping among “Subtopics” but I still need to “measure” them somehow…
Is there any “cool and easy” Elixir library that can be useful in my case?
Please share any tips!
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ai
- #ecto-query
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #metaprogramming
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
**Building the future of software with AI, trust, and human connection.**
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 10- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
thiagomajesk
Could you provide a real example of the data structure you have and how you expect the output to be?
This will improve the quality of the answers you’ll get from the community.
MihailPertsev
Sure, I have a bunch of Excel files with this kind of structure:
For Topic: “New Technologies”
Subtopic: “Artificial Intelligence”
Subtopic: “Internet of Things”
Subtopic: “Machine Learning”
And this is for every Subtopic inside “New Technologies” Topic.
As you can see a lot of keywords are overlapping. For example: “artificial intelligence”, “big data”, “machine learning”.
Now I want to create a combined keywords list for “New Technologies” topic .
But I don’t know how to “measure” a value of the given keyword. I am really lacking in theory about this kind of stuff… For example: the keyword “big data” is in every single one those Subtopics, so I need to rank it lower because it is not unique, but I don’t know the formula or algorithm to do it…
So, it would be really nice to use some library (maybe not an Elixir library) to achieve it. But I don’t know about them as well…
MihailPertsev
And there is also a problem with the keywords themself… Some of them are meaning the same thing, like “internet things iot”, “iot”, “internet things”. All of them are basically the same thing, yet in my dataset they are different. But currently I want to treat them as unique keywords for the purpose of simplicity and MVP.
MihailPertsev
Any tips?
kokolegorille
First step would be to normalize your data…
After this, sure You can transform your data, Elixir is good at that
MihailPertsev
Hi!
I can’t understand what do you mean by “transform your data”. Can you please specify what exactly do you mean?
kokolegorille
@pragdave explains it much better than I would do
TLDR in FP, almost everyhing is expressed as input → function → output, which is transforming input into output.
MihailPertsev
I am still confused… I understand that I need to write a bunch of functions, which would transform my data. I am just clueless what algorithms should I use in those functions…
So, I have a lack of understanding “what” to do, instead of “how”.
Should I use something like GloVe for “measure” and “ranking”? Or maybe it is easier to use some python library (that I dont know about yet)… Thats kind of questions is my concern right now.
kokolegorille
I don’t know what You really want to achieve… but what about something like?
MihailPertsev
I want to create some simple “measure” system which can rank keywords in each Subtopic by it “importance”. My current thinking about “importance”: “if given keyword is appearing more frequently across different products inside Subtopic, then this keyword is more important…” But I don’t know what to do about keywords that are overlapping across many Subtopics, for example keyword “big data” is probably more important inside “Artificial Intelligence” than "Internet of Things. But I don’t know how to mathematically “measure” it.