Curated HEX packages

Chat / Discussions
,
1

Everything that I shared is based on my experience, please don’t be offensive.

npm, ruby gems and now hex. I always have to deal with the same problem: find the best package for my project.

I notice that engineers in general like to just go and reimplement everything from scratch when something doesn’t work as them spec, or even sometimes because they don’t understand the current one.
That’s why you go and search in those websites for some dependencies you find out a huge amount of packages that basically try to resolve exactly the same issue.

I never understand why we (programmers in general) don’t work together as a team, because will always be better to work together and think more than one person in the same situation.

Yes, I understand that sometimes we got so deep in one implementation that we can’t use some packages or the trade off is huge compared with recreate your own.

I also understand that sometimes I will find some project driven by dictatorships, or even people don’t want to change something because somebody else will break.
I am not saying that having a LEADER in projects is bad, also I am not saying that we shouldn’t help “legacy” people

but,

we can’t hold some implementation, or even don’t change something that TECHNICALLY is better because somebody else will break, that’s why we have versioning and is their responsibility to upgrade or not.

All of those and many more problems can be resolve and I would like to find the solution

Anyway

I would like to see a little badge in HEX packages, that means that those packages are well supported and maintained.
Also I will group the packages in categories and I will only give the badge to the one that is the common and mainly use .This way we help people like me and the community.

I don’t believe in having a lot of packages that literally fix the same problem. I prefer to work together.
I would like to continue writing but I think you get the main idea.

And please share your opinion because this is really important for the Elixir community, now that we are smaller enough for curated HEX.

Let’s think together and work together

cc: @josevalim @chrismccord @ericmj

2 Likes
2

There is already awesome-elixir and also there has been another list that had an interactive search AFAIR and also tagged its packages, but I can’t remember the URL.

Also I do not like to have some kind of batch on hex. Hex is the “central authority” when it comes to packages. If it were advertising one package over another it would have a huge impact on the community and how new packages will be created or found.

6 Likes
3

I wouldn’t mind to support the idea of badges or something. Yes I don’t want Hex to hide any package at all but I would like to see features that I allow me to filter Hex packages.

We will end like npm where is pretty much useless to find anything on it because is just bad unless you are familiar with the packages. I just want to create a community mindset where we work together instead of going alone and creating packages that fix the same issue.

Sometimes is good to have some common vision (like Ecto), and there I relay on the more experienced developers.
This is about something like @h4cc package that have 1,580 issues related to “Evaluate Package”. Let’s bring that to Hex instead.

Even go father and create subset where one package could compete each other and the community decide about the final decision

4

I think it’s better to let Hex focus on serving packages. It would be too much work for me to go through 2800 packages and evaluate if they are good or bad and then decide which ones are the “best” in each category. The other alternative would be to have the “community” vote, but it’s not clear from your post how that should actually work. I don’t think it should be part of Hex either since Hex should focus on its core features, which is publishing and serving packages. There’s a lot of work running and maintaining a package manager and the core team has a long list of more important enhancements and features to work on before we can focus on curation or community voting.

It’s not a bad idea to have a place where you can find recommended packages. There is GitHub - h4cc/awesome-elixir: A curated list of amazingly awesome Elixir and Erlang libraries, resources and shiny things. Updates: and http://toolbox.elixir.pm/ already that does similar things. Maybe you can help out on those efforts? Both those projects use the HTTP API from the specifications and you should be able to build a service with the API that would be just as good as if the service was part of Hex.

9 Likes
5

Right now, either inside Hex or outside but a place that allow us to go and pretty much know that we have a curated list. It’s more around that idea.

I can’t give you the final answer in how to curated, that’s why I created this post and expecting people to participate and try to resolve a legit problem in packages management systems.

We as a community can contribute to this list but as well have some management and inputs from the “leaders” of Elixir community.

2 Likes
6

I think sorting by popularity (downloads) has often served as a really good way to see what is ‘hot’.

I like the idea of voting and reviews/comments - however I am not sure Hex.pm would be the best place for it as it would add a lot of work for Eric and the maintainers (because they will require some sort of moderation, and not just for spam).

What could be really handy is whenever a package has an associated github then the number of watchers and stars are added (updated daily or weekly).

3 Likes
7

What could be really handy is whenever a package has an associated github then the number of watchers and stars are added (updated daily or weekly).

:+1:
I agree the idea.

1 Like
8

I don’t know how any of these measurements of package «popularity» would work. I fear that anything we could come up with would become a skewed popularity contest.

Had we had a system that let people vote for their favourite package back when Chris started Phoenix it would have been shadowed by the popularity of Dynamo, which was build by Jose himself, and would gain tons of up-votes because of that. (Not saying this is how it would have panned out—this example is based on a big «what-if», and is pure fiction)

Besides that, stars on Github is not a good measure of what is good about the project. Say we have a project with a very funny Readme; tons of people would hit that star because of the lols. It might not solve the problem it claims to do, but it has a really funny Readme…

…and what if some popular twitter account post a link to a project on Github? Hoard of people would come to the page, each with a different idea about what it means to star something; some of them might use stars for «Interesting, should look into later», others might think «I have vetted this project and I use it in production.»

Not to mention all the projects that are not hosted on GitHub.

The best thing I’ve seen is something some of my friends whipped together for NPM called http://node-modules.com/ If you personalise your account the «popularity contest» is based on what your friends/followed on Github likes—not the entire world. This is not perfect, as it require the project to be on Github, but one great benefit about this is that you can ask your friends about their experience with a given package.

Hex should not deal with popularity contests in my opinion—only serve packages. A thing like node-modules for hex would be great but it should be a stand-alone service.

6 Likes
9

This is why I need you (community) to decide which one is most proper package in specific group.

We have to either stop going crazy like NPM and create packages that resolve exactly the same issue in almost exactly the same or do something for at least help the community.

This is a rough task and we need to work together specially the Leaders of Elixir community. No everybody will get some badge identifier that easier.

And yes I dont want to relay on Stars on Github or stuff like that. I want to work with you (community) in a solution that at least help us.

I really don’t want to see what happen with NPM guys :icon_sad:

P.S: My searching for elasticsearch packages, I got this GitHub - Zatvobor/tirexs: An Elixir flavored HTTP client and DSL library for Elasticsearch and this one GitHub - werbitzky/elastix: A simple Elasticsearch REST client written in Elixir.
Who is the best one? At the end of the day no matter what they are trying to resolve the same issue. Just merge the projects, bring the best ideas of both sides and booom the best for the community.
But those contributors have to be open to feedbacks and make changes when are required, even when you break people (that’s why versioning exists) and …

Please, either in hex.pm or any website, let’s fix this issue, I take responsibility to implement it but I need your feedbacks about it in how could work.

10

For example, if I just type ORM keyword I will separate in two list the packages. I would put the “recommended by the community” on top and whatever else in the bottom.

As you can see, I never said: hide packages, but just at least is something that help the community. I prefer people contributing to some package than creating their own, but again Owners and Contributors have to be really open to feedbacks and changes.

11

Maybe it’s too late now but Ember does something like this. Every package has a score:

Ember Observer Click on “Explain this” under the score.

Package curation is tricky and imprecise though, especially in the early days of a language. Also, open source is not a job. I may write the best and most maintained package ever, but tomorrow I can stop working on it forever. What then?

I don’t believe in having a lot of packages that literally fix the same problem. I prefer to work together.
I would like to continue writing but I think you get the main idea.

Too late for that. Search for “cache” in Hex :stuck_out_tongue: I’m a pessimist so I think Hex will be a mess like every package manager is, but we’ll get on by using the same 10-20 dependencies in most of our apps so whatever’s in Hex won’t matter much. Hex’s problem is to serve and host packages, not decide which is “best”, which is hard to do and will lead to politics/drama anyway. Just let Hex be messy. I take the time to review possible dependencies and their repos to see if they’re maintained well, but I don’t think the community can do that as a whole.

Check out: npms We can at least have a search like that for Hex. It all comes down to search: I don’t care if there are 1000 bad cache implementations if search returns the best one.

12

Again, doesn’t have to be Hex because Hex is used fro package management but the idea is the same one. I would love to see in hex website because people truth more.

@ericmj I don’t want to accept that “Hex” will finish in the same. This is why I am calling this problem now when we are not that big. Whatever takes I am wiling to help but please don’t let that happen.

13

The problem is that there’s no solution.

Hex’s only job is to server and store packages. No one in the Hex team wants to review packages, especially as Elixir gains traction and we get hundreds+ a week. They also don’t want a high barrier of entry for people to publish stuff either. Remember, Hex is an open source project done for fun, not anyone’s day job (correct me if I’m wrong).

The community is made up of tons of people. People are impossible to get together to agree on many things, especially tools and software. It happens very rarely and only for very big things like Phoenix or Express in Node (even then, many want Koa or Hapi).

Maintainers come and go and projects die after a few years. It’s an extra burden to keep track of which projects go unmaintained. Some projects don’t need maintenance at all after 1.0.0 because they’re simple.

Also, it’s not that bad. I get stuff done in Node all the time and rarely have a problem finding good libraries. I use npms which I linked above—their algorithm takes into account stuff you’ve mentioned and returns great results. Someone could build the same for Hex. After a few projects you just know which packages and people are good.

Phoenix, HTTPoison, Poison, Plug, Poolboy, Ecto (just naming a few) are all set in stone for years to come and the community has already decided on them, but I can go and publish 20 mediocre JSON parsers on Hex right now if I want. But you’ll never find them or use them with a good search system because you’ll get Poison instead. That’s my only solution: Have a good search algorithm and keep up with the community. Or have your own site that lists curated packages based on some algorithm you come up with.

You can’t stop people from publishing trash unless you hire someone to review packages 24/7, and then you’ll be known as the language that’s not friendly to beginners/people experimenting with packages and new ways to do stuff. I’ve even seen people accidentally publishing their Phoenix web app on Hex! Some programmers hate using dependencies, some like reinventing the wheel. People will always be silly :slight_smile:

Anyway, that’s my rantish answer. Let me just say I do agree with you and I’d love to have a perfect package repository with only the best stuff on it, but it’s just not going to happen.

7 Likes
14

As long as hex lets me install packages or view their documentation in a timely manner, that’s all I care about. And honestly I be pretty against any kind of moderation/curation on hex itself, as it could just lead to whole other set of problems.

Now having a separate service that curates the packages is most welcome, currently I use awesome-elixir quite often. Although solving the problem of project curation certainly isn’t easy. There’s a few different problems that come to mind.

  1. How do you determine what is the better package? What might be the best package for one user’s use case, might not be the best for another. Look at two popular GraphQL implementations graphql-elixir and absinthe, both are great choices (and from what I’ve previously read, there’s no intention of them to be merged, they’ll both continue to be developed separately), but they take very different approaches to their API. Certain users will prefer the workflow of one of them over the other, while other users will think the opposite.

  2. Curated lists (unless actively maintained and use filtering criteria that considers this) often don’t allow for new ideas to flourish. While you could say, those ideas should be contributed to an already established package. That’s always easier said than done, the already established project might not be at the liberty to make/adopt those changes. Sometimes it really is easier to just establish it as a new project. Now imagine someone develops an alternative to Ecto that’s better than Ecto in every possible way (I know, I know, how could that be possible :dizzy_face:). Given that Ecto is already established as the goto library in that space, it’s most likely that new project will never get exposure on a curated list (unless this has been carefully considered for).

  3. Effective categorisation is a lot more tricky than it sounds. As someone gave the example of searching for “cache”. What would be the best cache package? What even is a cache? A cache related package could be applied to many different areas, solving many different problems. If the categorisation/search isn’t considerate of this, it could lead to suggested packages that don’t even solve what the person is looking for.

Now I don’t know the answer to… Well, any of those :sweat: But they are definitely issues I think need to be considered. And they are at least some of the reasons why I wouldn’t want hex to try tackle it, at least a separate service can afford to make those mistakes while discovering what’s the best way to go about solving this problem.

3 Likes
15

I agree with most of the people here that Hex is not the right place for this idea. But I think the difference between Hex and other package managers is that it is already backed by the Elixir community, largely and well-represented on this forum, previously on the mailing list, and also now on Slack.

Building on this awesome community, I have found out that we have sticky “wiki” posts for editors and IDEs: Spacemacs, Emacs, Atom, and others. If you are researching any particular packages with a specific use case, could you not create a specific post for that use case and explain the packages you’ve found? I know I was recently involved in a discussion on “happy pipes”/railway-oriented programming libraries. Each of the different libraries had strengths, weaknesses, flavors that could conceivably be grown into/turned into a wiki?

If you create said post, and it is well researched, then the sticky post could show the various packages that exist for the use case, profiling any package’s strengths. Then over time, it grows and evolves, depending on the need for the use case. You could include any information you find relevant there, and it would be driven by moderation from the community, just as this forum is in general.

But I’m not a mod and I don’t know if this is really feasible. :smile:

1 Like
16

We definitely encourage Wikis here on the forum - anyone at Trust Level 1 or higher can edit them and the wiki owner always gets notified of any changes. Have a look here for more info:

2 Likes
17

@ibgib: Yes, it frequently happens, both on the Slack chat group and on this forum, that people have questions like ‘I am facing problem X, what is the best package to help me solve it?’

The nice thing about that, is that multiple people can give a very specialised opinion, based on the state of the packages at that point in time, and based on the given problem’s exact circumstances. I think this is really valuable, and one of the things I love the most about this community :).

2 Likes
18

There is a solution to the problem. I glad that somebody made it

1 Like
19

So, here’s my issue with this kind of thing. I did a bit of searching for libraries related to 2 areas I know a lot about: AWS and GraphQL.

Hex.pm

Search term: results

AWS: ExAws at 188k+ downloads comes in first, followed by AWSAuth and aws at 3k downloads.
Graphql: Absinthe comes in first at 24.6k downloads, the graphql package comes in 3rd at 14k downloads.

Both of these give the correct answer. ExAws is wildly more feature complete and tested than the aws library. The holds true for Absinthe and GraphQL, and the latter project hasn’t had a commit in 3 months.

elixir.libhunt.com

Search term: results

AWS: AWSAuth comes in first for… no particular reason as far as I can tell? All the numbers on the page are lower than they are for other libraries. Next is aws, finally followed by ex_aws
Graphql: graphql package is up top, followed by their parser, followed by Absinthe.

How is this supposed to be better? Someone using this site to try to use AWS or start with GraphQL would be directed towards two projects neither if which constitutes the best the Elixir community has to offer in that area.

3 Likes
20

Searching “web” ranks sugar higher than pheonix. Grouping things by category is of no value if the results in a category send people in the wrong direction.

4 Likes