AI-based Pull Request reviews

Never tried out anything similar , but it looks like it could be useful. Just installed it on our repo. My first impression is that is is quite noisy e.g. chore: Convert login page to react by shiroyasha · Pull Request #1638 · operately/operately · GitHub, but maybe I have to get used to it.

Did your team leave all the config options enabled or you made some modifications?

Yes it is very verbose and I think we will opt out some options (the generated graph doesn’t seem very useful)

This could be another alternative:

https://github.com/brave/pull-merge

I’ve never personally tried it but the maintainer is a friend of mine and he told me it’s generic so it should work on any type of repo.

Reporting back after a couple of weeks of using Sourcery that @cblavier shared. Initially the tool was a nuisance, it generated too much content and interfered with human-to-human communication. We had to silence most of its options, and to add a couple of extra handcrafted prompts to make it behave in a reasonable way.

With those additions, the generated comments started to be actually useful. It is not able to detect larger problems, wrong architecture decisions, but it is good at spotting typos, missing validations, edge cases, etc… It reduces the burden of manual reviews and gives feedback faster to the team.

Overall, the tool is in the nice-to-have category, but not yet in the must-have category.

I think this should be highlighted, as this is the silver lining. Such tools are just linters on steroids, even though I doubt they are that consistent, it would be a mistake to assume that they can replace the human factor in code reviews.

I also think it’s also important to point out that this might be a shortcut for companies that never understood how to setup and use already existing static analysis tools like credo (especially with custom made check rules).

For typo checks you should use GitHub - crate-ci/typos: Source code spell checker · GitHub. I’m using it almost every day and it simply works every time.

It even has a GitHub action if you don’t want to only use it as an offline CLI tool (which is how I use it exclusively but a GitHub action is super useful for CI/CD).

We eventually unsubscribed from Sourcery, it was too talkative for our taste.
And a bit too pricey for the value

I am curious: what would make you pay for automated code reviews? What are the top 3 features that would make you go “I want to have this now”?

I can’t answer since I had no expectations for such a tool.
My decision to try it was purely opportunistic and driven by curiosity.

I guess I could pay for it if it was more concise, and 100% accurate (not ChatGPT accurate )

i wrote my own (corerev.q32.com), here’s why:

i want it to:

• inline comment
• draft only
• in my name (for me to look at and approve/change/edit)

this:

• gives me focus
• saves me time

i use it every day.