byu
Phoenix `phx.new` generator and AGENTS.md
@chrismccord : I just saw the Extract AGENTS.md from Phoenix.new into phx.new generator commit to the phoenix project.
My initial shotgun all-over the place unorganized reactions:
- I like the idea of the community having a set of AI instructions that works well for elixir projects, which brings together experimentation from all parts of the community… As someone that just wants to “be productive” and doesn’t want to waste time “prompt engineering” an AGENTS.md file from my own frustrations… seeing what others have done/shared is nice.
- The merged AGENTS.md reads like an “Elixir + Phoenix + Ecto” best practices or a conventions document… Phoenix has great guides, Credo is like a best practice enforcer.
- Is the generator a place for this? Or would a separate community contributed “Phoenix AI Suggestions Package” (maybe credo?) that
usage_rulesuses to add toAGENTS.mdbe a better place? - Is there a Phoenix project’s AI Policy, like Ash Framework: AI Policy - #6 by zachdaniel ?
I’m just curious about what the community thinks about elixir plus AI agents instructions in general.
Trending in Discussions
Other Trending Topics
Latest Phoenix Threads
Chat & Discussions>Discussions
Latest on Elixir Forum
Sponsor Spotlight
Develop your skills with books, videos, and courses.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #performance
- #security
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Most Liked
zachdaniel
It already exists, and the Phoenix AGENTS.md will be compatible with the format. usage_rules v1.2.6 — Documentation
chrismccord
AGENTS.md is for LLMs so I wouldn’t necessarily call it a best practice or convention doc, but there will of course be overlap. The goals with this are gap-filling for SOTA LLMs to supplement their “world knowledge” only, rather than an exhaustive set of guidelines (which exist as phoenix guides). Less capable models will need more gap filling, but also being less capable they will have less world knowledge and I haven’t yet seen lower tier/local models cross be able to really cross the gap regardless of context stuffing. On the other hand, most state of the art models are closely on par with one another gap wise. Surprisingly so even. You can shop the AGENTS.md around to gpt-4.1/claude/gemini 2.5 pro/grok4 and get more or less similar results.
The generator is indeed the place for this
usage_rulesis great, and zach is doing some great work there! It’s hard to beat a dependency free static file value wise at this point. As folks want to customize and extend their own rules they can bring in usage_rules to manage that process and aggregate their other deps that also provide rules. I don’t think credo makes sense at all in this flow. I do think that a mix task that aggregates AGENTS.md across your deps makes total sense, and it’s one thing that fell out of the Elixir contributors summit that Zach kindly took on the burden of running with. As the community standardizes on this stuff and libraries adopt AGENTS.md I could absolutely see deep integration with a mix task/lib out of the box, but in any case, the file must be there when you run phx.new, so even if we made it library generated, it would to be programmatically run insidephx.newto ensure it’s there out of the box.Not at this time
anibal
I just generated a new phoenix app and noticed the AGENTS.md was pretty much the same as a few months ago and after doing some digging, as afar as I understand, it is basically one year old. Spent the weekend Fable tokens running evals on it and it looks like 90+ percent of it is no longer relevant, the model priors work just fine.
This experiment took 234 runs, for 3 models, with 7 tasks with and without Usage Rules, not all rules were tested, but for those the up-to-date models (Sonnet 5, Opus 4.8 and Fable 5) scored 94% of compliance, removing 23 of the 29 rules rules had no effect.
I am familiar with Elixir/Phoenix… but that’s it, so I am not suggesting an alternate version.
The harness Claude Code built to run the tests is available in Github is fully vibed, full disclaimer, in case you find it useful. And here is an artifact with the annotated AGENTS.mdfile that provides a quick overview of what was found.
Last Post!
josevalim
This is really cool @anibal, awesome job!
When Chris generated those rules, he did so by creating different apps. Given agents can run tasks for longer, what I would do is to have the tasks themselves be the creation of a new app, with features that exercise those paths.
So what I would do:
Ask your agent to generate the specification of three small apps that exercise the features there: authentication, forms, etc. The apps should be small, a todo list that can be shared with teams, a water ingestion tracker, etc.
Evaluate the generation with and without the rules
Would you be willing to run such evals again? Thank you!