Phoenix `phx.new` generator and AGENTS.md

AGENTS.md is for LLMs so I wouldn’t necessarily call it a best practice or convention doc, but there will of course be overlap. The goals with this are gap-filling for SOTA LLMs to supplement their “world knowledge” only, rather than an exhaustive set of guidelines (which exist as phoenix guides). Less capable models will need more gap filling, but also being less capable they will have less world knowledge and I haven’t yet seen lower tier/local models cross be able to really cross the gap regardless of context stuffing. On the other hand, most state of the art models are closely on par with one another gap wise. Surprisingly so even. You can shop the AGENTS.md around to gpt-4.1/claude/gemini 2.5 pro/grok4 and get more or less similar results.

The generator is indeed the place for this

usage_rules is great, and zach is doing some great work there! It’s hard to beat a dependency free static file value wise at this point. As folks want to customize and extend their own rules they can bring in usage_rules to manage that process and aggregate their other deps that also provide rules. I don’t think credo makes sense at all in this flow. I do think that a mix task that aggregates AGENTS.md across your deps makes total sense, and it’s one thing that fell out of the Elixir contributors summit that Zach kindly took on the burden of running with. As the community standardizes on this stuff and libraries adopt AGENTS.md I could absolutely see deep integration with a mix task/lib out of the box, but in any case, the file must be there when you run phx.new, so even if we made it library generated, it would to be programmatically run inside phx.new to ensure it’s there out of the box.

Not at this time

Sounds like a great idea—we could start by implementing some kind of task. I actually have a couple of ideas we could use to create our own version.

I’m thinking we could begin with something pretty basic, maybe a Bash script that clones the project guide and installs it into your Phoenix project.

Also, here’s a source of inspiration we could look at: https://youtu.be/CTMyzeKKb0o

It already exists, and the Phoenix AGENTS.md will be compatible with the format. usage_rules v1.2.6 — Documentation

https://github.com/just-another-rule-craft/download-phoenix-guides
I think you were referring to something like this? Or was it something else?

I made a small Bash script that downloads all the guides into your project—you just need to be in your project directory and run the script.

I just generated a new phoenix app and noticed the AGENTS.md was pretty much the same as a few months ago and after doing some digging, as afar as I understand, it is basically one year old. Spent the weekend Fable tokens running evals on it and it looks like 90+ percent of it is no longer relevant, the model priors work just fine.

This experiment took 234 runs, for 3 models, with 7 tasks with and without Usage Rules, not all rules were tested, but for those the up-to-date models (Sonnet 5, Opus 4.8 and Fable 5) scored 94% of compliance, removing 23 of the 29 rules rules had no effect.

I am familiar with Elixir/Phoenix… but that’s it, so I am not suggesting an alternate version.

The harness Claude Code built to run the tests is available in Github is fully vibed, full disclaimer, in case you find it useful. And here is an artifact with the annotated AGENTS.mdfile that provides a quick overview of what was found.

CleanShot 2026-07-21 at 16.01.45@2x1200×924 73 KB

This is really cool @anibal, awesome job!

When Chris generated those rules, he did so by creating different apps. Given agents can run tasks for longer, what I would do is to have the tasks themselves be the creation of a new app, with features that exercise those paths.

So what I would do:

1. Ask your agent to generate the specification of three small apps that exercise the features there: authentication, forms, etc. The apps should be small, a todo list that can be shared with teams, a water ingestion tracker, etc.

2. Evaluate the generation with and without the rules

Would you be willing to run such evals again? Thank you!