dogweather
My first LiveView app, "IndexFlow" - Automatic IndexNow site submissions
The idea is: Decoupled IndexNow URL submissions, using a website’s XML Sitemaps as input.
Here’s a walkthrough. It works, but there’s not a ton to see yet: https://www.youtube.com/watch?v=DJSjUHUb3Rs
I started making this last week: I have thousands of legal content web pages. These get updated weekly, but only a few hundred at a time. Since I have so many pages, I have the problem of telling search engines efficiently about the updates. I need to be very careful about my “crawl budget” (I have too many pages for crawlers to re-crawl my entire site at once.)
Google has stopped allowing pings to notify them about XML Sitemap updates. But I have a ton of investment in getting sitemaps working. IndexNow is a newer alternative. It’s push, rather than pull.
I was in the middle of custom coding IndexNow support when I realized I could cut down on the work by just using the sitemaps I’m already generating. And then I realized that this could be a standalone app. A week later, here we are.
I’m looking for beta testers, too. DM me if you’re interested.
Thanks for the feedback!
Most Liked
cgraham
Cool! Is this open-source or will be free? I was just about to vibe code an open-source version of this but if you are doing that, happy to participate in your beta!
cgraham
Nice!! Roll with it! I think it might have legs! In between our conversations, I rolled my own (vibe coded) for my own site(s) but I can see this having legs with a lot of people - especially new vibe coders!
Rooting for you for this to take off!
dogweather
Yeah, I realized that a pretty simple script could do this - just pull down your xml sitemaps, loop through the entries, and submit the ones with lastmod later than the last time the script ran. Like, literally a 5-line Ruby script.
And of course, I underestimated how much more work it is to make a SaaS app that other people would want to pay for. The biggest additions I’m adding are:
- Full logging & accountability: E.g., exactly what’s the status of this one particular URL?
- Content Intelligence: Targeted, smart analytics. I realized that since the app already has the robots.txt, list of sitemaps, and all the URLs, it can help me make sure that I’ve actually set the
lastmods correctly — or that I’m publishing on the schedule I think I am. In development I’m working on a histogram for each website showing URLs in buckets by age. It’s already helped highlight issues I didn’t know I have: I thought I’d have URLs many years old, but it shows them all as < 30 days, which is not what I expected.
Popular in Discussions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance













