dogweather

dogweather

My first LiveView app, "IndexFlow" - Automatic IndexNow site submissions

The idea is: Decoupled IndexNow URL submissions, using a website’s XML Sitemaps as input.

Here’s a walkthrough. It works, but there’s not a ton to see yet: https://www.youtube.com/watch?v=DJSjUHUb3Rs

I started making this last week: I have thousands of legal content web pages. These get updated weekly, but only a few hundred at a time. Since I have so many pages, I have the problem of telling search engines efficiently about the updates. I need to be very careful about my “crawl budget” (I have too many pages for crawlers to re-crawl my entire site at once.)

Google has stopped allowing pings to notify them about XML Sitemap updates. But I have a ton of investment in getting sitemaps working. IndexNow is a newer alternative. It’s push, rather than pull.

I was in the middle of custom coding IndexNow support when I realized I could cut down on the work by just using the sitemaps I’m already generating. And then I realized that this could be a standalone app. A week later, here we are.

I’m looking for beta testers, too. DM me if you’re interested.

Thanks for the feedback!

Most Liked

cgraham

cgraham

Cool! Is this open-source or will be free? I was just about to vibe code an open-source version of this but if you are doing that, happy to participate in your beta!

cgraham

cgraham

Nice!! Roll with it! I think it might have legs! In between our conversations, I rolled my own (vibe coded) for my own site(s) but I can see this having legs with a lot of people - especially new vibe coders!

Rooting for you for this to take off!

dogweather

dogweather

Yeah, I realized that a pretty simple script could do this - just pull down your xml sitemaps, loop through the entries, and submit the ones with lastmod later than the last time the script ran. Like, literally a 5-line Ruby script.

And of course, I underestimated how much more work it is to make a SaaS app that other people would want to pay for. The biggest additions I’m adding are:

  • Full logging & accountability: E.g., exactly what’s the status of this one particular URL?
  • Content Intelligence: Targeted, smart analytics. I realized that since the app already has the robots.txt, list of sitemaps, and all the URLs, it can help me make sure that I’ve actually set the lastmods correctly — or that I’m publishing on the schedule I think I am. In development I’m working on a histogram for each website showing URLs in buckets by age. It’s already helped highlight issues I didn’t know I have: I thought I’d have URLs many years old, but it shows them all as < 30 days, which is not what I expected.

Where Next?

Popular in Discussions Top

Donovan
Hello everyone, I’m so glad to have discovered this awesome community. Thanks for creating it! This is my second post, and apologies for...
New
jeramyRR
This is an interesting article to read. Elixir’s performance, like usual, is excellent. However, it seems like the high CPU usage is co...
New
laiboonh
Hi all, I am trying to convince my team to use liveview over the current react. What are some of the points where one should consider us...
New
mmmrrr
Just saw that dhh announced https://hotwire.dev/ Is it just me or is this essentially live view? :smiley: Although I like the “iFrame-e...
New
lorenzo
Hey everone! I created a prototype for my app using Nodejs for the api. But the framework I chose wasnt great (in general theresnt any g...
New
AstonJ
Are there any Elixir or Erlang libraries that help with this? I’ve been thinking how streaming services like twitch have exploded recentl...
New
Ankhers
Just a little information upfront. Generally speaking, if I feel like I need to either break a pipe chain or use an anonymous function in...
New
New
acrolink
How does the two languages compare when it comes to server side application development? Any experiences or ideas? Thank you.
New
dogweather
I wrote this comment on r/haskell, and it’s not popular there. :wink: But I think I’m on to something… Haskell reminds me of Java, and e...
New

Other popular topics Top

danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 29377 241
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
jaysoifer
Is there a way to rollback a specific migration and only that one (“skipping” all the other ones)? Would mix ecto.rollback -v 200809061...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
gausby
I asked this very same question on twitter and got some interesting feedback, but I thought it would be a good question to ask here as we...
1207 39297 209
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
AstonJ
Seen any cool LiveView demos, sample apps or examples? Please post them here! :003:
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement