sitemap.xml is old hat, use IndexNow instead, apparently

by mark | 24 May 2022, 9:04 p.m.

Updated by mark | 27 May 2022, 5:07 p.m.

So after going to all that hassle of getting sitemap.xml working it seems I have to implement IndexNow to ping Bing and other web greppers whenever I post stuff. Urgh. Not too bad, really.

• Generate a key, some random hex string that is quite long and hard to guess. You can even generate a key online.
• Make a key.txt file. Not too bad in Django. Store your key as an environment secret; load it in settings; create a url that looks for key.txt; have a really simple template that renders it. I called the key INDEXNOW_KEY.
  
  The URL I added to urls.py is:    
  
  path(f"{settings.INDEXNOW_KEY}.txt", TemplateView.as_view(template_name='INDEXNOW_KEY.txt', extra_context = {'key': settings.INDEXNOW_KEY}))
  
  Template file INDEXNOW_KEY.txt is really simple: {{ key }}
• Make a ping_bing() function, almost exactly like ping_google(), but you'll need to construct your new object's URL rather than just telling it to read the sitemap (this is all ping_google() does). The bing URL is https://www.bing.com/indexnow? ; you need two things in the query string, a url pointing to your new object and the key. You can use bing_url + urlencode({'url' : your_object_url, 'key' : settings.INDEXNOW_KEY }), and then feed this to urlopen()
• Amend relevant save() methods to ping bing when called. (I am not going to submit everything; bing already has sitemaps and any other web crawler should pull those as well the first time they come across my site)
• That is all that you need to do. It's nice watching the logs as <key>.txt is pulled, and then you can use Bing webmaster tools to verify that your object has been submitted.  
• And then I found a bug and had to recommit!