Skip to content
EnterraHost
Products
Tools
Hosting Email Support Client Area Find a domain

The sitemap, robots.txt and IndexNow

Three files decide what a crawler reads and where it goes next, and all three are text a person can check. The plugin serves them from the site rather than from a static file, so they follow the content.

The sitemap

The plugin serves its own XML sitemap at its own address and lists published pages, posts and the post types you include. It is generated from the content rather than written by hand, so a page that is published appears and a page that is deleted leaves.

WordPress serves a sitemap of its own at /wp-sitemap.xml. Two sitemaps are not a problem as long as one of them is declared and the other is not competing for the same query. The dashboard says which one is answering, which is the honest way to find out rather than guessing from the settings screen.

robots.txt

Rules are served by the plugin when you switch it on, and a real file in the site root wins over anything a plugin can add. That is not a bug in the plugin, it is how a web server works. If a file exists, fetch it and read it, because the screen cannot tell you what the file says.

The rules are the ordinary ones. Disallow the admin paths, allow the sitemap, and keep the paths a store needs. The one line worth explaining is the content signal, which is a declaration about how the content may be used. It is a statement in a text file rather than an access control, and a crawler that does not recognise it is required to ignore the line and keep reading.

IndexNow

IndexNow tells a search engine that a URL changed instead of waiting to be crawled. It needs a key, and the key has to be readable at the site root as a text file, because that file is the proof that whoever submitted the URL controls the domain.

The plugin writes the key file and disables WordPress's canonical redirect for that one path, so the file answers 200 rather than 301. A key file that redirects is a key file that fails verification.

Prove it

Fetch each address and read the response rather than trusting the screen.

curl -sI https://yoursite.com/sitemap.xml
curl -s https://yoursite.com/robots.txt
curl -sI https://yoursite.com/your-key.txt

The sitemap should be XML, the robots file should contain your rules, and the key file should be the key itself with no redirect in front of it. If the sitemap lists a page that returns a 404, the file is worse than absent, because a crawler will spend budget on the link.

The Blue Raven plugin page lists the rest of the content tools, and setting up redirects is the other half of telling a crawler what moved.

Summarize this guide with

You need an account with the assistant you choose. It fetches the guide itself, and you send the prompt, not us.

Every guide in this category is on the Blue Raven page.

Something here wrong, or out of date? Tell us at editorial@enterrahost.com.