On one of my WordPress sites, I was confused as to why it was not getting indexed in Google or other search engines.
I verified my site in Google Webmaster Tools and tried to submit a sitemap. In the webmaster tools panel, it told me that my robots.txt file was blocking Google from getting my sitemap.
Which I thought was strange, as I know I would never knowingly make a robots.txt file to block search engines.
So what was the problem and more importantly, how did I fix it?
Well first of all I checked out my robots.txt file and saw the following:
User-agent: *
Disallow: /
As if disallowing all bots wasn’t enough, I saw all the pages of my site had the following meta tag in it.
<meta name=’robots’ content=’noindex,nofollow’ />
Okay, so now my SEO was killed on two different levels – search engines can’t index or follow links, if they could somehow first get past the robots.txt.
How did I remedy this catastrophic mistake? It’s a simple fix.
I navigated to my privacy settings in the WordPress administration panel and selected I would like my blog to be visible to everyone, including search engines (like Google, Sphere, Technorati) and archivers.
So to prevent it happening to you.
When installing WordPress, make sure you don’t deselect the checkbox that reads: Allow my blog to appear in search engines like Google and Technorati.
So in conclusion, you should always verify your site(s) in Google Webmaster Tools to see if there are any problems with indexing. I hope you found this quick WordPress tip informative, because although this is a simple mistake, it’s something to be aware of when SEO’ing WordPress sites.
Related posts:
Hi Steve
Great post. Someone told me that “The latest versions of wordpress have that set to “block search engines” which seems mad if true!
I went to check my settings and they were fine, in my dashboard.
Then I went to Google Webmaster Tools in my Google account and entered my NicolaCairncross.com blog website, to verify it, and it was indexed ok, but with another site BusinessSuccessFactory.co.uk, I had to add a bit of code to the blog theme header (which was easy, in theme editor, I think I’ve done it ok).
Then I tried to add my XML sitemap URL which I havent’ managed to do. Do I put /blog/xmlsitemap in or just the xmlsitemap bit?
What else should we pay attention to in the Webmaster Tools, in your opinion?
Cheers, and keep the WP / SEO tips coming!
Nicola