Googlebot-News Bot Says Good-Bye

America’s human work force isn’t the only one affected by the desire of companies to streamline operations. Google has put Googlebot-News, its news site crawler, into retirement, too. The good-byes for Googlebot-News were brief, and Googlebot, the general web crawler, promptly took over the job of crawling news sites.

Information in this post gathered in association with a NY Personal injury attorney

If you’re not a news publisher, this won’t mean much to you. But if you are, you may want to know what—if anything—you need to do to ensure continuity in the way Google handles your content. Vanessa Fox discusses the bot change and how it might affect your site in “Google Retires The Googlebot-News Bot.”

First the good news: You probably don’t have to do anything. Googlebot will crawl your site and index your content to both news and web search. If you’ve been using a robots.txt file or meta robots tag to block Googlebot-News, Googlebot will also respect that.

The bad news is for people who were allowing Googlebot-News, but were blocking Googlebot. Now that Googlebot is crawling for both news and web search, blocking it means that Google cannot index your site for either service. You will have to allow Googlebot in order to appear in Google News.

There’s more bad news for troubleshooting pages that aren’t being indexed. Without separate bots, you can’t see whether particular pages are being crawled for web search, news or both. You can still see pages that aren’t crawled at all, but most of the time that’s not likely to be the problem. And Google’s webmaster tools aren’t much help in this area. They do provide crawl error information that’s either news- or web-specific, but the available information isn’t enough to figure out if you might accidentally be blocking news indexing.

Vanessa Fox considers this change a step in the wrong direction. Do you agree?