Google has announced that Googlebot, its web crawler, now executes and indexes some content in JavaScript. This change could impact negatively search results, unless a few basic rules are taken into account.
In a post on Google's Webmaster Central Blog, developers Erik Hendriks and Michael Xu, and Webmaster Trends Analyst Kazushi Nagayama, advise web developers that some changes that have taken place in the way Google handles JavaScript content when indexing web pages could impact negatively search results and provide a few hints as to how to prevent that.
According to Hendriks, Xu, and Nagayama, "in the past few months, Google's indexing system has been rendering a substantial number of web pages more like an average user’s browser" would when JavaScript has been turned on. This differs from the way Google traditionally indexed pages, i.e., by "only looking at the raw textual content that we’d get in the HTTP response body and didn't really interpret what a typical browser running JavaScript would see."
Sometimes, they write, JavaScript rendering will not have the expected outcome, "which may negatively impact search results for a site". To prevent this from occurring, they are offering some hints at potential problems and possible ways around them:
- Sites blocking JavaScript or CSS files from access will not allow Google's indexing system to see a site like an average user. The recommendation is to allow for JavaScript and CSS files access in robots.txt. This is especially relevant for mobile sites, since JavaScript and CSS files would allow Google algorithms to understand that a page is optimized for mobile.
- Web servers should also be able to handle the volume of crawl requests for resources, otherwise rendering results may be affected.
- JavaScript code which is too complex or arcane could also prevent rendering the page fully and accurately.
- Sometimes, JavaScript is used to remove content from a page rather than adding; this will make the removed content inaccessible to Google indexing engine.
Finally, suggest Hendriks, Xu and Nagayama, it's always a good idea to have a webpage degrade gracefully. This will make its content accessible to search engines that can't execute JavaScript yet.
Google is working on a tool that should be available soon to help webmasters better understand how Googlebot renders JavaScript and CSS content.