Welcome to WebmasterWorld Guest from

Message Too Old, No Replies

Robots.txt vs. JavaScript for Robot Control

5:48 pm on Aug 5, 2008 (gmt 0)

New User

5+ Year Member

joined:Dec 11, 2006
posts: 14
votes: 0

I got a message in the Google Webmaster console the other day about a site having too many URLs, encouraging me to look at the URL structure and to robots.txt off duplicate URLs. However, they provided a list of problematic URLs and approximately 60% of the URLs were already robots.txt disallowed. I used the robots.txt analyzer tool in Google Webmaster console to verify that everything was configured right and it is.

What should I do here? Do I need to code every URL that I don't want Googlebot to crawl in JavaScript? That's a pretty old school approach, but I've noticed a few competitors do it recently.

Has anyone done any tests on using robots.txt vs. JavaScript for robot control? What works better?

6:40 pm on Aug 5, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
votes: 0

If those URLs are disallowed in your robots.txt then you can safely disregard the WMT messages. As to the comparison between robots.txt and javascript, robots.txt wins hands down. It's the official standard, where as Google has never promised that they "won't" use links in javascript. in fact, these days they sometimes do use javascripted links for discovery.