- - Search Engines
- -- Search Engine Spider and User Agent Identification
- ---- And Now Google's Doing It. JS Stats Show GoogleBot
TheMadScientist - 7:02 am on May 14, 2011 (gmt 0)
Then you misunderstood:
|In a nutshell |
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:
The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
Thread source: http://www.webmasterworld.com/search_engine_spiders/4312058.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com