Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- And Now Google's Doing It. JS Stats Show GoogleBot


TheMadScientist - 8:14 pm on May 14, 2011 (gmt 0)


This is the whole robots.txt file:
Feel free to point out the errors I'm missing if there are any.
I copied & pasted (examplified 2 of the names, but that's it) from the live site, so I know it's there and accessible too.


User-agent: *
Disallow: /link-out.php
Disallow: /directoryname-pics
Disallow: /img
Disallow: /accounts
Disallow: /functions.js
Disallow: /directory.js


Sitemap: http://www.example.com/sitemap-index.xml


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4312058.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com