- - Search Engines
- -- Search Engine Spider and User Agent Identification
- ---- Snapbot
youfoundjake - 8:06 am on Jun 3, 2006 (gmt 0)
thanks for the link, yet another thread i feel at home on.
|If you have a 'pest' UA or IP address that repeatedly fetches robots.txt, then you can feed it a smaller 'cloaked version' containing only |
to minimize wasted bandwidth.
how do you mean cloaked version?
should i just do a wildcard in the robots.txt file for 5 minutes or actually specify the bot?
Thread source: http://www.webmasterworld.com/search_engine_spiders/3268.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com