- - Search Engines
- -- Search Engine Spider and User Agent Identification
- ---- Faxobot/1.0
Lord_Majestic - 10:38 am on Oct 14, 2004 (gmt 0)
|It was here today and requested robots.txt after sucking "/". |
Was the time difference between two requests very small (1 second or so), or not?
Thread source: http://www.webmasterworld.com/search_engine_spiders/2527.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com