Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Scooter


wilderness - 8:43 pm on Jul 6, 2003 (gmt 0)


Your robots.txt seems to be quite long at 2390 bytes, might a syntax error have crept in somewhere? There are several robots.txt syntax checkers out there.


SearchEngineWorld Robots.txt Validator

113 Field names of robots.txt maybe case insensitive, but do capitalize field names to account for challenged robots.
user-agent: szukacz
114 warning Field names of robots.txt maybe case insensitive, but do capitalize field names to account for challenged robots.
disallow: /
131 warning Field names of robots.txt maybe case insensitive, but do capitalize field names to account for challenged robots. (eg: User-agent)
User-Agent: Whizbang

reagrding lines 113 & 114?
szukacz honors the request to disallow and yet simple syntax error should make the entire file invalid and allow Scooter to override?

(edited by wilderness 07/06/03 17:00 EST)
I might add that this image Scooter grabbed doesn't even show on the page it linked from. Rather the page has a thumbnail which links to this larger image.

[edited by: wilderness at 8:56 pm (utc) on July 6, 2003]


Thread source:: http://www.webmasterworld.com/search_engine_spiders/2068.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com