Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Getting ERROR_DISALLOW with Inktomi after adding new robots.txt


bebox - 8:44 am on Dec 28, 2002 (gmt 0)


Thanks guys...yikes! :))
I got the robots.txt from Search Engine World (example 4) and thought that it would do as it said. I have now fixed the wrapped lines and tidied the file up and it validates just fine. With regards to the last line that doesn't allow anything, won't that only be read once the others above it have been read so it will allow any of the listed bots but nothign else?

If it isn't right perhaps I need to alert the people at Search Engine World too.

Thanks for your help and will try to fix now.

B


Thread source:: http://www.webmasterworld.com/robots_txt/50.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com