Welcome to WebmasterWorld Guest from 126.96.36.199 , register , free tools , login , search , subscribe , help , library , announcements , recent posts , open posts Subscribe to WebmasterWorld
empty robots.txt yes or no HuhuFruFru msg:1527577 7:51 am on Mar 5, 2003 (gmt 0) one more question about this:
i found a website, where someone says this:
"If you don't have any restricted area on your site (like me) you don't need a robots.txt file. Don't even put an empty robot.txt file as some robots interpret a blank robots.txt file as meaning "don't crawl any pages on this website."
he is wrong, right? because i have an empty robots.txt on my site and i want all sites to get spidered.
HuhuFruFru msg:1527578 7:54 am on Mar 5, 2003 (gmt 0)
uhuuuu, i found a lot of more sites where they say this!
(search for "blank robots.txt")
Macguru msg:1527579 8:27 am on Mar 5, 2003 (gmt 0)
I dont know about the blank file. But if you want to use the file and tell all search engines that your site is wide open, I suggest that you copy paste this in it.
Here is a great ressource. [
...] searchengineworld.com [ edited by: Macguru at 8:30 am (utc) on Mar. 5, 2003] HuhuFruFru msg:1527580 8:30 am on Mar 5, 2003 (gmt 0)
thank you macguru. i thought the same, but then i found a lot of sites about the robots.txt-file where they say that this code
could be misinterpreted by some robots in the way that they stop crawling your whole site.
Macguru msg:1527581 8:32 am on Mar 5, 2003 (gmt 0)
>> they say
I suggest you stop waisting your time over there. ;)
HuhuFruFru msg:1527582 8:32 am on Mar 5, 2003 (gmt 0)