Welcome to WebmasterWorld Guest from 22.214.171.124 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Pubcon Platinum Sponsor 2014
empty robots.txt yes or no HuhuFruFru
one more question about this:
i found a website, where someone says this:
"If you don't have any restricted area on your site (like me) you don't need a robots.txt file. Don't even put an empty robot.txt file as some robots interpret a blank robots.txt file as meaning "don't crawl any pages on this website."
he is wrong, right? because i have an empty robots.txt on my site and i want all sites to get spidered.
uhuuuu, i found a lot of more sites where they say this!
(search for "blank robots.txt")
I dont know about the blank file. But if you want to use the file and tell all search engines that your site is wide open, I suggest that you copy paste this in it.
Here is a great ressource. [
...] searchengineworld.com [ edited by: Macguru at 8:30 am (utc) on Mar. 5, 2003] HuhuFruFru
thank you macguru. i thought the same, but then i found a lot of sites about the robots.txt-file where they say that this code
could be misinterpreted by some robots in the way that they stop crawling your whole site.
>> they say
I suggest you stop waisting your time over there. ;)