Welcome to WebmasterWorld Guest from 54.157.222.62

Forum Moderators: goodroi

Message Too Old, No Replies

empty robots.txt

yes or no

   
7:51 am on Mar 5, 2003 (gmt 0)

10+ Year Member



one more question about this:

i found a website, where someone says this:

"If you don't have any restricted area on your site (like me) you don't need a robots.txt file. Don't even put an empty robot.txt file as some robots interpret a blank robots.txt file as meaning "don't crawl any pages on this website."

he is wrong, right? because i have an empty robots.txt on my site and i want all sites to get spidered.

7:54 am on Mar 5, 2003 (gmt 0)

10+ Year Member



uhuuuu, i found a lot of more sites where they say this!

(?)

(search for "blank robots.txt")

8:27 am on Mar 5, 2003 (gmt 0)

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Hi HuhuFruFru,

I dont know about the blank file. But if you want to use the file and tell all search engines that your site is wide open, I suggest that you copy paste this in it.

User-agent: *
Disallow:

Here is a great ressource. [searchengineworld.com...]

[edited by: Macguru at 8:30 am (utc) on Mar. 5, 2003]

8:30 am on Mar 5, 2003 (gmt 0)

10+ Year Member



thank you macguru. i thought the same, but then i found a lot of sites about the robots.txt-file where they say that this code

User-agent: *
Disallow:

could be misinterpreted by some robots in the way that they stop crawling your whole site.

8:32 am on Mar 5, 2003 (gmt 0)

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member



>>they say

I suggest you stop waisting your time over there. ;)

8:32 am on Mar 5, 2003 (gmt 0)

10+ Year Member



okay :)