homepage Welcome to WebmasterWorld Guest from 54.167.138.53
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
empty robots.txt
yes or no
HuhuFruFru




msg:1527577
 7:51 am on Mar 5, 2003 (gmt 0)

one more question about this:

i found a website, where someone says this:

"If you don't have any restricted area on your site (like me) you don't need a robots.txt file. Don't even put an empty robot.txt file as some robots interpret a blank robots.txt file as meaning "don't crawl any pages on this website."

he is wrong, right? because i have an empty robots.txt on my site and i want all sites to get spidered.

 

HuhuFruFru




msg:1527578
 7:54 am on Mar 5, 2003 (gmt 0)

uhuuuu, i found a lot of more sites where they say this!

(?)

(search for "blank robots.txt")

Macguru




msg:1527579
 8:27 am on Mar 5, 2003 (gmt 0)

Hi HuhuFruFru,

I dont know about the blank file. But if you want to use the file and tell all search engines that your site is wide open, I suggest that you copy paste this in it.

User-agent: *
Disallow:

Here is a great ressource. [searchengineworld.com...]

[edited by: Macguru at 8:30 am (utc) on Mar. 5, 2003]

HuhuFruFru




msg:1527580
 8:30 am on Mar 5, 2003 (gmt 0)

thank you macguru. i thought the same, but then i found a lot of sites about the robots.txt-file where they say that this code

User-agent: *
Disallow:

could be misinterpreted by some robots in the way that they stop crawling your whole site.

Macguru




msg:1527581
 8:32 am on Mar 5, 2003 (gmt 0)

>>they say

I suggest you stop waisting your time over there. ;)

HuhuFruFru




msg:1527582
 8:32 am on Mar 5, 2003 (gmt 0)

okay :)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved