homepage Welcome to WebmasterWorld Guest from 54.204.90.135
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
no robots.txt
textex




msg:1528132
 1:13 pm on Feb 6, 2005 (gmt 0)

Will having no robots.txt effect the ability being spidered?

Never really had a problem before, just trying to figure out why pages may have been dropped.

 

Sanenet




msg:1528133
 1:19 pm on Feb 6, 2005 (gmt 0)

No. If robots.txt doesn't exist, then it's treated as if the right to spider to given.

terk




msg:1528134
 2:50 pm on Feb 7, 2005 (gmt 0)

I have robot.txt. and got dropped this weekend from yahoo..

Sanenet




msg:1528135
 2:55 pm on Feb 7, 2005 (gmt 0)

Unless you told Yahoo/Slurp to stop spidering your site, it has nothing to do with you being dropped.

terk




msg:1528136
 4:01 pm on Feb 7, 2005 (gmt 0)

Being new at this, Iíve been adding a page a day and the only thing I change was the robot.txt , with was restore to last week version and added a commercial software to monitor my click. Well now I know that it wasínt the robot.txt
Thanks Sanenet

engine




msg:1528137
 5:50 pm on Feb 8, 2005 (gmt 0)

For the record, the format should be robots.txt

Have a look at the robots tutorial [searchengineworld.com]

You only need a robots.txt file if you want to provide some control on which parts of your site is not spidered and by which spider. Often you won't need a robots.txt file at all.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved