Welcome to WebmasterWorld Guest from 54.159.44.227

Forum Moderators: goodroi

Message Too Old, No Replies

no robots.txt

     

textex

1:13 pm on Feb 6, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Will having no robots.txt effect the ability being spidered?

Never really had a problem before, just trying to figure out why pages may have been dropped.

Sanenet

1:19 pm on Feb 6, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



No. If robots.txt doesn't exist, then it's treated as if the right to spider to given.

terk

2:50 pm on Feb 7, 2005 (gmt 0)

10+ Year Member



I have robot.txt. and got dropped this weekend from yahoo..

Sanenet

2:55 pm on Feb 7, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Unless you told Yahoo/Slurp to stop spidering your site, it has nothing to do with you being dropped.

terk

4:01 pm on Feb 7, 2005 (gmt 0)

10+ Year Member



Being new at this, Iíve been adding a page a day and the only thing I change was the robot.txt , with was restore to last week version and added a commercial software to monitor my click. Well now I know that it wasínt the robot.txt
Thanks Sanenet

engine

5:50 pm on Feb 8, 2005 (gmt 0)

WebmasterWorld Administrator engine is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



For the record, the format should be robots.txt

Have a look at the robots tutorial [searchengineworld.com]

You only need a robots.txt file if you want to provide some control on which parts of your site is not spidered and by which spider. Often you won't need a robots.txt file at all.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month