Welcome to WebmasterWorld Guest from 126.96.36.199 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Pubcon Platinum Sponsor 2014
no robots.txt textex msg:1528132 1:13 pm on Feb 6, 2005 (gmt 0) Will having no robots.txt effect the ability being spidered?
Never really had a problem before, just trying to figure out why pages may have been dropped.
Sanenet msg:1528133 1:19 pm on Feb 6, 2005 (gmt 0)
No. If robots.txt doesn't exist, then it's treated as if the right to spider to given. terk msg:1528134 2:50 pm on Feb 7, 2005 (gmt 0)
I have robot.txt. and got dropped this weekend from yahoo.. Sanenet msg:1528135 2:55 pm on Feb 7, 2005 (gmt 0)
Unless you told Yahoo/Slurp to stop spidering your site, it has nothing to do with you being dropped. terk msg:1528136 4:01 pm on Feb 7, 2005 (gmt 0)
Being new at this, Iíve been adding a page a day and the only thing I change was the robot.txt , with was restore to last week version and added a commercial software to monitor my click. Well now I know that it wasínt the robot.txt Thanks Sanenet engine msg:1528137 5:50 pm on Feb 8, 2005 (gmt 0)
For the record, the format should be robots.txt
Have a look at the
robots tutorial [ searchengineworld.com]
You only need a robots.txt file if you want to provide some control on which parts of your site is not spidered and by which spider. Often you won't need a robots.txt file at all.