Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: goodroi
Thanks, so do I erase from <html>to </html> completely? Then what do I put there, if I want the entire site to be spidered now.
So the robots.txt has been mentioned as the possible problem. I recently changed from a shared ip to a dedicated ip hoping that may be the problem. Thanks
The robots.txt is for exclusion. If you don't have one, thats ok. No spiders will then be excluded.
That said you should have one because it's shows you pay attention to details. The only thing in your file should be is the following
Have you validated your website html? That might show up problems? The banned question is a hard one. Lots of factors involved.
I tried to do that initially but it kept saving it in robots.txt.htm
If this doesnt work simply make a text file in notepad and call it robots.txt then copy paste it into your mywebs/sitename folder on your computer. It will then be visible as robots.txt within frontpage.