Forum Moderators: open

Message Too Old, No Replies

robots.txt the reason for poor indexing?

         

DXL

12:20 pm on Aug 23, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I just redesigned a site and noticed that it wasn't indexed in Google properly. I tried to look at an archived version of the site through the wayback machine and noticed that its robots were blocked. I found a robots.txt file in the root of the redesigned site and it read:

User-agent: *
Disallow: /

Does this mean that the site has blocked every attempt to spider it in the last few years, and that I have to resubmit it everywhere? I deleted the robots.txt file because I wasn't sure how I would change it to allow spidering from major search engines.

pageoneresults

1:01 pm on Aug 23, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Does this mean that the site has blocked every attempt to spider it in the last few years, and that I have to resubmit it everywhere?

Yes, the site has been Disallowed from indexing.

No need to resubmit anywhere. Just remove the robots.txt file or remove the / from the Disallow line and you'll be fine. Although you may want to look at Disallowing other parts of the site.

If there are links pointing to that site, which there probably are after a couple of years, you'll start to see the various bots traversing the site over the next few weeks (typically).

robots.txt Forum [webmasterworld.com]

DXL

10:40 pm on Aug 23, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks for the reply. Yes, there are plenty of backlinks, so hopefully the site starts getting crawled again this week.