homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Robots.txt Error

10+ Year Member

Msg#: 291 posted 2:37 pm on Feb 17, 2004 (gmt 0)

Some time during the last few weeks I made the unbelievably stupid mistake of uploading the wrong robots.txt file (dissallowing everything!) to my website. I didn't realise this until about 2 weeks ago when I noticed that my traffic had fallen to about 1/4 of what it was from one day to the next.

My web host has also recently changed the way log files are handled, which means I can't fully access my raw log files until the end of the month so at the moment I can't tell exactly what's going on!

What I'm wonderng is how do the spiders behave in this situation? How long is it likely to take for things to return to normal?

So far I have just carried on as normal in the hope that the spiders will eventually come back - is there anything else I should be doing?

I've noticed on Google that if I search allinurl for my site all my pages are listed. The listing for my home page however just lists the URL, without any title or description - what does that mean? I've also noticed that for the few pages that are listed with a cache link the cached page is at least a month old (probably more, it's hard to say). Most pages no longer have a cache at all.

I'd be very grateful for any advice or insight anyone has to offer.




WebmasterWorld Senior Member 10+ Year Member

Msg#: 291 posted 5:24 pm on Feb 18, 2004 (gmt 0)

Welcome to WebmasterWorld - you really need to examine the logs to determine exactly what is happening w/spiders on your site - talk to your host, raw access once a month is ridiculous.

In the meantime, keep working on new content and link development with other sites.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved