homepage Welcome to WebmasterWorld Guest from 54.204.67.26
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Accredited PayPal World Seller

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt Error
Takshaka




msg:1529232
 2:37 pm on Feb 17, 2004 (gmt 0)

Some time during the last few weeks I made the unbelievably stupid mistake of uploading the wrong robots.txt file (dissallowing everything!) to my website. I didn't realise this until about 2 weeks ago when I noticed that my traffic had fallen to about 1/4 of what it was from one day to the next.

My web host has also recently changed the way log files are handled, which means I can't fully access my raw log files until the end of the month so at the moment I can't tell exactly what's going on!

What I'm wonderng is how do the spiders behave in this situation? How long is it likely to take for things to return to normal?

So far I have just carried on as normal in the hope that the spiders will eventually come back - is there anything else I should be doing?

I've noticed on Google that if I search allinurl for my site all my pages are listed. The listing for my home page however just lists the URL, without any title or description - what does that mean? I've also noticed that for the few pages that are listed with a cache link the cached page is at least a month old (probably more, it's hard to say). Most pages no longer have a cache at all.

I'd be very grateful for any advice or insight anyone has to offer.

Thanks

 

bcolflesh




msg:1529233
 5:24 pm on Feb 18, 2004 (gmt 0)

Welcome to WebmasterWorld - you really need to examine the logs to determine exactly what is happening w/spiders on your site - talk to your host, raw access once a month is ridiculous.

In the meantime, keep working on new content and link development with other sites.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved