homepage Welcome to WebmasterWorld Guest from 54.198.139.141
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Accredited PayPal World Seller

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Why is googlebot asking for a directory that doesn't exist anymore?
johnlim9988




msg:3106429
 7:00 am on Oct 3, 2006 (gmt 0)

Hi,

why googlebot always spider currently no exist directory?

Beofore we have a links directory then we delete that directory about seven months ago. But googlebot always spider that directory until now.

Such behavior of googlebot has any side effect? Is this the reason our site pages SERP always go down for many kws? Should I ban the googlebot to spider that no exist directoy at robots.txt?

 

johnlim9988




msg:3106693
 12:34 pm on Oct 3, 2006 (gmt 0)

anybody have any idea for this?

dramstore




msg:3106701
 12:40 pm on Oct 3, 2006 (gmt 0)

Usually happens to me when I have left some link on a page buried somewhere.

johnlim9988




msg:3106765
 1:38 pm on Oct 3, 2006 (gmt 0)

before I have a links directory then I delete that links directory over seven months already, but robots still want to "eat" that directory, so have a lot of 404 error and I don't know how to do now. I don't know if it will affect the site ranking.

hutcheson




msg:3106823
 2:17 pm on Oct 3, 2006 (gmt 0)

Assuming dramstore is right (which I would!), you'd want to start by looking for pages that still have links to that vanished directory.

Then you'd want to remove any links you found.

RoyalBiscuit




msg:3106877
 2:52 pm on Oct 3, 2006 (gmt 0)

I have the same problem.

There are hundreds of pages listed in google that haven't existed for at least a month. They are not listed in my sitemap.txt and I am 99.999% certain that there are no links to them from any other pages on my site.

I am trying to recover my hits, which have recently plummeted to almost zero after about 10 good years.

g1smd




msg:3107557
 11:48 pm on Oct 3, 2006 (gmt 0)

Google will spider every URL that they have ever seen to see if the status of that URL ever changes again.

This is not a problem.

A problem occurs if they start showing and ranking that page in the normal index again.

They do often show 404 pages in the Supplemental index for a year after the content is removed.

Continue to serve a 404 for that URL.

johnlim9988




msg:3107571
 11:58 pm on Oct 3, 2006 (gmt 0)

The problem is the robot query too often for those removed directory and pages, sometimes several hundred times per day. Looks like they want those pages first before spider other pages.

g1smd




msg:3107577
 12:02 am on Oct 4, 2006 (gmt 0)

Make sure that they really do get served a 404 status code for each one.

dcheney




msg:3107579
 12:04 am on Oct 4, 2006 (gmt 0)

You could always add the non-existant directory to your robots.txt and block it from being scanned.

johnlim9988




msg:3107591
 12:18 am on Oct 4, 2006 (gmt 0)

Yes, every time got 404 for the delted directory and files.

Now I have three ways to deal with the issues,

1) find the links to those deleted directory and files and ask webmasters to remove those links, this costs much time to do so.

2) Block the deleted directory and files at robots.txt, but have any side effects to do so? the robots maybe will think why you don't allow me to spider those pages while links to those directory exist.

3)don't care about those to let the robots got 404. But have any side effect to do so? Will it affect the whole site ranking?

Pls kindly advise which way is the best way.

tedster




msg:3107610
 12:37 am on Oct 4, 2006 (gmt 0)

Just let all requests for urls in that directory be 404 and let it alone. No side effects that I can think of, just some lines in your server error log.

But here's another thought: if there are a lot of links to pages in that directory, then maybe you want to place some different content on those urls instead of just letting them all be 404. Last year I took over a site that (years before) had removed heavily backlinked url. As soon as I noticed this, I put new content at that url and it showed up at #9 for a one-word keyword search after just a couple days.

walkman




msg:3107704
 2:24 am on Oct 4, 2006 (gmt 0)

it could be that someone still links to that page

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved