Forum Moderators: open

Message Too Old, No Replies

Links Directory not crawled by GoogleBot

New GoogleBot behaviour

         

nileshkurhade

5:24 pm on Dec 31, 2003 (gmt 0)

10+ Year Member



HAPPY NEW YEAR - 2004.

My site is regularly crawled by GoogleBot. But recently I have noticed - it happened yesterday also - that GoogleBot is not crawling the Links Directory maintained on my site. Has anyone else also noticed this. Has Google found out a way to filter out links directory.

ThomasB

6:41 am on Jan 2, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



What is a "links directory"? Do you mean a sitemap? Please describe it first.

steveb

7:23 am on Jan 2, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



"Has Google found out a way to filter out links directory."

I certainly hope so... if its the type of "Links Directory" I'm thinking of.

seofreak

10:43 am on Jan 2, 2004 (gmt 0)

10+ Year Member



"Has Google found out a way to filter out links directory."

I certainly hope so... if its the type of "Links Directory" I'm thinking of.

Yes, i have seen nileshes site .. and u r right about thinking that. I too hope so thats the trend google will pick ..

steveb

11:29 am on Jan 2, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Several hours ago one of these sites that employs one of those bileous link directories moved ahead of me for my number one keyword, so I looked around a little. I saw many (but not most) of its pages in the database showing as just the URL, meaning googlebot saw the link but didn't crawl it and/or crawled it previously but not now.

However, most of the pages showing for site:domain.com -qwertyw command do have full titles and caches. The significance of this is hard to tell though since the content of the domain is under ten pages... while the bileous link directory section amounts to over 23,000 pages! Only about 1000 show, so perhaps the other 20,000+ would all just show up as those URL-only links.

Someday Google will take out this trash, but as of today it still LOVES these link directory based sites.

dirkz

11:44 am on Jan 2, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



> Has Google found out a way to filter out links directory

How did you name your directory? Is it hosted on your workspace?

wanna_learn

12:42 pm on Jan 2, 2004 (gmt 0)

10+ Year Member



Nilesh,
more details plz..
name of Dirtory page? is it a Dynamically generated page/pages?

does the Page contains link only or have quite an amount of text?

seofreak

4:08 pm on Jan 2, 2004 (gmt 0)

10+ Year Member



Several hours ago one of these sites that employs one of those bileous link directories moved ahead of me for my number one keyword...

I have noticed it too .. be it florida, new york or texas update .. they just keep rising. In my case .. a particular site uses the directory technique and it has about 4 to 5 domains (not with duplicate content but you know its him) .. they all have reciprocal links with each other with all kinds of anchor text links and all of them rank at top for my industry related *keywords*. ATM, this technique is at boom.

corpuscle

4:44 pm on Jan 2, 2004 (gmt 0)

10+ Year Member



I really can't abide those "directories". I keep finding terms which I *coined* appearing in "directories" that rank higher than me due to their pagerank.

Since all the content seems to be sourced from the same place, you'd think someone at google would be able to figure out how to apply some sort of bayesian analysis to identify pages using this obvious data source.

Also, aren't these "directory" owners opening themselves up to massive litigation threats from copyright infringement? If so, why are they still so prevalent after 3 months? You'd have thought they'd have been forced out of business by now.

Incidentally, I assume no-one is going into more detail about this to try to avoid encouraging others to jump on the bandwagon. Please let's keep it this way!

seofreak

4:49 pm on Jan 2, 2004 (gmt 0)

10+ Year Member



I think the opposite, the more it is noticed by google .. the more is the likeliness of google getting rid of such sites :)

pageoneresults

4:53 pm on Jan 2, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



These discussions pop up regularly at WebmasterWorld. There have been many really good topics on this in the past. From those various topics I was able to summarize this...

Avoid naming your links pages links.htm or whatever underlying technology you are using (file extensions). Avoid naming your sub-directory /links/. I wouldn't be surprised if the term links is now classified as a poison word.

/bookmarks/
/favorites/
/friends/
/partners/
/resources/

You'll need to be a little more creative in developing these types of directories. Unfortunately many of them have a similar footprint and can be filtered out easily. Its now a matter of getting around the filters. The problem is, it may be a short term strategy because at some point, the filters catch up.

P.S. From my perspective, there are more than enough link directories on the net. ;)

dirkz

7:49 pm on Jan 2, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



> Please let's keep it this way!

Okay :-)