homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

Strange accesslog finding

 7:01 am on Jan 26, 2011 (gmt 0)


i noticed a very strange behaviour with my accesslog today.

99% of the pages which are crawled by all bots (google,yahoo,bing) are all pages which have words in link with 3 letters or less

for example:

http://www.example.com/ men-gifts/
http://www.example.com/ ipod-mp4/
www.example.com/ mlb-jerseys/
http://www.example.com/ -mlb-logo/
http://www.example.com/ -nike-af/

is it a natural behaviour or could it be due to some error? please guide me, i think something is terribly wrong



 9:37 am on Jan 26, 2011 (gmt 0)

one more thing

for words with length 3 0r less than 3, i search with LIKE in database,... for greater than 3 i use full text search.... i dont know how this could be affecting the crawling of bots, but maybe it is


 9:59 am on Jan 26, 2011 (gmt 0)

There is little probability, that all bots can start to crawl 3-letters words in one day. Problem is on the site's engine side.


 10:36 am on Jan 26, 2011 (gmt 0)

this problem i have seen in all my previous accesslogs too, my site went from being one of top 40000 site to being obscure


 2:11 pm on Jan 26, 2011 (gmt 0)

Googlebot usually recrawls pages with a frequency proportional to the pagerank of the page. However, I have seen googlbot do a much deeper crawl. In this mode it visits urls that it may not have visited in years, or may not have visited at all in the past. It kind of looks like they found a whole box of urls in the basement and open it up like a kid a christmas. Interestingly, googlebot seems to crawl urls by url length, shortest first, in this mode. I haven't seen it target urls with short words so much as just short urls first.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved