Welcome to WebmasterWorld Guest from

Message Too Old, No Replies

Strange accesslog finding



7:01 am on Jan 26, 2011 (gmt 0)

5+ Year Member


i noticed a very strange behaviour with my accesslog today.

99% of the pages which are crawled by all bots (google,yahoo,bing) are all pages which have words in link with 3 letters or less

for example:

http://www.example.com/ men-gifts/
http://www.example.com/ ipod-mp4/
www.example.com/ mlb-jerseys/
http://www.example.com/ -mlb-logo/
http://www.example.com/ -nike-af/

is it a natural behaviour or could it be due to some error? please guide me, i think something is terribly wrong


9:37 am on Jan 26, 2011 (gmt 0)

5+ Year Member

one more thing

for words with length 3 0r less than 3, i search with LIKE in database,... for greater than 3 i use full text search.... i dont know how this could be affecting the crawling of bots, but maybe it is


9:59 am on Jan 26, 2011 (gmt 0)

5+ Year Member

There is little probability, that all bots can start to crawl 3-letters words in one day. Problem is on the site's engine side.


10:36 am on Jan 26, 2011 (gmt 0)

5+ Year Member

this problem i have seen in all my previous accesslogs too, my site went from being one of top 40000 site to being obscure


2:11 pm on Jan 26, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Googlebot usually recrawls pages with a frequency proportional to the pagerank of the page. However, I have seen googlbot do a much deeper crawl. In this mode it visits urls that it may not have visited in years, or may not have visited at all in the past. It kind of looks like they found a whole box of urls in the basement and open it up like a kid a christmas. Interestingly, googlebot seems to crawl urls by url length, shortest first, in this mode. I haven't seen it target urls with short words so much as just short urls first.

Featured Threads

Hot Threads This Week

Hot Threads This Month