Welcome to WebmasterWorld Guest from 188.8.131.52 , register , free tools , login , search , subscribe , help , library , announcements , recent posts , open posts Subscribe to WebmasterWorld
Strange accesslog finding epmaniac msg:4258225 7:01 am on Jan 26, 2011 (gmt 0) hi i noticed a very strange behaviour with my accesslog today. 99% of the pages which are crawled by all bots (google,yahoo,bing) are all pages which have words in link with 3 letters or less for example: http://www.example.com/ men-gifts/ http://www.example.com/ ipod-mp4/ www.example.com/ mlb-jerseys/ http://www.example.com/ -mlb-logo/ http://www.example.com/ -nike-af/ is it a natural behaviour or could it be due to some error? please guide me, i think something is terribly wrong
epmaniac msg:4258258 9:37 am on Jan 26, 2011 (gmt 0)
one more thing for words with length 3 0r less than 3, i search with LIKE in database,... for greater than 3 i use full text search.... i dont know how this could be affecting the crawling of bots, but maybe it is cls_wired msg:4258264 9:59 am on Jan 26, 2011 (gmt 0)
There is little probability, that all bots can start to crawl 3-letters words in one day. Problem is on the site's engine side. epmaniac msg:4258276 10:36 am on Jan 26, 2011 (gmt 0)
this problem i have seen in all my previous accesslogs too, my site went from being one of top 40000 site to being obscure deadsea msg:4258334 2:11 pm on Jan 26, 2011 (gmt 0)
Googlebot usually recrawls pages with a frequency proportional to the pagerank of the page. However, I have seen googlbot do a much deeper crawl. In this mode it visits urls that it may not have visited in years, or may not have visited at all in the past. It kind of looks like they found a whole box of urls in the basement and open it up like a kid a christmas. Interestingly, googlebot seems to crawl urls by url length, shortest first, in this mode. I haven't seen it target urls with short words so much as just short urls first.