Welcome to WebmasterWorld Guest from 107.20.34.173

Message Too Old, No Replies

Strange accesslog finding

     

epmaniac

7:01 am on Jan 26, 2011 (gmt 0)

5+ Year Member



hi

i noticed a very strange behaviour with my accesslog today.

99% of the pages which are crawled by all bots (google,yahoo,bing) are all pages which have words in link with 3 letters or less

for example:

http://www.example.com/ men-gifts/
http://www.example.com/ ipod-mp4/
www.example.com/ mlb-jerseys/
http://www.example.com/ -mlb-logo/
http://www.example.com/ -nike-af/


is it a natural behaviour or could it be due to some error? please guide me, i think something is terribly wrong

epmaniac

9:37 am on Jan 26, 2011 (gmt 0)

5+ Year Member



one more thing

for words with length 3 0r less than 3, i search with LIKE in database,... for greater than 3 i use full text search.... i dont know how this could be affecting the crawling of bots, but maybe it is

cls_wired

9:59 am on Jan 26, 2011 (gmt 0)

5+ Year Member



There is little probability, that all bots can start to crawl 3-letters words in one day. Problem is on the site's engine side.

epmaniac

10:36 am on Jan 26, 2011 (gmt 0)

5+ Year Member



this problem i have seen in all my previous accesslogs too, my site went from being one of top 40000 site to being obscure

deadsea

2:11 pm on Jan 26, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Googlebot usually recrawls pages with a frequency proportional to the pagerank of the page. However, I have seen googlbot do a much deeper crawl. In this mode it visits urls that it may not have visited in years, or may not have visited at all in the past. It kind of looks like they found a whole box of urls in the basement and open it up like a kid a christmas. Interestingly, googlebot seems to crawl urls by url length, shortest first, in this mode. I haven't seen it target urls with short words so much as just short urls first.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month