| Brand new website, popular topic
|
unlimited
msg:249208 | 8:42 am on Aug 29, 2005 (gmt 0) |
My brank new website, about jokes, funny pictures, games, etc. got online 2 days ago. I checked the log file, find google, yahoo, and msn bot came. google and yahoo bots crawled thousands pages, but why MSN bot only crawled robots.txt, index.html and rss.xml?
|
robotsdobetter
msg:249209 | 6:33 pm on Aug 29, 2005 (gmt 0) |
There's no telling why bots do the things they do sometimes. It's not like they have a mind like ours (For now anyway), that can learn as it goes. Sometimes they will spider 100s of web pages and other days only a few. Give it some time and it will get around to your other web pages. Just be happy it knows your site is there.
|
Gargen
msg:249210 | 3:47 am on Aug 30, 2005 (gmt 0) |
Once google just wouldnt leave site alone 50k pages in a single day that means it was spidering multiple times, it was madness msn has done a similar thing but not on the same level, you will see very soon SE like G and msn are very unpredictible
|
|
|
