Welcome to WebmasterWorld Guest from

Forum Moderators: mademetop

Message Too Old, No Replies

Brand new website, popular topic

8:42 am on Aug 29, 2005 (gmt 0)

New User

5+ Year Member

joined:Mar 28, 2009
votes: 0

My brank new website, about jokes, funny pictures, games, etc. got online 2 days ago. I checked the log file, find google, yahoo, and msn bot came. google and yahoo bots crawled thousands pages, but why MSN bot only crawled robots.txt, index.html and rss.xml?
6:33 pm on Aug 29, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 31, 2004
votes: 0

There's no telling why bots do the things they do sometimes. It's not like they have a mind like ours (For now anyway), that can learn as it goes. Sometimes they will spider 100s of web pages and other days only a few. Give it some time and it will get around to your other web pages. Just be happy it knows your site is there.
3:47 am on Aug 30, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:July 3, 2005
votes: 0

Once google just wouldnt leave site alone 50k pages in a single day that means it was spidering multiple times, it was madness msn has done a similar thing but not on the same level, you will see very soon SE like G and msn are very unpredictible