homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

To use or not to use a robots.txt file

 12:11 pm on May 20, 2008 (gmt 0)

I have never used a robots.txt file. My site is 5 years old and I've have had no problems. However in the last month or so I am seeing some strange things. Yahoo is hitting my site 5X as much as google and I'm wondering if it would be any benefit for me to add a robots.txt?

Also In my errors log it shows google is unable to crawl some pages do to a robots.txt file of which I do not have one....

Any ideas



 6:01 pm on May 20, 2008 (gmt 0)

well using robots.txt file is sometimes better and sometimes it reflects bad impact if we can see a pure crawling. the basic purpose of robots.txt file is blocking site's restricted areas otherthan no benefit you can get from it. You must re-study your pages for which google felt busy while crawling, might there is some script you are using that reflects out nature in respect of crawling. One more thing search engines doesn't cares if you are using robots.txt file, they jump over it and pass through without knowing what you are seeking for.
Now a days google agent with Mozila 5.0 with a dispute, that most of the people saying that it revels in respect of spamming.




 6:03 pm on May 20, 2008 (gmt 0)

I'm still looking Goodroi comments....

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved