homepage Welcome to WebmasterWorld Guest from 54.227.41.242
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
To use or not to use a robots.txt file
con771

5+ Year Member



 
Msg#: 3654405 posted 12:11 pm on May 20, 2008 (gmt 0)

I have never used a robots.txt file. My site is 5 years old and I've have had no problems. However in the last month or so I am seeing some strange things. Yahoo is hitting my site 5X as much as google and I'm wondering if it would be any benefit for me to add a robots.txt?

Also In my errors log it shows google is unable to crawl some pages do to a robots.txt file of which I do not have one....

Any ideas

 

bilalseo

5+ Year Member



 
Msg#: 3654405 posted 6:01 pm on May 20, 2008 (gmt 0)

well using robots.txt file is sometimes better and sometimes it reflects bad impact if we can see a pure crawling. the basic purpose of robots.txt file is blocking site's restricted areas otherthan no benefit you can get from it. You must re-study your pages for which google felt busy while crawling, might there is some script you are using that reflects out nature in respect of crawling. One more thing search engines doesn't cares if you are using robots.txt file, they jump over it and pass through without knowing what you are seeking for.
Now a days google agent with Mozila 5.0 with a dispute, that most of the people saying that it revels in respect of spamming.

thanks,

bilal

bilalseo

5+ Year Member



 
Msg#: 3654405 posted 6:03 pm on May 20, 2008 (gmt 0)

I'm still looking Goodroi comments....

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved