Welcome to WebmasterWorld Guest from 54.145.208.64

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
need help blocking a site
not sure if this is possible
11 willeffects 5:23 am Mar 15, 2005
PriceGrabber UserAgent
1 driwashsolutions 1:59 am Mar 15, 2005
crawl confusion
Understanding my hits log
5 cardpro 8:59 am Mar 14, 2005
i see the bigger site have the bots not crawling them
i see the bigger site have the bots not crawling them
2 indodpendo 7:12 pm Mar 13, 2005
im using this sites robits.txt file but my main page gets psa's
im using this sites robits.txt file but my main page gets psa's
2 indodpendo 7:10 pm Mar 13, 2005
PHP site
Disallowing pages
3 dj_lysuc 12:30 pm Mar 13, 2005
is it necessary for dynamic pages
aspx pages
3 qais77 2:38 pm Mar 12, 2005
Only Allowing Googlebot, Msnbot and Slurp
is it possible?
4 globay 4:28 pm Mar 11, 2005
disallow all from subdirectory
11 Driven 1:26 pm Mar 10, 2005
Robots.txt will it be ignored
1 azzbacwordz 6:39 pm Mar 9, 2005
robots.txt
I need a robots.txt
4 indodpendo 1:35 am Mar 9, 2005
simple question concerning adsense
1 esllou 10:26 am Mar 7, 2005
Need help with robot.txt for .php forums
robot.txt, .php, forums,
2 Freakindj 2:44 pm Mar 5, 2005
Bot blocking versus IP throttling for bad bots
Dealing with rippers and their ilk
1 Webwork 4:10 am Mar 3, 2005
Does this cancle out the robots.txt
Does this cancle out the robots.txt
3 indodpendo 2:16 pm Mar 1, 2005
Multiple Directories
robot.txt, subdirectories, protected directories
1 Ansuz 1:47 pm Feb 28, 2005
Robot terms defined
The differences and similarities between nofollow, noindex and robots.txt
1 u_to_the_sixth 7:29 am Feb 26, 2005
Maximum size for robots.txt
Does any one know what it is?
6 blend27 9:40 pm Feb 25, 2005
robots.txt
robots.txt
4 indodpendo 8:59 pm Feb 25, 2005
How can I get my whole site out of search engines and Google cache?
2 mig1234 5:19 pm Feb 23, 2005
Banning Affiliate Links from being Indexed
1 bears5122 11:57 pm Feb 22, 2005
Advanced robots.txt question
About blocking subdomain on different server.
4 Buddha 12:57 am Feb 21, 2005
robots.txt wildcard matching
How wild is it?
8 kwasher 12:51 am Feb 21, 2005
Creating A Robots.Txt File
How Do I Create A Correct robots.txt File?
3 Jackal 4:27 pm Feb 14, 2005
ban all except index.htm
2 rebelo 3:08 pm Feb 12, 2005