Welcome to WebmasterWorld Guest from 54.226.25.231

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Getting correct user agent
getting from access log the correct user agent for robots.txt
2 hessodreamy 9:03 pm Jan 11, 2008
Scrapers checking robots.txt
Spiders arrived on heels of robots.txt deployment
4 timster 1:29 am Jan 4, 2008
404 Errors in Webmaster Console
What to do with these pages?
2 rowtc2 11:35 pm Jan 3, 2008
directory and subdirectory access
allowing access to subdirectory but not main directory
3 Mr_Cool_Dude 6:11 pm Jan 2, 2008
robots.txt question
robots.txt examples
2 jim_knopf 5:43 pm Jan 2, 2008
Reference robots.txt file?
Does anyone have one?
3 mmiller 11:30 pm Dec 25, 2007
how to manage cname in robot
don't was to be indexed
4 experienced 2:10 pm Dec 21, 2007
disallow vs ban in .htaccess
12 joergnw10 11:10 am Dec 16, 2007
How do I prevent the SEs from crawling any links to my site w/HTTPS
I don't want the crawlers to index anything w/ HTTPS protocol
3 ZydoSEO 8:37 pm Dec 9, 2007
yahoo google msn etc.
other searches
1 ComputerHermit 5:56 pm Dec 7, 2007
Proposal for robots.txt To Have Greater Flexibility
12 engine 4:58 pm Nov 30, 2007
set crawl rate permanently to faster
1 agushalim 5:39 pm Nov 28, 2007
Wildcards on robots.txt disallows
11 MindTwist 5:59 pm Nov 27, 2007
Site title and description missing in search results
I had made some changes in robots.txt file
2 ecmedia 2:12 pm Nov 27, 2007
perfect robot text files
good robot text files
2 likwidholdings 12:57 am Nov 27, 2007
Google showing site despite blocking in robots.txt
The page is not cached though
3 ecmedia 4:28 pm Nov 20, 2007
Disallow all except adsense
6 thegreatpretender 4:40 pm Nov 19, 2007
How to avoid double indexing
3 mavi 4:43 pm Nov 13, 2007
Robots.txt Clarification
5 smartcard 10:44 am Nov 13, 2007
Robot from colo is on behalf of Google?!
Colo ISP claims robot crawling my site is from 3rd party Google contractor
5 yodokame 4:03 am Nov 12, 2007
robots.txt file
how it affects the page rank of any website
2 monikagaba 6:21 pm Nov 1, 2007
Exclude ALL bots but major search engines
Trying to make the perfect robots.txt
5 serpmaster 12:03 pm Oct 25, 2007
blocking all except root (index.php)
would disallow ALL , allow index work?
1 bkode 7:56 am Oct 24, 2007
http code 200 mystery
is the page served or not!?
3 dailyclicker 5:14 pm Oct 13, 2007
Do robots get cookies?
robots cookies possible dumb question
4 Michel_Samuel 10:55 am Oct 11, 2007

Featured Threads

Hot Threads This Week

Hot Threads This Month