Welcome to WebmasterWorld Guest from 54.144.21.195

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Denying access to everything but the home page?
Using robots.txt to only allow robot access to the home page
3 monkeylytics 8:00 pm Jan 28, 2008
Control Robots with Meta or robots.txt
3 driller41 12:21 pm Jan 15, 2008
Robots.txt disallowed.
12 g1smd 12:18 am Jan 15, 2008
robots.txt with wildcard?
3 tranceformer 12:39 am Jan 13, 2008
Disallowing with robots.txt
robots.txt, disallow certain urls
3 jim_knopf 12:36 am Jan 13, 2008
Getting correct user agent
getting from access log the correct user agent for robots.txt
2 hessodreamy 9:03 pm Jan 11, 2008
Scrapers checking robots.txt
Spiders arrived on heels of robots.txt deployment
4 timster 1:29 am Jan 4, 2008
404 Errors in Webmaster Console
What to do with these pages?
2 rowtc2 11:35 pm Jan 3, 2008
directory and subdirectory access
allowing access to subdirectory but not main directory
3 Mr_Cool_Dude 6:11 pm Jan 2, 2008
robots.txt question
robots.txt examples
2 jim_knopf 5:43 pm Jan 2, 2008
Reference robots.txt file?
Does anyone have one?
3 mmiller 11:30 pm Dec 25, 2007
how to manage cname in robot
don't was to be indexed
4 experienced 2:10 pm Dec 21, 2007
disallow vs ban in .htaccess
12 joergnw10 11:10 am Dec 16, 2007
How do I prevent the SEs from crawling any links to my site w/HTTPS
I don't want the crawlers to index anything w/ HTTPS protocol
3 ZydoSEO 8:37 pm Dec 9, 2007
yahoo google msn etc.
other searches
1 ComputerHermit 5:56 pm Dec 7, 2007
Proposal for robots.txt To Have Greater Flexibility
12 engine 4:58 pm Nov 30, 2007
set crawl rate permanently to faster
1 agushalim 5:39 pm Nov 28, 2007
Wildcards on robots.txt disallows
11 MindTwist 5:59 pm Nov 27, 2007
Site title and description missing in search results
I had made some changes in robots.txt file
2 ecmedia 2:12 pm Nov 27, 2007
perfect robot text files
good robot text files
2 likwidholdings 12:57 am Nov 27, 2007
Google showing site despite blocking in robots.txt
The page is not cached though
3 ecmedia 4:28 pm Nov 20, 2007
Disallow all except adsense
6 thegreatpretender 4:40 pm Nov 19, 2007
How to avoid double indexing
3 mavi 4:43 pm Nov 13, 2007
Robots.txt Clarification
5 smartcard 10:44 am Nov 13, 2007
Robot from colo is on behalf of Google?!
Colo ISP claims robot crawling my site is from 3rd party Google contractor
5 yodokame 4:03 am Nov 12, 2007