Welcome to WebmasterWorld Guest from 54.159.255.164

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Disallowing with robots.txt
robots.txt, disallow certain urls
3 jim_knopf 12:36 am Jan 13, 2008
Getting correct user agent
getting from access log the correct user agent for robots.txt
2 hessodreamy 9:03 pm Jan 11, 2008
Scrapers checking robots.txt
Spiders arrived on heels of robots.txt deployment
4 timster 1:29 am Jan 4, 2008
404 Errors in Webmaster Console
What to do with these pages?
2 rowtc2 11:35 pm Jan 3, 2008
directory and subdirectory access
allowing access to subdirectory but not main directory
3 Mr_Cool_Dude 6:11 pm Jan 2, 2008
robots.txt question
robots.txt examples
2 jim_knopf 5:43 pm Jan 2, 2008
Reference robots.txt file?
Does anyone have one?
3 mmiller 11:30 pm Dec 25, 2007
how to manage cname in robot
don't was to be indexed
4 experienced 2:10 pm Dec 21, 2007
disallow vs ban in .htaccess
12 joergnw10 11:10 am Dec 16, 2007
How do I prevent the SEs from crawling any links to my site w/HTTPS
I don't want the crawlers to index anything w/ HTTPS protocol
3 ZydoSEO 8:37 pm Dec 9, 2007
yahoo google msn etc.
other searches
1 ComputerHermit 5:56 pm Dec 7, 2007
Proposal for robots.txt To Have Greater Flexibility
12 engine 4:58 pm Nov 30, 2007
set crawl rate permanently to faster
1 agushalim 5:39 pm Nov 28, 2007
Wildcards on robots.txt disallows
11 MindTwist 5:59 pm Nov 27, 2007
Site title and description missing in search results
I had made some changes in robots.txt file
2 ecmedia 2:12 pm Nov 27, 2007
perfect robot text files
good robot text files
2 likwidholdings 12:57 am Nov 27, 2007
Google showing site despite blocking in robots.txt
The page is not cached though
3 ecmedia 4:28 pm Nov 20, 2007
Disallow all except adsense
6 thegreatpretender 4:40 pm Nov 19, 2007
How to avoid double indexing
3 mavi 4:43 pm Nov 13, 2007
Robots.txt Clarification
5 smartcard 10:44 am Nov 13, 2007
Robot from colo is on behalf of Google?!
Colo ISP claims robot crawling my site is from 3rd party Google contractor
5 yodokame 4:03 am Nov 12, 2007
robots.txt file
how it affects the page rank of any website
2 monikagaba 6:21 pm Nov 1, 2007
Exclude ALL bots but major search engines
Trying to make the perfect robots.txt
5 serpmaster 12:03 pm Oct 25, 2007
blocking all except root (index.php)
would disallow ALL , allow index work?
1 bkode 7:56 am Oct 24, 2007
http code 200 mystery
is the page served or not!?
3 dailyclicker 5:14 pm Oct 13, 2007