Welcome to WebmasterWorld Guest from 54.144.243.34

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
How to Handle Cname URL
Indexed in Google
7 experienced 7:47 am Feb 12, 2008
Google problem with robots.txt
Webmaster tools isn't helping.
21 wesg 8:16 pm Feb 5, 2008
*/? */index.*? is a must in robots.txt
I see /?widgetid=45 indexed
3 AjiNIMC 8:17 am Jan 30, 2008
Robots.txt (/index.jsp?pg name=ccpm/lob/sublob/pgname&site=a1)
Pattern on parameters
2 Surendra 12:20 am Jan 29, 2008
Denying access to everything but the home page?
Using robots.txt to only allow robot access to the home page
3 monkeylytics 8:00 pm Jan 28, 2008
Control Robots with Meta or robots.txt
3 driller41 12:21 pm Jan 15, 2008
Robots.txt disallowed.
12 g1smd 12:18 am Jan 15, 2008
robots.txt with wildcard?
3 tranceformer 12:39 am Jan 13, 2008
Disallowing with robots.txt
robots.txt, disallow certain urls
3 jim_knopf 12:36 am Jan 13, 2008
Getting correct user agent
getting from access log the correct user agent for robots.txt
2 hessodreamy 9:03 pm Jan 11, 2008
Scrapers checking robots.txt
Spiders arrived on heels of robots.txt deployment
4 timster 1:29 am Jan 4, 2008
404 Errors in Webmaster Console
What to do with these pages?
2 rowtc2 11:35 pm Jan 3, 2008
directory and subdirectory access
allowing access to subdirectory but not main directory
3 Mr_Cool_Dude 6:11 pm Jan 2, 2008
robots.txt question
robots.txt examples
2 jim_knopf 5:43 pm Jan 2, 2008
Reference robots.txt file?
Does anyone have one?
3 mmiller 11:30 pm Dec 25, 2007
how to manage cname in robot
don't was to be indexed
4 experienced 2:10 pm Dec 21, 2007
disallow vs ban in .htaccess
12 joergnw10 11:10 am Dec 16, 2007
How do I prevent the SEs from crawling any links to my site w/HTTPS
I don't want the crawlers to index anything w/ HTTPS protocol
3 ZydoSEO 8:37 pm Dec 9, 2007
yahoo google msn etc.
other searches
1 ComputerHermit 5:56 pm Dec 7, 2007
Proposal for robots.txt To Have Greater Flexibility
12 engine 4:58 pm Nov 30, 2007
set crawl rate permanently to faster
1 agushalim 5:39 pm Nov 28, 2007
Wildcards on robots.txt disallows
11 MindTwist 5:59 pm Nov 27, 2007
Site title and description missing in search results
I had made some changes in robots.txt file
2 ecmedia 2:12 pm Nov 27, 2007
perfect robot text files
good robot text files
2 likwidholdings 12:57 am Nov 27, 2007
Google showing site despite blocking in robots.txt
The page is not cached though
3 ecmedia 4:28 pm Nov 20, 2007

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month