Welcome to WebmasterWorld Guest from 54.226.147.190

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Custom rule for google, url args and wordpress feeds
6 Kurt 2:28 pm May 24, 2007
Wordpress - Robots.txt to fix Duplicate Content
I need a robots.txt file to prevent dup content from being crawled
7 jman123 2:16 pm May 23, 2007
Alexa robot
issue with alexa bot & alexa toolbar users
6 Michel_Samuel 10:45 am May 20, 2007
Robots.txt
Can you exclude a parameter in robots.txt?
3 North_Shore 9:22 pm May 18, 2007
robots.txt mess-up
4 strawbleu 9:12 pm May 18, 2007
XML Stemap
6 ajitaryan 11:35 am May 16, 2007
Is this valid?
3 aha7 12:34 pm May 9, 2007
Disallowing content previously indexed
What happens to indexed content subsequently blocked by robots.txt
3 daveozzz 11:22 am May 9, 2007
duplicate content indexing
11 coburn 6:34 pm May 7, 2007
Denying one page with multiple variables
page.html?item=1, page.html?item=2, page.html?item=3
7 wfernley 6:21 pm May 7, 2007
Can anyone post a complete robots.txt file here?
4 PowerUp 2:29 pm May 7, 2007
How do I disallow bots from my Wordpress year Archive Folders
I guess this is a common question but I can't find an answer
4 Drum 5:26 pm May 5, 2007
Add Sitemap To robots.txt For Autodiscovery
Ask.com, Google, Microsoft Live Search and Yahoo! [2] ( 1 2 )
32 OldWolf 7:56 am May 4, 2007
How is alexa getting past my stoppers
stop alexa index
1 Michel_Samuel 3:31 pm May 2, 2007
how to disallow all robots except Googlebot, Yahoo Slurp and MSNBot?
8 d6rth7ader 5:48 am May 2, 2007
Htaccess
3 harish_s 2:17 pm May 1, 2007
Limiting the depth of Google's crawl?
Can robots.txt be used to limit depth
6 daveozzz 7:21 pm Apr 27, 2007
Disallow: /
How did googlebot index my web server dir structure?
3 thinkweb 5:24 am Apr 26, 2007
URLs restricted by robots.txt (41) AGH!
I'm a bit stumped about what's best to do re my robots.txt
8 wrafter 8:38 pm Apr 25, 2007
Is a robots.txt file necessary?
if not trying to block bots?
3 Jakes_Redding 5:52 am Apr 20, 2007
Googlebot ignoring robots.txt?
Or is my robots.txt file wrong...
3 designaweb 9:28 am Apr 17, 2007
Blocking Google Only
From certain sub domains
5 Moncao 7:25 pm Apr 13, 2007
unknown robot
robot
4 kool002 1:59 pm Apr 11, 2007
Not being found by the search engines
noindex,nofollow
4 gtate 6:24 pm Apr 6, 2007
Efficient way to block multiple bots?
11 JAB_Creations 2:33 pm Apr 3, 2007

Featured Threads

Hot Threads This Week

Hot Threads This Month