Welcome to WebmasterWorld Guest from 54.163.168.15

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Add Sitemap To robots.txt For Autodiscovery
Ask.com, Google, Microsoft Live Search and Yahoo! [2] ( 1 2 )
32 OldWolf 7:56 am May 4, 2007
How is alexa getting past my stoppers
stop alexa index
1 Michel_Samuel 3:31 pm May 2, 2007
how to disallow all robots except Googlebot, Yahoo Slurp and MSNBot?
8 d6rth7ader 5:48 am May 2, 2007
Htaccess
3 harish_s 2:17 pm May 1, 2007
Limiting the depth of Google's crawl?
Can robots.txt be used to limit depth
6 daveozzz 7:21 pm Apr 27, 2007
Disallow: /
How did googlebot index my web server dir structure?
3 thinkweb 5:24 am Apr 26, 2007
URLs restricted by robots.txt (41) AGH!
I'm a bit stumped about what's best to do re my robots.txt
8 wrafter 8:38 pm Apr 25, 2007
Is a robots.txt file necessary?
if not trying to block bots?
3 Jakes_Redding 5:52 am Apr 20, 2007
Googlebot ignoring robots.txt?
Or is my robots.txt file wrong...
3 designaweb 9:28 am Apr 17, 2007
Blocking Google Only
From certain sub domains
5 Moncao 7:25 pm Apr 13, 2007
unknown robot
robot
4 kool002 1:59 pm Apr 11, 2007
Not being found by the search engines
noindex,nofollow
4 gtate 6:24 pm Apr 6, 2007
Efficient way to block multiple bots?
11 JAB_Creations 2:33 pm Apr 3, 2007
Blocking Google
I need to block google from my index page
4 loner 4:25 pm Apr 2, 2007
help needed
1 hugh 4:03 pm Apr 2, 2007
What's the Proper Way to Block Sub Folders
Looking for the best way to block 2nd level sub folders
5 graywolf 7:43 pm Mar 28, 2007
ROBOT and PR rank?
how does the robot effect your pr rank?
7 moppy56 1:48 pm Mar 28, 2007
Blocking Partial Contents of Folder Without Using Robots Tag
10 files (7 want indexed, 3 want blocked)
5 SandySEO 8:49 pm Mar 27, 2007
Another bad bot to ban
this one is using 302 redirects .. bad bad
1 Bewenched 3:43 am Mar 25, 2007
excluding a subdomain
Using the robots.txt file to exclude a subdomain
10 joebray 9:47 am Mar 23, 2007
G. ignores robots.txt at other ports, gives false page count
Google shows in its index pages that it never saw, even prohibited
1 OnetSzukaj 2:58 pm Mar 22, 2007
How to use Robots.txt as an INCLUSION method
It's exclusion only if you use it that way
1 incrediBILL 12:32 am Mar 22, 2007
TripAdvisorBot
Anyone seen this?
3 bobbarnes 10:28 pm Mar 16, 2007
problem with msnbot and wildcard matching
2 Achernar 1:36 pm Mar 16, 2007
How Can I Make Bots Visit My Web Site?
3 emadido 2:34 pm Mar 15, 2007

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month