Welcome to WebmasterWorld Guest from 107.20.5.156

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Blocking Google Only
From certain sub domains
5 Moncao 7:25 pm Apr 13, 2007
unknown robot
robot
4 kool002 1:59 pm Apr 11, 2007
Not being found by the search engines
noindex,nofollow
4 gtate 6:24 pm Apr 6, 2007
Efficient way to block multiple bots?
11 JAB_Creations 2:33 pm Apr 3, 2007
Blocking Google
I need to block google from my index page
4 loner 4:25 pm Apr 2, 2007
help needed
1 hugh 4:03 pm Apr 2, 2007
What's the Proper Way to Block Sub Folders
Looking for the best way to block 2nd level sub folders
5 graywolf 7:43 pm Mar 28, 2007
ROBOT and PR rank?
how does the robot effect your pr rank?
7 moppy56 1:48 pm Mar 28, 2007
Blocking Partial Contents of Folder Without Using Robots Tag
10 files (7 want indexed, 3 want blocked)
5 SandySEO 8:49 pm Mar 27, 2007
Another bad bot to ban
this one is using 302 redirects .. bad bad
1 Bewenched 3:43 am Mar 25, 2007
excluding a subdomain
Using the robots.txt file to exclude a subdomain
10 joebray 9:47 am Mar 23, 2007
G. ignores robots.txt at other ports, gives false page count
Google shows in its index pages that it never saw, even prohibited
1 OnetSzukaj 2:58 pm Mar 22, 2007
How to use Robots.txt as an INCLUSION method
It's exclusion only if you use it that way
1 incrediBILL 12:32 am Mar 22, 2007
TripAdvisorBot
Anyone seen this?
3 bobbarnes 10:28 pm Mar 16, 2007
problem with msnbot and wildcard matching
2 Achernar 1:36 pm Mar 16, 2007
How Can I Make Bots Visit My Web Site?
3 emadido 2:34 pm Mar 15, 2007
why i can't reaply the topic
why i can't reaply the topic
3 unknown 8:43 am Mar 14, 2007
robotsxx.txt not found.
2 benevolent001 1:16 pm Mar 13, 2007
Something a little different
2 futureX 1:56 pm Mar 12, 2007
Why is WebmasterWorld robot validator giving back HTML
3 Clair 7:11 pm Mar 10, 2007
When Robots.txt And Meta Robots Collide
What happens?
13 Easy_Coder 1:57 am Mar 10, 2007
OSC, dupe content, robots.txt and 301 redirects
Need to ensure bots 301 to new SEO URL
5 spina45 8:23 pm Mar 9, 2007
Disallow / access to our directory
disallow
2 drew68 3:47 pm Mar 6, 2007
Block the main page of a directory but not its contents?
6 bresso 7:39 am Mar 6, 2007
Robots.txt and pointing domains
3 Shanada 6:58 am Feb 28, 2007