Welcome to WebmasterWorld Guest from 54.166.227.36

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
ROBOT and PR rank?
how does the robot effect your pr rank?
7 moppy56 1:48 pm Mar 28, 2007
Blocking Partial Contents of Folder Without Using Robots Tag
10 files (7 want indexed, 3 want blocked)
5 SandySEO 8:49 pm Mar 27, 2007
Another bad bot to ban
this one is using 302 redirects .. bad bad
1 Bewenched 3:43 am Mar 25, 2007
excluding a subdomain
Using the robots.txt file to exclude a subdomain
10 joebray 9:47 am Mar 23, 2007
G. ignores robots.txt at other ports, gives false page count
Google shows in its index pages that it never saw, even prohibited
1 OnetSzukaj 2:58 pm Mar 22, 2007
How to use Robots.txt as an INCLUSION method
It's exclusion only if you use it that way
1 incrediBILL 12:32 am Mar 22, 2007
TripAdvisorBot
Anyone seen this?
3 bobbarnes 10:28 pm Mar 16, 2007
problem with msnbot and wildcard matching
2 Achernar 1:36 pm Mar 16, 2007
How Can I Make Bots Visit My Web Site?
3 emadido 2:34 pm Mar 15, 2007
why i can't reaply the topic
why i can't reaply the topic
3 unknown 8:43 am Mar 14, 2007
robotsxx.txt not found.
2 benevolent001 1:16 pm Mar 13, 2007
Something a little different
2 futureX 1:56 pm Mar 12, 2007
Why is WebmasterWorld robot validator giving back HTML
3 Clair 7:11 pm Mar 10, 2007
When Robots.txt And Meta Robots Collide
What happens?
13 Easy_Coder 1:57 am Mar 10, 2007
OSC, dupe content, robots.txt and 301 redirects
Need to ensure bots 301 to new SEO URL
5 spina45 8:23 pm Mar 9, 2007
Disallow / access to our directory
disallow
2 drew68 3:47 pm Mar 6, 2007
Block the main page of a directory but not its contents?
6 bresso 7:39 am Mar 6, 2007
Robots.txt and pointing domains
3 Shanada 6:58 am Feb 28, 2007
Will Googlebot be able to crawl these pages of my website?
4 saqitude 3:40 pm Feb 25, 2007
robots.txt question
2 seomack 8:14 am Feb 19, 2007
Is correct this robots.txt?
2 MalastiC 9:44 pm Feb 18, 2007
Sitemap and robot.txt
3 spuddy 3:53 pm Feb 15, 2007
How to exclude from crawling subdomain
exclude the crawling subdomain
4 fendy 1:00 pm Feb 15, 2007
Confused over Google spider and robots.txt
4 flowermark 12:56 pm Feb 15, 2007
Which bots to block?
robot.txt
2 cyclesolutions 9:51 pm Feb 12, 2007

Featured Threads

Hot Threads This Week

Hot Threads This Month