Welcome to WebmasterWorld Guest from 54.147.250.33

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
How to use Robots.txt as an INCLUSION method
It's exclusion only if you use it that way
1 incrediBILL 12:32 am Mar 22, 2007
TripAdvisorBot
Anyone seen this?
3 bobbarnes 10:28 pm Mar 16, 2007
problem with msnbot and wildcard matching
2 Achernar 1:36 pm Mar 16, 2007
How Can I Make Bots Visit My Web Site?
3 emadido 2:34 pm Mar 15, 2007
why i can't reaply the topic
why i can't reaply the topic
3 unknown 8:43 am Mar 14, 2007
robotsxx.txt not found.
2 benevolent001 1:16 pm Mar 13, 2007
Something a little different
2 futureX 1:56 pm Mar 12, 2007
Why is WebmasterWorld robot validator giving back HTML
3 Clair 7:11 pm Mar 10, 2007
When Robots.txt And Meta Robots Collide
What happens?
13 Easy_Coder 1:57 am Mar 10, 2007
OSC, dupe content, robots.txt and 301 redirects
Need to ensure bots 301 to new SEO URL
5 spina45 8:23 pm Mar 9, 2007
Disallow / access to our directory
disallow
2 drew68 3:47 pm Mar 6, 2007
Block the main page of a directory but not its contents?
6 bresso 7:39 am Mar 6, 2007
Robots.txt and pointing domains
3 Shanada 6:58 am Feb 28, 2007
Will Googlebot be able to crawl these pages of my website?
4 saqitude 3:40 pm Feb 25, 2007
robots.txt question
2 seomack 8:14 am Feb 19, 2007
Is correct this robots.txt?
2 MalastiC 9:44 pm Feb 18, 2007
Sitemap and robot.txt
3 spuddy 3:53 pm Feb 15, 2007
How to exclude from crawling subdomain
exclude the crawling subdomain
4 fendy 1:00 pm Feb 15, 2007
Confused over Google spider and robots.txt
4 flowermark 12:56 pm Feb 15, 2007
Which bots to block?
robot.txt
2 cyclesolutions 9:51 pm Feb 12, 2007
Accidentally blocked root access, doh!
3 ninjaNeen 6:51 pm Feb 8, 2007
robots.txt usage and examples
Using robots.txt to improve SEO
3 cduke250 3:50 pm Feb 7, 2007
disallowing subdomain access rule
1 coburn 1:37 pm Feb 7, 2007
can put two robots at one line at robot.txt to ban two spiders?
robots.txt, ban google, yahoo bots
4 johnlim9988 3:54 am Feb 7, 2007
Blocking subdir, but allowing crawl in deeper dir?
7 Vulcan315 3:52 pm Feb 6, 2007

Featured Threads

Hot Threads This Week

Hot Threads This Month