Welcome to WebmasterWorld Guest from 54.227.83.19

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Will Googlebot be able to crawl these pages of my website?
4 saqitude 3:40 pm Feb 25, 2007
robots.txt question
2 seomack 8:14 am Feb 19, 2007
Is correct this robots.txt?
2 MalastiC 9:44 pm Feb 18, 2007
Sitemap and robot.txt
3 spuddy 3:53 pm Feb 15, 2007
How to exclude from crawling subdomain
exclude the crawling subdomain
4 fendy 1:00 pm Feb 15, 2007
Confused over Google spider and robots.txt
4 flowermark 12:56 pm Feb 15, 2007
Which bots to block?
robot.txt
2 cyclesolutions 9:51 pm Feb 12, 2007
Accidentally blocked root access, doh!
3 ninjaNeen 6:51 pm Feb 8, 2007
robots.txt usage and examples
Using robots.txt to improve SEO
3 cduke250 3:50 pm Feb 7, 2007
disallowing subdomain access rule
1 coburn 1:37 pm Feb 7, 2007
can put two robots at one line at robot.txt to ban two spiders?
robots.txt, ban google, yahoo bots
4 johnlim9988 3:54 am Feb 7, 2007
Blocking subdir, but allowing crawl in deeper dir?
7 Vulcan315 3:52 pm Feb 6, 2007
robots noindex on an iframe page
What would this do?
2 dataguy 1:20 pm Feb 3, 2007
Slowing down MSN bot?
Doubling it's bandwidth usage each month.
4 bouncybunny 3:33 am Feb 3, 2007
Slurps and wildcards
5 Vimes 1:36 am Feb 3, 2007
robots.txt blocking my index page?
Hello
4 hola2 2:08 pm Feb 2, 2007
Robots.txt command? "do not search anything outside this."
5 OWizeOne 1:08 pm Feb 1, 2007
How do I block yahoo's SE spider and allow their image spider?
Does Yahoo send a different spider for the image search?
4 Clark 5:18 pm Jan 27, 2007
How robot.txt works
Robot.txt
3 walmslei 4:10 am Jan 27, 2007
Why is google indexing this banned directory?
5 ichthyous 4:55 pm Jan 26, 2007
https on google and robots.txt
2 concepts99 1:54 pm Jan 24, 2007
directory structure question
5 union_jack 7:43 pm Jan 22, 2007
block a subdirectory
2 concepts99 2:05 pm Jan 22, 2007
Google not respecting robots.txt
Found >500 pages in Google index that are forbiden by robots.txt
5 roots 5:54 am Jan 22, 2007
domo arigato
mr. roboto
3 ricnut 11:52 pm Jan 20, 2007

Featured Threads

Hot Threads This Week

Hot Threads This Month