Welcome to WebmasterWorld Guest from 23.23.46.20

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Quick Question
When something is disallowed, do bots ignore it completely?
5 jlander 2:43 am Sep 12, 2005
Am I correctly blocking diamondbot?
2 JAB_Creations 11:25 am Sep 11, 2005
Does excluding a cgi page automatically disable same page with paramet
paramters?
4 Clark 3:47 am Sep 11, 2005
Is it harmful if google indexes my /feed/ directory?
how to dissallow safely
3 brokenbricks 2:42 pm Sep 9, 2005
robots.txt wildcard
Will this work
7 gosman 1:59 pm Sep 6, 2005
Robots.txt and frontpage
14 sunflower12 1:32 pm Sep 4, 2005
The codes required to call the robots
2 benifer 10:04 pm Aug 31, 2005
Google Hammering My site (1500+ hits already today)
How to edit robots.txt to set delay
5 wfernley 12:05 pm Aug 31, 2005
Why do you need robots.txt?
5 gordele 6:38 pm Aug 30, 2005
Specifying pages for robots to crawl
Worth it or not?
1 sixdeep 6:14 pm Aug 29, 2005
Including specific robots necessary or not?
7 silverbytes 8:27 am Aug 29, 2005
how do i ban image robots
2 spivei 11:04 pm Aug 28, 2005
basic idea of robots.txt
8 moi_britphile 10:38 am Aug 28, 2005
Banning AskJeeves Bot
Need to ban askJeeves
2 Marcus_Aurelius 8:31 pm Aug 27, 2005
problem with robots.txt file
problem with robots.txt file
5 raybo75 5:36 pm Aug 27, 2005
How Often Should a Spider Come
What is the right amount of access a spider should have
4 sirjeffer 11:00 pm Aug 22, 2005
Bot cn exclude this directory?
2 tintin74 10:42 pm Aug 22, 2005
disallow all : including host
4 moua 3:57 pm Aug 22, 2005
Google Sitemap file name?
What should I call my Google Sitemap?
2 jetnovo 3:21 pm Aug 22, 2005
Spiders and redirects to disallowed directories
Are the disallowed pages indexed?
2 cigjonser 7:18 pm Aug 19, 2005
allow some and exclude the rest in robots.txt
9 jcmiras 5:24 am Aug 19, 2005
can you have 2 robots.txt files on site?
one in main and one in subdomain
8 stiksandstones 12:10 am Aug 19, 2005
What bots ignore robots.txt?
8 Jon12345 10:56 pm Aug 18, 2005
Validator Questions
Need Help Deciphering Validator Output
7 Jioffri 6:07 am Aug 18, 2005
Quick question on robots.txt
Can you separate the agent and the allows/disallows?
2 cmendla 11:19 pm Aug 17, 2005

Featured Threads

Hot Threads This Week

Hot Threads This Month