Welcome to WebmasterWorld Guest from 107.20.59.213

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
how long till search bot checks again?
2 editordude 11:47 am Sep 17, 2005
How to prevent hotlinking but allow one site using htaccess?
2 hersheyyu 11:38 pm Sep 15, 2005
Help with disallow
Help with disallow
4 SlugKing 10:21 pm Sep 14, 2005
Can Spiders see my dynamic "includes" pages?
3 zepper 11:57 pm Sep 13, 2005
picking up
5 stevelibby 7:04 am Sep 13, 2005
Need Help
Google spiders crawl 2 sites same content
1 milindnaik 6:54 am Sep 12, 2005
Allow search spider - error 503 socket.gaierror: (7, 'getaddrinfo fail
Allow search spider - error
5 shahindastur 5:39 am Sep 12, 2005
Quick Question
When something is disallowed, do bots ignore it completely?
5 jlander 2:43 am Sep 12, 2005
Am I correctly blocking diamondbot?
2 JAB_Creations 11:25 am Sep 11, 2005
Does excluding a cgi page automatically disable same page with paramet
paramters?
4 Clark 3:47 am Sep 11, 2005
Is it harmful if google indexes my /feed/ directory?
how to dissallow safely
3 brokenbricks 2:42 pm Sep 9, 2005
robots.txt wildcard
Will this work
7 gosman 1:59 pm Sep 6, 2005
Robots.txt and frontpage
14 sunflower12 1:32 pm Sep 4, 2005
The codes required to call the robots
2 benifer 10:04 pm Aug 31, 2005
Google Hammering My site (1500+ hits already today)
How to edit robots.txt to set delay
5 wfernley 12:05 pm Aug 31, 2005
Why do you need robots.txt?
5 gordele 6:38 pm Aug 30, 2005
Specifying pages for robots to crawl
Worth it or not?
1 sixdeep 6:14 pm Aug 29, 2005
Including specific robots necessary or not?
7 silverbytes 8:27 am Aug 29, 2005
how do i ban image robots
2 spivei 11:04 pm Aug 28, 2005
basic idea of robots.txt
8 moi_britphile 10:38 am Aug 28, 2005
Banning AskJeeves Bot
Need to ban askJeeves
2 Marcus_Aurelius 8:31 pm Aug 27, 2005
problem with robots.txt file
problem with robots.txt file
5 raybo75 5:36 pm Aug 27, 2005
How Often Should a Spider Come
What is the right amount of access a spider should have
4 sirjeffer 11:00 pm Aug 22, 2005
Bot cn exclude this directory?
2 tintin74 10:42 pm Aug 22, 2005
disallow all : including host
4 moua 3:57 pm Aug 22, 2005

Featured Threads

Hot Threads This Week

Hot Threads This Month