Welcome to WebmasterWorld Guest from 54.157.222.62

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Allow search spider - error 503 socket.gaierror: (7, 'getaddrinfo fail
Allow search spider - error
5 shahindastur 5:39 am Sep 12, 2005
Quick Question
When something is disallowed, do bots ignore it completely?
5 jlander 2:43 am Sep 12, 2005
Am I correctly blocking diamondbot?
2 JAB_Creations 11:25 am Sep 11, 2005
Does excluding a cgi page automatically disable same page with paramet
paramters?
4 Clark 3:47 am Sep 11, 2005
Is it harmful if google indexes my /feed/ directory?
how to dissallow safely
3 brokenbricks 2:42 pm Sep 9, 2005
robots.txt wildcard
Will this work
7 gosman 1:59 pm Sep 6, 2005
Robots.txt and frontpage
14 sunflower12 1:32 pm Sep 4, 2005
The codes required to call the robots
2 benifer 10:04 pm Aug 31, 2005
Google Hammering My site (1500+ hits already today)
How to edit robots.txt to set delay
5 wfernley 12:05 pm Aug 31, 2005
Why do you need robots.txt?
5 gordele 6:38 pm Aug 30, 2005
Specifying pages for robots to crawl
Worth it or not?
1 sixdeep 6:14 pm Aug 29, 2005
Including specific robots necessary or not?
7 silverbytes 8:27 am Aug 29, 2005
how do i ban image robots
2 spivei 11:04 pm Aug 28, 2005
basic idea of robots.txt
8 moi_britphile 10:38 am Aug 28, 2005
Banning AskJeeves Bot
Need to ban askJeeves
2 Marcus_Aurelius 8:31 pm Aug 27, 2005
problem with robots.txt file
problem with robots.txt file
5 raybo75 5:36 pm Aug 27, 2005
How Often Should a Spider Come
What is the right amount of access a spider should have
4 sirjeffer 11:00 pm Aug 22, 2005
Bot cn exclude this directory?
2 tintin74 10:42 pm Aug 22, 2005
disallow all : including host
4 moua 3:57 pm Aug 22, 2005
Google Sitemap file name?
What should I call my Google Sitemap?
2 jetnovo 3:21 pm Aug 22, 2005
Spiders and redirects to disallowed directories
Are the disallowed pages indexed?
2 cigjonser 7:18 pm Aug 19, 2005
allow some and exclude the rest in robots.txt
9 jcmiras 5:24 am Aug 19, 2005
can you have 2 robots.txt files on site?
one in main and one in subdomain
8 stiksandstones 12:10 am Aug 19, 2005
What bots ignore robots.txt?
8 Jon12345 10:56 pm Aug 18, 2005
Validator Questions
Need Help Deciphering Validator Output
7 Jioffri 6:07 am Aug 18, 2005