Welcome to WebmasterWorld Guest from 54.167.253.186

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Removing pages from Google's cache
Is it best to allow a "robots" "noarchive" crawl first?
1 Phil_Payne 2:05 am Jan 10, 2006
Disallow ALL folders?
but allow top level html
6 Reno 9:14 pm Jan 9, 2006
How to disallow widgets from being indexed
Order of User-agents
2 albatros 8:10 pm Jan 9, 2006
Empty line after User-agent:
Could this do any harm?
1 taps 5:31 pm Jan 7, 2006
Robot Validator Problem
7 bunky6767 1:13 pm Jan 6, 2006
Blocking all URLS to a website using the robots.txt file?
2 scubajared 10:05 am Jan 6, 2006
Turing keys
2 Ejap 5:57 pm Jan 5, 2006
ABC of robots.txt
Now that will be the best tutorial!
8 Monzilla 2:19 pm Jan 4, 2006
Define allowable spider times
MSN is going nutz![2] ( 1 2 )
31 Reno 1:44 pm Jan 4, 2006
New empty site beeing spidered
2 sren 3:08 am Dec 29, 2005
remove PHP from Google.com
remove PHP from Google.com
2 maimax 7:32 am Dec 28, 2005
paging google! robots.txt being ignored!
14 elvey 7:25 am Dec 27, 2005
How to block robots spidering certain pages?
4 zoltan 6:42 pm Dec 23, 2005
Correct Syntax for Blogs in robots.txt
3 vincevincevince 4:27 pm Dec 22, 2005
Blocking domain specific links
3 seasalt 11:33 pm Dec 21, 2005
Google appears to be ignoring robots file
No change after three months
5 Lokutus 9:50 pm Dec 16, 2005
parameter page to be banned from crawling
directory.asp?
4 idolw 1:18 pm Dec 15, 2005
About the robots tag for Googlebot
Is it necessary?
2 a1call 8:57 pm Dec 14, 2005
Remove part of your website
Should I use robots.txt or 301 redirects?
1 twist 7:27 pm Dec 13, 2005
Robots.txt and the Google sandbox
Does de-indexing put one back into the sandbox?
2 Rollo 7:05 pm Dec 13, 2005
Should I use the "?" or not.
2 kgormat 3:52 am Dec 12, 2005
robots.txt and the law
do they intersect
2 Powdork 10:19 am Dec 11, 2005
Newbie here require help
robots.txt
2 sim20 11:36 pm Dec 2, 2005
disallow URL with parameters
/?id=asdf vs /
5 Ratbert 11:18 am Dec 1, 2005
Ban Google from 1 directory
Ban Google from 1 directory
2 dunhill 6:47 am Nov 30, 2005