Welcome to WebmasterWorld Guest from 54.145.209.34

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Prohibiting folders by name regardless of root relation
Is this possible or not?
6 Jeremy_H 2:38 pm Jan 15, 2006
question about robots.txt, am I banning a directory by accident
17 arbitrary 3:32 am Jan 15, 2006
Quick 'Sanity Check'
3 markd 10:20 am Jan 11, 2006
directory
4 sylvieg 4:38 am Jan 11, 2006
Removing pages from Google's cache
Is it best to allow a "robots" "noarchive" crawl first?
1 Phil_Payne 2:05 am Jan 10, 2006
Disallow ALL folders?
but allow top level html
6 Reno 9:14 pm Jan 9, 2006
How to disallow widgets from being indexed
Order of User-agents
2 albatros 8:10 pm Jan 9, 2006
Empty line after User-agent:
Could this do any harm?
1 taps 5:31 pm Jan 7, 2006
Robot Validator Problem
7 bunky6767 1:13 pm Jan 6, 2006
Blocking all URLS to a website using the robots.txt file?
2 scubajared 10:05 am Jan 6, 2006
Turing keys
2 Ejap 5:57 pm Jan 5, 2006
ABC of robots.txt
Now that will be the best tutorial!
8 Monzilla 2:19 pm Jan 4, 2006
Define allowable spider times
MSN is going nutz![2] ( 1 2 )
31 Reno 1:44 pm Jan 4, 2006
New empty site beeing spidered
2 sren 3:08 am Dec 29, 2005
remove PHP from Google.com
remove PHP from Google.com
2 maimax 7:32 am Dec 28, 2005
paging google! robots.txt being ignored!
14 elvey 7:25 am Dec 27, 2005
How to block robots spidering certain pages?
4 zoltan 6:42 pm Dec 23, 2005
Correct Syntax for Blogs in robots.txt
3 vincevincevince 4:27 pm Dec 22, 2005
Blocking domain specific links
3 seasalt 11:33 pm Dec 21, 2005
Google appears to be ignoring robots file
No change after three months
5 Lokutus 9:50 pm Dec 16, 2005
parameter page to be banned from crawling
directory.asp?
4 idolw 1:18 pm Dec 15, 2005
About the robots tag for Googlebot
Is it necessary?
2 a1call 8:57 pm Dec 14, 2005
Remove part of your website
Should I use robots.txt or 301 redirects?
1 twist 7:27 pm Dec 13, 2005
Robots.txt and the Google sandbox
Does de-indexing put one back into the sandbox?
2 Rollo 7:05 pm Dec 13, 2005
Should I use the "?" or not.
2 kgormat 3:52 am Dec 12, 2005

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month