Welcome to WebmasterWorld Guest from 54.147.44.93

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Frequencey of GET /robots.txt
How often should a crawler/spider/robot ask for robots.txt?
9 Dijkgraaf 5:08 am Jun 18, 2005
Need Clarification On Directory Exclusions
10 jonrichd 5:00 am Jun 18, 2005
Robots.txt and Mod Rewrite
Which url does the robot read?
2 eflouret 4:52 am Jun 18, 2005
how to ban Google from indexing ALL .php files
I'm using rewrite and don't want a dupe penalty
16 walkman 5:12 pm Jun 15, 2005
boitho.com bot violating robots.txt
Specifically requested only forbidden files[4] ( 1 2 3 4 )
111 jazzguy 8:30 pm Jun 14, 2005
robots.txt and Google on intranet
robots.txt not working
4 John488 8:11 pm Jun 13, 2005
Specific Spiders do need more bandwidth?
Leaving Gbot or Msn does other robots take more bandwidth when crawling?
2 bnvrc 12:39 pm Jun 13, 2005
robots.text ..please help
help with robots.text file
2 dmiller63 12:35 pm Jun 13, 2005
Robot,txt Help
8 branmh 6:34 pm Jun 12, 2005
Google Robots.txt Wildcard Feature
Please take a look at my robot.txt
4 seunosewa 7:02 am Jun 11, 2005
Separate robots.txt file for secure site
How do i block one and not the other?
3 Vimes 3:34 am Jun 10, 2005
Bad bots
15 ownerrim 4:25 am Jun 8, 2005
If I disallow /filename.php ...
will google still index /filename.php?whatever
7 Trisha 7:05 am Jun 7, 2005
Agent Name for MSN
Where can I find this information.
8 Pat1975 5:55 am Jun 3, 2005
General questions
General robots.txt queries
7 makzan 11:49 pm Jun 2, 2005
Disallow /file.php same as Diallow file.php?
6 One_on_One 3:43 am Jun 2, 2005
Stopping spiders
from Competitors Analysis Tools...
4 ambiorix_rb 1:21 pm Jun 1, 2005
SEO / robots.txt question
unintentional cross-linking penalty
1 wolfgang 6:27 pm May 31, 2005
Blocking a Sub Domain Directory
3 is300 8:25 am May 30, 2005
Order allow,deny or deny,allow
Not sure which order is best...
3 whoisgregg 9:16 pm May 27, 2005
Search Engine Crawlers
2 dayo 8:39 pm May 26, 2005
Spidering of my code pages
Wrong pages being spidered and losing referals
5 djfelip 11:18 pm May 25, 2005
robots4.txt -- I'm confused
Confused about what is allowed in this file
3 Goose_68 8:46 pm May 24, 2005
newbie question
http://www.yourdomain.com/robots.txt
2 webmz 3:10 am May 22, 2005
Validation
18 Dexie 4:14 pm May 21, 2005

Featured Threads

Hot Threads This Week

Hot Threads This Month