homepage Welcome to WebmasterWorld Guest from 54.161.185.244
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconhow to ban Google from indexing ALL .php files
  I'm using rewrite and don't want a dupe penalty
16walkman 5:12 pm on Jun 15, 2005 (gmt 0)
newpost thread iconboitho.com bot violating robots.txt[4] ( 1 2 3 4 )
  Specifically requested only forbidden files
111jazzguy 8:30 pm on Jun 14, 2005 (gmt 0)
newpost thread iconrobots.txt and Google on intranet
  robots.txt not working
4John488 8:11 pm on Jun 13, 2005 (gmt 0)
newpost thread iconSpecific Spiders do need more bandwidth?
  Leaving Gbot or Msn does other robots take more bandwidth when crawling?
2bnvrc 12:39 pm on Jun 13, 2005 (gmt 0)
newpost thread iconrobots.text ..please help
  help with robots.text file
2dmiller63 12:35 pm on Jun 13, 2005 (gmt 0)
newpost thread iconRobot,txt Help
 
8branmh 6:34 pm on Jun 12, 2005 (gmt 0)
newpost thread iconGoogle Robots.txt Wildcard Feature
  Please take a look at my robot.txt
4seunosewa 7:02 am on Jun 11, 2005 (gmt 0)
newpost thread iconSeparate robots.txt file for secure site
  How do i block one and not the other?
3Vimes 3:34 am on Jun 10, 2005 (gmt 0)
newpost thread iconBad bots
 
15ownerrim 4:25 am on Jun 8, 2005 (gmt 0)
newpost thread iconIf I disallow /filename.php ...
  will google still index /filename.php?whatever
7Trisha 7:05 am on Jun 7, 2005 (gmt 0)
newpost thread iconAgent Name for MSN
  Where can I find this information.
8Pat1975 5:55 am on Jun 3, 2005 (gmt 0)
newpost thread iconGeneral questions
  General robots.txt queries
7makzan 11:49 pm on Jun 2, 2005 (gmt 0)
newpost thread iconDisallow /file.php same as Diallow file.php?
 
6One_on_One 3:43 am on Jun 2, 2005 (gmt 0)
newpost thread iconStopping spiders
  from Competitors Analysis Tools...
4ambiorix_rb 1:21 pm on Jun 1, 2005 (gmt 0)
newpost thread iconSEO / robots.txt question
  unintentional cross-linking penalty
1wolfgang 6:27 pm on May 31, 2005 (gmt 0)
newpost thread iconBlocking a Sub Domain Directory
 
3is300 8:25 am on May 30, 2005 (gmt 0)
newpost thread iconOrder allow,deny or deny,allow
  Not sure which order is best...
3whoisgregg 9:16 pm on May 27, 2005 (gmt 0)
newpost thread iconSearch Engine Crawlers
 
2dayo 8:39 pm on May 26, 2005 (gmt 0)
newpost thread iconSpidering of my code pages
  Wrong pages being spidered and losing referals
5djfelip 11:18 pm on May 25, 2005 (gmt 0)
newpost thread iconrobots4.txt -- I'm confused
  Confused about what is allowed in this file
3Goose_68 8:46 pm on May 24, 2005 (gmt 0)
newpost thread iconnewbie question
  http://www.yourdomain.com/robots.txt
2webmz 3:10 am on May 22, 2005 (gmt 0)
newpost thread iconValidation
 
18Dexie 4:14 pm on May 21, 2005 (gmt 0)
newpost thread iconCan't find the robot.txt
  where is it?
27mmmwowmmm 10:56 am on May 21, 2005 (gmt 0)
newpost thread iconSonicWALL Firewall Blocks Spiders
  Settings will block spiders from indexing site.
26EliteWeb 3:16 pm on May 15, 2005 (gmt 0)
newpost thread iconRobots.txt Format
  Will this work?
4kbba04527 6:37 am on May 6, 2005 (gmt 0)
Displaying Page 47 of 72<< Back 1 ... 37 38 39 40 41 42 43 44 45 46 [ 47 ] 48 49 50 51 52 53 54 55 56 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved