homepage Welcome to WebmasterWorld Guest from 54.196.199.101
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconWhat will Inktomi do in this scenario?
 
18flobaby 6:02 am on Jan 31, 2004 (gmt 0)
newpost thread iconGoogle spidering past secure server
  Robots.txt file to boot the bot?
2xbase234 12:52 am on Jan 31, 2004 (gmt 0)
newpost thread iconLogs keeping showing 404 error
  I have a robots file and it looks ok
3eesnard 12:03 am on Jan 30, 2004 (gmt 0)
newpost thread iconneed robots.txt for pages that are not linked?
  robots.txt
5rld3 5:09 pm on Jan 29, 2004 (gmt 0)
newpost thread iconSEO Robot.txt tag and robot meta tags... need advice
  Help about robot.txt and robot meta tags
3jamieabbott 2:09 pm on Jan 28, 2004 (gmt 0)
newpost thread iconRobots.txt returning "test"
  Host has really messed me up
4GodLikeLotus 10:37 am on Jan 28, 2004 (gmt 0)
newpost thread iconNeed quick help
  Prevent robots from clicking on PPC links
2hostlead 11:11 am on Jan 27, 2004 (gmt 0)
newpost thread iconhow to let robot crawl the links at a page but not index this page?
  index, robot
2johnlim 11:39 pm on Jan 26, 2004 (gmt 0)
newpost thread iconWildcard use in robots.txt
  Need to disallow spidering of graphics files
2Probably 4:58 am on Jan 26, 2004 (gmt 0)
newpost thread iconrobots.txt and htaccess
  .htaccess circumvents to site root
4stefan009 5:35 pm on Jan 24, 2004 (gmt 0)
newpost thread iconAltavista Scooter eats up too much trafic
 
2maxidrom11 5:28 am on Jan 24, 2004 (gmt 0)
newpost thread iconDoes every site need a robots.txt?
  I have one, but I'm not sure I need it
2MatthewHSE 4:00 pm on Jan 23, 2004 (gmt 0)
newpost thread iconcreating robots.txt
 
5artdog 12:21 pm on Jan 23, 2004 (gmt 0)
newpost thread iconMy page has never been indexed
 
2kknusa 6:45 am on Jan 23, 2004 (gmt 0)
newpost thread iconSubdomains
 
2BBB_Guy 6:38 am on Jan 23, 2004 (gmt 0)
newpost thread iconIndexed pages that are disalowed by robots.txt
  is this normal?
10WebJoe 7:03 pm on Jan 20, 2004 (gmt 0)
newpost thread iconGoogle ignoring robots.txt
  No real surprise, but a real pain
9voice220 4:12 pm on Jan 19, 2004 (gmt 0)
newpost thread iconIs there any bots to block?
  Else than the infamous "Web Content International"?
3Daemon 11:56 pm on Jan 16, 2004 (gmt 0)
newpost thread iconRobots.txt help
 
5mikemcs 1:04 pm on Jan 16, 2004 (gmt 0)
newpost thread iconWho are these spiders and what do they want?
 
7obsos 4:51 pm on Jan 14, 2004 (gmt 0)
newpost thread iconRevisit instruction for robots
  I want to specify different revisit instruction different sections
3adityamodi 5:13 am on Jan 10, 2004 (gmt 0)
newpost thread iconHave I been blocking Altavista spider from spidering my site?
 
6irock 6:10 pm on Jan 9, 2004 (gmt 0)
newpost thread iconrobot.txt to remove an alias
  Afraid of duplicated content
2Ulkari 5:48 pm on Jan 8, 2004 (gmt 0)
newpost thread iconSpider Names
  I need the names of some SEs usernames
3The_Young_One 2:47 am on Jan 8, 2004 (gmt 0)
newpost thread iconBlocking Includes (SSI)
  blocking ssi
3direktor1 4:03 pm on Jan 7, 2004 (gmt 0)
Displaying Page 62 of 71<< Back 1 ... 52 53 54 55 56 57 58 59 60 61 [ 62 ] 63 64 65 66 67 68 69 70 71 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved