homepage Welcome to WebmasterWorld Guest from 54.227.159.165
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread icondisallow: /*
  robot.txt newbie, please help
2emm0uk 11:15 pm on Mar 29, 2004 (gmt 0)
newpost thread iconrobots.txt blocking foreign search engines
  google robots.txt block foreign engines
1jer61375 4:49 pm on Mar 29, 2004 (gmt 0)
newpost thread iconAllowing only search engines to crawl forums
  Only want search engines in my forums
7Psycho111 8:40 pm on Mar 27, 2004 (gmt 0)
newpost thread icondoes a search engine care if a robots.txt file is 403 instead of 404
 
4kahuna 6:20 pm on Mar 27, 2004 (gmt 0)
newpost thread iconrobots.txt and multiple domain names
  Using a Virtual Server...
1nerolabs 7:45 pm on Mar 26, 2004 (gmt 0)
newpost thread icongoogle images ignoring disallow
 
2apollo 10:43 pm on Mar 25, 2004 (gmt 0)
newpost thread iconValidation Error
  Ran the robots.txt validation checker and recieved nothing but errors.
3jseamless 5:06 pm on Mar 25, 2004 (gmt 0)
newpost thread iconWhy use Robots - Again?
  What are the benefits of using Robots.
4oltc 4:48 pm on Mar 25, 2004 (gmt 0)
newpost thread iconHow can I block Cyveillancebot(63.148.99.****)
  It doesn't look like a useful bot...
6itisgene 4:43 pm on Mar 25, 2004 (gmt 0)
newpost thread iconYahoo-MMCrawler
 
1simondev3 8:08 pm on Mar 24, 2004 (gmt 0)
newpost thread iconReferral log spamming: How use robots.txt to keep them out?
 
10Dan_Vendel 12:36 pm on Mar 24, 2004 (gmt 0)
newpost thread iconrobots.txt elementary question
 
3Vespasian 5:29 am on Mar 24, 2004 (gmt 0)
newpost thread iconHow to identify search engine robots?
  What's the best way for ASP developers to identify robots?
1vieth 3:48 pm on Mar 23, 2004 (gmt 0)
newpost thread iconRobot text for dissallow asp pages indexed
 
2DivinoStore 8:06 am on Mar 21, 2004 (gmt 0)
newpost thread iconAnyone recognize this agent?
  Can't find any info about it anywhere
2martyt 4:15 pm on Mar 20, 2004 (gmt 0)
newpost thread iconA new bot?
  Anyone else seeing this?
4walrus 6:48 am on Mar 19, 2004 (gmt 0)
newpost thread iconCan't prevent php files from being searched
  Is this possible
1Jack_mcs 1:58 pm on Mar 18, 2004 (gmt 0)
newpost thread iconnewbie robots.txt and forum Q
  how to write it
6buksida 3:27 am on Mar 18, 2004 (gmt 0)
newpost thread iconRobot.txt File
  What is it's importance?
7Enigmatic 8:52 pm on Mar 16, 2004 (gmt 0)
newpost thread iconQuepasaCreep Agent
  where is it from?
3barbeta 8:17 pm on Mar 16, 2004 (gmt 0)
newpost thread iconDoes robots exclusion prevent pr being passed
 
2djtaverner 5:38 pm on Mar 16, 2004 (gmt 0)
newpost thread iconShould I ban this user agent?
 
4jehoshua 5:13 pm on Mar 16, 2004 (gmt 0)
newpost thread iconsyntax for all php exclusion
 
2djtaverner 4:29 pm on Mar 15, 2004 (gmt 0)
newpost thread iconCannot find error
  Robots.txt syntax error
5webapache 5:04 pm on Mar 14, 2004 (gmt 0)
newpost thread iconrobots.txt and mod_rewrite
  just double checking
6hannamyluv 2:24 am on Mar 14, 2004 (gmt 0)
Displaying Page 59 of 72<< Back 1 ... 49 50 51 52 53 54 55 56 57 58 [ 59 ] 60 61 62 63 64 65 66 67 68 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved