Welcome to WebmasterWorld Guest from 54.146.246.4

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
How to avoid getting print-pages indexed?
What's the best way for Google?
1 simczak 3:42 pm Jul 24, 2005
Robots.txt stops automated link software
2 jdhuk 11:01 pm Jul 18, 2005
Which editor?
Notepad or editor in UNIX
3 Bali_Freak 12:15 pm Jul 18, 2005
selectively including folders
that are beneath a folder excluded
2 stu2 5:20 am Jul 17, 2005
Would this work? Robots exclusion or considered cloaking?
5 dvldvl3 5:11 am Jul 17, 2005
Previously Indexed Files-Want To Exclude Now
RobotsTxt Change
3 buview1 12:23 pm Jul 16, 2005
Google only hits robots and index
Google only hits robots and index
10 rostom 3:51 pm Jul 15, 2005
Disallowing & in url
2 Wolvereness 2:54 pm Jul 12, 2005
robots.txt verses sitemap.html verses OPML file
2 Neal 8:39 pm Jul 11, 2005
Just added robots.txt file
4 soccrt 4:41 am Jul 11, 2005
block all except
2 Radial 9:31 pm Jul 7, 2005
robots.txt with? in URL
16 jozomannen 11:17 am Jul 6, 2005
How do i add robot.txt?
10 genny 4:38 am Jul 6, 2005
Include files, spiders, and robots.txt
4 rottsnroses 11:24 pm Jul 3, 2005
How to Stop BecomeBot?
12 shilmy 3:09 pm Jun 29, 2005
Google ignoring my robots.txt?
9 homesby 12:17 am Jun 29, 2005
Robots meta tag *CORRECT* syntax
Space* or no space for "index,*follow", and case sensitive?
6 Clint 12:56 pm Jun 28, 2005
Question about Robots with multiple similar sites
Question about Robots with multiple similar sites
2 ccash 9:54 pm Jun 27, 2005
Robots/Slurp, etc - what does this mean
I am a newbie - not sure what this means
4 ramnyc 3:01 pm Jun 22, 2005
Does adding robot text help your site get crawled more often?
Advantages of robot text?
6 moneyangel13 8:04 am Jun 22, 2005
wildcards in page names
Disallow: /store/scripts/emailFriend.asp*
5 jimsthoughts 3:54 pm Jun 21, 2005
How can I allow robots to access my pages
inktomi is following links it should not
4 expat123 10:08 am Jun 21, 2005
Updating robots.txt and Crawlers behaviour
How to tell bots to come back when you changed robots.txt banning them
6 silverbytes 6:57 pm Jun 20, 2005
slurp ignoring disallow?
-or have i got it wrong?
4 tallis 12:09 pm Jun 20, 2005
Question about blocking 1 robot, but allow all others
8 capsmaster 3:26 pm Jun 18, 2005