Welcome to WebmasterWorld Guest from 54.226.238.178

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Spiders and redirects to disallowed directories
Are the disallowed pages indexed?
2 cigjonser 7:18 pm Aug 19, 2005
allow some and exclude the rest in robots.txt
9 jcmiras 5:24 am Aug 19, 2005
can you have 2 robots.txt files on site?
one in main and one in subdomain
8 stiksandstones 12:10 am Aug 19, 2005
What bots ignore robots.txt?
8 Jon12345 10:56 pm Aug 18, 2005
Validator Questions
Need Help Deciphering Validator Output
7 Jioffri 6:07 am Aug 18, 2005
Quick question on robots.txt
Can you separate the agent and the allows/disallows?
2 cmendla 11:19 pm Aug 17, 2005
Google's Nonstandard robots.txt is Not Valid
Googles robots.txt Does not validate
6 forak 1:59 pm Aug 17, 2005
Search engines picking up our backend?
Developement site picked up
3 financialhost 12:46 pm Aug 11, 2005
Still no PR for site
3 gnuman 8:39 pm Aug 10, 2005
Can excluding with one word also exclude phrases?
3 Marcia 10:47 pm Aug 7, 2005
New algo about Robots.txt
Is there any new algo introduced regarding robots.txt?
4 Graham_Hans 2:31 am Aug 7, 2005
Dmoz robots.txt
Is the dmoz.org/world disallowed?
4 angiolo 8:05 am Aug 6, 2005
How to exclude Oracle Portal Help pages using robots.txt
Need to exclude Oracle Portal Help Pages from search engines
10 atul_iiit 9:12 am Aug 5, 2005
http://<hostname.domainname>/robots.txt is access denied can robots
Can robots read robots.txt if I get access denied for this file
4 atul_iiit 4:25 pm Aug 4, 2005
What's Robot.txt?
What's the importance of Robot.txt?
3 Lowkei 9:02 am Jul 31, 2005
Don't want to index dynamic pages
But would it be a mistake to block them?
5 chopin2256 2:12 am Jul 30, 2005
Could 744 permissions on robots.txt cause problems
all other files are set to 644
5 dbar 7:41 pm Jul 29, 2005
Robots.txt viewable
viewing the robots.txt file
2 eleap 3:27 pm Jul 27, 2005
How to exclude a few pages?
2 jack38 10:25 am Jul 27, 2005
Blocking a file extension with robots.txt
Inherited a site with a problem
5 abbeyvet 5:45 pm Jul 26, 2005
How to avoid getting print-pages indexed?
What's the best way for Google?
1 simczak 3:42 pm Jul 24, 2005
Robots.txt stops automated link software
2 jdhuk 11:01 pm Jul 18, 2005
Which editor?
Notepad or editor in UNIX
3 Bali_Freak 12:15 pm Jul 18, 2005
selectively including folders
that are beneath a folder excluded
2 stu2 5:20 am Jul 17, 2005
Would this work? Robots exclusion or considered cloaking?
5 dvldvl3 5:11 am Jul 17, 2005