Welcome to WebmasterWorld Guest from 23.20.186.146

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Validation Error
Ran the robots.txt validation checker and recieved nothing but errors.
3 jseamless 5:06 pm Mar 25, 2004
Why use Robots - Again?
What are the benefits of using Robots.
4 oltc 4:48 pm Mar 25, 2004
How can I block Cyveillancebot(63.148.99.****)
It doesn't look like a useful bot...
6 itisgene 4:43 pm Mar 25, 2004
Yahoo-MMCrawler
1 simondev3 8:08 pm Mar 24, 2004
Referral log spamming: How use robots.txt to keep them out?
10 Dan_Vendel 12:36 pm Mar 24, 2004
robots.txt elementary question
3 Vespasian 5:29 am Mar 24, 2004
How to identify search engine robots?
What's the best way for ASP developers to identify robots?
1 vieth 3:48 pm Mar 23, 2004
Robot text for dissallow asp pages indexed
2 DivinoStore 8:06 am Mar 21, 2004
Anyone recognize this agent?
Can't find any info about it anywhere
2 martyt 4:15 pm Mar 20, 2004
A new bot?
Anyone else seeing this?
4 walrus 6:48 am Mar 19, 2004
Can't prevent php files from being searched
Is this possible
1 Jack_mcs 1:58 pm Mar 18, 2004
newbie robots.txt and forum Q
how to write it
6 buksida 3:27 am Mar 18, 2004
Robot.txt File
What is it's importance?
7 Enigmatic 8:52 pm Mar 16, 2004
QuepasaCreep Agent
where is it from?
3 barbeta 8:17 pm Mar 16, 2004
Does robots exclusion prevent pr being passed
2 djtaverner 5:38 pm Mar 16, 2004
Should I ban this user agent?
4 jehoshua 5:13 pm Mar 16, 2004
syntax for all php exclusion
2 djtaverner 4:29 pm Mar 15, 2004
Cannot find error
Robots.txt syntax error
5 webapache 5:04 pm Mar 14, 2004
robots.txt and mod_rewrite
just double checking
6 hannamyluv 2:24 am Mar 14, 2004
Several robots.txt?
2 brakkar 1:18 am Mar 14, 2004
Where is the "Top Level"?
is this important these days?
4 chewy 9:46 pm Mar 13, 2004
Does this work?
Disallow: /somepath/*/file/*
6 DoppyNL 5:36 pm Mar 13, 2004
CCGCrawl -- custom crawler/harvester software
New crawler software potential for abuse
2 privacyman 1:38 pm Mar 13, 2004
How to exclude bots from specific domains if all point to same place?
We have foreign domains that all point to the same folder
2 alpine 6:06 am Mar 13, 2004
Yahoo Images
2 RammsteinNicCage 5:56 am Mar 13, 2004

Featured Threads

Hot Threads This Week

Hot Threads This Month