Welcome to WebmasterWorld Guest from 126.96.36.199
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
I asked whether does this, but the engine spider ip address has any to use
|2||ericchui||11:18 am Nov 22, 2004|
|Protecting robots.txt file from bad guys
Is it enough to passwoed protect those dirs?
|3||damlag||4:10 pm Nov 21, 2004|
|How to Exclude from Yahoo/MSN results!
I want to exclude underconstruction site
|2||fm101||1:49 am Nov 21, 2004|
|How to block all subdirectories?
||2||TerryMc||7:05 pm Nov 17, 2004|
|How to exclude dynamic URLS?
So Search engines dont get bogged down.
|7||Tech2004||4:54 am Nov 16, 2004|
|Effect of Robots.txt
||5||seolancer||5:31 am Nov 15, 2004|
|robots.txt - newbie needs help
||5||geezer_uk||5:39 am Nov 10, 2004|
is this right
|2||rbacal||2:59 am Nov 10, 2004|
|Is robot.txt a must for good SEO?
--whats the purpose of robot.txt
|4||redleaf||12:24 am Nov 9, 2004|
|Weird 404 error hitting on robot.txt
anyone experienced this before?
|3||redleaf||9:30 am Nov 8, 2004|
|robots.txt and free web hosting
||3||rhodopsin||7:46 am Nov 8, 2004|
|Possible to have a page disalowed by robots.txt but indexed by adsense
Possible to do this?
|2||brakkar||9:16 pm Nov 5, 2004|
Peerbot or I misunderstood the robot exclusion protocol
|3||WebJoe||9:11 pm Nov 5, 2004|
|Need help about File Robots.txt
how do I know spiders visit to my site or not?
|3||viva||10:15 am Nov 4, 2004|
|what happen with robot exclusion and iframes?
could it be a problem to exclude content that appears inside a home iframe?
|2||mgorricho||5:03 pm Nov 3, 2004|
|protocol extension resource control
proposed extension to robots.txt protocol allowing resource control
|1||copsewood||11:29 am Nov 2, 2004|
||2||papamaku||9:36 am Nov 2, 2004|
|robots and nav bars
excluding links in nav bars ...
|2||f00fighter||11:38 pm Oct 29, 2004|
|Need assistance with robot controls
need help with a robot .txt file
|2||borkboing||11:31 pm Oct 29, 2004|
|Robots.txt not working
bots not obeying robots.txt
|4||cyberjunkie||11:05 pm Oct 29, 2004|
|Use .htaccess to serve correct robots.txt?
Using htaccess and robots.txt to elimiate file duplication and 404 errors
|2||iceman||12:05 am Oct 26, 2004|
|Robots detection tool?
||2||obsos||1:17 am Oct 25, 2004|
|robots.txt vs index,follow
Who will win?
|3||brakkar||6:13 pm Oct 24, 2004|
|Removed 'deny all' robots.txt but no change
||2||kapow||11:03 pm Oct 23, 2004|
|What name does msn's spider use?
Lots of visits from yahoo and google....but not msn
|2||David_Bruning||7:05 am Oct 19, 2004|