Welcome to WebmasterWorld Guest from 54.145.13.215

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
stop Yahoo's crawler but for specific domains
how to stop yahoo's crawler for specif domains
2 nrueda 11:40 pm Nov 8, 2005
wildcard for googlebot
4 fzx5v0 4:02 pm Nov 7, 2005
Proposing updates to the robots.txt standard
It is time for a change
16 volatilegx 9:20 pm Nov 4, 2005
what i did wrong?
5 fluxer 3:45 pm Nov 4, 2005
protected ffiles and robots
2 StickeR 7:48 pm Nov 3, 2005
Exculde php url with vars using robots.txt
Exclude urls with vars using robots.txt
2 alexmart 7:44 pm Nov 3, 2005
Is Temporary Disallow a Problem Later?
6 bumpaw 2:10 pm Nov 3, 2005
disallow problem
7 Autoweb 8:25 am Nov 3, 2005
Added robots.txt - 3 days later rankings gone!
11 Block19Row13 3:45 pm Nov 2, 2005
OK now I am baffled - question from new to SEO
4 Iwrite 12:38 pm Nov 2, 2005
Robots.text question, disallowing files
10 mikeD 1:37 am Nov 2, 2005
crawl delay in google, msn, yahoo etc.
crawl delay
4 freegk 7:03 pm Nov 1, 2005
Is robots.txt That necessary?
5 makzan 7:02 pm Nov 1, 2005
after google indexs is it too late to use Disallow
2 watercrazed 8:51 pm Oct 30, 2005
Exclude by filename, not directory?
19 joergnw10 7:36 pm Oct 30, 2005
Robots.txt File to Block from Dynamic URLs
Trying to eliminate duplicate content...
3 BillyS 2:07 am Oct 28, 2005
disallowing a folder with its own domain name
subdomain being spidered although it has its own domain name
2 chandrika 8:52 pm Oct 26, 2005
robots.txt in a folder not root
will this work?
3 fatpeter 6:27 am Oct 26, 2005
Robots.txt and Query Strings
Can they block non directory paths?
3 soquinn 4:45 am Oct 26, 2005
Spider and registred access to web
5 andreabg 8:07 am Oct 19, 2005
google accessing blocked image files
why does it appear that google is not abiding robots.txt?
3 latimer 8:01 am Oct 19, 2005
Disallow all Directories
2 Omega1Megs 8:48 pm Oct 14, 2005
Disallow: line in robots.txt
2 stefan009 5:33 pm Oct 14, 2005
Can a robots.txt be too long?
3 gingerspice 3:32 am Oct 13, 2005
Log file is robot.txt file?
Log file is robot.txt file?
5 i_am_dhaval 9:20 pm Oct 12, 2005