Welcome to WebmasterWorld Guest from 54.242.94.72

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Disallowing specific query strings
3 duke33 3:05 pm Jun 8, 2004
Is it a spider?
2 topher 5:38 am Jun 8, 2004
Preventing robot access to all subdirectories within a directory
2 Leonardi 12:16 am Jun 7, 2004
Robots.txt triggers 404 error
Robots.txt triggers 404 error
2 Nout 11:10 am May 29, 2004
Is this robots.txt ok?
5 esllou 12:32 pm May 26, 2004
Removing "query-stringed" pages from Google
And other search engines using robotx.txt
3 MatthewHSE 2:23 pm May 25, 2004
Stop Spiders Crawling Message Board Edit
Cgi Factory Message Board
7 mn1dbp 12:36 pm May 25, 2004
Google still lists disallowed pages
How long do disallowed pages stay in the index?
5 jam13 10:56 am May 23, 2004
large robots.txt file
What size will cause a crawler to choke in the file
7 DoppyNL 9:43 pm May 21, 2004
Using robot.txt to keep unwanted backlinking
Is this an effective means
2 Daily_Sparring 5:14 pm May 21, 2004
importance of a robots.txt file
11 hughie 12:27 am May 21, 2004
Logic Error
Wildcard userAgent last entry in robots.txt file?
2 pageoneresults 2:34 pm May 20, 2004
Question about robots meta tag, as well as robots.txt....
3 xsirrionx 11:48 am May 20, 2004
How to validate robots.txt on "https" site?
Please help me validate my "https" site
2 CodeCrunch 12:29 am May 20, 2004
Wild chars *.* allowed in "Disallow" of Robots.txt?
or anything else like *.txt, etc.
2 CodeCrunch 12:26 am May 20, 2004
robots.txt restrictions conflictual with site's Terms
Which of them should prevail?
4 cristiscu 6:52 am May 19, 2004
Comments at the end of a line bad style, why?
3 zachariah 6:52 am May 18, 2004
Spider Testing
6 cabowabo 3:44 am May 13, 2004
Meta Tags & Robots.txt
4 tcustom 1:57 am May 13, 2004
Why is Google still indexing my SWF files?
2 ichthyous 6:10 am May 10, 2004
Deny or Not to Deny /images
Denying /images folder
9 quebrada 12:19 am May 6, 2004
proper way of changing site
2 jmdb71 5:11 pm May 1, 2004
How to get spider to visit parts of site more frequently
Is this possible?
5 Pimpernel 10:35 am Apr 30, 2004
robots.txt missing?
2 zyshen 5:19 pm Apr 27, 2004
Google ignoring robots.txt
I'm seeing lots of my 'Disallowed' pages in their index
4 Amygdala 4:54 am Apr 24, 2004

Featured Threads

Hot Threads This Week

Hot Threads This Month