Welcome to WebmasterWorld Guest from 54.234.45.10

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Logs keeping showing 404 error
I have a robots file and it looks ok
3 eesnard 12:03 am Jan 30, 2004
need robots.txt for pages that are not linked?
robots.txt
5 rld3 5:09 pm Jan 29, 2004
SEO Robot.txt tag and robot meta tags... need advice
Help about robot.txt and robot meta tags
3 jamieabbott 2:09 pm Jan 28, 2004
Robots.txt returning "test"
Host has really messed me up
4 GodLikeLotus 10:37 am Jan 28, 2004
Need quick help
Prevent robots from clicking on PPC links
2 hostlead 11:11 am Jan 27, 2004
how to let robot crawl the links at a page but not index this page?
index, robot
2 johnlim 11:39 pm Jan 26, 2004
Wildcard use in robots.txt
Need to disallow spidering of graphics files
2 Probably 4:58 am Jan 26, 2004
robots.txt and htaccess
.htaccess circumvents to site root
4 stefan009 5:35 pm Jan 24, 2004
Altavista Scooter eats up too much trafic
2 maxidrom11 5:28 am Jan 24, 2004
Does every site need a robots.txt?
I have one, but I'm not sure I need it
2 MatthewHSE 4:00 pm Jan 23, 2004
creating robots.txt
5 artdog 12:21 pm Jan 23, 2004
My page has never been indexed
2 kknusa 6:45 am Jan 23, 2004
Subdomains
2 BBB_Guy 6:38 am Jan 23, 2004
Indexed pages that are disalowed by robots.txt
is this normal?
10 WebJoe 7:03 pm Jan 20, 2004
Google ignoring robots.txt
No real surprise, but a real pain
9 voice220 4:12 pm Jan 19, 2004
Is there any bots to block?
Else than the infamous "Web Content International"?
3 Daemon 11:56 pm Jan 16, 2004
Robots.txt help
5 mikemcs 1:04 pm Jan 16, 2004
Who are these spiders and what do they want?
7 obsos 4:51 pm Jan 14, 2004
Revisit instruction for robots
I want to specify different revisit instruction different sections
3 adityamodi 5:13 am Jan 10, 2004
Have I been blocking Altavista spider from spidering my site?
6 irock 6:10 pm Jan 9, 2004
robot.txt to remove an alias
Afraid of duplicated content
2 Ulkari 5:48 pm Jan 8, 2004
Spider Names
I need the names of some SEs usernames
3 The_Young_One 2:47 am Jan 8, 2004
Blocking Includes (SSI)
blocking ssi
3 direktor1 4:03 pm Jan 7, 2004
Subdomains
How to exclude
4 gosman 2:54 pm Jan 6, 2004
Does the leading / make any difference?
Sorry, I'm new to robots.txt
2 AsleepATheWheel 1:07 am Jan 6, 2004

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members