homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

How To Tell Robots to Index My Site Everyday
I need it to come everyday and update my website content everday

 12:08 pm on Mar 8, 2004 (gmt 0)

well i have a big website which is indexed everyday
i wish to know why the google bot doesn't goes to the pages that my main page points to.

we really add up more than 15 news everyday so the google should get them on their site, but there is no indexing of the other pages?

Can anyone suggest remedy , please write the meta tag i should place on the pages to ask google to index all pages of my website everyday as the content on the pages changes everyday. Also the pages are dynamically generated

waiting for reply



 9:01 am on Mar 9, 2004 (gmt 0)

Also the pages are dynamically generated

... and that may well be where the problem lies. It may simply be that spiders cannot follow your URL's, especially if you are using Session ID's and/or have lots of odd characters (question marks, ampersands and the like) in your URL strings.

Do a search for Simspider or something similar - lots of tools/utilities will help you view your site and pages as an SE spider/bot would, and subsequently help you identify any blockages or red lights.

Forget trying to tell the spiders to come - you can't - and concentrate on making sure that your site can easily be navigated by both users and bots, if and when they show up. If your URL's do turn out to be a problem, you may need to read-up on such things as mod_rewrite and ISAPI_rewrite, depending on your platform.

(you may also find the following thread very useful:)


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved