Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

How To Tell Robots to Index My Site Everyday

I need it to come everyday and update my website content everday

12:08 pm on Mar 8, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 3, 2004
votes: 3

well i have a big website which is indexed everyday
i wish to know why the google bot doesn't goes to the pages that my main page points to.

we really add up more than 15 news everyday so the google should get them on their site, but there is no indexing of the other pages?

Can anyone suggest remedy , please write the meta tag i should place on the pages to ask google to index all pages of my website everyday as the content on the pages changes everyday. Also the pages are dynamically generated

waiting for reply


9:01 am on Mar 9, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Apr 5, 2002
votes: 0

Also the pages are dynamically generated

... and that may well be where the problem lies. It may simply be that spiders cannot follow your URL's, especially if you are using Session ID's and/or have lots of odd characters (question marks, ampersands and the like) in your URL strings.

Do a search for Simspider or something similar - lots of tools/utilities will help you view your site and pages as an SE spider/bot would, and subsequently help you identify any blockages or red lights.

Forget trying to tell the spiders to come - you can't - and concentrate on making sure that your site can easily be navigated by both users and bots, if and when they show up. If your URL's do turn out to be a problem, you may need to read-up on such things as mod_rewrite and ISAPI_rewrite, depending on your platform.

(you may also find the following thread very useful:)