noindex -> index

changing noindex to index

[blade2050]

Hi!

Half a year ago, I've got some (security) reasons to set a "noindex" metatag. But now the site is a bit securer and it seems better to appear in some searchengines. So I switched the tags to "index,follow" in July.

But the site wasn't found by any engines. I updatet the description tag and killed the frameset at the index.html! I've set the lycos/google bots to visit the page, but nothing happens!

Any ideas? Thanks!
Kevin

[tedster]

Are you saying that you submitted your site to Google and Lycos using their "Add URL" function, and you haven't seen any spider visits in your server logs?

You may need to submit again. And sometimes again and again. I always give a 2-3 week gap before resubmitting the same page.

However, if your logs show that the spiders have visited you but you aren't in the index, it may have something to do with the way your pages are coded. So many possibilities that it's hard to answer in a general way.

[engine]

Also, I've found "index/follow" to be a waste of time. A spider visting a site is there to index the page and follow links. Save some code and remove the index/follow instruction.

[tedster]

I couldn't find the thread, but I remember that Brett once mentioned this. Inktomi's default for all their spidering is index/nofollow. So an index/follow meta can help get some deeper crawls into your site from Ink.

[engine]

Interesting, so, if you don't have follow, how does it index the site? Or is it limited to only x number of pages indexing?

[tedster]

If you don't specify "follow", I imagine that you need to submit each page you want spidered. The site would be theme ranked according to however many pages you let the spider have.

I admit my conclusion involves some guesswork. But I was struggling to get position in Ink for one client who didn't have "follow" meta tags. Ink never crawled beyond page 1 of the site - until I read Brett's comment and then put the meta's up.

[Brett_Tabke]

It will still spider the whole site, but I am convinced it does it faster with index follow.

[blade2050]

Hmm...

I think I'll try it some weeks, submitting the site sometimes to be spidered.
The log shows some IP adresses, but I did'nt find them at Searchengineworld.com. Are there some spiders using NT5 IE4 compatible? Is there a NT5 using IE4 ;)?
At the end of these log lines I found the words "Nikoma" and "DigEx"!

Was it a failure, to give the site a "nofollow" tag?

Kevin

[engine]

Tedster, I've never used the follow/index on any sites, except a link page. All sites have been indexed satisfactorily, so far.
It is interesting that it may speed up indexing, Brett. Saving a lttle bit of code is not worth it if it brings faster indexing.

[lizzie]

What about the TAG "revisit after X days?"

[engine]

Hi Lizzie,
That tag has been around for a long time and there is some discussion here [webmasterworld.com]