Part of the site isnt spidered

Forum Moderators: DixonJones

Message Too Old, No Replies

Part of the site isnt spidered

Need some advice

snuif

8:20 am on Jan 20, 2004 (gmt 0)

A part of my site isnt spidered can some one tell me wht the reason might be

the url is [****xx.nl...] (oppss my mistake changed it)and the scripting is

<head>

<title>Massaal protest tegen verbod op hoofddoek in Frankrijk :: Airtel</title>

</head>

does anyone have a clue for this, the rest does get spidered well but this part isnt.If have to add more info let me know wht to add im sort of new to this.:)

thks for your help in advanced :)

[edited by: snuif at 8:43 am (utc) on Jan. 20, 2004]

bill

8:38 am on Jan 20, 2004 (gmt 0)

Welcome to WebmasterWorld snuif

You should remove the URL...it's against our TOS

Could you tell us what spider you're using? I can't see anything in your example that would stop a spider.

snuif

8:47 am on Jan 20, 2004 (gmt 0)

well im using robot text editor to track spiders,with my daily logs, im not using a spider myself.But when i try to search on certaining keywords added in my news section on days before i can not find any of the added material back on googles search engine.Its weird cause the rest does get spidered but the news directory isnt.

bill

9:05 am on Jan 20, 2004 (gmt 0)

Oh, if you're talking about Google I would suggest you visit our Google Forum [webmasterworld.com]. Essentially they will tell you that Google doesn't spider every site on the Internet, and only certain pages are spidered frequently. It can take up to several months for content to show up...it all depends on a number of factors.