Forum Moderators: open

Message Too Old, No Replies

Problems with crawler page and google submission

Getting Crawler Page out of search engines

         

Hot_Nose

9:57 pm on May 14, 2003 (gmt 0)

10+ Year Member



Hello everyone. Great board. First time poster.

I've been lurking for quite some time now, learning as I go and making mistakes as well.

Website has 10K+ sku's. Crawler Page was developed.
URL looks like this:
www.root_url.com/dir3xhtml/secid=0/ subsecid=155/catid=1696/root_url.com

Before this the site was unindexable (sp?).

Prior to recent Google submission i submitted the Crawler Page to Position Tech. After submission found that PT only indexes one page. I took my chances and gave the Crawler Page <Title> and <meta> tags. Now that crawlwer page is showing up before anything else in Google (and in Yahoo) and in MSN as well.

Problem 1:
Postion tech admin gives all kinds of great keyword combos...things where we wouldn't have been found if we weren't a big amalgamation of keywords from a crawler page. And PT gives me about 20-25 clickthroughs a week.

Problem 2:
I have placed the Crawler page behind a "site map" link on the bottom of each page. Also, after reading that Google might not get to the bottom of the page, I placed it as Link on the home page logo. Not sure of the legality of this in Google's eyes.

Google's eaten up about 500 or so links. I'm beginning to show in the index but invariably the Crawler Page comes in first and much higher than the specific keyword page.

So question is do I want the crawler page in there and will it's relevant rank always mean that it comes up before the other page in the site that has specific keyword density. Or will it be surpassed as I move my way up in the Search Engines.

Thanks again for all your postings. Looking forward to membership on the board

jeremy goodrich

10:40 pm on May 12, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Welcome to WebmasterWorld.

First, let me clear up a bit of confusion in your post: position tech has *nothing* to do with Google.

They offer paid inclusion, but Google does NOT have a paid inclusion program. :) So if you got into Google, it was for free - from the changes to your site perhaps.

As to those, I'm a bit 'fuzzy' because of your references to 'crawler page' what is that, exactly?

Hot_Nose

6:34 pm on May 13, 2003 (gmt 0)

10+ Year Member



Crawler Page is a page with nothing but links on it. Usually does not have a <title> tag or <meta> tag. Just a bunch of links with url strings that have been filtered to leave out query strings.

My Crawler Page however has <title> tags and <meta> tags because of the single submission on PT.

So Google also hits this Crawler Page. Takes a few hundred of links but also takes the page itself, search_index.asp.

So I'm getting search_index.asp showing up now ahead of the other pages.

Guess I'm wanting to know how to get a page delisted?
Just take off the tags and wait until the next update?

Editor

8:57 pm on May 19, 2003 (gmt 0)



Greetings, Hot_Nose...

Wonder if we know of one another...

Theoretically, crawler page is a clear form of spam. If Jscript "browser detect" at top, G may have discovered this as a spammy tactic at best...

Real world- the crawler eventually becomes subjegated to the corresponding content page as one indent below home page.

We maintained #1 ranking for years until competitor reported a series of minor infractions on our site that added up to what appeared to be spam (since deleted, but no return to index. :-(

As the famous Squint says- "Do you feel lucky, punk? Well, do ya?" (No inference meant).

A risky tactic that targets the G algo well. The risks outweigh the benefits. Don't mess with G.