Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

spidering ..please help

only my index page is there. And sometimes not

8:23 pm on Feb 11, 2003 (gmt 0)

Here is the problem, I can only find my site on the search engines if i type in my domain name. when i do that, only the index page can be found. The worst thing of all is sometimes i cant find my page anywhere, It seems like google adds me, than kicks me out again. Can someone please tell me whats going on?

[edited by: o0_cops_0o at 9:47 pm (utc) on Feb. 11, 2003]

9:12 pm on Feb 11, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member


Welcome to WebmasterWorld [webmasterworld.com]!

Be sure to read the WebmasterWorld Terms of Service (link at bottom of page), and the welcome page linked above - Personal URL drops are not allowed here, and I imagine a moderator will remove yours shortly. However, the most common reason for what you are seeing is that your site is new, has few incoming links, has not been deep-crawled yet, and only appears due to what we call Google's "freshbot". These "fresh" listings are unstable until your site gets deep-crawled and is listed in the monthly index.

The best way to get and keep Google's attention is to get plenty of incoming links from well-ranked related-topic sites using relevant link text. A listing in the Open Directory Project (dmoz.org) is a great way to get into just about all important search engines and stay there.

Do a site search (link at top of page) here on WebmasterWorld for "everflux", "freshbot", "my site dropped/disappeared", etc. for a lot more info on why you are experiencing problems.


9:20 pm on Feb 11, 2003 (gmt 0)

jdmorgan Thanks for the heads up! I edited my post and took out the link..sorry about that. Im glad i posted here because i have been reading for days now, and no where did i find an answer like yours. thanks again
9:37 pm on Feb 11, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member


Try these two links as well. There is a lot of information here on WebmasterWorld, and a lot of it is worth the time spent reading!



9:39 pm on Feb 11, 2003 (gmt 0)

10+ Year Member


You are still announcing your domain (by telling what to search for). I suggest removing that too. Ironic that you take pains to discourage people from inspecting your source code (with a kind of challenge on view source), yet you announce your domain in what is arguably one of the most dangerous places on the web for such things...

In addition to what jdMorgan said, after having looked at your code:

1. Most likely, google is only seeing your home page because it is a frameset with an empty noframes section. To rank your other pages, remove the frame set. I may be wrong, but the only justification I see for the frame set is a mis-guided attempt to hide your code from competitors. Really, you can't do it anyway. And, the cost of your attempt is to seriously harm the seo potential of your site. Google does not penetrate frame sites well. It will follow links in the noframes, but with a severe PR penalty for the child pages.

2. Your content page (main.asp) is probably not visible to google because it his blocked by the frameset, but, if it is, it may still have trouble because of the javascript magic you are attempting. If you want the page to rank well, ditch the fancy javascript tricks and present a clean page for spiders to read without triggering flags that you may be attempting seo tricks.

9:53 pm on Feb 11, 2003 (gmt 0)

Jack-straw, You visited the wrong site, im pretty sure you went to the .com where mine is the .ca Thanks for the quick responces im getting. This is my fist site and my first time on these forums. Im learning alote by reading and such. this is what i found on my stats.
Company or ISP : Google Inc. US,CA
IP Range : -
Total Visits : 107
so i know google came to my site but only found my index page.
10:10 pm on Feb 11, 2003 (gmt 0)

10+ Year Member

never mind then.
12:22 am on Feb 12, 2003 (gmt 0)

Now im more confused than ever! I have read the links you all gave me, and now knowing that that the deep spider ip is which has visited my site 107 times, It should of indexed all or at least some of my other pages. It hasent indexed anything but my index.html page. I followed all the rules of building a webpage. I dont spam, theres no frames, no picture links, what am i doing wrong?
12:25 am on Feb 12, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

How old is your site?
When was the first link from a site having PageRank 4 or higher to your site added?


12:36 am on Feb 12, 2003 (gmt 0)

The site is 2 months old, and the first link to my site that had a pr4 was about a month ago
1:29 am on Feb 12, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member


OK, in that case, take a deep breath and relax... :)

The first time your site was deep-crawled was either early January, or maybe just this week. Google has not had time to recognize your site fully, and probably has not yet seen any of your incoming links. You are probably only showing up in Google because the freshbot has picked you up and lists you for a few days at a time.

107 hits from the deepcrawler is a good sign. The results of these hits will show up in late February and not before, if the usual pattern holds.

Get more solid incoming links, work on content, and don't sweat getting banned unless you are violating one of Google's published no-no's - See their "Information for Webmasters" pages. It is not easy or common to get penalized or banned by Google. In very rare cases, new Webmasters take very bad advice and manage it, but it's rare.

After several years, I've found that I'm a little better at writing decent content, doing some server-side tricks with Apache server modules, and doing "lite" search engine optimization. The one skill I have really developed, though, is the "chill skill". It's really tough at first, but work on that one, too! :) (One of the best pieces of advice posted here by GoogleGuy was, "Go outside, enjoy the day." - Advice meant to say that the Google update was not going to start that day. I took that advice, and appreciated it.)


1:43 am on Feb 12, 2003 (gmt 0)

jdmorgan .....wheeeeeeeeew I feel much better. last coupl of months i been on this pc 24/7 and i guess time passes by very slowly when your looking for this spider to come along. Thank you and all who replied to this post. I guess ill go out side for a few minutes ;)

Featured Threads

Hot Threads This Week

Hot Threads This Month