homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / WebmasterWorld / New To Web Development
Forum Library, Charter, Moderators: brotherhood of lan & mack

New To Web Development Forum

Why are Spiders only indexing my home page?
How do spiders work?

 2:31 pm on Feb 13, 2004 (gmt 0)

Hello everyone,

I am relatively new to the world of website development, but I want to learn how it all works. I hope you can bear with me if my questions are a bit 'second grade'.

I run my own business and I want to be in control of what is happening. Therefore, I would rather take my time and learn about web developing than just let some geek build my site. (apologies to any geeks reading)

Anyway, as I am only learning the basics for now I would really appreciate it if someone could tell me why Google spiders keep visiting my home page but fail to follow any links to any other page within the site.

I have a basic html text link from the home page to my site map, and on that page are further text links to 25 other pages. So far, not one other page has been found by a spider.

Educate me.



 6:14 pm on Feb 13, 2004 (gmt 0)

Check your logs to see if the spiders have even visited the other pages in your site.


 6:31 pm on Feb 13, 2004 (gmt 0)

No they haven't visited the pages, this is my concern. I was under the impression that they would find all the site pages from links on the main page. However, the log shows google regularly coming to my main page, then high tailing it home without looking elsewhere.

I have submitted all my pages to google one at a time as they were built (not all at once though, one per day) hoping that they would come and find them, but so far no joy.

I don't mind waiting if that is all I have to do, so long as I am not missing something basic. If all is well then I will just leave these spider things to get on with it.

They must be very busy and overworked.



 6:41 pm on Feb 13, 2004 (gmt 0)

I think it might just be a case of waiting, 2-4 weeks and google usually seems (in my experience) to get 2nd level pages of reasonably well linked sites.

It's a good move having a site map that google can find easily, make sure your URL's are spider friendly i.e. clean HTML links with 2 or less parameters in them (if you pass parameters between pages).

You could perhaps speed it up by getting links to your internal pages from other sites....

and if it hasn't been said already welcome to WebmasterWorld :)


 7:13 pm on Feb 13, 2004 (gmt 0)

It hasn't been said, so thank you for saying it :)


 11:20 pm on Feb 13, 2004 (gmt 0)


1. When did you submit your site to Google?
2. How many links from sites you don't own are linking to you?
3. Is your site in any major directories?
4. How much content do you have on those additional pages?
5. Why are you only linking to those pages from a site map and not your home page?
6. What is your home page like?

>but I want to learn how it all works

Glad you are learning this stuff yourself, it will pay off.


 11:59 pm on Feb 13, 2004 (gmt 0)


Thanks for taking so much trouble to answer my questions. I do appreciate it. :)

I will respond to your points;

1. When did you submit your site to Google?

Site was submitted in December last year and the home page was first spidered on 31st Dec. 2003

2. How many links from sites you don't own are linking to you?

Six links at present. However, when I do a link check (link:*****) it shows no results. Not sure why.

3. Is your site in any major directories?


4. How much content do you have on those additional pages?

Loads. I am including good rich content to build trust with any visitor.

5. Why are you only linking to those pages from a site map and not your home page?

I am not, I have links on the home page too. I just built the site map because I thought it might help.

6. What is your home page like?

Sorry, I am not quite sure what you are asking here James, but if you have a spare moment take a look, ****

Thanks for all the help so far.


[edited by: JamesR at 8:44 pm (utc) on Feb. 16, 2004]
[edit reason] sorry, no personal URLs per Terms of Service [/edit]


 2:38 am on Feb 14, 2004 (gmt 0)

6 links is not very many incomming links. Google only shows links to your site if those sites have a PR of 4 or higher. But saying that all of your pages should have been indexed by this time.

Submit to as many major directories as you can. Dmoz and yahoo are the biggest.

Seeing that you are fairly new to web design as you said your pages may contain many html errors. These can unsettle robots (spiders) and they will leave your site instead of going on their merry way. Try running your pages through a html validator.

I guess that all web designers are geeks in the old terms...... and you are aiming to become one. I think that computers are so common now that the word geek has very little meaning.

Just an example of what you need to aim for. The #1 position on google that you'd be competing against has 761 incomming links from PR4+ pages


 11:20 am on Feb 14, 2004 (gmt 0)

Thanks for the feedback on this topic. It is always good when people share information and help us newcomers to climb the ladder.

I will run a html validator, that's something I hadn't thought about so thanks for that.

I am dedicating some time each day to reading all the archive articles in the newbies section. I can gain a lot of tips in those old postings and then maybe I won't need to ask 'what have I done wrong?', because I will learn how to 'do it right' from the beginning.

At least that is the theory, but it's all a learning curve.

With regards to my use of the work 'geek', I apologise if it was taken disparagingly. I was just trying to break the ice with a tongue in cheek comment;

"just let some geek build my site. (apologies to any geeks reading) "

However, I tend to mix mainly with (non computer) business people and the word geek does still tend to be used a lot when referring to web designers, software engineers, IT people, etc..., but I do take your point.

If it sounds a bit 'old hat' then you must forgive me for my naivety.



 8:35 pm on Feb 14, 2004 (gmt 0)

I know a number of people who could build you a nice looking site, but as they have ZERO understanding of search engines the site would never do very well.

In the long term you might be better off doing it all yourself. Valid code, with good use of title tags, meta description, and headings properly used, are a good start. Build more incoming links from diverse (ownership and location) but on-topic sites, and see what happens.


 5:42 pm on Feb 15, 2004 (gmt 0)

Hello Gary

First, Google is updating at the moment AGAIN. they just went through 2 updates the "Florida" and "Austin" this new updated from what I see is being called "Brandy". But there are changes coming in the search engine business.

Google is planning on going public, in which case Yahoo will drop then and get their results from the companies that have been purchasing (Overture and Inktomi). If Yahoo kicks Google, then it will also kick Looksmart Australia over the fence from supplying paid sponsored results to them. Yahoo, owning Overture, will utilize Overture results in Australia very soon as they already do in most other countries.

The moral, the search engine world will be going through changes pretty rapidly. Like I said above Google has had 3 updates happing pretty close to each other. What I tell my clients is that, you are running a business. It isn't what your PR is that makes you money, its what you do with the visitors after they are in your site.

If you have a people outside pulling visitors into your brick and mortar retail store, but no salesmen inside how much money you think you will make?



 6:21 pm on Feb 15, 2004 (gmt 0)

First of all you need a lot more quality links relating to your web site. Currently you have none that google find really interesting.

Secondly you should use some text links on your first page that are clean html pointing to most of your main pages with in the web site.

Thirdly which is something that is strange, i know your robots.txt and error pages are providing a proper 404 error message however in Google we find pages indexed with


This -download.net/ at the end is very strange but because your on an Apache server perhaps their is a share hosting type thing that will be related to the domain name, but what is puzzeling they both have completely different IP addresses.

I don't think you will have a problem you just need more backlink pointing to your web site and text links on your main page.

Can't really see anything else a miss.

Hope it helps and Good luck sometimes it takes time but you got to get a link from a PR4 web site that will get the ball rolling...


 12:13 am on Feb 16, 2004 (gmt 0)

Thanks lasko, all good advice.

With regards to the download site I have opened a new thread. This point is a concern to me I must admit.



 8:47 pm on Feb 16, 2004 (gmt 0)

If there are any niche directories or info sites that categorize translation services, I would think about getting links from those.

Also, consider swapping links with other, non-competing translation service sites that are already indexed.

Global Options:
 top home search open messages active posts  

Home / Forums Index / WebmasterWorld / New To Web Development
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved