Welcome to WebmasterWorld Guest from

Forum Moderators: mademetop

Message Too Old, No Replies

Pages not being crawled!

crawling pages



5:40 pm on Apr 16, 2004 (gmt 0)

10+ Year Member

Hi All,

A small problem.....

As my clients are not willing to opt fot the new Overture Site Match it is up to me to get the pages crawled for free.

Although some of the pages have been crawled and appear high in the SERPS it is apparent that other pages (on the same directory level) are not.

I have added sitemaps and crosssed linked each page but they still do not get indexed.

Any ideas how to 'persuade' the SE's to crawl the additional pages?

Many thanks in advance, hope you all have a wonderful weekend.


6:22 pm on Apr 16, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Assuming you have enough PR to go around, you can just create a "What's New" link on your home page and link to these sites. Presumably, they are already worked into the site map, and the site is not too large for the amount of PR it has.

There is a "Handy Dandy" guide in the Supporters Forum [webmasterworld.com] that ThomasB posted that gives a great breakdown of how much PR you need to get X pages crawled. It's a rough estimate, but a good approximation for a site with evenly distributed PR.


12:17 pm on Apr 17, 2004 (gmt 0)

10+ Year Member

do some bloging for these pages you want to be crawled with the sites/forums crawled ever 24-36 hours.


2:49 pm on Apr 20, 2004 (gmt 0)

10+ Year Member

If a page with PR 5 or above offers a link to your client's site's homepage, it helps to fasten the process.


2:59 pm on Apr 22, 2004 (gmt 0)

Indexing of your pages won't mean that they will appear in SERPs. Google may index you page - but it is probably that it won't be shown in search results for several weeks...


3:18 pm on Apr 22, 2004 (gmt 0)

10+ Year Member

Thanks all,

I identify whether a page is indexed by searching for the url (i.e. www.mydomain.com/specificpage.htm)

Thats right isn't it!


4:13 pm on Apr 22, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

I have a much easier time getting pages in any search engine when I get multiple inbound links from different sites that are already in that engine index.


5:30 pm on Apr 22, 2004 (gmt 0)

10+ Year Member

Cheers graywolf,

I'd love to hear how you are so succesful in getting those links..do you know something we don't?

To me, link building is a very time consuming task and often a very unsucessful one!


6:15 pm on Apr 22, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

I'm nowhere near the link building casanova martinibuster is, but almost all of my linkbuilding tricks I learned here.

I use the searches mentioned in this thread on Y! and G


In fact I'm so lazy I wrote ASP script where I just put the keywords in and it generates the links to the pages for me. For the sites with form pages I fill those out, for the websites that don't I use methods mentioned here.


Thats should be enough to get you started.


Featured Threads

Hot Threads This Week

Hot Threads This Month