homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Marketing and Biz Dev / General Search Engine Marketing Issues
Forum Library, Charter, Moderators: mademetop

General Search Engine Marketing Issues Forum

Pages not being crawled!
crawling pages

 5:40 pm on Apr 16, 2004 (gmt 0)

Hi All,

A small problem.....

As my clients are not willing to opt fot the new Overture Site Match it is up to me to get the pages crawled for free.

Although some of the pages have been crawled and appear high in the SERPS it is apparent that other pages (on the same directory level) are not.

I have added sitemaps and crosssed linked each page but they still do not get indexed.

Any ideas how to 'persuade' the SE's to crawl the additional pages?

Many thanks in advance, hope you all have a wonderful weekend.



 6:22 pm on Apr 16, 2004 (gmt 0)

Assuming you have enough PR to go around, you can just create a "What's New" link on your home page and link to these sites. Presumably, they are already worked into the site map, and the site is not too large for the amount of PR it has.

There is a "Handy Dandy" guide in the Supporters Forum [webmasterworld.com] that ThomasB posted that gives a great breakdown of how much PR you need to get X pages crawled. It's a rough estimate, but a good approximation for a site with evenly distributed PR.


 12:17 pm on Apr 17, 2004 (gmt 0)

do some bloging for these pages you want to be crawled with the sites/forums crawled ever 24-36 hours.


 2:49 pm on Apr 20, 2004 (gmt 0)

If a page with PR 5 or above offers a link to your client's site's homepage, it helps to fasten the process.


 2:59 pm on Apr 22, 2004 (gmt 0)

Indexing of your pages won't mean that they will appear in SERPs. Google may index you page - but it is probably that it won't be shown in search results for several weeks...


 3:18 pm on Apr 22, 2004 (gmt 0)

Thanks all,

I identify whether a page is indexed by searching for the url (i.e. www.mydomain.com/specificpage.htm)

Thats right isn't it!


 4:13 pm on Apr 22, 2004 (gmt 0)

I have a much easier time getting pages in any search engine when I get multiple inbound links from different sites that are already in that engine index.


 5:30 pm on Apr 22, 2004 (gmt 0)

Cheers graywolf,

I'd love to hear how you are so succesful in getting those links..do you know something we don't?

To me, link building is a very time consuming task and often a very unsucessful one!


 6:15 pm on Apr 22, 2004 (gmt 0)

I'm nowhere near the link building casanova martinibuster is, but almost all of my linkbuilding tricks I learned here.

I use the searches mentioned in this thread on Y! and G


In fact I'm so lazy I wrote ASP script where I just put the keywords in and it generates the links to the pages for me. For the sites with form pages I fill those out, for the websites that don't I use methods mentioned here.


Thats should be enough to get you started.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Marketing and Biz Dev / General Search Engine Marketing Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved