Strange listing issues

Can someone spare 2 mins to help me?

[badass101]

Hi all,

I've got a problem with a fairly new site I've made.
It's got a few good backlinks, and there are more coming.
The 'bots have all been around to say hello, but they only seem to be grabbing / and robots.txt
Googlebot did crawl a few more pages the other day (5 or 6) but nothing else.

My listing hasn't appeared in MSN at all yet.
It's in Google, but it's currently URL only (it was URL + description last night! and URL only a few days before).

I was wondering if someone could spare 2 mins to have a look at my robots.txt or my page and let me know I'm not doing something wrong.

Stickymail me for the URL, or let me know some suggestions/thoughts if you have any.

Thanks a million

Craig

[MHes]

Hi

Try this:

[searchengineworld.com...]

[jschmitz]

You need to check your pages, not just your robots.txt... and sometimes they come, sniff, and then come back. Robots.txt doesnt really guarantee anything in particular....some sites that I have put up have done well right away without it, and some with it take quite a will to get crawled deeply. Still, you shoudl be showing some ranking if your internal links are good and your pages are coded well.

j

[badass101]

Hi guys,

Thanks for your feedback. The robots.txt validates ok - I'd checked that before posting.
The site is built with a popular CMS, so I'm assuming the internal linking is ok - the site works ok!
Everything is mod-rewritten to be search engine friendly (except a few sections which I can't adjust, but they aren't major.)

Anything else I should check?

Thanks

Craig

[kpaul]

how new is fairly new?

it seems like you've covered all the basic bases so i would concentrate on building new content each day and getting relevant links.

check out Brett's classic 26 steps to website success for other things you can be doing.

patience is a virtue ;)

[shri]

It can take upto a month or more for the first deepcrawl to start.

[badass101]

I should try and be more patient, but you all know what it's like! ;-)

Just seemed a bit odd, as most of the other sites I've done get deep crawled very quickly.
I guess it's just the law of the Google!

I noticed Teoma came by last night and took a page that wasn't the index, so some spidering is happening.
Does it usually only take the robots.txt and one page?
(I've never concentrated on it much before)

Thanks

Craig

[badass101]

Ok, here's the update for you all - with a few questions!

I'm getting hit every day without fail by Googlebot for my robots.txt and /
Last week Googlebot (a different one, with Mozilla 5 compatible agent in Urchin) came by and deep crawled. This was last Thurs.
In Google, I'm still just showing my homepage.

I'm also being hit by Slurp from Yahoo! - In fact, it's gobbling me up - It grabs a large chunk of pages daily and did a huge hit a few days ago (more pages than I have, so it must have gone a bit haywire)
In Yahoo! I have 2 entries if I search for #*$!.com and press 'More results from this site' and both are my homepage - one with my Meta description and one with my content snippet.

Oh, and same story from MSNBot - it loves spidering me but won't give me anything on the SERPS.

I'm also being slowly grabbed by Teoma, but I have no idea how to check if I'm in the SERPS there.

So, I'm still not appearing in search results - I know it's a finger in the air job to predict timescales, but does it sound like something isn't quite right?

Craig

[jdhuk]

Hey Craig,

Just a thought, is the domain brand new never used before?

[The Combsinator]

I went to the link that Mhes just posted and every file said Error..I'm thinking that isn't good :(.

[badass101]

Hey Craig,

Just a thought, is the domain brand new never used before?

As far as I know - what's the most accurate way to check?

[badass101]

I went to the link that Mhes just posted and every file said Error..I'm thinking that isn't good :(.

Huh?
Sorry, I don't understand what you mean... (me being stupid!)

[jdhuk]

As far as I know - what's the most accurate way to check?

You can check if your domain was previously owned here....

[archive.org ]

[badass101]

Thats the way I knew - no, nothing comes up on a search there.