Welcome to WebmasterWorld Guest from 126.96.36.199
I've got a problem with a fairly new site I've made.
It's got a few good backlinks, and there are more coming.
The 'bots have all been around to say hello, but they only seem to be grabbing / and robots.txt
Googlebot did crawl a few more pages the other day (5 or 6) but nothing else.
My listing hasn't appeared in MSN at all yet.
It's in Google, but it's currently URL only (it was URL + description last night! and URL only a few days before).
I was wondering if someone could spare 2 mins to have a look at my robots.txt or my page and let me know I'm not doing something wrong.
Stickymail me for the URL, or let me know some suggestions/thoughts if you have any.
Thanks a million
Thanks for your feedback. The robots.txt validates ok - I'd checked that before posting.
The site is built with a popular CMS, so I'm assuming the internal linking is ok - the site works ok!
Everything is mod-rewritten to be search engine friendly (except a few sections which I can't adjust, but they aren't major.)
Anything else I should check?
it seems like you've covered all the basic bases so i would concentrate on building new content each day and getting relevant links.
check out Brett's classic 26 steps to website success for other things you can be doing.
patience is a virtue ;)
Just seemed a bit odd, as most of the other sites I've done get deep crawled very quickly.
I guess it's just the law of the Google!
I noticed Teoma came by last night and took a page that wasn't the index, so some spidering is happening.
Does it usually only take the robots.txt and one page?
(I've never concentrated on it much before)
I'm getting hit every day without fail by Googlebot for my robots.txt and /
Last week Googlebot (a different one, with Mozilla 5 compatible agent in Urchin) came by and deep crawled. This was last Thurs.
In Google, I'm still just showing my homepage.
I'm also being hit by Slurp from Yahoo! - In fact, it's gobbling me up - It grabs a large chunk of pages daily and did a huge hit a few days ago (more pages than I have, so it must have gone a bit haywire)
In Yahoo! I have 2 entries if I search for #*$!.com and press 'More results from this site' and both are my homepage - one with my Meta description and one with my content snippet.
Oh, and same story from MSNBot - it loves spidering me but won't give me anything on the SERPS.
I'm also being slowly grabbed by Teoma, but I have no idea how to check if I'm in the SERPS there.
So, I'm still not appearing in search results - I know it's a finger in the air job to predict timescales, but does it sound like something isn't quite right?