homepage Welcome to WebmasterWorld Guest from 54.161.202.234
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Website not showing in any search engine
jesstp




msg:4294008
 4:23 am on Apr 8, 2011 (gmt 0)

Hi,

I have recently built a website and published it a week ago. I also put the google code into the HTML, which I have now removed and added as an external file.

It is not showing up in any search engines, even when I search the URL.

I have never had this problem before and really not sure what is going on, could it be a problem with the host provider?

Thanks

 

Leosghost




msg:4294067
 10:18 am on Apr 8, 2011 (gmt 0)

I also put the google code into the HTML,

What do you mean by
the google code into the HTML

What
google code

where was it .where is it now ..how is it "called".

jesstp




msg:4294073
 10:44 am on Apr 8, 2011 (gmt 0)

In google webmaster it asks to verify the site through code in the meta tag. I tried this, inside the <head> tags and as this didn't work I deleted the meta tag and upload the HTML file instead. This was only done today, but I don't get a result when searching for the URL.

Leosghost




msg:4294080
 10:58 am on Apr 8, 2011 (gmt 0)

Do your logs say if any search engines have actually crawled the pages of the site ? and if so ? when ?..Lack of, or errors in the implementation of G's "verify" would not keep you out of other search engines.

jesstp




msg:4294114
 12:21 pm on Apr 8, 2011 (gmt 0)

Don't have any access to logs from the web host, but have just looked at analytics & only have my direct visits. On google webmaster it just keeps coming up with no data available.

Also doesn't come up on page ranking, when looking at google, bing or yahoo.

Leosghost




msg:4294160
 1:51 pm on Apr 8, 2011 (gmt 0)

What does your "robots.txt" look like ? just in case you've inadvertently banned all the search engines :)

scooterdude




msg:4294164
 1:59 pm on Apr 8, 2011 (gmt 0)

it takes a lot longer nowadays for google an others to index most new sites, unless you've got a lot of indexed sites linking to your new site

seeing as you've got gwmt, a decent sitemap often helps

pageoneresults




msg:4294169
 2:03 pm on Apr 8, 2011 (gmt 0)

What does your "robots.txt" look like ? just in case you've inadvertently banned all the search engines.


If the OP were doing site: searches, the site would normally show up as a URI only listing along with any other URIs Googlebot was able to detect from the robots.txt file.

I don't see that the OP have mentioned doing any site: searches so the above could be the cause.

I'd also look at your <head></head> and make sure there are no metadata directives for noindex.

Check the Server Headers of your pages and make sure they are returning the proper HTTP Status Codes, in this instance, 200 OK is what you want to see for pages you want indexed.

I have recently built a website and published it a week ago.


Most likely a case of not enough time. Patience Grasshopper. :)

Leosghost




msg:4294172
 2:08 pm on Apr 8, 2011 (gmt 0)

it takes a lot longer nowadays for google an others to index most new sites, unless you've got a lot of indexed sites linking to your new site

Huh?

I get a new site listed in under 48 hours with just one inbound from a an old PR 5 I keep hanging around just for that..and even if I don't link in ( while playing around with dev, if I forget to ban them or "protect" the site within the first 24/48 hours anyway G and the others have always been by when I check the logs )..getting listed is easy ..getting listed for what you want where you want, and staying there, takes a wee bit more effort ..sometimes ;-)

jesstp




msg:4294486
 2:37 am on Apr 9, 2011 (gmt 0)

Ok, maybe a dumb question, but I don't have a robot.txt. Is that the problem?

phranque




msg:4294537
 8:02 am on Apr 9, 2011 (gmt 0)

no robots.txt makes everything free game so that's good for now.
do you have a .htaccess file?
do you have access to your server access log file?
i would start looking there for clues.

jesstp




msg:4296884
 1:34 pm on Apr 13, 2011 (gmt 0)

Ok, so I have called the hosting company & they said that I need to upload the google dns code into the "zone" and go to google for help. I called back to ask where the "zone" is to upload the code and I was asked to put it in an email. So I asked how putting a google code would index me on bing & yahoo.

The reply I received was: For your domain to be indexed with the different search engines, you will need to index it with each search engine.

Hmm, I've not heard this before and I have always been under the impressions that if you type in the URL in the search field the site should come up!

I have checked the logs and there is no activity.

phranque




msg:4297410
 6:56 am on Apr 14, 2011 (gmt 0)

"google dns code" in your zone file only applies if google provides your hosting (as in google apps or gmail)

For your domain to be indexed with the different search engines, you will need to index it with each search engine.

that sounds like someone is completely clueless.

have you tried the site:example.com search for your domain?

when you verify a site for GWT using the HTML file method you are supposed to click the link to test for the file existence.
what happens when you click that link?

jesstp




msg:4297430
 8:10 am on Apr 14, 2011 (gmt 0)

When I search site:mysite I receive the following result:
Your search did not match any documents.

It says that I'm verified & in the dashboard it says no data available!

Thanks all for your help! I love webmasterworld! I tried explaining the problems to the hosting company and they talked to me like I was an idiot & had no idea what I was talking about!

phranque




msg:4297460
 9:33 am on Apr 14, 2011 (gmt 0)

ok i didn't realized you had been verified.

what happens when you "fetch as googlebot" in GWT?

jesstp




msg:4297478
 10:54 am on Apr 14, 2011 (gmt 0)

Status is successful.

scooterdude




msg:4297514
 12:55 pm on Apr 14, 2011 (gmt 0)

Have you recieved any inbound links, and have the pages linking to you appearing in the search engines?

have you submitted sitemaps to google/yahoo/bing ?

jesstp




msg:4298164
 10:05 am on Apr 15, 2011 (gmt 0)

No inbound links and I haven't done a sitemap. Would it help to do one?

lucy24




msg:4298763
 3:50 am on Apr 16, 2011 (gmt 0)

If you are only just setting up the site, a sitemap-- assuming you mean the sitemap.xml file, not a "sitemap" page visible to humans-- may well be worse than useless. (Voice Of Experience.) When you move things around and submit a fresh sitemap, google doesn't purge the information from the old one. They keep right on trying to crawl those nonexistent pages. So you get a bunch of spurious "crawl errors" that you have to waste time investigating.

If everything on your site can be reached by following links from the front page, a sitemap shouldn't be necessary at all. Unless you have a large Russian audience. Yandex simply adores my sitemap; it's their first stop on every visit.

scooterdude




msg:4298851
 10:39 am on Apr 16, 2011 (gmt 0)

my 2 cents

submit your site about 2 highly regarded directories

beg a couple of your website owning friends for a couple of links from related sites

submit a sitemap to google/yahoo/bing if you're confident that your site is complete and no major changes are imminent

jesstp




msg:4299418
 3:20 pm on Apr 17, 2011 (gmt 0)

I have built his site for someone who is a wholesaler. His product is already being sold by other companies on the web, so I'm not sure how he would get links from sites as they would want customers to buy from them.

What does it mean that no results came up on the site:mysite result?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved