Site not indexed?

[kingbottler]

Hi,

I'm looking for some help. Our site recently just fell right out the rankings with the homepage missing for our brand search which was previously number 1.
What is the best most effective way of ensuring site is indexed? I think we've done everything


robots.txt looks like this...

Sitemap: http://www.example.com/sitemap-index.xml
User-agent: EtaoSpider
Disallow: /
User-agent: baiduspider
Disallow: /
User-agent: *
Crawl-delay: 2
Allow: /
Disallow: /ajax/
Disallow: /basket/
Disallow: /checkout/
Disallow: /errors/
Disallow: /quickbuy/
Disallow: /product_personalisation/
Disallow: /products/
Disallow: /register/
Disallow: /register
Disallow: /vs/
Disallow: /images/products/yofla/
Disallow: /customer_groups/choose_group/
Disallow: /formbuilder/
Disallow: /callbacks/

To me, that looks ok perhaps someone might check.
Have submitted the sitemap through search console
Have used Fetch on sitemap and on homepage.

What else can I check?

Many Thanks for any help given!

KB

[edited by: engine at 10:23 am (utc) on Mar 10, 2017]
[edit reason] please use example.com [/edit]

[lucy24]

robots.txt looks like this...

I hope the missing blank lines are just an artifact of cut-and-paste, since blank lines have syntactic meaning in robots.txt.

In any case it should not make any difference, because indexing and robots.txt are different things. At worst, if your site was indexed for a long time and then you accidentally changed something in robots.txt that prevented search engine crawling, you'd get a search snippet saying something about "The site's robots.txt prevents blahblah". (Google and Bing have near-identical lines. Don't know about other search engines.)

There are many unanswered questions.

Is Google still crawling your site? When they visit, do they request files other than robots.txt? Do these requests receive a 200/304 response?

If you do an exact-text search for some phrase from your site, does it come up?

What are you not telling us? There's always something.

[not2easy]

Could just be like webmasters around the world are noticing here: [webmasterworld.com...] - the latest update has made major changes. If your site was getting traffic from images, that also recently went away across many countries. 40% (or more) drops seem to be common right now. If you have not made recent changes to robots.txt or sitemaps, you may be seeing the effects of "Fred".

[kingbottler]

Many thanks for your replies folks. If I do a search for info:sitename.co.uk another site using our meta tags come up. Ours is not visible using this method of search

[edited by: goodroi at 8:22 pm (utc) on Mar 13, 2017]
[edit reason] fixed typo [/edit]

[phranque]

welcome to WebmasterWorld, kingbottler! [webmasterworld.com]

Our site recently just fell right out the rankings with the homepage missing for our brand search which was previously number 1

If I do a search for info:sitename.co.uk another site using our meta tags come up. Ours is not visible using this method of search

have you checked to see if your site has been hacked?
have you examined these other sites?
perhaps they have scraped your content or are using a proxy to serve your content.
have you tried "fetch as googlebot" to see if google is receiving the document you expect for a sample set of urls?

[edited by: goodroi at 8:22 pm (utc) on Mar 13, 2017]
[edit reason] fixed typo [/edit]

[kingbottler]

Hi,
Thanks for getting back. We believe we have been scraped and our content search strings and title tags are showing in SERPS for sites not related to our content.
We have also found multiple thousand links pointing to our site which has been negatively added.We have disavowed much of these. and attempted dmca takedown requests and copyright infringement requests.

However the concerning thing is our home page falling out the sky. Have checked Fetch and Render, it does render as our homepage.


Thanks again

[keyplyr]

If you haven't already done so, I would add canonical tags to the head section of all, especially the index, pages:

<link rel="canonical" href="https://www.example/page.html">

Then I would resubmit via Fetch As... or using you sitemap.xml.

[kingbottler]

Thanks for coming back on this :)

We do have <link rel="canonical" href="http://www.mysite.co.uk" /> in the head :(

[not2easy]

There are some cases where your content can be displayed on other sites using iframes or displaying your rss feed. If you are finding your content in snippets within other content, it is more likely scrapers. But if you visit a site showing your fresh content as it appears on your site, you can prevent that. See this recent discussion: [webmasterworld.com...] for some ideas if that is the case.

[kingbottler]

Thanks for this not2easy will look into this... Looks complex.
What other reasons might the main page not be indexed?

[kingbottler]

Site finally appears to be back up and has recovered all listings. No idea what is going on with G but relief is palpable.
Thanks to all who contributed to this thread

[keyplyr]

Good work :)