Welcome to WebmasterWorld Guest from 54.92.197.82

Forum Moderators: goodroi

Message Too Old, No Replies

Site not indexed?

     
9:17 am on Mar 10, 2017 (gmt 0)

New User

joined:Mar 10, 2017
posts: 6
votes: 0


Hi,

I'm looking for some help. Our site recently just fell right out the rankings with the homepage missing for our brand search which was previously number 1.
What is the best most effective way of ensuring site is indexed? I think we've done everything


robots.txt looks like this...

Sitemap: http://www.example.com/sitemap-index.xml
User-agent: EtaoSpider
Disallow: /
User-agent: baiduspider
Disallow: /
User-agent: *
Crawl-delay: 2
Allow: /
Disallow: /ajax/
Disallow: /basket/
Disallow: /checkout/
Disallow: /errors/
Disallow: /quickbuy/
Disallow: /product_personalisation/
Disallow: /products/
Disallow: /register/
Disallow: /register
Disallow: /vs/
Disallow: /images/products/yofla/
Disallow: /customer_groups/choose_group/
Disallow: /formbuilder/
Disallow: /callbacks/

To me, that looks ok perhaps someone might check.
Have submitted the sitemap through search console
Have used Fetch on sitemap and on homepage.

What else can I check?

Many Thanks for any help given!

KB

[edited by: engine at 10:23 am (utc) on Mar 10, 2017]
[edit reason] please use example.com [/edit]

6:40 pm on Mar 10, 2017 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:14429
votes: 576


robots.txt looks like this...

I hope the missing blank lines are just an artifact of cut-and-paste, since blank lines have syntactic meaning in robots.txt.

In any case it should not make any difference, because indexing and robots.txt are different things. At worst, if your site was indexed for a long time and then you accidentally changed something in robots.txt that prevented search engine crawling, you'd get a search snippet saying something about "The site's robots.txt prevents blahblah". (Google and Bing have near-identical lines. Don't know about other search engines.)

There are many unanswered questions.

Is Google still crawling your site? When they visit, do they request files other than robots.txt? Do these requests receive a 200/304 response?

If you do an exact-text search for some phrase from your site, does it come up?

What are you not telling us? There's always something.
10:39 pm on Mar 10, 2017 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:3561
votes: 197


Could just be like webmasters around the world are noticing here: [webmasterworld.com...] - the latest update has made major changes. If your site was getting traffic from images, that also recently went away across many countries. 40% (or more) drops seem to be common right now. If you have not made recent changes to robots.txt or sitemaps, you may be seeing the effects of "Fred".
9:44 am on Mar 13, 2017 (gmt 0)

New User

joined:Mar 10, 2017
posts: 6
votes: 0


Many thanks for your replies folks. If I do a search for info:sitename.co.uk another site using our meta tags come up. Ours is not visible using this method of search

[edited by: goodroi at 8:22 pm (utc) on Mar 13, 2017]
[edit reason] fixed typo [/edit]

10:34 am on Mar 13, 2017 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:11161
votes: 116


welcome to WebmasterWorld, kingbottler! [webmasterworld.com]

Our site recently just fell right out the rankings with the homepage missing for our brand search which was previously number 1

If I do a search for info:sitename.co.uk another site using our meta tags come up. Ours is not visible using this method of search


have you checked to see if your site has been hacked?
have you examined these other sites?
perhaps they have scraped your content or are using a proxy to serve your content.
have you tried "fetch as googlebot" to see if google is receiving the document you expect for a sample set of urls?

[edited by: goodroi at 8:22 pm (utc) on Mar 13, 2017]
[edit reason] fixed typo [/edit]

10:49 am on Mar 13, 2017 (gmt 0)

New User

joined:Mar 10, 2017
posts: 6
votes: 0


Hi,
Thanks for getting back. We believe we have been scraped and our content search strings and title tags are showing in SERPS for sites not related to our content.
We have also found multiple thousand links pointing to our site which has been negatively added.We have disavowed much of these. and attempted dmca takedown requests and copyright infringement requests.

However the concerning thing is our home page falling out the sky. Have checked Fetch and Render, it does render as our homepage.


Thanks again
12:01 pm on Mar 13, 2017 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:10642
votes: 630


If you haven't already done so, I would add canonical tags to the head section of all, especially the index, pages:

<link rel="canonical" href="https://www.example/page.html">
Then I would resubmit via Fetch As... or using you sitemap.xml.
12:19 pm on Mar 13, 2017 (gmt 0)

New User

joined:Mar 10, 2017
posts: 6
votes: 0


Thanks for coming back on this :)

We do have <link rel="canonical" href="http://www.mysite.co.uk" /> in the head :(
12:41 pm on Mar 13, 2017 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:3561
votes: 197


There are some cases where your content can be displayed on other sites using iframes or displaying your rss feed. If you are finding your content in snippets within other content, it is more likely scrapers. But if you visit a site showing your fresh content as it appears on your site, you can prevent that. See this recent discussion: [webmasterworld.com...] for some ideas if that is the case.
2:52 pm on Mar 13, 2017 (gmt 0)

New User

joined:Mar 10, 2017
posts: 6
votes: 0


Thanks for this not2easy will look into this... Looks complex.
What other reasons might the main page not be indexed?
9:57 am on Mar 14, 2017 (gmt 0)

New User

joined:Mar 10, 2017
posts: 6
votes: 0


Site finally appears to be back up and has recovered all listings. No idea what is going on with G but relief is palpable.
Thanks to all who contributed to this thread
10:11 am on Mar 14, 2017 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:10642
votes: 630


Good work :)