Welcome to WebmasterWorld Guest from

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Effect of Staging Environment Indexing + Search Console Stats

10:11 am on Sep 11, 2017 (gmt 0)

New User

joined:July 5, 2017
posts: 29
votes: 1

1. Development site was indexed by accident before launch causing numerous indexing issues (see old thread on this - [webmasterworld.com...] )
2. I spent the whole of July cleaning up the mess, undertaking all the 301s, 410s and working with the R+D department on the CMS which was creating funky URLs

Situation Now
1. Site now has a clean index minus 2 or 3 URLs. 95% of the real HTML URLs (around 130) are indexed
2. No of pages crawled per day over the last 90 days is 465, but this is inflated by some deep crawls when I requested indexing of pages. Real average is around 200.
- The average pages crawled per day of the development/staging site was 20
- But a PPC site in my company with minimal SEO attention gets 600 pages crawled per day, (its a 2 year old site)
3. Organic traffic is minimal but i'm operating in a very competitive affiliate niche and its only been 2 months since launch

1. Is the pages crawled per day low? And thus crawl budget low? (there not being any crawl rate limit issues now)
2. To what extent is the indexing of the development site, i.e. a site which wasn't launch ready, with numerous 'bad' URLs, and which was kind of static, dead for a couple of month impeding the site's progress now? (From Google's perspective there was this poor site online for 2-3 months where nothing was happening, it would come and do a thin crawl every day, see nothing is changed and leave and then in July there's suddenly all this on and off site activity)

[edited by: goodroi at 11:55 am (utc) on Sep 11, 2017]
[edit reason] fixed url [/edit]

3:54 am on Sept 12, 2017 (gmt 0)

New User

joined:July 13, 2017
posts: 3
votes: 0

Crawl Budget and thus Pages crawl per day completely depends upon how frequently page content is being updated.
Over the time, if crawler does not see much changes it will reduce the crawl budget
95% is good indexable rate, for rest, you have to identify them and make sure that they are navigable from the high priority pages
10:58 am on Sept 12, 2017 (gmt 0)

New User

joined:July 5, 2017
posts: 29
votes: 1

thanks, yes I know that updating my content will have an impact on crawl budget.
At the moment I'm wondering what impact having the indexing of the development site with all the technical problems not fixed is impacting the site's crawl budget?
9:06 am on Sept 14, 2017 (gmt 0)

New User

joined:July 5, 2017
posts: 29
votes: 1

Any one else can help with this?
Bumping this thread, think it contains 2 important SEO questions

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members