SEO Hosting 101 – Don’t Leak Your Staging URLs into Google!

Your web design company has sold you an “advanced” SEO deal with your brand new website and hosting package and everything looks perfect. The site architecture, url handling, migration redirects and everything else SEO is apparently perfect. You go live, safe in the knowledge of a job well done. Months later, nothing. You ranked better before the new site went live!

What went wrong? Sometimes duplicate content problems are not at all easy to find and here’s a classic example. A group of recruitment websites leaking their staging server urls into Google’s, Yahoo’s and Live’s index. In live’s case, the problem is quite severe with more than 6000 indexed urls at the staging site hosting.hotlizard.co.uk.





The moral of the story? There are some highly important checks to carry out and fixes to implement when you’re migrating a website to a different hosting / web design agency. These are tests related to the integrity of your hosting setup rather than the usual “make sure all old urls 301 redirect” stuff.

Here we go:

1) Can bots access the site at the test url? Why not specify the IP range from which access to your test urls will be allowed by the firewall. Most likely, your office and / or your web design agency office will do. That way a crawler will be blocked straight away.

2) Have you leaked some urls already? Check by doing a site:”your-test-url” query in Google. If there’s a problem, detect the incoming user agent on the staging site URLS and redirect appropriately. If your visitor is a search engine crawler, do the right thing. Canonicalise! For example: http://ellisfairbank.hotlizard-hosting.co.uk/transportation should 301 redirect to: http://www.ellisfairbank.co.uk/transportation/

3) Insert a robots.txt file at the root of the url you’ve been testing to disallow all spiders

4) Use a noindex,nofollow in the test platform meta code. Don’t forget to change this when you migrate the live site!

5) Just to be on the safe side, you could register your test domain at Google webmaster tools and totally remove the site from Google’s index. I prefer 301s myself.

One last thing. Make sure every content url in your new site has been changed to the correct live url prior to migration. If that old test url isn’t anywhere on the site, you might be less at risk.