Every time Googlebot arrives at your site, it will first check to see if you have a robots.txt file. If the robots.txt file blocks any pages, Google won’t crawl them.

For years, website owners and web developers have used the robots.txt file to block Google from accessing duplicate content. From blocking URLs that use tracking parameters, blocking the mobile or print version of sites or just to fix flaws in CMS’s, I’ve seen a lot of duplicate content blocked with robots.txt in my time.

Why blocking URLs doesn’t help

But the robots.txt file is a terrible way to deal with duplicate content. Even if you’re 301 redirecting the duplicate URL to the real one, or using the canonical tag to reference the proper URL, the robots.txt file works against you.

If you have a 301 that redirects to the proper page, but you block the old URL with robots.txt, Google isn’t allowed to crawl that page to see the 301. For example, have a look at Ebooker’s listing for ‘flights’:

The URL that’s ranking (on page 1 of Google for ‘flights’) is blocked in robots.txt. It’s got no proper snippet because Google can’t see what’s on the page, it’s had a guess at the title based on what other sites have linked to it with. And here’s the reason why Google can’t crawl that URL:

If Ebooker unblocked that URL, Google would be able to crawl it to discover the 301, and the page would most likely have a better chance of ranking higher (as it wouldn’t just appear to be a blank page to the search engines).

If you block Google from seeing a duplicate page, it’s not able to crawl it and see that it’s duplicate. If there’s a canonical tag on that page, it may as well not be there as Google won’t be able to see it. If it redirects elsewhere, Google won’t know.

If you have duplicate content, don’t block the search engines from seeing it. You’ll just prevent the links to those blocked pages from fully counting.

Flickr image from Solo.

Robots.txt & Duplicate Content is a post from: Shark SEO. Have you played The Search Game?

Patrick Altoft had an interesting tip about leaving the brand name out of the title tag – while this may not work for everyone, the idea is that for a generic keyword search (like “red widgets”), Google may display a title that’s optimised for that term. If the search term is branded, however (“Brand name”), then Google will most likely use the Dmoz title.

In a similar way, you can actually have multiple meta descriptions – potentially one for the keyword, and one for the brand name. This isn’t recommended for everyone, and I wouldn’t recommend it for many pages on your site, but it’s possible. The regular limit for meta descriptions to be displayed in full in Google is 156 characters (although I tend to stick to around 154 characters). I recently experimented with having a double length meta description – with the first snippet being designed to be well written for a generic keyword, and the second snippet written for a brand search. The full meta description for my homepage is this:

“Dave is a freelance SEO consultant, specialising in creative link building and in-depth technical site audits. To find out more, feel free to get in touch. Shark SEO is a search marketing blog with free advice on ranking your site better in Google, Bing & Yahoo. Check out the SEO blog today at SharkSEO.com.”

That’s twice the length of a regular meta description. Now check out the snippet for “freelance SEO consultant”:

And here’s the snippet for the search “Shark SEO blog”, which again returns the homepage:

When you put multiple snippets in the same meta description tag, it looks as if Google will use the snippet that’s most suitable for the query.

Flickr image by Amir K.

Multiple Meta Descriptions is a post from: Shark SEO. Have you played The Search Game?

The set-up process is very straight-forward. You first add your URL, and then sync it up with your Google Analytics account – this step is optional, you can skip the GA import and just copy and paste the keywords you want it to focus on (useful if you’re using a different analytics package to Google Analytics). The tool then runs through all of the selected keywords – the ones you’ve manually put in, or the ones that have driven traffic to the site if you’ve used the Google Analytics option – and checks your ranking for that term, along with the estimated search volume (on exact match) from Google’s keyword tool. The ranking data can take a while to be gathered, depending on how many keywords you’ve added, but the search volume appears remarkably quickly. You are then presented with your keywords, graphed with where they rank, what their rough potential is based on the expected search volume and by how much traffic they’ve actually sent you.

You’re then free to start creating categories and filters.

Categories

One of the main features is being able to group sections of your keywords into categories. For example, the default category is ‘Brand’, and automatically includes all of your brand terms. You can create your own categories if you want to further seperate things out – for example, if you’re a clothing retailer, you can create a category for things like ‘shoes’ and ‘boots’, one for ‘jeans’, and so on. As an example, the graphs used here are from a snowboard site that I own – I have categories for “brand name”, “boots” and “boards”.

The one thing that I’d really like to see is a negative option, so you could create a category that *doesn’t* include your brand terms. That change would mainly benefit sites that have a huge amount of longtail brand traffic that are using the GA import option, although it’s far from essential.

Setting up categories is ridiculously easy, and gives you the ability to quickly drill down into particular product areas (especially useful if you’re dealing with a huge data set). This is particularly useful for retail sites, as you can seperate out product types easily and quickly.

Filters

Filters allow you to manipulate your data by setting up rules – rules like “show me keywords that have a search volume greater than 1000 according to the AdWords keyword tool, that I rank at the bottom of page 1 for and that sent me at least 50 visits”. This is what that rule would look like:

Easy.

You’re then free to slice and dice the data to how you see fit – as an example, I now have a list, in order of priority, of which terms I should go after if I want to sell more snowboards:

The graph has been filtered to only show keywords that mention “boards” or “snowboards”, that have sent traffic to the site already and where I’m not ranking in the top 5.

The real beauty in this is the speed at which you can slice and dice the data. If I wanted to focus on boots, or snowboard bindings, I can be up and running within seconds. That’s especially useful if you have a large site with a lot of potential categories. The best way to see it in action, though, is to actually play with it – Richard has been kind enough to give out a coupon. At the sign up page, if you use the coupon code SHARKSEO, you can get 1 month free.

Flickr image by C.SooHoo.

Clever Keyword Research is a post from: Shark SEO. Have you played The Search Game?

Depending on which option you choose you can run the risk of splitting up your link equity, you might find that you’re not ranking as well as you should in local versions of Google and you might still find that your US based page is ranking above your local listing, no matter what you do.

I once heard a Google representative say that to properly geo-target your site, you just needed to set it up in Webmaster Tools. That was it. Google would work out the rest. From everything else I’ve seen, that’s not quite true.

For country specific versions of Google, I’ve found that it looks at a few things.

1. Your country specific top-level domain (.co.uk, .fr, etc.)
2. The country in which your site is hosted
3. The language used on your site (if applicable)
4. The location of the sites that link to you (their tlds, host countries, etc)
5. Your geo-targetted settings in Webmaster Tools

There’s a few ways you can use this to your advantage, if you’re designing an international site from scratch.

Use Country Specific TLDs

If you want to target the UK you can grab yourdomain.co.uk. If you’re targetting Germany, you can put your site up on yourdomain.de. This is arguably (although not definitely) the most effective, yet it’s often the least doable. It’s rare that you’ll be able to get the tlds in all of your brand names. Having said that, if you can grab them all, you have the freedom to host each domain in it’s own country. You’ll also, possibly, have a higher click-through rate (although that’s more speculation than proof) because it’s more likely that someone in the UK will want to click on your .co.uk, compared to a .com. The bad news, though, is that you’ll split up your link equity. If your US site gets 10,000 links, and your UK site gets 10,000 links, then it could be a slightly missed opportunity. You can interlink them, sure, but 10,001 links isn’t the same as 20,000 links.

Having said that, the links that each site gets will be more likely to be from the right country, so your .co.uk is more likely to attract links from other UK sites. This will, obviously, help the site rank well in Google UK.

Pros: Boost from having the country specific tld. You can host the site in the targetted country. Possibly higher CTR.

Cons: Hard to get all the domain names. You split up your links more.

Example: www.amazon.co.uk

Use Subdomains

Subdomains, such as uk.domain.com, fr.domain.com etc, have the bonus of letting you host each subdomain within the target country. That’s pretty cool. With each domain geo-targetted in Webmaster Tools AND hosted in that country, you should get a decent enough boost (if you deserve to rank) in country specific versions of Google. The drawback is that you’re again splitting up your link equity – but again, the links that you get naturally are more likely to be from sites in that country, which will help.

Pros: You can host subdomains in the targetted countries. It can be easier to manage.

Cons: You split up your links.

Example: uk.pokernews.com

Use Your Folder Structure

You could structure your site so that it uses folders, such as domain.com/uk. You can geo-target that folder to the UK in Webmaster Tools, but the drawback is that you can’t host the folders in seperate countries. You will, however, be able to use all of the links you get more effectively. Any links that the /uk site gets will significantly help to boost everything that’s on that domain, including the /fr site and the /de site. That’s a huge bonus.

Pros: The links you get will benefit the whole domain.

Cons: You can’t host the folders in their target countries.

Example: www.apple.com/uk

It’s very difficult to say which is the best structure to use, and in all honesty it often comes down to other things like the nature of the site itself, your resources and how much you’re prepared to pay, how your CMS works, whether you use load balancing, how you need to manage your files and things like that. But if you’re designing a new site, it’s always worth considering how you’re going to structure your site to get the most out of international search traffic.

(Flickr image from bytezh)

Building Your Site For International Traffic is a post from: Shark SEO. Have you played The Search Game?

This is a bad thing.

I discovered WordPress’ love for 302s just after I wrote the post on how to make your own URL shortener in WordPress so if you’re using this (awesome) method of shortening your own URLs, then make sure your redirects are using 301s instead of 302s. Mine weren’t, and it made Google pretty much throw up.

How Can You Tell If WordPress Is 302ing Posts?

You can either run a server header checker and search for a URL on your domain that you know will redirect – usually something like either the www or non-www version of your domain. I also highly recommend the Live HTTP Headers Firefox plugin, because it’s awesome. Run that, browse around, and it’ll show you the server headers.

How do I make WordPress 301?

As far as I’m aware there’s no way to change this in the WordPress dashboard normally, but the very nice people at Urban Giraffe have made a WordPress redirection plugin which fixes it. Activate the plugin and it’ll automatically make the 302s become 301s. It also lets you do a whole lot of other stuff, which is all useful.

Apparantly Platinum SEO Pack will let you manage redirects too, although I’ve not tried it out yet.

Why Does WordPress Do This?

I really, really don’t know. But honestly, other than this, WordPress is the best. Let’s never fight again WordPress.

(Flickr image from Eric M Martin)

What The Hell, WordPress? is a post from: Shark SEO. Have you played The Search Game?

I’ve been designing it on my smaller laptop screen so there’s still a few bugs and tweaks that I should really be making – if you have critical eyes and spot anything that’s not quite right with the site, please let me know in the comments on by tweeting @SharkSEO.

Cheers folks

Big Sharky Redesign is a post from: Shark SEO. Have you played The Search Game?

The RT @tweetmeme is pretty unneccessary. Tweetmeme are taking your content and attaching their brand name to it instead. This makes me sad.

The styling of the button annoys me because it doesn’t always fit into your theme. Sometimes you want your page to look how you want it to look, rather than have buttons tacked onto it, looking all green and out-of-place.

And the URL shortener is bad. If Google treats bit.ly in the same way as it treats TinyURL, then any links that get passed around because of @tweetmeme won’t give you any link juice. This is pretty much a crime.

But Here’s How To Fix All That

You can create your own URL shortener, exclusive to your blog. Easily, if you’re running WordPress at least. You’ll be able to have URLs that look like http://sharkseo.com/go/154. Check out the Twitter icon at the top of this post , see where that links to if you hit it.

For WordPress, create a folder within the root of your domain. Name it “go”, or something equally short. Then make a .htaccess file and include the following code:

RewriteEngine On

RewriteRule ([0-9]+) /index.php?p=$1 [R=301,L]

RewriteRule (.*) / [R=301,L]

The easiest way to make that a .htaccess file is to post that code into notepad, save it as a .txt and then upload it to the “go” folder on your server. Then rename it to .htaccess. Windows sometimes has a bit of a tantrum if you try to change it on your machine locally.

This code basically says “Whenever the URL /go/ is accessed, and the file specified is numbered, add that to index.php?p=. If it’s not a number, go to the homepage.

Put that .htaccess file in your new /go folder.

To test that your 301s are working, open your WordPress admin panel, go to Manage > Posts and then hover over one of your posts. You should see in the small link-to box a post number.

An earlier examples of linkbait post, for example, is number 190. Try going to http://www.yourdomain.com/go/190 (or whichever number of you found). It should redirect you to the relevant post (like here). That redirect is a 301, by the way, so all the link juice will be passed to the right page when people link to that URL. Hat tip to planetmike.com for revealing this simple but awesome tip.

Next, you’ll want to create and style your own Twitter button so that, when it retweets, it retweets your Twitter name instead of @techmeme. The good news is, you can use any image or text to retweet, it’s just a link to Twitter.

To do this, go into your theme editor in WordPress, open up the single.php (or whichever file you’re using for your single post theme), find the button that you want to be your Twitter button, and add an a href code around it to make it a link to Twitter. Here’s the code for the link:

<a href=”http://twitter.com/home?status=RT @yourname <?php the_title(); ?> – <?php echo get_option(‘home’); ?>/go/<?php echo $post->ID ?>”></a>

The @yourname you obviously change to whatever your Twitter username is. The <?php the_title(); ?> is simply the title of the blog post. The <?php echo get_option(‘home’); ?> is the root domain of your blog, followed by the /go/ which is your new, personalised 301ing URL shortener. The <?php echo $post->ID?> is the number of your post, so it’ll complete the URL shortened link. That will give a result, for my previous TinyURL page, as a link that looks like http://twitter.com/home?status=RT @SharkSEO TinyURL Doesn’t Pass Value – http://sharkseo.com/go/154.

And there you have it, you get to keep your brand name instead of passing out Techmeme’s. If people want to follow you from retweets, it’s easier for them. More importantly, instead of risking losing your well deserved link authority, you’ll get it all going straight to your blog. Enjoy, and don’t forget to retweet this post

(Flickr image from jenteach123)

Reclaim Your Twitter Links is a post from: Shark SEO. Have you played The Search Game?

Good luck.

Check out the SEO Search Game.

Know Your SEO? is a post from: Shark SEO. Have you played The Search Game?