Comments on: Robots.txt & Duplicate Content http://sharkseo.com/whitehat/robots-dot-text/ Black, White, Grey. Whatever. Thu, 25 Apr 2013 09:38:23 +0000 hourly 1 http://wordpress.org/?v=3.4.1 By: san kay http://sharkseo.com/whitehat/robots-dot-text/comment-page-1/#comment-2618 san kay Thu, 18 Apr 2013 10:13:21 +0000 http://sharkseo.com/?p=1222#comment-2618 Ya This file is very important for indexing website in google Ya This file is very important for indexing website in google

]]> By: neil http://sharkseo.com/whitehat/robots-dot-text/comment-page-1/#comment-2498 neil Mon, 12 Sep 2011 16:40:41 +0000 http://sharkseo.com/?p=1222#comment-2498 I use the robots.txt file to dissallow all when a website is in developement stages then allow all when the website has gone live, is this ok ? Best Wishes, Neil. I use the robots.txt file to dissallow all when a website is in developement stages then allow all when the website has gone live, is this ok ?

Best Wishes,
Neil.

]]> By: martin ray http://sharkseo.com/whitehat/robots-dot-text/comment-page-1/#comment-2415 martin ray Tue, 31 May 2011 08:41:55 +0000 http://sharkseo.com/?p=1222#comment-2415 Can anyone explain all the queries used in this robots.txt file...? Can anyone explain all the queries used in this robots.txt file…?

]]> By: Jeremy Referencement http://sharkseo.com/whitehat/robots-dot-text/comment-page-1/#comment-2402 Jeremy Referencement Wed, 20 Apr 2011 03:49:25 +0000 http://sharkseo.com/?p=1222#comment-2402 Totally makes sense mate! Robots.txt is a very powerful file and the old pages redirection gets messed up so much times with a client. Totally makes sense mate!

Robots.txt is a very powerful file and the old pages redirection gets messed up so much times with a client.

]]> By: admin http://sharkseo.com/whitehat/robots-dot-text/comment-page-1/#comment-2400 admin Tue, 19 Apr 2011 15:00:10 +0000 http://sharkseo.com/?p=1222#comment-2400 @Tom - I've seen it happen on tracking parameters before quite a lot too. The problem is, to a web dev it sounds quite logical on the surface, but in reality it's actually pretty harmful. @Malcolm - I actually did steal the idea from you after seeing yours! @Tom – I’ve seen it happen on tracking parameters before quite a lot too. The problem is, to a web dev it sounds quite logical on the surface, but in reality it’s actually pretty harmful.

@Malcolm – I actually did steal the idea from you after seeing yours!

]]> By: malcolm coles http://sharkseo.com/whitehat/robots-dot-text/comment-page-1/#comment-2399 malcolm coles Tue, 19 Apr 2011 14:41:04 +0000 http://sharkseo.com/?p=1222#comment-2399 I see you paid homage to http://malcolmcoles.co.uk/robots.txt with http://sharkseo.com/robots.txt ...! I see you paid homage to http://malcolmcoles.co.uk/robots.txt with http://sharkseo.com/robots.txt …!

]]> By: Tom http://sharkseo.com/whitehat/robots-dot-text/comment-page-1/#comment-2398 Tom Tue, 19 Apr 2011 13:23:58 +0000 http://sharkseo.com/?p=1222#comment-2398 Just had this discussion internally last week after tracking parameters were added to nav links - blocking content using Robots.txt should not be taken lightly! Just had this discussion internally last week after tracking parameters were added to nav links – blocking content using Robots.txt should not be taken lightly!

]]>