No; Mars has not attacked! One element of SEO work, be it in Travel SEO or elsewhere, that is often overlooked is the Robots Txt File which should be found as an extension to any home page. For instance at domain.com/robots.txt.
This file is very important as it tells Google et al what to index; i.e. add to their search results and what not to index; these could be private folders and files on the server or even a development test area.
While Searching for Cheap Cruises
The other day I was searching for Cheap Cruises; no; I don’t want a cruise; it was as part of my SEO Consultancy work for a client. Anyway if you go far enough down the serps and I mean a long way down you will come across two of websites that have “A description for this result is not available because of this site’s robots.txt – learn more.”
What this means is the websites have the wrong mark up in their robots.txt file that is telling Google not to index the pages. Saying that; the pages are in the serps but lack the snippet seen below the web page title that helps attract visitors. I will just say that Google does sometimes ignore the robots file and also that the two websites I mention have different robot files.
User-agent: *
Disallow: /
What this does is tells Google et al not to index the website; the mistake is the trailing slash saying you are disallowed from indexing anything. It should be written like this:
User-agent: *
Disallow:
If they had wanted to block Google from certain parts of their server/website then it could be written like this:
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/
Should they make a little adjustment then they may well move up the rankings; but not far as there are many other issues!
Since writing this, 1st4Cruising have changed their robots file and are now on page one in the UK for cheap cruises
User-agent: *
Allow: /cruises/promos/1st4cruising/
Disallow: /
Now their website automatically redirects the home page to (www.)1st4cruising(.com/cruises/promos/1st4cruising/index.jsp?cobrand=85) so technically the robots txt file is correct as it is disallowing Google et al to index other pages other than those after the /1st4cruising bit. BUT the home page is the most important page on any website; why are they not using it? Google wants to show it; we know this as it is always in the search results.
I guess the reason for the redrect is that they are using some form of Content Distribution Network (CDN) or cloud hosting that is set up incorrectly. Anyway they are loosing out on selling cheap cruises because of it!
I mentioned on twitter (from my personal account) a while back that their robots file was wrong; I am sure it is currently different from then however they are hindering themselves.
Perhaps I am wrong to point out Poor SEO to my former clients competitors; but then I love a challenge and to be honest; both websites have other issues. But there is a moral to this; make sure that your Robots don’t block the selling of Cheap Cruises!