How to set up the robots.txt file in Joomla

The robots.txt file is a small file located in the root folder of your Joomla site. The file contains instructions to the search engines on what to index and what to leave out.

Some people confuse the robots.txt file with the .htaccess file. The difference is significant. The first one only gives instructions to search engines - and most search engines respect it. The robots.txt file is not used for anything related to security. Nevertheless, it can have an impact on Search Engine Optimization.

The .htaccess file, on the other hand, is used to reconfigure the settings of your Apache server, redirect URLs and other server related tasks. More about that in a later post.

The robots.txt file and SEO

As mentioned, the robots.txt file is in your site root folder. It contains info on which folders should be indexed and not. I can also include information about your XML sitemap.

There are just two tips I would recommend regarding SEO and the robots.txt file:

1. Remove exclusion of images

For reasons I don't understand, the default robots.txt file in Joomla is set up to exclude your images folder. That means your images will not be indexed by Google and included in their Image Search. And that's something you would want, as it adds another level to your sites search engine visibility.

To change this, open your robots.txt file and remove the line that says:

Disallow: /images/

By removing this line, Google will start indexing your images on the next crawl of your site.

2. Add a reference to your sitemap.xml file

I've talked about the Sitemap XML file previously, in my post on How to get your Joomla site indexed in Google. If you have a sitemap.xml file (and you should have!), it will be good to include the following line in your robots.txt file:

sitemap:http://www.domain.com/sitemap.xml

Naturally, this line needs to be adjusted to fit your domain and sitemap file. In my case, I use the Xmap component to create the Sitemap XML file automatically.

So, the line looks like this for Joomlablogger.net:

sitemap:http://www.joomlablogger.net/component/option,com_xmap/lang,en/sitemap,1/view,xml/
Other than that, the robots.txt file can live happily at peace in your site's root folder.
Kristoffer Sandven, JoomlaBlogger

Kristoffer Sandven is a Joomla! and SEO consultant living in Horten, Norway. He's the founder and author of JoomlaBlogger.

Website: www.joomlablogger.net/about
Read 22028 times Last modified on Friday, 18 February 2011 09:43

Want more information like this?

Buy the 287 page e-book Blogging with Joomla - only $9!

  • Understand and configure Joomla extensions like K2 and EasyBlog
  • Optimize your web site for search engine visibility
  • Create and organize content
  • Track and analyze traffic
  • Integrate advertising, social functions and video
  • Optimizing a web site for speed and much more 
  • Great info for anyone with a Joomla website
buy
comments powered by Disqus
Powered by Joomla
Download Joomla for free