No monthly payments or hidden costs.
Websites from R2200 + vat
Hosting and e-mails for a year.
Websites with Hosting
from R2550 + vat
Online store. Platform allows unlimited products.
E-commerce website
from R3980 + vat
Directories, property websites and more...
Custom and advance websites
price on application
By the only website design company first on google in SA.
Website Marketing
from R2075 + vat
Hosting solutions.
Hosting, E-mails and Domains
from R37 p/m + vat

Search Engine Optimization - Robots.txt.

People are often confused by robots.txt file and what they are. They are a text file on your website that instructs a robot what to index and how to crawl through your website. Sometimes companies do not want certain web pages from their website to display on search engines, the robots.txt file can help prevent that. These text files can also save on bandwidth by excluding photos, videos, and JavaScript.

It is not a requirement to include a robot.txt file on your website but it is incredibly beneficial to an SEO campaign as it helps search engine robots to index the correct pages of a website. It does not prevent a robot from crawling your webpage, however it does instruct them on how to do so correctly. By doing so, your search engine result ranking can improve quite a bit. Correct indexing is one of the key factors of a successful SEO campaign.

Immediate Action Request

Please note: you will receive a response in 4 working hours or less.

Please complete the reCAPTCHA to help us reduce spam and to complete the form sending process.

Don't want to wait?

Click here to do an immediate purchase online.

Website Design Shop


More on Robots.txt.

Search Engine Optimization - Robots.txt

A robot.txt file is found in the main directory of your website to help the search engine robots find it. These robots are not specifically looking for these files but will obey them when they come across them in the main directory. In your robot.txt file, you can specify which pages to ignore, and this means that the search engine robot will focus solely on those files that have not been excluded.


Search Engine Optimization explained.

Keyword Assessment | Meta Data | Webmaster Tools | Google Analytics | Monthly Reporting | Robots.txt | .htaccess | Favicon | Site Map | Search Engines | Google Maps | Forms and E-mails | Footer Link | Google Search Engine | Picture Tagging | Directory Page | Social Media | Heading Tags | Indexed Pages | Robots | Sitemap | HTML Errors | Search Engine Errors | Broken Links | Pages and Content | Search Engines | Picture Tagging | Flash and iFrames | Hierarchy | Forms and Emails | Add Media | Static URLs | Descriptive URLs | Duplicate Content | Plagiarism Check | Positive Content | Negative Content | Character Coding | Downloadable Files | Optimize Graphics | Optimize Navigation | Incoming Links | Outgoing Links | Page Content | Meta Data | Rich Content | Reduce Bounce Rate | Social Media | Analyze Competitors | Website Security | 3rd Party Tracking