Posts Tagged ‘Search Engines’
Robots.txt file
Sep 09 22
Have you heard of a robots.txt file? Do you have a robots.txt file on your website? 
Search engine spiders and similar robots will look for a robots.txt file, located in your main web directory. It’s very simple and can help with your site’s ranking in the search engines.
What is a robots.txt file?
A robots.txt file is a small text file that you place in the root directory of your web site. This file is used to fence off robots from sections of your web site, so they won’t read files in those areas. Search engines often call these spiders and send them out to look for pages to include in their search results.
How do I create a robots.txt file?
Using a text editor such as Notepad, start with the following line:
User-agent: *
This specifies the robot we are referring to. The asterisk addresses all of them. You can be more specific by entering the bot name but in most cases you would use the asterisk.
The next line tells the robot which parts of your website to omit in their crawl:
User-agent: *
Disallow: /cgi-bin/
This would fence off any path on your website starting with the string /cgi-bin./
Multiple paths can be added using additional disallow lines:
User-agent: *
Disallow: /cgi-bin/
Disallow: /private/
Disallow: /test/
This robots.txt file instructs all robots that any files in directories /cgi-bin/, /private/ and /test/ are off limits.
If you want search engines to index everything in your site, you don’t need a robots.txt file (not even an empty one).
Using meta tags to block access to your site.
You can also use Meta tags to the same effect.
To prevent search engines from indexing a page on your site, place the following meta tag into the header of the page:
<meta name=”robots” content=”noindex”>
Not all robots may support the robots.txt directive or the Meta tag, so it is advisable to use both.
You can learn more about robots.txt files here.
Google Webmaster Tools
Sep 09 18
Do I need a Google Webmaster Tools account?
Google Webmaster Tools is a great resource for any site owner. and a must for anybody who is serious about making a search engine friendly site.
Creating content for the web is easy, but getting people to discover it is the hard part. You want your site to be as search friendly as possible. You want to know how your website interacts with the search engines and you want to be made aware of any problems with your site that may prevent people from reaching you. Webmaster Tools gives you all this information, it’s Google’s way of giving site owners the best information they need to have a successful website.
Features:
- First of all it’s free! A wealth of useful information at your fingertips and all you have to do is sign up.
- Top search queries tool which gives information about user search queries which have returned pages from your site.
- A Sitemap tool which allows you to submit an XML Sitemap with details on pages you would like Google to index and the priority of your pages so Google knows which ones to crawl first. If you do not have a site map create one and submit it to the site configuration sitemap section of your account.
- Diagnostics section which reports errors that were encountered when your site was crawled such as problems with the sitemap, pages not found etc. It also gives you HTML suggestions to improve your visitor’s experience and performance.
- Change of address option, which is a special tool to tell Google about your new domain name and helps to move your old site users to you new domain easily.
- Top search queries tool has information on the keywords and phrases that your website appeared for, the percentage of clickthroughs to your website, along with the keyword position.
- Links to your site tool provides information on inbound links with anchor text for your site pages.
- Keywords and internal links tool shows the common keywords Google sees when it crawl’s the site and internal links shows the total number of pages linked by other pages in your site.
- A vast amount of extra help topics, articles, forums resources are also available, all there to help you make the most of your website.
Setting up your Google account.
Getting started with Google Webmaster Tools is not to difficult. The first thing you’ll need is a Google account. Google has combined, or converged, their different accounts such so if you have If you use Gmail, Adwords, Analytics, iGoogle or one of the many other services, then you are ready to go. If not, sign up for a free Google Account. Once you have an account and sign in, the top right corner of Google will now have a “My Account” link. Click on that link and from the page that loads, click on “Webmaster Tools”.
Veryfying your domain.
Once you have added your domain to the dashboard, you need to verify it. To verify, you either upload a text file to the root of the website or add a meta tag to the index page. Once you upload the file to your web site (or modify the home page with the meta tag), you click the “Verify” button and Google will check to see if the file or meta tag is there. Once Google finds it, you are verified and able to access all the information about your site.
Do I need a sitemap?
You can also submit sitemaps using Google WebMaster Tools. A sitemap is a file that list pages on your site that you want Google to know about. Follow the “more information” links in the site map section to find out more information about creating a site map. According to www.Sitemaps.org a sitemap will allow search engines to more intellingently crawl your website. More details and FAQ’s about sitemaps can be found here.
Robots.txt file
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
Web robots often go by different names. You may hear them called:
- Spiders
- Bots
- Crawlers
To find out more about the robots.txt file click here.
There are many features to WebMaster Tools. Hopefully this overview has given enough information for you to get it set up and start benefitting from it.