SEO - URLs, Crawlers and Site-Maps

Have you ever wondered how Google searches all of the world’s websites and gives us the best results so quickly? This is organised by and intricate system of Spiders and Crawlers, and we’re going to have a look into how your website fits into this system! We’ll also show you quick and easy steps you can do with your URL organising and sitemap to make sure Google is indexing your website!

URLs

What you see in an Address Bar is called a URL (Universal Resource Locator). It is the address to the webpage you are currently on. This is easy to see.  It is ideal to have URLs containing wording that is related to your page rather than a string of numbers or characters.

Figure 1 - This is the address to the Psych Press About Us page.

This example is for the Psych Press about us page. Notice how the URL has “about us” in it. Your users should be able to quickly recognise what the URL will bring them to as the page loads and when they see it again, if retyping your address. The influence on search rankings isn’t huge, but it is still an influence that is easy to change! Google also reads hyphens as a space but underscores as linking two words together, so keep this in mind. Apart from these considerations, having comprehensible URLs will help greatly with analysis with Google Analytics (which we’ll get into later!).

Spiders and Crawlers

Your website will be crawled by a spider or crawler. These robots index web pages for Google’s search database. If they cannot crawl your website you have a problem. This means your website will not appear as high in Google’s search results. The best way to do this is to create an account with Google Webmaster Tools and check your website’s crawl health. You will need to insert a Meta tag into the HTML of your front page in order for Google to verify that you are the owner. Google will provide the Meta tag for you. Once this is set up you can go to your webpage’s dashboard on Webmaster Tools and see if there are any errors. You should also get accustomed to the ‘Traffic’ and ‘Optimisation’ areas of this tool as this is where you can see which key terms are becoming more effective in attracting clicks to your site!

Figure 2 - Webmaster Tools Crawl Errors page. Detailed reports on all the crawl errors are available through webmaster tools. There would not be green ticks if the entire website was unable to be crawled.

 

 

 

 

Site Map

A quick and easy way to ensure all your pages on your website are seen by the crawlers is to upload a site map and then tell Webmaster tools where to find it on your website. A free site mapping tool can be found at XML-sitemaps.com which is quick and simple to use and will give you the site map file you need. Find it here at http://www.xml-sitemaps.com/.   It has a max of 500 pages, which is more than enough for most websites. Once the map is complete upload it to your website and connect it to webmaster tools. The site map will assist crawlers in finding all the pages on your website and improve site visibility to searchers interested in your services!

 

To Do For My Business

        Are your website’s URLs logical and user friendly?

        Is your website connected to a webmaster tools account?

        What is the health of your webpage?

        Does your webpage have a site map registered with Webmaster Tools?

For Psych and HR

In Webmaster Tools, don’t become too concerned with common errors such as the “404 not found” error if you’re often removing parts of the site. If you follow the error links given and realize it is no longer required or linked to on site it shouldn’t affect your SEO. Another good use of Webmaster Tools for you will be to check how often your page is being crawled, which will usually be around once a week, and backlinks tend to take longer to show up. If you’re making changes and want to check on backlinks or errors the crawl times will give you an idea of when you can expect results. Just keep in mind that all Google ‘tools and rules’ are being further developed all the time!