XML sitemaps are files that provide Google and other search engines a map to your site and its various URLs and pages.

Consider when a crawler arrives on your site. Without an XML sitemap, it has to follow links around your site to locate new content for crawling and eventually, indexing.

With an XML sitemap, the crawler is signposted to the parts of your site that you want it to crawl and index.

XML sitemaps are signposts that urge crawlers on the right path round your site.

Without a sitemap, your site will still get indexed so long as you have proper internal linking, so don’t panic if you’ve never used an XML sitemap before.

As far as SEO goes? XML sitemaps are supportive of good SEO but will only directly influence it under specific circumstances.

You’ll need to set your site up with Google Search Console and other search engine webmaster tools to point them to your sitemap.

Crawlers can also find the XML sitemap in your robots.txt file if you specify the path to the sitemap there.

Do You Need an XML Sitemap?

XML sitemaps aren’t strictly essential – your site will still get indexed without one, so long as pages are properly linked by internal and/or external URLs.

But that doesn’t provide the whole picture as XML sitemaps certainly improve the efficiency and quality of indexing and this does impact SEO.

Google themselves state that XML sitemaps are only 100% essential if your site is reasonably large (over 500 pages) or features a lot of rich media.

The problem is, by modern standards, sites with 500+ pages can still seem small, and many ‘small’ sites will exceed this if they have a store with quite a few products, a blog, etc.

Other situations when you definitely need an XML sitemap is when your site has a complex structure with a lot of disparate or isolated content that might get overlooked by crawlers if you don’t provide a sitemap.

In these situations, crawlers may never reach your newly created content if you don’t have a sitemap to direct them there. Obviously, this is a problem; they’ll arrive on your homepage, explore a little and pretty much get lost crawling the same pages until your crawl budget depletes.

This could leave parts of your website uncrawled and therefore, unindexed!

Why You Should Create an XML Sitemap

Even if Google states that you don’t need an XML sitemap, particularly if your site is small (though you might be surprised as to how large your site really is), it’s definitely good practice to create one for any site. 

Creating an XML sitemap is pretty easy, and once you upload your sitemap into Google Search Console, you’ll be able to check when Google visits it and crawls your URLs, which is very useful in itself.

Providing a sitemap to Google will also help get your site indexed quickly and you’ll be able to check and track the crawl status of your URLs from the Sitemaps section of Google Search Console.

For this alone, creating a sitemap is well worth it.

When You Should Create a Sitemap

It depends on how you’re constructing your website or what platform or theme you’re using.

If you go down the WordPress route, your theme will already have some sort of built-in URL structure, e.g. you might have a homepage, blog, about, shop, portfolio and other related pages.

Shopify and other platforms will auto-generate XML sitemaps, but you still might need to modify them at some point.

Generally speaking, once you build your site out with some content and get your site set up with Google Search Console, you should create and add a sitemap.

When you make small changes to your site, there’s no need to update the sitemap unless it’s very large and you’re concerned about your crawl budget.

Dynamic Sitemaps

Some sitemaps generated using plugins like Yoast SEO are known as ‘dynamic sitemaps’ and will auto-update when you add and remove content from your site.

These plugins ‘ping’ search engines to alert them of updates (which may get new content indexed quicker, but it’s hard to tell and you can’t count on it.

Where Do You Put the XML Sitemap?

The XML sitemap should be placed at your site route, so www.example.com/sitemap_index.xml.

You place your sitemap manually with the file manager of your web hosting software (such cPanel), but you can also use a plugin to do this for you (and many will also generate the sitemap also).

Once you’ve placed your sitemap file at the root, you can head into Google Search Console and head to Sitemaps > Add a New Sitemap.

For other search engines, e.g. Bing, Yandex or Baidu, you’ll have to use their equivalent webmaster tools to upload your sitemap. You can also add the sitemap to your robots.txt so they can find the sitemap automatically when they crawl your site.

Step-by-step:

1. Create a sitemap using a sitemap generator tool or plugin
2. Add the sitemap to your site’s root (manually or with a plugin)
3. Add that sitemap to Google Search Console
4. Wait for confirmation that Google can see your sitemap, your URLs should be mapped and crawled soon after
5. You can also add your sitemap to any robots.txt files you create for your site

Advanced Sitemap Attributes and Controls

XML sitemaps can contain complex attributes that provide in-depth signposts for crawlers visiting your site.

XML sitemap attributes include priority, last modified and change frequency, all of which are designed to instruct crawlers on what your important URLs are and how often they change.

You can alter these if you like (using in-depth XML sitemap generators like Screaming Frog), but many search engines pay little or no attention to them.

For example, Google stated that they do not consume priority attributes in sitemaps.

XML Sitemap Tools

Screaming Frog XML Sitemap Generator

The Screaming Frog XML sitemap generator is part of their SEO Spider tool that crawls your site, locates its URLs and analyzes various on-page and technical SEO characteristics.

In the process, it will crawl your URLs for XML sitemap generation.

Screaming Frog provides a full breakdown of your site’s URL structure, allowing you to create an XML sitemap that is modifiable with attributes such as priority and change frequency.

You’ll also be able to select any URLs you don’t want to include in the sitemap (though they’ll still be crawled if there are internal or external links pointing to them).

The free version of the SEO Spider will crawl up to 500 URLs which you can use to create your XML sitemap.

Features

• Crawls your site for URLs
• Allows you to add many types of sitemap attributes (e.g. priority, change frequency)
• Up to 500 URLs for the free version of the SEO Spider (which does a lot more than create sitemaps)

XML Sitemaps

This longstanding XML sitemap generator tool has been around since 2005 and is a quick and easy way to build a standard XML sitemap for any site.

Simply insert your site into the field and the tool will crawl its various URLs to generate a downloadable XML sitemap to place in your website root and Google Search Console.

The free version of the tool will map up to 500 pages. There is a premium version of the tool that creates dynamic sitemaps for larger websites, and it will ping them to search engines so you don’t have to upload them yourself (but you should anyway).

An excellent, simple free tool for creating XML sitemaps for upload to your site’s root and Google Search Console.

Features

• Free tool crawls up to 500 pages
• Downloadable XML sitemaps
• Premium version ideal for super-sized sites

XML Sitemap Generator

Another super-simple, free XML sitemap generator, this free tool will crawl up to 2,000 URLs, more than broadly equivalent free tools.

There are also several options for creating advanced XML sitemaps with change frequency, priority and exclusion or filtering settings. It’s only worth fiddling with these settings if you’re an advanced SEO user or want to manage a crawl budget problem.

The company also offers a WordPress XML sitemap plugin that lets you choose whether or not to add new content you create for your site.

Features

• Generate XML sitemaps for up to 2,000 pages
• Advanced attribute settings
• WordPress plugin also available

My Sitemap Generator

This is another super-simple sitemap generator suitable for the vast majority of standard websites. Simply enter your root URL to create your sitemap.

You’ll be able to set priority and change frequency for specific URLs if you like and can add attributes to instruct crawlers whether or not to crawl certain content (e.g. javascript).

The free tool accommodates sites with up to 500 pages. There’s plenty of advanced settings here if you’re looking to manage a crawl budget problem (unlikely for small sites).

Features

• Free for up to 500 pages
• Set advanced sitemap instructions and attributes
• Set URL change frequency and priority

Yoast SEO

The fantastic and widely known Yoast SEO plugin includes an XML sitemap feature that is free to use.

The plugin will automatically update your sitemap as you add/remove/change content on your site, which is obviously great.

The XML sitemap feature is very easy to activate, and once you’ve done it from within the plugin menu in WordPress, you’ll be able to view your sitemap and submit it to Google Search Console and other webmaster tools.

You’ll probably notice that there are no additional settings to adjust priority or change frequency, but as stated, you really don’t need to worry about these too much.

Features

• Auto-generate dynamic XML sitemaps for any WordPress site
• Sitemaps are auto-updated
• Simply point Google Search Console to the sitemap that you can open from within the plugin

Google Sitemap Generator

Though named Google Sitemap Generator, this sitemap generator plugin is compliant with most search engines and allows you to auto-generate sitemaps for WordPress sites.

When you update your site or create new content, the plugin ‘pings’ Google and other search engines with an alert that your sitemap has been updated.

This plugin does allow you to edit priorities, change frequency and other parameters if you want to. It’s extremely well-rated, reliable and easy-to-use for anyone.

Features

• Auto-generate dynamic XML sitemaps for any WordPress site
• Auto-update sitemap and alerts Google and other search engines to changes
• Allows for changes to advanced sitemap parameters and attributes

Rank Math

Rank Math is an extremely sophisticated WordPress SEO plugin that provides unparalleled control over numerous on-page and technical SEO settings ranging from schema to sitemaps.

It doesn’t just cover XML sitemaps but also news and video sitemaps that can help Google index your content and media to Google Videos and Google News.

The XML sitemap generator is exceptionally easy to use and modify, and you’ll be able to allocate URLs that you don’t want to include in the sitemap.

In general, though, Rank Math is a gem of a WordPress SEO tool that has been featured by many top SEO brands. The XML sitemap feature is available in the Standard version at $59 a year (and is also obviously available in the more advanced, expensive tiers).

Features

• WordPress SEO plugin
• Powerful sitemap features
• Tons of other tools available in the plugin

Site Robot for Shopify

This is an XML and HTML sitemap generator plugin for Shopify.

It essentially creates an XML sitemap for any Shopify site for upload to Google Search Console.

The only real advantage it lends over auto-generated Shopify sitemaps is that you can choose URLs to exclude (and that it lets you create HTML sitemaps too, which are a different type of sitemap that pull all your site URLs onto one page of your site).

Features

• Auto-generate XML sitemaps for Shopify
• Exclude URLs
• Also generates HTML sitemaps

Magento XML Google Sitemap Generator

For the Magento CMS, this sitemap generator is designed to quickly and efficiently create very large sitemaps for super-sized sites (typically enterprise-level or larger business sites with thousands of pages).

It’s a powerful tool that allows total control over sitemaps and it’s capable of mapping the maximum number of URLs Google permits for a sitemap (50,000 URLs or 50MB).

This is very useful for large Magento stores with potentially thousands of products and many dynamic pages. Essential for enterprise-level crawler budget management for Magento.

Features

• Sitemap generator for Magento
• Works with XL websites
• Plenty of control over attributes

Summary

Creating and implementing XML sitemaps for Google Search Console and other webmaster tools is straightforward and whilst the jury is out on whether or not they confer direct SEO advantages, XML sitemaps definitely qualify as SEO ‘good practice’.

For some sites, there’s no doubt of the necessity of XML sitemaps and properly implementing them can get your site indexed properly, also helping you manage crawl budget problems.

It’s also very useful to be able to view and navigate your site in the Search Console under the Sitemaps tab.

This provides you with an easy way to monitor the URL structure and indexing status of your site and its pages.

The post XML Sitemap Generators appeared first on SEO Administrator.

Without accurate and proper HTML, web pages would just be chaotic blocks of unformatted text. HTML guides the web browser to display the document as the designer or creator intends it to be displayed.

HTML is fairly simple and is one of the first languages web developers learn – though you cannot technically call it a ‘programming language’, as its sole intent is to markup documents.

Here, we’ll be providing a brief example of what HTML and validation is, overviewing some HTML validation tools, and explaining how HTML errors can affect your site’s SEO performance.

HTML Example

An HTML copy of the first portion of this document would be:

<h1>HTML Analyzers & Validators</h1>

<p>HTML is the number one foundational document markup language.</p>

Here, <h1> marks up the first line as a title and <p> marks up the first paragraph. Notice how the opening tags are different to the closed tags, which are closed with </.

There are around 100 total HTML markup tags – not too many in the grand scheme of things.

HTML and W3C

The W3C, World Wide Web Consortium, is a quasi-regulatory organization for the internet. They set out codes of best practices for web designers and developers to follow.

Over time, HTML has needed to adapt to different devices, browsers, and assistive technologies. HTML errors can cause readability issues, particularly on certain devices such as screen readers (which render text documents as speech for the blind or visually impaired).

Poorly constructed HTML can cause a number of issues that sabotage the readability and compatibility of your web pages.

This could even render the entire webpage unreadable or uncrawlable in exceptional circumstances.

The W3C has laid out best practices for HTML – if you follow those then, in theory, your site is more likely to be compatible cross-browser, device and technology – both now and in the future.

This is why we have HTML validators – they check the quality and ‘cleanliness’ of your HTML.

Why Do HTML Errors Occur?

HTML errors and bugs can result when you’re editing pages in HTML format and make a mistake, or when you use a buggy plugin or other tool to create and markup web pages.

Codeless web development apps and WYSIWYG editors mean that we generally don’t need HTML knowledge to create documents – the software does that for us.

But it’s not always perfect, and errors can occur, harming the technical quality of your webpages, in turn damaging SEO.

HTML Validation

To analyze HTML documents for errors and bugs, you can use an HTML validator.

HTML validators check and validate your HTML against W3C best practice guidelines. They’re simple online tools that load an HTML file or URL before parsing the HTML markup and analyzing it for errors and bugs versus W3C guidelines.

Some HTML guidelines are pretty trivial, e.g. W3C says you should use <strong> and not <b> to embolden text.

Both markup a document in a near-identical fashion, but it just so happens that some plugins or editors might use <b> instead of <strong> for whatever reason.

That said, HTML validation will route out genuine errors and bugs.

Some key examples include unclosed tags, incorrectly written tags, and formatting errors which cause compatibility issues, particularly with regards to tables.

In fact, you might have experienced dodgy HTML in the past when surfing a webpage that clearly doesn’t render properly, particularly old web pages that haven’t been optimized for mobile.

Yoast SEO draws attention to an example of a Dutch news site whose homepage failed to crawl properly, and couldn’t be rendered in Google’s cache.

They were using an XMP tag, similar to the PRE tag, but instead of rendering tags internally, it outputs them instead. The XMP tag wasn’t closed correctly, so GoogleBot failed to render and crawl the page.

This is a pretty unique example, though, and you’re very unlikely to experience page-breaking HTML errors.

What Does Google Think About HTML Validation?

It’s worth remembering that W3C is not really affiliated with Google, and that you’re under no obligation to conform every component of your documents to their standard.

So what does Google say on the issue?

Google Search Central states:

“The best way to make sure that your page looks the same in all browsers is to write your page using valid HTML and CSS, and then test it in as many browsers as possible. Clean, valid HTML is a good insurance policy,”

– Google.

Meanwhile, Google’s Matt Cutts admits that HTML validation is not an organic ranking factor, also stating that a minuscule percentage of sites have perfect HTML – and a large portion of global giant websites actually have pretty poor HTML on balance.

“Google does not penalize you if you have invalid HTML because there would be a huge number of webpages like that”

– Matt Cutts, Former Google Head of Webspam.

HTML, like many areas of SEO, is a grey area. You can leave messy code alone and hope it never matters – and it probably won’t unless there are serious errors – or you can strive long and hard to build the perfect HTML markup for every page on your site.

Largely, this comes down to time and resources. HTML gurus will whizz through validation, but it could still be time-consuming.

So What’s the Point in Validation?

Validation is non-essential but it’s still good practice, but the truth is, few SEOs take it particularly seriously unless they encounter errors.

W3C themselves do put forward a strong case for checking HTML, particularly as it will future proof your pages to new browser software and accessibility tools.

Debugging HTML errors and bugs are made simpler by HTML validation tools, and they can direct you towards serious issues. If you have knowledge of HTML then it should be fairly simple to use an HTML validator to check your code, but if not, then it can be pretty bewildering, especially if it throws up a lot of errors.

It’s also worth highlighting that HTML validation is not the only way to check page errors. SEO tools such as Semrush Site Audit and Surfer SEO offer a multitude of other on-page and technical SEO checkers that can crawl your site and will highlight errors that Google will see – and which could then affect your SEO.

You can also check for browser compatibility, Google highlights this themselves:

“Once you’ve created your web design, you should review your site’s appearance and functionality on multiple browsers to make sure that all your visitors are getting the experience you worked so hard to design.”

HTML Validator Tools

Without further ado, let’s suggest some HTML validator tools to plug your HTML in for markup analysis and validation.

W3 Validator

By W3C themselves, this validator is very easy to use via URL input, HTML file upload, or direct input. It handles all HTML, XHTML, and other web document formats.

Once you’ve input your URL and clicked validate, the tool will parse your HTML and display both errors and warnings. Errors are what you should focus on first, warnings usually related to unnecessary attributes or other markups that could be removed – but that’s usually unnecessary, or at least very low priority.

It is almost guaranteed that you’ll find some errors and likely many warnings. Try some competitor sites, friend’s sites, or other major sites and see how they compare – you’ll likely find many errors there also.

If you do find reams and reams of errors then don’t panic, and don’t rush into making any changes. Check your site on multiple browsers and use an SEO audit tool like Semrush SEO Audit to check the crawlability and technical SEO of your site.

If Google can crawl it and people can read it without issues, then you’re probably fine!

You can do your own research into errors – there are too many possibilities to describe but the likelihood is, if it’s a genuine issue then people will have discussed it many times already.

JSON Formatter HTML Validator

A powerful set of tools for formatting, converting, and validating HTML as well as JSON and CSS, the JSON Formatter HTML Validator works like any tool – you simply plugin your URL or upload a URL file.

You can hit ‘Run’ to render the HTML and Validate to check the HTML for errors. Again, the same caveats apply, and you’ll likely find errors – possibly many

The interface here isn’t as good as other HTML validator tools and doesn’t give you a detailed breakdown of errors and warnings, though it’s still fine if you have HTML knowledge and know what you’re looking at.

Freeformatter HTML Validator

Another simple HTML validator tool. The Freeformatter HTML Validator only lets you copy and paste HTML or upload an HTML file. You can choose from all ISO language codes for parsing HTML in different languages.

You’ll find that errors and warnings are clearly differentiated here, allowing you to look at common errors that appear on your pages and delve into further explanations online.

Infohound Tidy

This tool lets you check and validate HTML and then create and download a tidied copy. There are many options

Like other tools, it will display both warnings and errors. You can then tidy errors using the controls provided before downloading the tidied HTML.

DO NOT just copy HTML from your site into here and replace it with the tidied HTML, at least not without creating a duplicate to test it first, or using a backup.

Like with other HTML validators, it’s likely that many errors are negligible or pretty much meaningless – always research them before taking action.

Code Beautifier for HTML

Code beautification is slightly different from validation. With this tool, you can beautify your code, meaning deleting unnecessary components and minimizing down to what is needed.

This makes it easier to work with the code.

Again, this has a pretty niche use but will be useful for some

SonarSource

SonarSource is a professional suite of paid code analysis tools. They’re oriented towards creating highly robust, secure codes. The software helps tidy code but also identifies secure blackspots and other backdoors or other security issues.

For sure this is the most advanced HTML validator and analyzer around and each and every code or warning is broken down with an explanation of what’s going on.

For clients that demand perfectly validated error-free code, SonarSource is tough to beat.

What Do I Do If I Find HTML Errors?

It’s likely that you’ll find HTML errors when using a validation too.

Feature-rich pages that run lots of plugins will likely be the worst, whereas simple pages with little content, or just plenty of simple block text, will usually be fine or totally error-free.

The first thing to do is to check and see if any errors appear multiple times. Look them up to gain more info on what they mean and whether they matter – it might be jargon but you might find some simple-worded insight!

You can also consider hiring an HTML specialist on a freelance website. Checking your HTML should be easy for someone who knows HTML inside out and it’s likely that any serious errors can be fixed in seconds.

If all else fails, then you’ll need to check your pages using SEO audit tools and browser compatibility apps. If your pages load fine on every browser and device, and you sort technical SEO issues highlighted by an SEO audit, then you’ll probably be fine!

Make sure you monitor your site in Search Console, Google Analytics, or whatever other tools you use, and keep an eye for changes in traffic or other anomalies that could indicate more serious site bugs or UX problems.

The only other thing to do is to keep up-to-date with Google’s algorithm changes to make sure they never add HTML validation signals or other new rules that could lead to negative SEO results for sites with messy HTML – but this seems unlikely.

Summary

Clean HTML is a hot topic, with the purists arguing that fully clean, precise and valid HTML as validated by validation services is totally necessary for now and the future.

But, checking the HTML of many sites reveals a different story and Google themselves admit that ranking sites on their HTML would be likely destructive and antithetical to what they’re trying to do – i.e. serve the user.

But, therein lies the crux of the issue.

If HTML does affect user experience on your site, then it will affect SEO.

This is why HTML validation is a good way of tracking down HTML issues, and together with technical SEO audit tools, you can build a clean, fast and robust site.

The post 7 Free HTML Code Analyzers & Validators appeared first on SEO Administrator.

Analyzing log files is an often-overlooked but important aspect of technical SEO, and it really comes into its own when dealing with huge websites with thousands, or hundreds of thousands of pages.

Here, we’ll be explaining what a log file is, where to find it, what it means for your site, and how you can squeeze SEO juice out of it.

What is a Log File?

Every request made to a web server is logged and saved.

The file is output and saved to your hosting web server – and you’ll be able to find it and analyze it.

When you look at a raw log file, it’ll be densely packed with written info and data – but don’t panic!

Some logged request information you’ll see may include:

• IP Address
• Type of HTTP request (GET/POST, etc)
• User Agent
• Requested URL
• Timestamp
• HTTP Status Code
• Referrer

Other attributes might include:

• Host name
• Bytes downloaded
• Request/Client IP
• Duration

This information can be useful for troubleshooting issues, but for SEO, we’re primarily interested in HTTP requests – this recorded information tells us about how GoogleBot and other crawlers crawl a site.

The log is a 100% accurate representation of how GoogleBot and other search engine bots crawl a site and its pages.

Log files essentially log the activity of crawlers, providing SEOs clues as to how a site and its pages are being crawled and indexed, and by who.

How Do I Find a Log File?

It depends on your server setup and configuration.

For those who have access to/have configured their own server, the following guides can be followed to download a log file. This is often a convoluted process – you can ask your web developer or tech team for log files if you are not responsible for the server.

Using cPanel to Download a Log File

cPanel is a web hosting control panel software that simplifies web and server management.

If you have access to your site’s cPanel then downloading log files is much simpler (thankfully!).

Log into cPanel and you’ll typically need to head to Metrics and Raw Access Logs. You’ll need to tick the Archive box to ensure that logs will be saved from then on.

After a new log is created, you’ll then be able to download the log for analysis.

This is by far the easiest DIY method to retrieve logs and anyone can do it, provided you can access your site’s cPanel.

Once you have the log file, you’ll need to look into log analyzer tools to analyze the data.

You can manually convert and filter the file, but there’ll be tons of info in there which is totally irrelevant to SEO. In fact, some log files can reach hundreds of megabytes, and parsing and sorting the info DIY is painstaking.

It’s worth noting that you may need multiple log files to cover longer periods of time – one day’s worth of data is not likely to be enough to analyze how GoogleBot crawls your site.

Why Analyze Log Files?

It’s a lot of hassle right?!

Well, if you use cPanel then it’s not too bad!

But still, what do you do with all that data?!

Identify Crawl Bots

SEO is usually all Google, Google, and Google, but other crawl bots matter too, especially if you’re looking to tap into emerging markets or audiences in China for example, via Baidu.

Your server log will tell you if requests have been made by GoogleBot as well as BingBot, Baidu, Yahoo, and Yandex, and any other user agents.

So, for example, you may find that you’re not getting crawled by Baidu, but you want to appear in China. The log file contains this sort of information.

You’ll also be able to see if your robots.txt file is doing its job in disallowing certain crawlers.

Crawl Budget

Search engines don’t crawl your site indefinitely, and they don’t have unlimited resources either.

Crawls are periodical and only a dedicated quantity of crawling resources will be dedicated to your site. Log files let you find and assess how many pages are being crawled across your site over a given period of time – usually daily.

More authoritative, healthy, and optimized sites will be dedicated greater crawling resources.

Consider a site like Wikipedia which has some 6 million pages, with over 600 newly added articles each day. That’s going to need a lot of crawling and as such, GoogleBot probably crawls many thousands or even millions of Wiki pages every day to check for updates and changes.

Wikipedia is one of the most authoritative domains in the world and thus, it has a huge crawl budget to suit its vast quantity of pages and ultra-high authority.

But your site might be different, you still might have thousands of pages and if your site isn’t being allocated the appropriate crawl budget, then your newly created or updated pages might sit there uncrawled for days, weeks, months, or even years in exceptional circumstances!

You could also find that crawler bots are indexing old pages, redirects, orphaned pages, and other useless URLs when you want them to crawl your new or updated content.

Log analysis lets you calculate how many pages of your site are being crawled – and which ones – so you can make necessary changes to your site, robots.txt and sitemap to make sure your top pages are prioritized.

Once you’ve assessed crawl budget and timing, you can modify your XML sitemap to prioritize which pages Google crawls regularly.

This might be your blog or similar, whereas other parts of your site that remain the same for long periods of time will not need to be crawled so often.

Temporary 302 Redirects and Duplicate URLs

Temporary 302 redirects direct users and crawlers to a temporary page. They’re not harmful to SEO but do eat into crawl budget as the search engine will be continually crawling to see if the redirect is still there.

You can also narrow down the content that you can instruct Google not to crawl at all, i.e. duplicate content. This can also help reduce analytics errors.

Check Page Crawl Times

You can analyze log files to discover the time to first byte (TTFB) and time to last byte (TTLB) associated with each page, helping you quickly find your fastest and slow loading pages.

This is a quick and easy way to check page load stats for big sites, and you’ll be able to sort your largest web pages so you can check those for particularly slow content (e.g. high-res images).

Discover Orphaned Pages

Orphaned pages are pages still linked to your site and crawled, but aren’t internally linked. They exist to Google (if they’re being instructed to crawl the page from the sitemap, or external links point to it) but not to you or the user (unless external links still point to them).

These can be created by site structure changes, internal linking errors, and old redirects.

Either way, they won’t rank as they’ll have no internal links. Log analysis finds orphaned pages so you can inspect them and deal with them.

Best Log File Analysers

Here is a compilation of log file analyzer tools that take the graft out of analyzing complex log files for SEO purposes.

Semrush Log File Analyser

In Semrush’s mighty set of SEO tools, you’ll find the Semrush Log File Analyser which was actually only released in 2018. This tool is available with Semrush’s main set of tools, costing $119/mo for the Pro package.

You’ll be able to upload your log files for analysis where the tool will break down all of the important SEO-related metrics and display them in graphs.

The tool will breakdown:

• Requests from various search engine bots (e.g. GoogleBot)
• HTTP status codes found each day
• The different file types crawled each day

This easily lets you assess your crawl budget, and shows how often pages and/or files are crawled. This enables you achieve the main goal; to analyze the crawlability of your site and point GoogleBot towards the pages that require crawl priority (e.g. a regularly updated blog).

Features

• Analyze bot activity
• Discover most crawled and least crawled pages
• Filter by file type, e.g. HTML, PHP, amp, CSS, Javascript, and JSON

Screaming Frog SEO Log File Analyser

The Screaming Frog Log File Analyzer is probably one of the most complete log analyzers around and doubles up as an excellent technical SEO tool.

Costing just £99.00 per year, it’s one of the cheaper tools around and the free version still lets you analyze 1000 lines of log, plenty for single-site owners and SEO novices.

This tool will breakdown crawler activity in full, filterable by the bot, e.g. GoogleBot, Baidu, Yandex, etc. It’ll point you towards broken links, redirects, and orphaned pages. Sorting by page size is simple too, so you can quickly ascertain page load speed and size.

For SEO purposes, the tool breaks down crawl budget and reveals the most crawled pages and content, enabling you to discover how efficiently your site makes use of its crawl budget.

Features

• Discover and verify bots
• Analyze crawl frequency
• Find your most crawled pages to analyze crawl budget
• Find orphaned pages

Paper Trail by Solar Winds

This is more of a log management tool for DevOps and tech teams. It aggregates many types of logs, including server log files, for centralized accessibility across an organization.

Log alerts can be sent via Slack or other team management platforms and data can be imported into software utilities such as Hadoop.

Features

• Aggregates logs for engineers and dev teams
• Email alert system for anomalies
• Access and download log files for further analysis
• Wide technical remit (not really an SEO tool)

Graylog

Graylog is a sophisticated log and data analysis platform that enables aggregation of logs of virtually any type, from any source. It’s designed for enterprise and organization-level businesses that require rigorous analysis of many data log outputs.

Graylog is an advanced tool for engineers and dev teams, it’s designed for analyzing data from many inputs or outputs – not just servers. 

Features

• Can store huge quantities of data
• Designed for enterprise-wide data analysis
• Real-time analysis
• Unprecedented scalability

Loggly

Loggly aggregates log data across large enterprise-level or organization-wide networks. It’s an enterprise-level platform that allows for real-time analysis of server logs as well as log data from near-limitless sources.

Loggly enables the analysis of all server data, but it’s also an interdisciplinary tool for log aggregation and analysis.

Features

• Store and analyze server logs across large distributed systems
• Secure, cloud-enabled analytics platform
• Near-limitless scalability
• Enables cross-department collaboration

Log Entries by Rapid1

Another enterprise-level server log file and aggregator. Designed to pool and centralize server info from distributed networks, Log Entries enables advanced analysis of server resources and system health.

A professional engineering/dev tool that breaks down server metrics in minuscule detail to audit the health of colossal networks.

Features

• Collect server logs across large networks
• Real-time alert system
• Server anomaly detection
• Enables cross-department collaboration

GoAccess

A fast, terminal-based log analyzer, GoAccess breaks down key server requests in detail for real-time analysis. It shows visitor numbers as well as crawlers and spiders so you can analyze how often your site is crawled. It’s also great for assessing page response time and server load.

Features

• Terminal-based dashboard
• Analyze web traffic and crawlers in real-time
• Analyze server resources and bandwidth
• Free

SEOLyzer

Optimized for SEO log analysis, SEOLyzer provides seamless live log tracking tools to detect crawling errors rapidly. It has an easy-to-use interface and the focus here is on speed; it enables you to locate and home in on errors before your other systems (e.g. Search Console) pick up the error.

SEOLyzer has built an impressive repertoire of clients and has proven a superb technical SEO tool for auditing useful server logs for key information. The aim of SEOLyzer is to detect logged errors and work to fix them before site coverage declines.

The software also allows you to aggregate information into graphs and KPIs.

Features

• Log analysis for SEO specifically
• Aggregate log data for analysis
• Find errors before they hit your site coverage
• Free version for single-site users (limited analysis capacity.

Summary

SEO log analysis may seem fairly niche and complex, but it’s a powerful weapon for your technical SEO armory.

SEO log analysis’s main strengths lie in debugging and troubleshooting, but it’s also an excellent tool for analyzing crawl budget in the natural habitat of the crawlers themselves.

The log is 100% accurate, data-rich, and hard-linked to the requests made by crawlers – that is its main advantage.

The post Log File Analyzers appeared first on SEO Administrator.

Bots typically include web crawlers, such as GoogleBot, which will look at the robots.txt file on your site to learn what it should and shouldn’t crawl on your site.

Robots.txt files are not just for GoogleBot, but will be ‘read’ by many other web crawlers from other search engines and services, as well as web scrapers that are hunting your site for information.

It’s worth bearing in mind that the instructions contained in the robots.txt aren’t mandated – they’re not legally binding (though violating them could still result in breaking the law or violating copyright, etc).

Robots.txt files just pleasantly nudge bots in the right direction and kindly ask them to not crawl certain parts of your site.

Robots.txt files are created by default in most WordPress sites and other sites that use Wix or other site builders, and will automatically be set to allow web crawlers to crawl everything on your site.

With many site builders or platforms like Shopify, you can’t edit the robots.txt file (but can use other techniques to control the crawling of your site.

When To Adjust the Robots.txt File

If you’re an SEO beginner, novice, or an owner/designer of a personal or individual small site or blog (that has already been published), you likely won’t need to adjust the robots.txt file for some time, but checking it is still good practice.

One potential use some advocate for the robots.txt file is preventing your site from being crawled whilst it’s still under construction.

This is ineffective if links point to the page(s), and Google recommends you use a noindex meta tag or password protection instead if you want to prevent a page from being indexed during construction/migration.

You can also use a plugin that covers your site with an ‘under construction’ notice.

There are many other legit reasons to check and edit the robots.txt, though:

Crawl Budget and Robots.txt

There are legitimate motives for editing the robots.txt file and these can confer SEO benefits.

Most of these benefits are linked to a site’s crawl budget. Crawl budget refers to the resources Google, Bing, Yahoo, and other search engines allocate to crawling and indexing your site.

For example, whilst you might assume Google has an infinite army of bots ready to crawl each and every intricate detail of the internet at will, 24/7 around the clock, this isn’t strictly true.

Even Google has finite resources (for now!), and they allocate these resources to sites based on their size/reputation/authority and other factors (many of which are not totally known or understood).

SEOs analyze crawl budgets by checking out their site’s log files, which provide evidence of how many pages crawlers are crawling on their site.

In short, editing the robots.txt directly influences how crawlers interact with your site, which directly influences crawl budget.

Case Example: Robots.txt and an eCommerce Store

Before moving onto some robots.txt tips and tools, let’s run through a simple case study of when you should consider editing your robots.txt file.

eCommerce stores typically use dynamic pages and contain a lot of duplicate content.

Dynamic pages and duplicate content are typically changed via user interaction, like when a user filters product categories or customizes a product. When a user filters a group of products, a page with duplicate content is produced – there’s nothing useful there for anyone apart from the user.

You can disallow bots from crawling duplicate content and filter content, or any other content that is practically useless for SEO and might occupy your crawl budget.

Content to potentially disallow from crawlers includes:

Whilst you might feel your site is way too small to be affected by crawl budget issues – and you’d probably be right – being aware of how you can edit the robots.txt file for the future is still very important.

If your site becomes bloated with content that is useless to Google then retrospectively fixing that is more exhaustive than preparing your robots.txt file early on in your web design and SEO journey!

Additionally, you may have pages with sensitive information, copyright material, and other files you don’t want to be crawled. Editing the robots.txt file can help keep these files off search engines.

Mostly, though, editing the robots.txt file is useful for managing crawl budget.

The Main Five Main Robots.txt Directives

There are five main robots.txt directives.

When you go to edit or generate your robots.txt you’ll see/use some of the following:

When you edit your robots.txt, you’ll be providing it with directives on URLs and directories.

The robots.txt file will sit in the root of your site, e.g. for site www.example.com, the robots.txt file lives at www.example.com/robots.txt.

A simple example of a robots.txt directive would be:

User-agent: Googlebot

Disallow: /thank-you/

This tells Googlebot to disallow crawling at: www.example.com/thank-you/

Note, whilst some tools ask you to input ‘directories’ and others ‘URLs’, you can input either.

Directories will be useful if you want to tell bots to ignore specific folders or files within your site’s root, e.g. /wp-content/plugins/ (which contains likely useless data on WordPress plugins).

URLs are best for singling out specific pages that you don’t want bots to crawl (e.g. thank-you [for ordering, buying, your interest, etc] pages as above).

Advanced Robots.txt Controls/Directives

There are some advanced robots.txt directives that go beyond allow/disallowing.

One key example is the wildcard, used to bulk-block files.

So, Disallow: /copyright-material/*.jpg would block all .jpg images located in the copyright-material directory.

Where To Put the Robots.txt File?

The robots.txt is placed at the root of your domain.

So, if your site is example.com, the robots.txt would be placed at: http://www.example.com/robots.txt.

You can locate this using the cPanel file manager (common for WordPress websites).

Once you open the file (literally a text file), you can write your new directives straight into it and save it.

There are different instructions for site builders like Wix, but Shopify and Squarespace don’t allow you to edit the file – though there are other options for hiding your site or its pages and preventing them from being indexed.  

Some SEO suites and plugins such as Better Robots.txt and AIO SEO semi-automate the process of creating robots.txt files for your site.

Robots.txt Generator Tools

Small SEO Tools Robots.txt Generator

Small SEO tools provide a fantastic array of free SEO tools, and the robots.txt generator does what it says on the tin (and is obviously 100% free).

This is a simple robots.txt generator with a selection of 15 bots and crawlers that can be set to be allowed or refused from your site. You can point it to your site’s sitemap.

You can also set the default for all bots and choose crawl delay settings which will roll out to bots that take notice of this (i.e. not GoogleBot).

You’ll then be able to add restricted directories and/or URLs using the form at the bottom. The /cgi-bin is pre-filled (a commonly blocked directory that does not need to be crawled), but you can edit that field if you want to.

Features

• Allow/refuse 15 common bots/crawlers
• Add sitemap
• Block directories/URLs
• Crawl-delay

Ryte Robots.txt Generator

An intuitive robots.txt generator with step-by-step instructions. The Ryte Robots.txt Generator is an excellent little tool for quickly creating robots.txt files with a selection of 11 bots. It’s a bit strange to not see other search engines like Baidu or Yandex in the bot selection, but you can add these user agents yourself.

To block crawling of URLs or directories, simply input them into the fields. You can also allow or disallow all bots from crawling your site.

Features

• Good interface
• Add sitemap
• Block directories/URLs

SEOBook Robots.txt File Generator

Another simple robots.txt generator with allow/disallow all and per-bot control of basic robots.txt directives. There are 9 bots here and it’s straightforward to add URLs or directories to block.

A super-simple tool that lets you copy and paste your new robots.txt straight into the old one.

Features

• Simple interface
• Add sitemap
• Blocks URLs/directories

SEOptimer Free Robots.txt Generator

With a selection of 15 bots, this is one of the more complete robots.txt generators available. It also has crawl delay settings. You can allow/refuse certain bots or set all bots to allow/disallow by default.

The fields allow you to add the URLs or directories you don’t want to be crawled.

Features

• Allow/refuse 15 common bots/crawlers
• Add sitemap
• Block directories/URLs
• Crawl-delay

Internet Marketing Ninjas Robots.txt Generator Tool

With a comprehensive selection of 22 user agents, this is a quality tool for allowing/refusing crawling from many major bots. You can add URLs/directories to allow/disallow per bot.

A very simple, easy-to-use robots.txt generator tool with plenty of bots. Unfortunately, there are no crawl delay settings.

Features

• Allow/refuse 22 common bots/crawlers
• Add sitemap
• Block directories/URLs

LinkGraph Robots.txt Generator

With over 40 bots built-in to the tool, this is a comprehensive robots.txt generator. However, it only lets you add 5 URLs/directories for disallowing, which is a minor letdown.

Still, though, with crawl delay and plenty of user agents in the list, this is yet another solid free robots.txt generator.

Features

• Allow/refuse 40+ common bots/crawlers
• Add sitemap
• Block directories/URLs
• Crawl-delay

Summary

Editing the robots.txt file is actually pretty straightforward!

Even as an SEO novice, it’s very handy to know what can and can’t be done with the robots.txt file.

As Google recommends, using the robots.txt file for completely hiding a site and its pages from the SERPs is ineffective compared to using a noindex meta tag.

The primary reason you’ll want to edit the robots.txt file is to prevent the crawling of non-publicly accessible site data that potentially eat into your crawl budget, or to control the activity of individual user agents that crawl your site.

The post Robots.txt Generators appeared first on SEO Administrator.