Backlinks have been a key ranking signal for many years, but Google has been cracking down on dodgy or spammy backlinks for a long time now. 

Previously, it was exceptionally easy to hop from forum to forum, building up a bit of posting history and then posting some content with links for instant SEO juice.  

Prior to 2005, Google followed pretty much every link which meant it was a case of quantity over quality. 

Fast forward to now and we now realize that Google wants quality over quantity. They’ve understandably made it harder to jump between forums posting backlinks embedded in thin or low-quality content. 

So, are forum backlinking tactics dead? Not quite!

The Chronology of Backlinking

Backlinks have a turbulent history. Google has long considered them an important ranking signal since they provide a means for one site to vouch for another by means of a referral. 

However, the tricky element is, Google needs to discern when a site is referring to itself. Prior to recent updates, it was quite easy to self-submit stacks of links to various indexes, databases, spammy forums, and directories. These tactics could provide instant SEO juice to any site, regardless of the value it offers internet users. 

In 2005, Google added the nofollow attribute so webmasters flag poor quality or user-generated links to prevent Google from crawling them. This gave power to credible website forums that could be abused by link spammers looking for quick-win backlinks. However, according to Moz, data shows that even nofollow links are considered a ranking signal when other factors are also favorable (e.g. the authority of the site). 

The 2011 Panda update further clamped down on so-called ‘article marketing’ where SEOs published low-quality, often templated content to article databases to embed some quick-win backlinks. This terminated the rankings of article submission sites that had been taking a handsome sum in return for hosting poor-quality articles solely for the purpose of backlinking. 

The war on spammy backlinks has continued, with Google announcing a series of algorithm changes that gave webmasters the ability to add two new attributes to links on their site in addition to nofollow. Webmasters can now mark up sponsored or paid links as well as links embedded in user-generated content. This gives webmasters another way to provide Google with clues as to what a link means and how they should rank it. 

Google has even taken a dim view of backlink exchanges which are a genuine white hat link building technique for the most part. 

Generally speaking, the quality of a backlink still depends on:

It’s clear that most backlinks likely yield some benefit for SEO, even those earned via backlink exchanges or legitimate quick-win methods. 

For forum posting, what does matter is developing a strategy that both Google and the forum owner respect. 

That means sticking to relevant, higher authority forums, and not skimping out on content quality and posting authenticity. 

These principles are fundamental to building backlinks. 

Why Bother With Forum Posting?

So why bother with forum posting for SEO; what are some of the benefits of forum backlinking tactics?

1. There are thousands of forums

There are thousands of still-active forums on the internet, many of which receive thousands of posts and comments every single day. 

Forums cover pretty much every single niche, subject, industry, or topic you can think of which provides you with an opportunity to find forums that are directly relevant to your site(s). 

2. It’s quick and easy

So many sites don’t have any backlinks – some 66.31% in fact, according to Ahrefs. 

Building backlinks undoubtedly takes time and effort, which is why quick-win tactics like forum posting, directories, and link exchanges have always been popular. 

Whilst it’s true that Google has broadly devalued synthetically-earnt backlinks in recent years, they still matter, especially for smaller sites. 

3. Many forums are high-authority sites

Many forums are part of high-authority sites. That’s not to say that Google will treat your backlink like any other link embedded in content created by the site itself, but still, there’s some genuine value to be extracted from forum posting. 

You can check the domain authority (DA) using various SEO tools – Moz has a free one here. 

4. Network and drive traffic to your site

Engaging in forums is a solid way to drive traffic to your site and build community engagement and awareness. 

Many forums are targeted at startups, DIY entrepreneurs, and SEO beginners or novices. Being active in forums relevant to your niche is valuable beyond backlinks. 

Finding Dofollow Forums

Whilst there probably is some (limited) value to nofollow links, there are plenty of forums out there that don’t mark links as nofollow. These are the forums you want to hunt down. 

It’s worth mentioning that the vast majority of social media sites, Q&A sites, and major messaging boards like Reddit mark all links as nofollow by default. On rare occasions, sites like Reddit do allow Google to crawl links. BacklinkSEO provides a good explanation of this here. 

Aside from searching manually, there are many sites that post lists of dofollow forums:

• This dofollow forum database by Shoutmeloud contains plenty of dofollow forums. 
• Woblogger provides a similar list here. Note how there’s plenty of high-authority sites here like Overclockers and TechRepublic. 
• Here’s another option by WebSetNet. 

Search for ‘dofollow forums,’ ‘link follow forums,’ ‘backlink forums,’ etc, yourself and you’ll find tons of similar databases and lists of dofollow forums. 

Another option is to manually search Google for forums specific to your niche. 

Use the inurl:/ search string. 

For example, if your industry or niche is photography then search for:

“Photography inurl:/ forum” 

This particular search string returned tons of photography forums. It’s worth mentioning that many of these will be ridden with spam whereas others are filled with a decent level of quality content. 

Hunt down the more authoritative, relevant forums and target those first. The more authoritative and the more relevant, the better. 

Next, you’ll need to check the forum to make sure their links are follow and not nofollow. 

One solid strategy is to use the forum’s search box to search for ‘www.’ This will locate links in forum posts. You can see the result for searching ‘www.’ on PhotoForum here.

We can see a couple of links here – probably people posting for backlinks.

Now we need to check if the links are follow or whether they’re marked with nofollow. 

1. (on Chrome) Right click > View page source.
2. Hit Ctrl+F and search ‘www.’, or search some text from the link (in this case, I typed in ‘417’ to find the link pictured in the screenshot.
3. Find the link and look for a nofollow tag. If there’s no tag then you’re good to go – the link should be crawled by Google, unless there’s something else going on with the site. 
4. Backlinko has a good guide to finding follow/nofollow links here. There are a few methods you can use. 
5. You can also use a Chrome browser add-on like this one which automatically scans backlinks as you browse.
6. You can also analyze your competitor’s backlinks using a competitor analysis SEO tool. You’ll be able to quickly find forum backlinks and other quick-win backlinks that you can then go and claim yourself. 

Finding whether or not links are marked with nofollow should be quite straightforward. 

Once you’re certain that the forum is good for linking then it’s time to make a profile and get posting. 

Posting Strategy For Forum Backlinks

Once you’ve located a solid forum, it’s time to make a profile. You should be able to insert your site(s) into your profile signature and possibly even a bio/about section. This is well worth doing. 

Next, it’s time to engage with some posts. 

If you’ve found a genuinely good, active forum, then genuine engagement shouldn’t be too difficult. Some very general forums (like SiteOwnersForums) contain all sorts of posts on almost any topic imaginable – you’re bound to find a few topics to get stuck into immediately. 

It’s good practice to build up some posting history before you embed any links whatsoever. It can’t do any harm – making one to three short comments a day across say, 10 to 15 forums, should take just 15 minutes or so.

Bear in mind that many forums (especially the better ones) will have robust anti-spam bots and moderators in place to clamp down on spammy backlinks. 

After all, allowing spammy backlinks will damage the authority of the host site. 

Embedding Links

One common tactic is asking for others to ‘critique their site’. It’s a fair tactic and it’s hardly unreasonable to want to collect some feedback on your site. Make sure your site is absolutely secure before you place your first link, as you can’t be sure as to who is lurking in the forum. 

It’s probably best to avoid being too direct with your links if possible. This might seem tricky but you’re not looking to place your link loads of times – placing it in a couple of active, high-volume conversations should do the trick. 

All forums have different tolerance to linking, and many users will be none-the-wiser that you’re inserting your links for backlink purposes anyway, so long as you’re being authentic. 

Analyzing your Backlinks

Once you’ve placed some links, you can use an SEO tool to check your site’s backlink profile. 

Your backlink profile should reveal whether or not any links pointing to your site are indeed ‘follow’ and not ‘nofollow’. You’ll also receive a breakdown of the quality of the backlink and whether or not Google trusts it. 

Any spammy/dodgy backlinks that might harm your SEO are also highlighted by most SEO tools’ backlink analyzers. If you’ve accidentally placed your link on a dodgy site then you should remove it. 

Summary: Guide to Forum Posting sites for SEO

Posting links to forums remains a solid quick-win backlink strategy today. 

Quite how far these links will go to rank your site is up for debate, but the general consensus amongst SEOs is that forum posting is a solid strategy for smaller or medium-sized sites at least. 

Once you’ve identified some forum candidates, setting up your profile and posting some content doesn’t take long. 

Always pay attention to the relevance and authority of the site, and make sure that any links posted by users are marked to follow and not nofollow. 

That’s about it – it’s time to win some quick backlinks!” 

The post A Guide to Forum Link Building appeared first on SEO Administrator.

Everybody from marketers to advertisers, writers, CRMs, eCommerce professionals, and startups of every shape and size use SEO in their daily digital strategy.

But what are the real advantages and disadvantages of SEO?

Does SEO have any disadvantages? Or is it just a mantra that we drone on about through mere habit and repetition?

What is SEO?

It’s easy to get lost in the many varying definitions of SEO.

In a nutshell, SEO is the process and method of optimizing websites and other online entities (like social media accounts), to enhance their visibility in search engines. 

SEO endeavors to shuffle sites and their content to the top of the pile where users are most likely to find them.

SEO is a simple concept, but it has numerous sub-disciplines.

The core components of SEO are:

What Is The Point of SEO?

But why does SEO exist, and why is it even important?

Because search engines are the main gateway to the internet. SEO has a strange relationship with Google and other search engines.

Whilst it might be tempting to think that Google and SEO have a closely related or even symbiotic relationship, this doesn’t paint the whole picture.

Consider Google’s main aims; to make Google products and services the best in the world, to sell adverts, and to bolster their online influence. Google wants to be in control, and for the most part, they are.

The aim of SEO, in some sense, is to take back control from Google and engineer effective strategies that benefit businesses, organizations, and any other entity that requires an online presence.

In the beginning, this used to be about gaming Google’s algorithms to ‘cheat’ the system, but now it’s much more about serving the user – this is what search engines want you to do. 

Google’s focus on entirely honest and transparent organic SEO was enhanced in 2021 when Google decided to crack down on some forms of backlink exchange.

In the early days of SEO, gaming Google was pretty easy. By purchasing a few pricey backlinks and creating a quality site, your position at the top of the SERPs was all but guaranteed, especially given the relatively low competition at the time.

Google fought back, engineering new algorithms that placed increasing focus on the genuine credibility of sites. Google wants their search to work for the user, if they gained a reputation of pushing dodgy, scammy, or spammy sites to the top of the SERPs then Google would not be the massive business it is today.

SEO is now far-evolved and more complex.

The upshot is, it’s now easier for any and all businesses, organizations and individuals to harness the power of SEO.

The Advantages and Disadvantages of SEO

Without further ado, let’s look at the advantages and disadvantages of SEO.

Advantages of SEO

1)    Organic Search Reigns Supreme

Organic search is fuel for businesses. The more that people find your content via organic search, the more traffic you’ll get. You need the traffic to make conversions. SEO is top-of-the-funnel, it’s your opportunity to capture people from the SERPs.

According to BrightEdge, 68% of online experiences start with a search whereas 53.3% of all website traffic comes from organic search. Moreover, 1000% more traffic comes from organic search than social media.

SEO is a surefire way to improve digital visibility. SEO is the only tool you have to work your way up the SERPs. Paid traffic is another game entirely and whilst it can be effective, it doesn’t have the same longevity as SEO.

2)    SEO Has High ROI

But is all the money and time spent on SEO worth it in terms of ROI? Data suggest that it is – 60% of marketers told HubSpot that SEO is their main source of leads. These SEO leads had a close rate of 14.6%.

SEO converts traffic into leads efficiently, especially if the search terms have strong conversion or buying intent. For example, ranking highly for a phrase like “buy chamomile tea” gives you a strong chance of capturing leads who have their wallets in their hands already.

In comparison, running PPC campaigns can be very expensive, especially for competitive keywords.

SEO can be done DIY or with the service of an SEO professional – there’s a lot of choices out there and getting started doesn’t need to cost you anything at all. Some 70% of marketers view SEO as having a superior ROI to PPC.

3)    Long-Term Results

A solid SEO strategy will set your site up for long-term results. SEO does change as Google’s algorithms shift and add new features and improvements, but the core tenets of SEO such as content, UI/UX, backlinks, and technical SEO are not subject to a great deal of variation.

By building SEO steadily, your site will ascend the SERPs slowly but surely. Once you’ve obtained a decent position, it’s just about sustaining it. In a sense, SEO becomes easier with every rung you ascend in the SERPs. Once you’ve built authority and presence, it’s hard to lose it.

By comparison, PPC is quite unstable and volatile. SEO is much more stable and has greater longevity, especially if you create excellent evergreen content.

Disadvantages of SEO

1)    SEO is Long-Term Strategy

The advantage of SEO’s longevity doubles up as a disadvantage because of how time-consuming it is.

SEO strategies take some time to implement and though some gains can occur as quickly as it takes Google to index the page, it can take several months for multiple efforts to accumulate their benefits.

SEMRush says it can take 6 to 12 months for SEO strategies to reap rewards. It depends on your work rate, expenditure, and the competitors you need to beat to ascend the SERPs.

2)    No Real Guarantees

Your SEO efforts are not guaranteed to work, at least not as quickly as you’d like them to.

Google expressly warns against SEO companies that guarantee first-page results. SEO is not a get-rich-quick scheme, even though there are a vast number of success stories and rags-to-riches tails of startups and SMEs blossoming into much bigger companies via the power of SEO.

So long as you moderate your expectations, this is not a real issue.

3)    You’ll Have Competitors

Even if you think you have a very strong niche with low competition, it’s only a matter of time before new competitors enter your space. The internet is the most competitive space in the world. Some keywords like ‘car insurance’ or ‘mobile phone brands’ are likely to be absolutely dominated by huge sites with enormous SEO clout.

This doesn’t mean there are no opportunities, even in competitive areas. But, the more competitive the niche or subject, the more shrewd you’ll have to be, and you’ll have to invest more time into your content, link building, social media, and other SEO juice.

SEO is an ongoing competition and you’ll need to stay on the ball to prevent competitors from leapfrogging you.

Once you’ve found some decent rankings, maintaining them is crucial. Sometimes, there’s very little you can do to remain competitive for a specific keyword or phrase, but you’ll still be able to work around that keyword or phrase to out-maneuver competitors and stay on-trend.

Summary: The Advantages and Disadvantages of SEO

SEO is a unique discipline in many ways. It works both with and against Google and other search engines.

The advantages are clear – search engines are the machines between global web traffic and SEO is your opportunity to feed them what they need to rank you higher in the SERPs.

SEO does take time, dedication, and skill, but it’s not rocket science either, even if it might seem like it is at times.

When in doubt, keep it simple and focus on your bottom line; content, on-page, and off-site SEO.

Usually, it’s only a matter of time before you start to see the rewards.

Good luck!

The post The Advantages and Disadvantages of SEO appeared first on SEO Administrator.

The passive and active voice are both ways of phrasing a sentence that includes a subject, object, and verb.

Both active and passive voice have their place in writing, but the active voice is more direct, impactful, and rhythmic with a stronger narrative flow.

The vast majority of advertising and marketing copy exclusively targets the active voice for these reasons.

Take these marketing slogans:

Clearly, the active form of these short phrases is more impactful. The passive formation has identical meaning but throws in unnecessary words that clouds the phrase’s impact.

But how does this fit in with SEO?

Language and Google

Google’s algorithm has understood natural language to some extent for nearly a decade, but natural language processing – or NLP – has become much more advanced in recent years.

With BERT, SEOs can assume that Google is very competent at analyzing most elements of natural language.

So, granted the knowledge that Google understands the language very well, could this be an SEO ranking factor?

Language and SEO: A Complex Relationship

Language is something we often take for granted, but it’s hugely complex and thus, SEO and language have a complicated relationship.

There are numerous ways to construct sentences with near-identical meaning. This is why building NLP algorithms is so difficult.

Consider the following sentence:

“I saw a man on a hill with a telescope.”

It’s a seemingly simple sentence, but there are many different potential meanings here:

1. I was on a hill when I saw a man with a telescope
2. I’m watching a man on a hill with my telescope
3. The man on the hill has the telescope
4. There’s both a man and a telescope on the hill

For SEO purposes, Google has to try and unpack the meaning of these sentences to properly rank the phrase.

Since the phrase is so ambiguous, it will have to use the surrounding context of the phrase to understand what the passage is really about. This is consequential for SEO purposes.

SEO, as we may know, is about making Google’s life easier – writing is a big part of that.

Writing unambiguous text is potentially important for SEO as it helps algorithms understand the meaning of the text, e.g. that this blog post is about my sighting of a man on a hill using my telescope and not my sighting of a hill with a man and telescope on it.

Writing content that is clear, articulate, and error-free may directly or indirectly impact SEO performance.

What Google Has to Say

Google Webmaster guidelines reference the quality of written content in several places:

Google states: “Create a useful, information-rich site, and write pages that clearly and accurately describe your content.”

Google states: “Avoid writing sloppy text with many spelling and grammatical mistakes.”

Google states: “Creating high-quality content takes a significant amount of at least one of the following: time, effort, expertise, and talent/skill. Content should be factually accurate, clearly written, and comprehensive.”

Even back in 2011, SearchEngineLand found that content littered with spelling and grammar issues often ranked lower than accurately written pieces.

If there are two similar pieces competing for a hotly contested space in the SERPs, the better-written piece might just edge it.

However, Google has also said that they do not actively penalize sites with poorly written content so long as it doesn’t impact user experience.

As ever in SEO, the waters here are pretty murky. But, it’s still safest to assume that you need to produce excellent, clear, and error-free copy

1. For your users
2. For Google

And that’s why you need to think about passive vs active voice.

Passive vs Active Voice

If you’ve produced web copy with writing tools such as Yoast SEO, you’ll likely have come across the passive and active voice already. You can use the Hemingway editor to quickly check content for passive voice.

Simply defined, in an active voice sentence, the subject acts on the verb.

The subject in a passive voice sentence is a recipient of the verb’s action.

Here are a few examples:

Active: The team laid the road quickly.

Passive: The road was quickly laid by the team.

—

Active: I will clean the house on Monday.

Passive: The house will be cleaned by me on Monday.

—

Active: The writer created the article.

Passive: The article was created by the writer.

—

In some of these examples, you could strip away the ‘by me’:

Active: I will always remember my first trip to London

Passive: My first trip to London will always be remembered by me

Passive 2: My first trip to London will always be remembered

The last example here is ambiguous – it doesn’t specify that you are remembering that trip and not someone else.

When Passive Voice is Preferable

Overuse of passive voice does impact the readability of a passage, but it can sometimes be unavoidable.

If you’re reflecting on a process or events that occurred in the past tense then you’ll probably rack up more uses of the passive voice than if you’re writing about something that is happening right now.

Active voice is also easier to use when describing something that happens on an ongoing basis, e.g:

Active: Global warming caused a 3% rise in sea level in 2015.

Passive: A 3% rise in sea level was caused by global warming in 2015.

Here, the passive version of the sentence is clearly inferior. It doesn’t change the meaning of the sentence, it just throws in an extra couple of words that affect its readability.

However, sometimes, you’ll need to use passive voice to emphasize the correct part of a sentence:

Here’s an example of when passive voice best communicates the meaning of a sentence:

Active: An elephant injured grandad

Passive: Grandad got injured by an elephant

The second sentence, phrased using passive voice, describes the incident more clearly than the first.

The first focuses more on the elephant, whereas the second focuses more on the grandad.

Here’s another example:

Active: The burglar attacked an elderly couple at 8 pm

Passive: The elderly couple was attacked by a burglar at 8 pm

The passive version of the phrase emphasizes the elderly couple. If you’re reporting on this incident, e.g. in a newspaper, then it’d be more sensical to use the passive voice.

Passive vs Active Voice: The Verdict

Overuse of the passive voice is likely inconsequential for SEO unless it clouds the meaning of a passage or otherwise affects its readability.

But, there’s a thin line here, because losing readers due to ambiguous writing will affect SEO.

Away from optimization, the active voice can encourage action which may yield higher conversion rates, engagement, and enhanced shareability of content.

For marketing, promotional content, and advertising, a strong command of the active voice is essential. 

The active voice is also encouraged in scientific writing, medical writing, healthcare, therapy, and self-help.

It’s also worth mentioning that passive voice is by no means the only factor that can affect the readability of written text. Proper spelling, grammar, and formatting are all UX factors that Google understands.

The subject, message, and tone of a passage should also be well-aligned with the subject.

Whilst a post encouraging users to try or buy a product should be packed full of short, snappy, and motivational content, a reflective post on sensitive emotional themes ought to evoke a slower overall pace.

There are all potential ranking factors if they affect UX, so think carefully about written content and the role of passive and active voice.

The post Does the Use of Active vs Passive Voice Affect SEO? appeared first on SEO Administrator.

Not only did Google require engineering prowess and a unique penchant for digital innovation, but it also required long-term strategy, agility, and foresight.

To remain ahead of the curve, Google has had to undergo huge development.

Now, Google serves some 90,000 searches each second – since you’ve been reading this, Google has served millions of searches for millions of people worldwide!

Of course, many of these stats are indicative of how technological capacity has grown exponentially on the whole, but it also begs the question of how Google manages to do it.

What Are Data Centers?

Data centers are buildings that house data infrastructure and other technology, including the various software stacks required to run the technology.

Data centers are comprised of servers, data storage devices, and processor systems, plus a vast plethora of supportive technologies, power grids, and security systems.

Google’s data centers are unique in that they house the hardware and software required to maintain Google’s services, including Google Search and the Google search index which is over 100,000,000 gigabytes in size.

The data center has to store this data, but also has to serve it to the user.

Data Center Technologies

Data center technologies include:

• Routers
• Servers
• Application-delivery controllers
• Software security (e.g. firewalls)
• Hardware security (e.g. biometric or passcode entrances and exits, blast-proof walls)
• Storage systems
• Switches
• Environmental control (e.g. temperature, dehumidification)
• Cooling
• Load balance and fault tolerance software for managing issues and errors
• Predictive maintenance tools to measure system decline and forecast/schedule engineering/maintenance works

How Many Google Data Centers Are There?

Google currently has 21 data center locations.

Between these 21 data centers, in 2016, it was predicted Google had some 2.5 million servers.

The current figure is likely much higher, especially as Google has invested considerably in providing cloud computing services to rival AWS and Microsoft Azure.

There has been considerable speculation that Google owns backup or other secret data centers, but these have not been verified to date.

It’s worth mentioning that Google’s cloud data centers are not the same as the data centers they use to serve Google services, though there is likely some cross-over.

Google Cloud data centers are available in 24 regions right now in 2021.

Google Data Center Technology

Google’s Data Center technology can be broken down into two broad categories:

Indexation

Google says they know about some 130 trillion pages or more, but this doesn’t represent how many pages are actually in their index.

The index is different from the internet. Imagine the index as the collections of books that the librarian has read, or analyzed, for placing in the library database. Aside from these books, there are stacks of other books the librarian hasn’t looked indexed – they may have had a look and decided the content is not worthy of the index.

This information resides in what has come to be known as the “deep web”. Whilst the connotations of the deep web is often dark or even criminal, this really isn’t the case.

The deep web is mostly full of unpublished or half-finished or half-dead material, any content that resides between paywalls or sign-in credentials, content that the owners have blocked from indexing, and much more.

Google’s index works via an inverted index system where the index obtains documents listed by query. When you use a query on Google, it will retrieve the relevant material from where it’s stored in the index and serve it to you.

Documents IDs are divided into what are known as database shards, divided and replicated between many servers.

In 2010, Google designed a new index called “Caffeine” that continuously crawled the search index to update it at rapid periodical intervals. These days, the entire search index is created by crawlers that work in tandem with indexation algorithms to locate, parse, save and rank internet data.

Google Data Center Server Types

Google uses many hardware and software technologies, some are publicly known, others are secret.

There are several server types:

• Web servers form the backbone of the data center’s networking tech. These execute the queries of internet users. Queries are sent from Google Search to index servers. These are coordinated with the ranking algorithms and also receive information on auto-suggest and other search features. The servers will deliver search results and all other relevant information.

• Google stores internet documents and files in document servers. These return documents relevant to queries.

• Google employs spiders and crawlers that live inside the web. Google’s crawler is known as GoogleBot. This robot network crawls the web to locate new and existing web pages for potential ranking in the index.

• Other servers manage other Google services ranging from Maps to Shopping and Ads.

Environment and Security

Google Data Center security was hush-hush until Google published some posts on the interior of their data centers. Indeed, they’re not as secretive as they once were.

Security is provided by a multitude of entry protocols and security tiers controlled using badges. Retinal scanning and multi-stage doors are also used at some data centers.

Hardware and software security is obviously a huge priority and there are numerous in-house-developed and third-party technologies used here to safeguard both data and hardware from all forms of vulnerability and attack.

Some of the most common criticisms of data centers revolve around energy usage and environmental impact. Some data centers are thought to use billions of gallons of water daily to cool servers and other hardware.

Back in 2016, Google pledged to use 100% renewable energy and is one of the world’s biggest corporate investors in renewables technologies today. It’s unclear what progression has been made on that earlier claim, but progress has certainly been made.

How Google Serves Your Query

The data center is the brain and lifeblood of Google Search and other Google services.

Google has become phenomenally intelligent over the years and is now able to serve queries with extreme accuracy.

This involves a few steps:

• Google uses natural language processing (NLP), a branch of machine learning (ML), to understand your query. Different operative words you use will alter the results.
  
  For example, ‘change’ can be an entirely different operative word to ‘replace’, despite them being interchangeable in some contexts. Google will understand when ‘replace’ is being used differently to ‘change’ to modify the query.
  
• Once Google understands your query, it will calculate the most relevant responses. This is where SEO comes into play as Google will have already ranked responses for that query.
  
  Google uses a priority system to allow it to serve users with the best results. Optimized pages that Google thinks will help the user will be at the top of the pecking order in the search engine results page (SERPs).

• Google will also take the context of the search into account, including your location and other personalized variables.
  
  For example, if you’re currently in Barcelona and Barcelona vs Real Madrid is on that night, the query “football” will bring up results about this match.

All of these actions are processed at Google’s data centers. They can be thought of as super-computers that are networked to the web via colossal numbers of servers.

Of course, Google’s data centers will also process other information and data related to their business, whether that’s data related to the Android mobile operating system or data related to their DeepMind AI platform.

The post Google Data Centers Explained appeared first on SEO Administrator.

Clickable tables of contents are one such ‘SEO gadget’.

There are so many indirect and direct SEO ranking factors ranging from off-page SEO such as backlinks to on-page SEO, e.g. content, and technical SEO.

But the way you design your pages and content is also important, and Google has placed great emphasis on user interaction and UX in recent years.

Adding a table of contents to your posts is a quick and easy way of adding a great-looking UX feature for users, as well as placing some keywords at the top of your text.

The Benefits of Using Tables of Contents for SEO

Building a table of contents is an easy way to augment your content for both SEO, UX, and the middle ground that exists between the two.

Here are some benefits of building tables of contents for your blog posts and articles:

How Do You Create Tables of Contents?

Your theme might have a table of contents plugin or content block already – it’s worth checking. 

If not, creating great-looking, user-friendly tables of contents on WordPress is pretty easy with the use of the following 4 plugins:

These should provide a breakdown of your headings along with clickable links that take users to that heading in your content.

On-Page SEO and UX

On-page SEO and UX work together. They refer to the characteristics and components of your pages and content.

UX probably didn’t matter much in SEO until the early 2000s or later. Prior to that, webpages were generally so basic anyway that ‘anything that worked’ probably provided a good UX.

When web page design became more diverse and the options grew, webpages started to look different and displayed more features to make themselves unique, add functionality and provide a stronger user experience.

UX then became a central focus of web design, and Google followed suit with a sequence of algorithm changes to reflect this.

Since the early days of SEO, as SearchEngineLand states, Google’s Panda, Penguin, and Hummingbird algorithm updates have reflected the metamorphosis of the web from raw textual data into a much richer, multimedia environment.

But with great media comes great responsibility, and it’s been a challenge to deliver the functionality of complex web pages through quality UX.

What is UX?

Consider social media giants like Twitter, Facebook, and Instagram. Twitter’s UX is more oriented around the tweets themselves – it’s a reel of continuous textual and photographic data, organized via hashtags and trends.

Instagram, on the other hand, is a more visual platform, and thus it has a more visually-oriented UX. Text is mostly replaced by images.

Facebook is somewhat of a mixture but it’s also oriented around networking. As a result, Facebook’s UX is optimized for navigating and interacting with people and brands.

UX For Your Site

But what does UX mean for your site?

It depends on the site, but in general, content needs to be attractive and easy to navigate. Users want to get to what they want almost immediately. That means fast, accurate, and well-signposted pages.

UX involves some technical SEO, but mostly on-page SEO.

In terms of technical SEO, load time is one of the greatest UX factors. A great user experience is a fast user experience, and tests by Pingdom found that bounce rates could jump by some 20% if a site loads for longer than 3 seconds or so.

Page speed is always worth tackling, and most fixes are straightforward. You can use SEO audit tools to get to the bottom of why your pages are loading slowly, if they are at all.

UX and Content Design

Regardless of how complex and flashy your website is, every site has one thing in common when it comes to UX:

Content has to be easy to read.

All the beautiful graphics and menus in the world can’t help you if your content is hard to read!

Content design is the process of making sure your content is well-optimized for the reader.

Google Wants People to Read Your Content

Written content is the main-stay of the internet, and Google wants to know that people are actually reading it on your site.

There are two signals here:

1. Bounce rate – the rate at which users click off your page/site soon after visiting it
2. Dwell time – the time users spend ‘dwelling’ or interacting/read with your page

It’s very difficult to quantify how Google looks into these signals and how they may/may not affect SEO.

Ahrefs concluded that dwell time is a valuable indicator of how engaging your pages are, but that it was unclear whether or not this would be a direct ranking factor.

CXL and Moz, however, argue that Google discerns between ‘short clicks’ i.e. bouncing clicks where someone spends little to no time on your site, and ‘long clicks’, when users spend more time on your site.

Regardless of whether content engagement, dwell time or long clicks are actual ranking factors, designing your content to fulfill the user’s query is very likely to yield SEO benefits.

Content your users actually read ought to be an indicator of ‘search success’ – a validating signal that lets Google know your page is earning its spot in the SERPs.

That’s why you should add a table of contents to your pages!

Tables of Contents for SEO

The concept of a table of contents is simple.

Placed at the top of a page, they let users view a quick breakdown of your article or content for easy navigation purposes.

If they click on a heading link, (also known as a jumplink), then they’ll be taken to that relevant heading – no need to scroll manually.

For UX this is very important. When someone lands on a post with a table of contents, they can easily find their answer quickly.

This at least reduces bounce rate and could increase dwell time or the chance of a ‘long click’ compared to if the user tried to scroll through the post for info and gave up before finding their answer.

When to Use a Table of Contents?

Tables of contents are most useful for longer content with plenty of headings.

You can add them to any content, though, and they do yield some technical and on-page benefits UX, as mentioned.

There’s no reason why you shouldn’t add a table of contents to all content with say, 3 or more headings, and over 500 or so words.

A quick win SEO gadget with potentially measurable results – tables of contents are a simple but effective tool to add to your SEO arsenal!

Summary

Adding a table of contents to your pages is simple but effective for both UX and SEO. UX and SEO are related in many ways, so it’s pretty much a win-win all round!

Tables of contents enhance the usability and navigation of your site, and that counts for a lot.

It’s pretty much as simple as that, so the answer is yes, click tables of contents are indeed good for SEO.

The post Is a Clickable Table of Contents Good for SEO? appeared first on SEO Administrator.

XML sitemaps are files that provide Google and other search engines a map to your site and its various URLs and pages.

Consider when a crawler arrives on your site. Without an XML sitemap, it has to follow links around your site to locate new content for crawling and eventually, indexing.

With an XML sitemap, the crawler is signposted to the parts of your site that you want it to crawl and index.

XML sitemaps are signposts that urge crawlers on the right path round your site.

Without a sitemap, your site will still get indexed so long as you have proper internal linking, so don’t panic if you’ve never used an XML sitemap before.

As far as SEO goes? XML sitemaps are supportive of good SEO but will only directly influence it under specific circumstances.

You’ll need to set your site up with Google Search Console and other search engine webmaster tools to point them to your sitemap.

Crawlers can also find the XML sitemap in your robots.txt file if you specify the path to the sitemap there.

Do You Need an XML Sitemap?

XML sitemaps aren’t strictly essential – your site will still get indexed without one, so long as pages are properly linked by internal and/or external URLs.

But that doesn’t provide the whole picture as XML sitemaps certainly improve the efficiency and quality of indexing and this does impact SEO.

Google themselves state that XML sitemaps are only 100% essential if your site is reasonably large (over 500 pages) or features a lot of rich media.

The problem is, by modern standards, sites with 500+ pages can still seem small, and many ‘small’ sites will exceed this if they have a store with quite a few products, a blog, etc.

Other situations when you definitely need an XML sitemap is when your site has a complex structure with a lot of disparate or isolated content that might get overlooked by crawlers if you don’t provide a sitemap.

In these situations, crawlers may never reach your newly created content if you don’t have a sitemap to direct them there. Obviously, this is a problem; they’ll arrive on your homepage, explore a little and pretty much get lost crawling the same pages until your crawl budget depletes.

This could leave parts of your website uncrawled and therefore, unindexed!

Why You Should Create an XML Sitemap

Even if Google states that you don’t need an XML sitemap, particularly if your site is small (though you might be surprised as to how large your site really is), it’s definitely good practice to create one for any site. 

Creating an XML sitemap is pretty easy, and once you upload your sitemap into Google Search Console, you’ll be able to check when Google visits it and crawls your URLs, which is very useful in itself.

Providing a sitemap to Google will also help get your site indexed quickly and you’ll be able to check and track the crawl status of your URLs from the Sitemaps section of Google Search Console.

For this alone, creating a sitemap is well worth it.

When You Should Create a Sitemap

It depends on how you’re constructing your website or what platform or theme you’re using.

If you go down the WordPress route, your theme will already have some sort of built-in URL structure, e.g. you might have a homepage, blog, about, shop, portfolio and other related pages.

Shopify and other platforms will auto-generate XML sitemaps, but you still might need to modify them at some point.

Generally speaking, once you build your site out with some content and get your site set up with Google Search Console, you should create and add a sitemap.

When you make small changes to your site, there’s no need to update the sitemap unless it’s very large and you’re concerned about your crawl budget.

Dynamic Sitemaps

Some sitemaps generated using plugins like Yoast SEO are known as ‘dynamic sitemaps’ and will auto-update when you add and remove content from your site.

These plugins ‘ping’ search engines to alert them of updates (which may get new content indexed quicker, but it’s hard to tell and you can’t count on it.

Where Do You Put the XML Sitemap?

The XML sitemap should be placed at your site route, so www.example.com/sitemap_index.xml.

You place your sitemap manually with the file manager of your web hosting software (such cPanel), but you can also use a plugin to do this for you (and many will also generate the sitemap also).

Once you’ve placed your sitemap file at the root, you can head into Google Search Console and head to Sitemaps > Add a New Sitemap.

For other search engines, e.g. Bing, Yandex or Baidu, you’ll have to use their equivalent webmaster tools to upload your sitemap. You can also add the sitemap to your robots.txt so they can find the sitemap automatically when they crawl your site.

Step-by-step:

1. Create a sitemap using a sitemap generator tool or plugin
2. Add the sitemap to your site’s root (manually or with a plugin)
3. Add that sitemap to Google Search Console
4. Wait for confirmation that Google can see your sitemap, your URLs should be mapped and crawled soon after
5. You can also add your sitemap to any robots.txt files you create for your site

Advanced Sitemap Attributes and Controls

XML sitemaps can contain complex attributes that provide in-depth signposts for crawlers visiting your site.

XML sitemap attributes include priority, last modified and change frequency, all of which are designed to instruct crawlers on what your important URLs are and how often they change.

You can alter these if you like (using in-depth XML sitemap generators like Screaming Frog), but many search engines pay little or no attention to them.

For example, Google stated that they do not consume priority attributes in sitemaps.

XML Sitemap Tools

Screaming Frog XML Sitemap Generator

The Screaming Frog XML sitemap generator is part of their SEO Spider tool that crawls your site, locates its URLs and analyzes various on-page and technical SEO characteristics.

In the process, it will crawl your URLs for XML sitemap generation.

Screaming Frog provides a full breakdown of your site’s URL structure, allowing you to create an XML sitemap that is modifiable with attributes such as priority and change frequency.

You’ll also be able to select any URLs you don’t want to include in the sitemap (though they’ll still be crawled if there are internal or external links pointing to them).

The free version of the SEO Spider will crawl up to 500 URLs which you can use to create your XML sitemap.

Features

• Crawls your site for URLs
• Allows you to add many types of sitemap attributes (e.g. priority, change frequency)
• Up to 500 URLs for the free version of the SEO Spider (which does a lot more than create sitemaps)

XML Sitemaps

This longstanding XML sitemap generator tool has been around since 2005 and is a quick and easy way to build a standard XML sitemap for any site.

Simply insert your site into the field and the tool will crawl its various URLs to generate a downloadable XML sitemap to place in your website root and Google Search Console.

The free version of the tool will map up to 500 pages. There is a premium version of the tool that creates dynamic sitemaps for larger websites, and it will ping them to search engines so you don’t have to upload them yourself (but you should anyway).

An excellent, simple free tool for creating XML sitemaps for upload to your site’s root and Google Search Console.

Features

• Free tool crawls up to 500 pages
• Downloadable XML sitemaps
• Premium version ideal for super-sized sites

XML Sitemap Generator

Another super-simple, free XML sitemap generator, this free tool will crawl up to 2,000 URLs, more than broadly equivalent free tools.

There are also several options for creating advanced XML sitemaps with change frequency, priority and exclusion or filtering settings. It’s only worth fiddling with these settings if you’re an advanced SEO user or want to manage a crawl budget problem.

The company also offers a WordPress XML sitemap plugin that lets you choose whether or not to add new content you create for your site.

Features

• Generate XML sitemaps for up to 2,000 pages
• Advanced attribute settings
• WordPress plugin also available

My Sitemap Generator

This is another super-simple sitemap generator suitable for the vast majority of standard websites. Simply enter your root URL to create your sitemap.

You’ll be able to set priority and change frequency for specific URLs if you like and can add attributes to instruct crawlers whether or not to crawl certain content (e.g. javascript).

The free tool accommodates sites with up to 500 pages. There’s plenty of advanced settings here if you’re looking to manage a crawl budget problem (unlikely for small sites).

Features

• Free for up to 500 pages
• Set advanced sitemap instructions and attributes
• Set URL change frequency and priority

Yoast SEO

The fantastic and widely known Yoast SEO plugin includes an XML sitemap feature that is free to use.

The plugin will automatically update your sitemap as you add/remove/change content on your site, which is obviously great.

The XML sitemap feature is very easy to activate, and once you’ve done it from within the plugin menu in WordPress, you’ll be able to view your sitemap and submit it to Google Search Console and other webmaster tools.

You’ll probably notice that there are no additional settings to adjust priority or change frequency, but as stated, you really don’t need to worry about these too much.

Features

• Auto-generate dynamic XML sitemaps for any WordPress site
• Sitemaps are auto-updated
• Simply point Google Search Console to the sitemap that you can open from within the plugin

Google Sitemap Generator

Though named Google Sitemap Generator, this sitemap generator plugin is compliant with most search engines and allows you to auto-generate sitemaps for WordPress sites.

When you update your site or create new content, the plugin ‘pings’ Google and other search engines with an alert that your sitemap has been updated.

This plugin does allow you to edit priorities, change frequency and other parameters if you want to. It’s extremely well-rated, reliable and easy-to-use for anyone.

Features

• Auto-generate dynamic XML sitemaps for any WordPress site
• Auto-update sitemap and alerts Google and other search engines to changes
• Allows for changes to advanced sitemap parameters and attributes

Rank Math

Rank Math is an extremely sophisticated WordPress SEO plugin that provides unparalleled control over numerous on-page and technical SEO settings ranging from schema to sitemaps.

It doesn’t just cover XML sitemaps but also news and video sitemaps that can help Google index your content and media to Google Videos and Google News.

The XML sitemap generator is exceptionally easy to use and modify, and you’ll be able to allocate URLs that you don’t want to include in the sitemap.

In general, though, Rank Math is a gem of a WordPress SEO tool that has been featured by many top SEO brands. The XML sitemap feature is available in the Standard version at $59 a year (and is also obviously available in the more advanced, expensive tiers).

Features

• WordPress SEO plugin
• Powerful sitemap features
• Tons of other tools available in the plugin

Site Robot for Shopify

This is an XML and HTML sitemap generator plugin for Shopify.

It essentially creates an XML sitemap for any Shopify site for upload to Google Search Console.

The only real advantage it lends over auto-generated Shopify sitemaps is that you can choose URLs to exclude (and that it lets you create HTML sitemaps too, which are a different type of sitemap that pull all your site URLs onto one page of your site).

Features

• Auto-generate XML sitemaps for Shopify
• Exclude URLs
• Also generates HTML sitemaps

Magento XML Google Sitemap Generator

For the Magento CMS, this sitemap generator is designed to quickly and efficiently create very large sitemaps for super-sized sites (typically enterprise-level or larger business sites with thousands of pages).

It’s a powerful tool that allows total control over sitemaps and it’s capable of mapping the maximum number of URLs Google permits for a sitemap (50,000 URLs or 50MB).

This is very useful for large Magento stores with potentially thousands of products and many dynamic pages. Essential for enterprise-level crawler budget management for Magento.

Features

• Sitemap generator for Magento
• Works with XL websites
• Plenty of control over attributes

Summary

Creating and implementing XML sitemaps for Google Search Console and other webmaster tools is straightforward and whilst the jury is out on whether or not they confer direct SEO advantages, XML sitemaps definitely qualify as SEO ‘good practice’.

For some sites, there’s no doubt of the necessity of XML sitemaps and properly implementing them can get your site indexed properly, also helping you manage crawl budget problems.

It’s also very useful to be able to view and navigate your site in the Search Console under the Sitemaps tab.

This provides you with an easy way to monitor the URL structure and indexing status of your site and its pages.

The post XML Sitemap Generators appeared first on SEO Administrator.

Whilst you can control many things associated with your brand, business, organization, or individual reputation, poor reviews, bad press, and other damaging stories can come out of the woodwork to haunt you in the SERPs.

According to The Harris Poll, 4 out of 5 consumers search a business on Google before purchasing from them. Meanwhile, 86% of people rate local business reviews as important, and over half look for businesses with 4-star reviews or over.

Clearly, trust is a massive priority and sharp-eyed internet users will always pick up on negative press, stories, and reviews in the SERPs.

In short, when bad press strikes your brand, you need a strategy to fight it.

Reverse SEO is one tactic you can use.

What is Reverse SEO?

Reverse SEO is a strategy whereby you identify problematic web pages associated with your brand and attempt to push them down the SERPs.

This could involve a combination of pulling up positive or neutral results that appear below the bad results in the SERPs to push the bad results down, creating new content to rank for the keywords of the negative content, or even creating new sites to take the space of the bad in the SERPs.

The negative content you’re targeting will usually return for queries involving your brand/business/personal name.

Pages that might impact your brand could include:

Locating this content is simple when your brand, organization, or personal name is used. But otherwise, it might rank for your products, services, employees, or other aspects of your business.

It’s worth bearing in mind that if the negative content itself is very poorly optimized then it’s not likely that many people will see it – that’s why it’s best to focus on content that ranks for branded or personal mentions.

Is Reverse SEO Black-Hat/Allowed by Google?

Reverse SEO is not a black-hat SEO tactic, but negative SEO is.

There is some crossover between the two and it’s important to bear in mind that a poorly executed reverse SEO campaign can easily become black-hat and serious repercussions are possible.

Fortunately, so long as you stay within the lines with reverse SEO, you’ll be fine.

Negative Vs Reverse SEO

Negative SEO is a black-hat technique, unpermitted by Google and other search engines.

Negative SEO ranges from producing fake reviews on a competitor’s products to flooding their pages with spammy links or holding sensitive information for ransom. Brand mentions could also be ranked alongside NSFW content.

Not only is negative SEO expressly disallowed by Google, but it can also be situationally illegal. 

Before You Consider Reverse SEO

Before you consider reverse SEO, you may be able to request that Google removes the harmful content from the SERPs for you.

Google’s removal policies revolve around legal issues, privacy breaches, security, and the potential for personal harm.

They keep free speech in-tact, meaning that they won’t apply to general negative press or other bad coverage that affects your brand.

Google has 6 main content removal policies:

In any of these situations, Google should help you remove the content in question from the SERPs.

Google can also remove fake Google My Business reviews, providing you have a strong case.

You can also have a look through the site and find out whether they’re breaking other Google guidelines, e.g. they’re engaging in negative SEO or are publishing illegal or spam content.

Contact the Site Owner/Manager

Secondly, it’s always worth contacting the site to ask what’s going on and get them to explain their side of the story. Tread carefully and kindly request they reconsider their views or otherwise take down the content.

This is a good shout if the negative press/review results from a rare defect product, a misunderstanding, or a complaint that is no longer relevant/has been dealt with internally by your company or similar.

So long as you’re professional, courteous, and have a strong, genuine case, then there’s a good chance this will permanently solve the issue!

Reverse SEO: Research Stage

The first stage of an SEO campaign is locating the target pages and analyzing them for competitive intel.

You’ll need an SEO audit tool or competitive research tool as provided in SEO suites such as Moz, Ahrefs, UberSuggest, and Semrush.

Trust Score and Domain Authority

You’ll first need to look at the site’s trust score and authority.

Needless to say, if you’re a small business and you’ve been unlucky enough to get slapped by a bad review or negative press from a major publication then there’s likely no easy way out.

If the domain has a low trust and authority score and the content itself is thin then you’ve got a strong chance of pushing them down the SERPs.

Keyword Analysis

The next step is to analyze the keywords used by the page in question.

These are the keywords you’ll have to target in your reverse SEO campaign. If you create on-site and off-site content around these keywords, you can rank for those keywords and resecure the SERPs.

Backlink Analysis

You’ll also want to analyze the backlinks of your target page.

This will enable you to contact those sites and request they backlink your own properties.

Reverse SEO: Organic SEO Comes First

The best way to perform reverse SEO is by bolstering your own domain’s SEO.

This technique has longevity – if you can rank in a few places across the SERPs then you’ll leave less room for negative press and reviews.

For example, if a poor review hits one of your new products hard then you can create new pages on your site dedicated to that product. These should rank themselves, thus pushing the negative reviews out of the picture.

Of course, this technique can generate traffic of its own, which is a bonus.

Always focus on building and consolidating your own quality content around your brand name and products first.

Brand Name Optimization

Negative content typically hits the hardest when someone searches your exact brand, business, or organization name (or you as an individual).

In this case, you might realize that you haven’t actually published enough optimized content for your brand name – publish more to fill the SERPs with your own positive branded content.

Press Releases and Building Positive Reviews or Feedback

If bad press or negative reviews hit you hard in the SERPs then you can build additional positive reviews and press to reoccupy the space.

Create new press releases for your target keywords and write more guest posts for other sites. The more the better! The long-term goal is to really monopolize the SERPs with your own positive content.

Build Links to New Sites

You can also build links to other positive content that appears around or below the negative page you’re targeting in the SERPs.

Look beyond page 1 of the SERPs and discover content you can link both on-site or off-site with the aim of boosting that up the SERPs.

Reverse SEO: When One Site Isn’t Enough

If you find that guest posting and publishing more keyword-optimized content on your own site isn’t doing the trick, then you can consider creating additional websites specifically for the purpose of reverse SEO.

Make sure these new sites don’t seem or look like competitors to your own site, though. They should be complementary where possible.

Summary

Reverse SEO can be very effective and eliminating thin, low-trust, or authority content from the SERPs is usually straightforward.

But if you’re dealing with a lot of negative content or content published by authoritative domains then reverse SEO can be very tricky indeed!

There are SEO reputation managers out there who can assist with high-profile reverse SEO, these could be worth looking into if the negative content is particularly damaging or malicious.

Remember to always focus on bolstering your own organic SEO around problematic keywords before resorting to other more complex tactics!

The post Reverse SEO appeared first on SEO Administrator.

The internet is a content-driven universe and words and language are its main ingredient.

The brilliant thing about words is that they’re so info-rich, they communicate everything from thoughts to feelings to topics, sentiments, opinions, and more.

This has made keywords a primary SEO ranking factor since the dawn of Google and other search engines.

Researching keywords is crucially important for the SEO of any website, and they’re quite straightforward to get started with.

Of course, any keyword strategy requires the use of some tips, tricks, and perhaps most importantly, keyword tools.

Exploring Keywords

Keywords are relatively simple to research and explore, so long as you know what you’re searching for and why.

There are also several key metrics to be aware of.

The first thing to be aware of is that there are two key ‘types’ of keywords, PPC and organic.

PPC

PPC (pay per click) are keywords you bid for. You bid for a keyword and pay each time a user clicks on that keyword – the cost per click (CPC). High competition keywords have higher bid prices.

An example would be ‘insurance’, which is ultra-competitive and could have bid prices in the $20 to $50 region – very expensive indeed!

Your bid helps you secure your ad for your targeted keywords in the SERPs.

PPC keywords are:

Organic

Organic keyword targeting should be the primary focus for long-term, sustained SEO.

These types of keywords are unpaid – they’re nothing to do with Google AdWords.

Organic keyword targeting revolves around content creation. The idea is to home in on keywords that people use to obtain certain results and rank your page in the SERPs for those words/terms.

Keywords include everything from single words such as “tea” to long-tail keywords that involve phrases such as “How do you make Chai tea?”

Organic/SEO keywords are:

Before we move onto the best keyword tools, let’s quickly review the key metrics you’ll need to bear in mind when conducting keyword research:

Important Keyword Metrics

The most important keyword metrics include search volume, search volume over time (trends), keyword difficulty (KD) and the commercial value of the keyword.

Sadly these rarley exist!

This means people are searching for these keywords but finding…well, relatively little as there are few pages competing in the SERPs for those keywords!

SearchEngineLand dubs these the “Holy Grail of SEO”.

Furthermore, you want to bear the purpose of the keyword in mind, as it will have to fit the content you embed it in.

There’s no point in using the keyword “buy Chai tea” to create content on making Chai tea – “how to make Chai tea” would be a much better option instead.

Similarly, if you’re selling Chai tea on the page you want to rank then targeting “how to make Chai tea” probably won’t work.

Without further ado, let’s look at the best keyword tools around, both paid and free!

The Best Paid Keyword Tools

Semrush is an SEO titan and they actually offer 6 powerful keyword tools in their full membership packages.

These include:

• Keyword Overview – For rapidly checking new keywords and assessing their viability
• Organic Research – Analyze competitor’s keywords and discover how others are using keywords in your niche
• Keyword Magic Tool – Super-easy keyword tool that lets you find semantically related keywords and search volume over time
• Keyword Gap – Compare your keyword profile with others to identify gaps you need to fill
• Keyword Manager – Manage keywords and view how well they perform in the SERPs
• Organic Insights – Analyse keyword traffic and discover more keyword opportunities

Semrush also includes a full PPC keywords research tool.

Semrush SEO tools are fully comprehensive and allow you to find and research keywords in granular detail.

Brainstorm new effective keywords for your content, check difficulty, search volume, search volume over time, CPC, and SERPs performance. Plan keyword targeting and compare your content to competitors to find a competitive edge.

Prices

From $119 for Pro membership. Ranges up to $449 for businesses.

Features

KWFinder by Mangools is specifically designed for keyword research. That makes it nice and streamlined for the job, and cost-effective too.

You’ll be able to search and analyze keywords per location, including local keywords – great for small local businesses and services. All your usual metrics are in there and well-marked, including search volume, search volume over time, and difficulty.

Competitor analysis is also well-implemented, so you’ll be able to look up what keywords your competitors are ranking for/targeting. Once you know, you can go and beat their content!

KWFinder is great for PPC keywords too and you can easily compare PPC vs organic keywords to discover what you should and shouldn’t consider bidding for, which is excellent.

Prices

Starts at €29.90 per month for Basic, Premium is €39.90 and Agency is €79.90. The Basic version lets you look up 100 keywords every 24 hours which should suffice for the vast majority of smaller businesses or DIY SEOs.

Features

Ahrefs is another top name in SEO and this in-depth keyword tool contains keywords from 171 countries, making it a go-to for international keyword research. It also allows you to research keywords for numerous platforms including other search engines such as Baidu and YouTube.

Brainstorming keywords is also super-easy and each keyword will display a ‘parent topic’ allowing you to move between more niche terms and words and broader topics or entities. You’ll also find questions relating to your keywords providing yet more content ideas and ways to rank.

SERPs analysis is also straightforward, so you’ll be able to drill down into what keywords make the SERPs tick and how well your competitors are targeting these.

PPC research is also fully comprehensive, analyzing the effectiveness of PPC keywords and their CPC over time.

Of course, all your standard keyword metrics are in there including search volume, search volume over time, CPC, difficulty, and many more.

It’s worth mentioning that Ahrefs includes a full suite of SEO tools and you’ll get access to all of their other coveted tools in the same membership (similar setup to Semrush).

Prices

Prices range from $99 for the Lite version with 175 keyword reports per week (plenty for most single sites). Standard is $179, Advanced is $399 and Agency is $999. Remember, you get access to the whole suite of cutting-edge Ahrefs tools.

Features

Longtail Pro is oriented around longtail keywords with 3 or more words. These are amongst the most dynamic and easily-targetable keywords, making them the most likely candidates for high volume and low difficulty.

Longtail Pro helps you discover and analyze longtail keywords in superb detail. It’s essentially a robust keyword checking tool with all features for both organic and PPC keyword research.

Track volume over time, SERP performance, and spy on your competitors to find gaps in your content strategy.

The SERPs analyzer here is particularly good and is updated daily with keyword performance and ranking changes. It does also have other tools such as backlink analysis for checking backlinked keywords.

Longtail Pro also has an excellent suite of helper videos and tutorials to assist you in your keyword research journey. Overall, a superb program that is trusted by many SEOs and agencies worldwide.

Prices

The Basic version is just $25 / month and allows for 800 keyword searches every 24 hours, easily enough for DIY SEOs and smaller sites. Even the Agency version is just $98 / month which is quite impressive for 6000 daily keyword lookups.

Features

Keywords are everywhere! This browser plugin allows you to conduct keyword research from directly within the SERPs.

This semi-automatic, semi-manual approach to keyword research allows you to search queries and terms directly from within Google or other search engines.

Keywords Everywhere enables you to discover accurate keyword data and metrics as you use the internet – it certainly makes sense, allowing you to collect keywords on the go without heading into an SEO tool.

The tool pulls up keywords from related terms and People Also Searched For, and you’ll also be able to pull up keyword data and traffic for any page or site ranking in the top-20 SERPs.

All the main keyword metrics are covered in the tool including search volume, CPC and competition data. You can also analyze keyword trend data.

You can create lists of keywords for export and further analysis in more detailed keyword tools or Google Keyword Planner, etc.

An excellent holistic keyword tool, Keywords Everywhere is very good for surfing competitors sites and lifting keywords from their blog posts and articles.

Prices

This tool has a credit pricing system where 1 credit = 1 keyword. $10 will net you 100,000 credits, which is a fair quantity, though you may be surprised how quickly you get through them. From there, pricing is linear, e.g. $50 for 500,000, $100 for 100,000, etc.

Features

Jaaxy does take a slightly leftfield approach to keyword research and it packs some innovative features for affiliate marketers. There’s also a nifty trend analysis tool that lets you find and track emerging trends, discovering how these are impacting search intent so you can remain current in your niche.

Jaaxy is perhaps on the obscure side, though some users will look through it and have a eureka moment – this could be the tool affiliate marketers and other niche web content creators are looking for.

For keyword research, the tools are fairly basic but all your core metrics are in there. There are caveats, e.g. difficulty is expressed as ‘SEO’.

There is an intriguing ‘Alphabet Soup’ tool for finding keywords with high buying intent, e.g. “Chai tea” could produce results like “Buy masala Chai Tea US” – a longtail keyword with strong buying intent.

The idea is to combine these high buying intent keywords with trending niche data – it makes sense on paper and could potentially be very profitable in the right hands.

Prices

The free trial here is unlimited time-wise but only lets you have 30 keyword searches. Pro at $49/mo lets you access more Alphabet Soup searches (which may be where its strengths lie).

Features

Working with major multinational brands such as Uber, L’Oreal, Shopify, and Deloitte, SerpStat is a professional all-in-one SEO software suite that delivers impeccable results across the board.

The Keyword Research tool delves into profitable and relevant keywords associated with your niche or topic. There’s a significant focus on long-tail keywords and SerpStat is very capable of delivering useful lists of long-tail keywords to incorporate into your content.

All of your usual keyword stats and metrics are represented including search volume, competition, CPC, average clicks, and volume throughout time.

Measuring on-page SEO reveals keyword and content gaps and competitive intelligence can be carried out to identify the keywords your competitors are ranking for but you aren’t.

By all accounts, SerpStat is a powerful SEO suite of more than 20 tools that link keyword tracking and research into many other SEO tools. It’s an advanced and upcoming SEO suite that is already trusted by global brands.

Prices

Pricing starts at $69 for the Lite version, aimed at small businesses, individual users, and freelancers. It allows for 4,000 keyword queries a day. From there, the Standard package represents a significant jump to $149 and enables logins from 3 users. The Advanced and Enterprise packages cost $299 and $499 respectively.

Features

The Moz SEO suite is incredibly popular and has proved its worth for businesses of every shape and size. The Keyword Explorer is a powerful tool that covers everything from organic and paid keyword research to keyword analysis, competitor keyword analysis, and predictive keyword metrics.

The Keyword Explorer is adept at finding long-tail and question-based keywords and phrases. Keywords can be searched by country, region, or locality. Keyword difficulty, volume, and organic click-through rates are all easily measurable from within the tool.

With the SERPs analyzer, you can view how keywords perform in the SERPs, and competitor keyword analysis helps you identify keyword gaps and suggestions.

Moz’s selection of SEO tools covers all the key bases including rank tracking, site audit, and reporting.

Prices

Moz Pro starts from $99 for the Standard package that allows you to track 300 keyword rankings and generate 5 keyword lists with 100 keywords per list, which is plenty for individual users and small businesses. The Medium, Large and Premium packages are priced at $149, $249, and $599 respectively.

Features

The Best Free Keyword Tools

Google Search Console is an SEO lynchpin and provides you with an excellent way to analyze the keywords you’re ranking for and what queries users search to find your site.

Most of this is done by inspecting URLs from your site and then heading to the Queries tab. This will show you a breakdown of the queries people use to reach your site.

When you link Google Search Console to Google Analytics, you’ll be able to delve into the search behavior of people that search for relevant queries to your site.

Under the Audience tab, you’ll also find Interests and Affinity Categories, which provide a report of what your site users tend to search for and interact with online.

Features

In many respects, Google Keyword Planner is your classic keyword, but it is oriented around Google Ads (PPC advertising).

Once you input your head terms, you’ll be provided with a list of related terms which are all broken down with volume forecasts, CPC, and impressions. You can also find keywords for product categories that Google has collated.

There is plenty of organic research potential here and you’ll be able to check out Organic Impression Share and Average Organic Position as well as Competition.

Overall, Keyword Planner combines with other free Google tools to provide impressive research capabilities. You do somewhat miss the ability to contextualize keyword performance in the SERPs, and competitive analysis isn’t really possible in the same way as it is in other paid SEO suites.

Features

Google Trends is a keyword research tool you can use immediately without any sort of sign-up or integration process.

It allows you to analyze the trends of what people are searching for online, and that of course extends to the keywords they’re searching.

Google Trends is particularly powerful for identifying how search interest changes over time. By typing in keywords related to your niche, you’ll see how their search volume has changed throughout time as broken down by nation, region, or locality.

The Related Topics and Related Queries tabs are very useful for providing additional keyword and topic ideas.

The strength of Google Trends is that it’s hard-linked to what people are searching for right now, but also in the past.

Features

This awesome tool lets you conduct keyword research whilst you surf the web, which is actually a very intuitive way to go about it

You can use it when manually researching your niche or competitors. When you search a query in Google, the tool will display monthly search volume, keywords suggestions, visibility metrics, and related searches all from the plugin interface.

This allows you to quickly build a list of analyzable keywords as you toy around with different search queries.

The tool also displays location and site traffic metrics as well as CPC.

Features

The majority of search queries are indeed queries or questions.

Answer the Public harnesses this fundamental search characteristic and enables you to discover the questions that people are asking in relation to a keyword or phrase.

It’s very easy to use and will break down your query into questions by question or preposition, e.g. how, what, where, why, can, where, and what?

Finding questions surrounding your niche provides you with long-tail keywords for further analysis. This is an excellent tool for researching ‘how to’ types of content or any other content that answers the public.

Features

Looking for instant, no-fuss, no-frills keyword ideas? Soovle provides a breakdown of top keywords by search engine/platform. YouTube, Amazon, Google, Bing, Yahoo, Wikipedia, and eBay are all represented.

Soovle is brilliant for providing keyword ideas with different intent. For example, you can see if keywords align between different search engines, allowing you to formulate a content strategy that takes into account Google search queries as well as eBay or Amazon buying intent.

Features

Keyword Sheeter takes your terms and spews out a massive list of related terms.

It’s a solid tool for providing a bulk of keywords for further analysis. You can also use the tool to buy different reports on your generated list, e.g. on search volume, CPC, and difficulty.

Overall, though, the best use of this tool is to generate a raw list of related terms for any given keyword.

Features

The aptly named Keyword Tool allows you to search for a limited number of keywords for free.

You can research keywords for Google, YouTube, Bing, Amazon, eBay, Play Store, Instagram, and Twitter, which is very useful.

The resulting list will be very similar to what Keyword Sheeter and Soovle provide and you’ll have to pay to view search volume, CPC, trend, and competition. You can also sort by questions and prepositions.

Another quality tool for finding a list of keyword starting points for further research.

Features

Similar in some respects to Answer the Public, Question DB provides a list of questions related to an input keyword. It does also display a list of standardly related keywords.

It will display 50 results for free and you can also discover topics related to your keyword.

Features

Summary

The SEO software market is hardly lacking for competition and choosing the right tools can be a tricky one!

Big brand tools from Moz, Semrush, and Ahrefs will rarely let you down, but many of the lesser-known suites and tools are also awesome.

There are some nice innovations here, like KWFinder’s feature that lets you compare organic keywords to PPC keywords to research whether or not it’s worth bidding.

It’s hard to go wrong as the standards are just so high, and even free SEO tools provide powerful starting points and can get your first pieces of content off the ground with minimal to no cost.

The post 17 Best Keyword Research Tools (Free & Paid) appeared first on SEO Administrator.

Here’s an analogy:

GoogleBot is strolling through a library, it encounters a new book it’s never seen before.

This book permits GoogleBot to see it – to look through it – GoogleBot then crawls the book, analyzing and following its content.

Once it’s analyzed the book, it can go about storing its information in the library database.

Without crawling, the book is never found, without indexing, it can never be searched in the database.

The Google Search Index is over 100,000,000 gigabytes in size – that’s a lot of information to index, and it does raise the question of how Google goes about crawling and indexing an ever-increasing resource.

It’s worth bearing in mind that only a small percentage of the internet is actually indexed, so SEOs can’t take indexation for granted.

Luckily, most of the unindexed or ‘deep web’ is paywall or password protected content, dead or dormant content, spam and other information that doesn’t need to be in the index.

How Do I Index My Site?

It goes without saying that to get anything indexed on Google, you have to create a functional website with pages and content.

Building a site with some content is a good idea before you take steps to go ‘live’ and get it indexed. There’s no rule of thumb here really, but you’ll want your site fit for users, error-free and functional on both mobile and desktop.

In fact, Google uses mobile-first indexing now anyway, meaning they crawl and index the mobile version of a site. That means your site needs to be absolutely cut-out for mobile UX.

Mobile-First Indexing

If you’re reading up about indexing then it’s possible that you’ve not built your site yet, or are still in the process of building it.

You might be using a site builder, like Wix, or might be developing your site yourself using WordPress and a theme from ThemeForest or similar.

Most modern site builders and themes are fully responsive, meaning they’re mobile-optimized and will seamlessly respond to mobile and tablet browsing.

But, mobile optimization is still something to watch out for when adding plugins and other content and elements – bear in mind that Google will index and rank your site based on the mobile version – this is very important!

Indexing Your Site

Once you’ve developed your site, you’ll need to take a few extra steps to prepare it for indexing and ranking.

The first step is to set yourself up with Google Search Console. This is crawling and indexing HQ – a one-stop-shop for basic SEO that is totally free.

Search Console will also let you access Google Analytics to measure and analyze site traffic.

There really is no trade-off or substitution for using Google Search Console – it’s pretty much essential and is thankfully very easy to set up and use.

Once you set up Search Console with your Google Account, you’ll have to add your site as a ‘Property’.

Not quite done yet – you need a sitemap.

Sitemaps

Sitemaps translate very literally – they’re maps of your site and its pages.

Once your site is hosted, Google will eventually crawl it and it should eventually index it, even without using Search Console. But, this really isn’t the best route to go down, and you’re best off uploading a sitemap to Search Console – indexing is much faster that way.

You can create sitemaps using Yoast or use an XML sitemap creator like XML Sitemaps. Once you create a sitemap file, you’ll need to add it to your site using Yoast or equivalent for site builders.

Then, head into Google Search Console > Sitemaps (under Index) > Add a Sitemap.

Once you’re done, Google will crawl your site and you’ll be able to analyze its status in Search Console – your first step to getting your site properly indexed in the SERPs!

After your sitemap has been crawled, you’ll be able to check the indexation status of your site’s URLs – be patient as it can take a few days.

Inspect URLs

Your base site and its main pages should be indexed shortly after loading up a sitemap.

When you add new content, your sitemap will be updated – GoogleBot will look there and find your new content and pages.

But, you can also use the Inspect URL field at the top. Paste URLs for new content into this field and you can inspect them – to find that they’re not indexed – and then hit Request Indexing.

This could save about a week or so versus simply waiting for Google to index your new page at its own accord.

Google has a guide on how to use the URL Inspection Tool here.

Does Google Crawl and Index All Content?

Google has its work cut out – it’s crawling and indexing millions of pages on a near-constant basis.

The short answer is no, Google does not guarantee that your pages will be indexed.

To get pages indexed, they need to provide some value to the user – they need to be worthy of Google’s resources.

The Search Console states:

“Note: Requesting a crawl does not guarantee that inclusion will happen instantly or even at all. Our systems prioritize the fast inclusion of high quality, useful content.”

This is pretty much why SEO revolves around content, and why the internet is indeed a library and not just a big box of random bits and pieces. Google wants to create a content-rich universe, not a stack of chaotic information that evidently provides no value.

There are other factors and variables too, including your site’s crawl budget and you robots.txt file.

But, for new sites with a small amount of content, Search Console + sitemaps should get your site indexed.

Checking Indexation Status

By far the easiest and best way to check and manage indexation status is with the Search Console.

The URL Inspection Tool will tell if a page is indexed. You can also see this in the Sitemap section of the Search Console.

However, there are other tools designed for checking indexation status, and some of them do display other handy information on your pages:

This nifty free tool provides a full SEO report – for free – with actionable tips. It’ll also grade your site with an SEO score out of 100.

An impressive free tool, the WebFX Google Indexing Checker will tell you whether or not your pages are indexed and well as breaking down all the core SEO features of your site alongside a pass or fail grade. Any errors will be explained and tips are given for fixing them.

Features

This extremely simple index checker does what it says on the tin – simply copy URLs into the field and it will check whether or not they’ve been indexed.

Whilst you may not need tools like this when you’re using Search Console, they’re useful for checking the indexing status of pages on the go, e.g. on your phone when you can’t or don’t want to log in to Search Console.

Features

Small SEO’s set of free tools are awesome for SEO beginners and novices. This simple tool allows you to check up to 5 URLs for indexing purposes. It can be re-run again with more URLs.

Features

From another excellent free SEO tool provider, Northcutt. The Northcutt index checker tells you whether a page or domain has been indexed or not – simple. Note that large sites will probably exceed their request limit (but most are fine).

Features

Link Indexing Tools

Not everything on the internet gets indexed. Google will only index pages that it deems relevant, valuable, and trustworthy – pages that deserve a spot in its library.

This can pose issues, and links you want to be indexed in Google may not be indexed in some scenarios.

Link indexing tools are tools that help ‘push’ links to Google and other search engines (but mainly Google), forcing (or trying to force) Google to index them.

They’re relatively grey-hat. The general idea is to ‘push’ links that Google would probably not index naturally.

They can be handy when you publish a guest post and don’t really know when it’ll be indexed but are usually used for indexing spammy or low-quality content that is ignored by Google.

There are some legit uses, say the site you publish a guest post on could be very bloated, or poorly optimized for crawl budget, meaning your content won’t be indexed for weeks, months, or at all!

The guest post itself might be very good, though, but is unfortunately lost in the noise, and Search Console can’t help here as the site isn’t your property.

A link indexer can help you in this scenario, though.

Other Legit Uses for Link Indexing Tools Include:

• Getting social profiles indexed – these might contain your link. Some social profiles on lesser-known sites have a larger area for writing bios – this provides you with an opportunity to link your site for some link juice. But you’ll need Google to index that page to pick up that link. Link indexers can help you do that.
• This also might apply to other social site content, for example, some people have used Omega Indexer to get their YouTube playlists indexed. This could yield traffic from people clicking onto your site from that playlist. Another example is Pinterest boards – Google simply might never index them and you can’t find out why. Another legit use for link indexing tools.
• If you create some good content for another site, they publish it, but you find it’s unindexed after some time, you can use a link indexer to push it to Google. Of course, there will likely be legitimate reasons why Google didn’t index naturally and spammy or manual actioned sites could be one reason. But, it’s possible that the post doesn’t get indexed due to crawl budget or other technical SEO issues.
• If you make changes to social profiles (e.g. site address, email, phone numbers, etc), then you can use link indexers to push these to Google. You don’t have control over these sites so Search Console would be of no use.
• If a site goes down and has a lot of links attached to it, link indexers are a quick way to get those links re-indexed. Basically, they just alert Google that this renewed site is back on the internet.

Tier 2 and Tier 3 Backlinks

he kind of links you push to a link indexer are known as tier 2 or tier 3 backlinks.

Tier 2 backlinks are links from comments, social profiles, posts, and other content where your link is placed in relatively thin content.

Tier 1, on the other hand, are your high-grade genuine content-based links that are designed for organic ranking as well.

Tier 3 links are basically spam. They can be posted on any page (that allows link following) with low trust and authority.

Obviously, these links are often not indexed at all, thus providing no link juice.

That’s why you use a link index tool to ‘push’ these links to Google – a common grey-hat or black-hat link-building method.

To do this, these sites likely have some sort of network where your link is placed temporarily until indexed, or somehow hit Google’s server with that link until it indexes it. They might use other ways to guide Google to your link also, like providing it with some (likely fake) social signals.

The following tools should be used with caution:

Omega Indexer is an indexing tool that works for the above use cases.

You create a ‘campaign’ and add your links then request Omega ‘drips feeds’ them to Google. This helps links stay under the Google Penguin radar – which downplays or refuses to index low-quality links.

Omega Indexer works over around 5 days. It’s not 100% successful but testimonials reveal a good rate of success with hard-to-index links.

Omega Indexer works via credits. 1 Credit = 1 Link = $0.02.

Features

Very similar to Omega Indexer, Elite Link Indexer is another paid tool for pushing hard-to-index tier 2 and tier 3 links to Google.

The claim indexation rates of around 64% (high for a service like this). They claim to use white-hat methods to get links noticed by Google.

Elite Link Indexer costs $10 a month for 5,000 links, enough for most users.

Features

Summary

Google and other search engines are choosey with their index, which is fair enough.

Without screening content and its links, Google would be rammed full of spam content.

For sure, the best method of link building is via the creation of quality content and content marketing. Use Search Console and the URL Inspection Tool to request indexing of genuine content you publish on your site (though it will be indexed naturally after some time – usually just days for small sites).

If you’re looking for methods of pushing other links to Google using grey-hat link-building tools, then there are options. These tools do have legitimate uses too, though, so shouldn’t always be frowned upon.

Proceed with caution and good luck!

The post 7 Google Indexation Tools appeared first on SEO Administrator.

Bots typically include web crawlers, such as GoogleBot, which will look at the robots.txt file on your site to learn what it should and shouldn’t crawl on your site.

Robots.txt files are not just for GoogleBot, but will be ‘read’ by many other web crawlers from other search engines and services, as well as web scrapers that are hunting your site for information.

It’s worth bearing in mind that the instructions contained in the robots.txt aren’t mandated – they’re not legally binding (though violating them could still result in breaking the law or violating copyright, etc).

Robots.txt files just pleasantly nudge bots in the right direction and kindly ask them to not crawl certain parts of your site.

Robots.txt files are created by default in most WordPress sites and other sites that use Wix or other site builders, and will automatically be set to allow web crawlers to crawl everything on your site.

With many site builders or platforms like Shopify, you can’t edit the robots.txt file (but can use other techniques to control the crawling of your site.

When To Adjust the Robots.txt File

If you’re an SEO beginner, novice, or an owner/designer of a personal or individual small site or blog (that has already been published), you likely won’t need to adjust the robots.txt file for some time, but checking it is still good practice.

One potential use some advocate for the robots.txt file is preventing your site from being crawled whilst it’s still under construction.

This is ineffective if links point to the page(s), and Google recommends you use a noindex meta tag or password protection instead if you want to prevent a page from being indexed during construction/migration.

You can also use a plugin that covers your site with an ‘under construction’ notice.

There are many other legit reasons to check and edit the robots.txt, though:

Crawl Budget and Robots.txt

There are legitimate motives for editing the robots.txt file and these can confer SEO benefits.

Most of these benefits are linked to a site’s crawl budget. Crawl budget refers to the resources Google, Bing, Yahoo, and other search engines allocate to crawling and indexing your site.

For example, whilst you might assume Google has an infinite army of bots ready to crawl each and every intricate detail of the internet at will, 24/7 around the clock, this isn’t strictly true.

Even Google has finite resources (for now!), and they allocate these resources to sites based on their size/reputation/authority and other factors (many of which are not totally known or understood).

SEOs analyze crawl budgets by checking out their site’s log files, which provide evidence of how many pages crawlers are crawling on their site.

In short, editing the robots.txt directly influences how crawlers interact with your site, which directly influences crawl budget.

Case Example: Robots.txt and an eCommerce Store

Before moving onto some robots.txt tips and tools, let’s run through a simple case study of when you should consider editing your robots.txt file.

eCommerce stores typically use dynamic pages and contain a lot of duplicate content.

Dynamic pages and duplicate content are typically changed via user interaction, like when a user filters product categories or customizes a product. When a user filters a group of products, a page with duplicate content is produced – there’s nothing useful there for anyone apart from the user.

You can disallow bots from crawling duplicate content and filter content, or any other content that is practically useless for SEO and might occupy your crawl budget.

Content to potentially disallow from crawlers includes:

Whilst you might feel your site is way too small to be affected by crawl budget issues – and you’d probably be right – being aware of how you can edit the robots.txt file for the future is still very important.

If your site becomes bloated with content that is useless to Google then retrospectively fixing that is more exhaustive than preparing your robots.txt file early on in your web design and SEO journey!

Additionally, you may have pages with sensitive information, copyright material, and other files you don’t want to be crawled. Editing the robots.txt file can help keep these files off search engines.

Mostly, though, editing the robots.txt file is useful for managing crawl budget.

The Main Five Main Robots.txt Directives

There are five main robots.txt directives.

When you go to edit or generate your robots.txt you’ll see/use some of the following:

When you edit your robots.txt, you’ll be providing it with directives on URLs and directories.

The robots.txt file will sit in the root of your site, e.g. for site www.example.com, the robots.txt file lives at www.example.com/robots.txt.

A simple example of a robots.txt directive would be:

User-agent: Googlebot

Disallow: /thank-you/

This tells Googlebot to disallow crawling at: www.example.com/thank-you/

Note, whilst some tools ask you to input ‘directories’ and others ‘URLs’, you can input either.

Directories will be useful if you want to tell bots to ignore specific folders or files within your site’s root, e.g. /wp-content/plugins/ (which contains likely useless data on WordPress plugins).

URLs are best for singling out specific pages that you don’t want bots to crawl (e.g. thank-you [for ordering, buying, your interest, etc] pages as above).

Advanced Robots.txt Controls/Directives

There are some advanced robots.txt directives that go beyond allow/disallowing.

One key example is the wildcard, used to bulk-block files.

So, Disallow: /copyright-material/*.jpg would block all .jpg images located in the copyright-material directory.

Where To Put the Robots.txt File?

The robots.txt is placed at the root of your domain.

So, if your site is example.com, the robots.txt would be placed at: http://www.example.com/robots.txt.

You can locate this using the cPanel file manager (common for WordPress websites).

Once you open the file (literally a text file), you can write your new directives straight into it and save it.

There are different instructions for site builders like Wix, but Shopify and Squarespace don’t allow you to edit the file – though there are other options for hiding your site or its pages and preventing them from being indexed.  

Some SEO suites and plugins such as Better Robots.txt and AIO SEO semi-automate the process of creating robots.txt files for your site.

Robots.txt Generator Tools

Small SEO Tools Robots.txt Generator

Small SEO tools provide a fantastic array of free SEO tools, and the robots.txt generator does what it says on the tin (and is obviously 100% free).

This is a simple robots.txt generator with a selection of 15 bots and crawlers that can be set to be allowed or refused from your site. You can point it to your site’s sitemap.

You can also set the default for all bots and choose crawl delay settings which will roll out to bots that take notice of this (i.e. not GoogleBot).

You’ll then be able to add restricted directories and/or URLs using the form at the bottom. The /cgi-bin is pre-filled (a commonly blocked directory that does not need to be crawled), but you can edit that field if you want to.

Features

• Allow/refuse 15 common bots/crawlers
• Add sitemap
• Block directories/URLs
• Crawl-delay

Ryte Robots.txt Generator

An intuitive robots.txt generator with step-by-step instructions. The Ryte Robots.txt Generator is an excellent little tool for quickly creating robots.txt files with a selection of 11 bots. It’s a bit strange to not see other search engines like Baidu or Yandex in the bot selection, but you can add these user agents yourself.

To block crawling of URLs or directories, simply input them into the fields. You can also allow or disallow all bots from crawling your site.

Features

• Good interface
• Add sitemap
• Block directories/URLs

SEOBook Robots.txt File Generator

Another simple robots.txt generator with allow/disallow all and per-bot control of basic robots.txt directives. There are 9 bots here and it’s straightforward to add URLs or directories to block.

A super-simple tool that lets you copy and paste your new robots.txt straight into the old one.

Features

• Simple interface
• Add sitemap
• Blocks URLs/directories

SEOptimer Free Robots.txt Generator

With a selection of 15 bots, this is one of the more complete robots.txt generators available. It also has crawl delay settings. You can allow/refuse certain bots or set all bots to allow/disallow by default.

The fields allow you to add the URLs or directories you don’t want to be crawled.

Features

• Allow/refuse 15 common bots/crawlers
• Add sitemap
• Block directories/URLs
• Crawl-delay

Internet Marketing Ninjas Robots.txt Generator Tool

With a comprehensive selection of 22 user agents, this is a quality tool for allowing/refusing crawling from many major bots. You can add URLs/directories to allow/disallow per bot.

A very simple, easy-to-use robots.txt generator tool with plenty of bots. Unfortunately, there are no crawl delay settings.

Features

• Allow/refuse 22 common bots/crawlers
• Add sitemap
• Block directories/URLs

LinkGraph Robots.txt Generator

With over 40 bots built-in to the tool, this is a comprehensive robots.txt generator. However, it only lets you add 5 URLs/directories for disallowing, which is a minor letdown.

Still, though, with crawl delay and plenty of user agents in the list, this is yet another solid free robots.txt generator.

Features

• Allow/refuse 40+ common bots/crawlers
• Add sitemap
• Block directories/URLs
• Crawl-delay

Summary

Editing the robots.txt file is actually pretty straightforward!

Even as an SEO novice, it’s very handy to know what can and can’t be done with the robots.txt file.

As Google recommends, using the robots.txt file for completely hiding a site and its pages from the SERPs is ineffective compared to using a noindex meta tag.

The primary reason you’ll want to edit the robots.txt file is to prevent the crawling of non-publicly accessible site data that potentially eat into your crawl budget, or to control the activity of individual user agents that crawl your site.

The post Robots.txt Generators appeared first on SEO Administrator.