Indexing your website and pages on Google so they’re locatable in the SERPs relies on two processes:
Here’s an analogy:
GoogleBot is strolling through a library, it encounters a new book it’s never seen before.
This book permits GoogleBot to see it – to look through it – GoogleBot then crawls the book, analyzing and following its content.
Once it’s analyzed the book, it can go about storing its information in the library database.
Without crawling, the book is never found, without indexing, it can never be searched in the database.
The Google Search Index is over 100,000,000 gigabytes in size – that’s a lot of information to index, and it does raise the question of how Google goes about crawling and indexing an ever-increasing resource.
It’s worth bearing in mind that only a small percentage of the internet is actually indexed, so SEOs can’t take indexation for granted.
Luckily, most of the unindexed or ‘deep web’ is paywall or password protected content, dead or dormant content, spam and other information that doesn’t need to be in the index.
How Do I Index My Site?
It goes without saying that to get anything indexed on Google, you have to create a functional website with pages and content.
Building a site with some content is a good idea before you take steps to go ‘live’ and get it indexed. There’s no rule of thumb here really, but you’ll want your site fit for users, error-free and functional on both mobile and desktop.
In fact, Google uses mobile-first indexing now anyway, meaning they crawl and index the mobile version of a site. That means your site needs to be absolutely cut-out for mobile UX.
If you’re reading up about indexing then it’s possible that you’ve not built your site yet, or are still in the process of building it.
You might be using a site builder, like Wix, or might be developing your site yourself using WordPress and a theme from ThemeForest or similar.
Most modern site builders and themes are fully responsive, meaning they’re mobile-optimized and will seamlessly respond to mobile and tablet browsing.
But, mobile optimization is still something to watch out for when adding plugins and other content and elements – bear in mind that Google will index and rank your site based on the mobile version – this is very important!
Indexing Your Site
Google Search Console
Once you’ve developed your site, you’ll need to take a few extra steps to prepare it for indexing and ranking.
The first step is to set yourself up with Google Search Console. This is crawling and indexing HQ – a one-stop-shop for basic SEO that is totally free.
Search Console will also let you access Google Analytics to measure and analyze site traffic.
There really is no trade-off or substitution for using Google Search Console – it’s pretty much essential and is thankfully very easy to set up and use.
Once you set up Search Console with your Google Account, you’ll have to add your site as a ‘Property’.
- Install Yoast SEO on your WordPress site (for Wix and other site builders you’ll use SEO Tools)
- Open Search Console
- On the Search menu, find ‘Add a property’
- Copy the HTML Tag (there are other methods, but this is the easiest)
- Head into Yoast (or SEO Tools or equivalent for your site builder)
- Copy the tag into the field (Webmaster tools on Yoast)
- This should verify your site – it can now be indexed
Not quite done yet – you need a sitemap.
Sitemaps translate very literally – they’re maps of your site and its pages.
Once your site is hosted, Google will eventually crawl it and it should eventually index it, even without using Search Console. But, this really isn’t the best route to go down, and you’re best off uploading a sitemap to Search Console – indexing is much faster that way.
You can create sitemaps using Yoast or use an XML sitemap creator like XML Sitemaps. Once you create a sitemap file, you’ll need to add it to your site using Yoast or equivalent for site builders.
Then, head into Google Search Console > Sitemaps (under Index) > Add a Sitemap.
Once you’re done, Google will crawl your site and you’ll be able to analyze its status in Search Console – your first step to getting your site properly indexed in the SERPs!
After your sitemap has been crawled, you’ll be able to check the indexation status of your site’s URLs – be patient as it can take a few days.
Your base site and its main pages should be indexed shortly after loading up a sitemap.
When you add new content, your sitemap will be updated – GoogleBot will look there and find your new content and pages.
But, you can also use the Inspect URL field at the top. Paste URLs for new content into this field and you can inspect them – to find that they’re not indexed – and then hit Request Indexing.
This could save about a week or so versus simply waiting for Google to index your new page at its own accord.
Google has a guide on how to use the URL Inspection Tool here.
Does Google Crawl and Index All Content?
Google has its work cut out – it’s crawling and indexing millions of pages on a near-constant basis.
The short answer is no, Google does not guarantee that your pages will be indexed.
To get pages indexed, they need to provide some value to the user – they need to be worthy of Google’s resources.
“Note: Requesting a crawl does not guarantee that inclusion will happen instantly or even at all. Our systems prioritize the fast inclusion of high quality, useful content.”
This is pretty much why SEO revolves around content, and why the internet is indeed a library and not just a big box of random bits and pieces. Google wants to create a content-rich universe, not a stack of chaotic information that evidently provides no value.
There are other factors and variables too, including your site’s crawl budget and you robots.txt file.
But, for new sites with a small amount of content, Search Console + sitemaps should get your site indexed.
Checking Indexation Status
By far the easiest and best way to check and manage indexation status is with the Search Console.
The URL Inspection Tool will tell if a page is indexed. You can also see this in the Sitemap section of the Search Console.
However, there are other tools designed for checking indexation status, and some of them do display other handy information on your pages:
WebFX Google Indexing Checker
This nifty free tool provides a full SEO report – for free – with actionable tips. It’ll also grade your site with an SEO score out of 100.
An impressive free tool, the WebFX Google Indexing Checker will tell you whether or not your pages are indexed and well as breaking down all the core SEO features of your site alongside a pass or fail grade. Any errors will be explained and tips are given for fixing them.
- Full free SEO audit tool with indexing checker
- Also checks for XML sitemaps and robots.txt files
- Displays free actionable tips
- Can be re-run again for free
Linkody Index Checker
This extremely simple index checker does what it says on the tin – simply copy URLs into the field and it will check whether or not they’ve been indexed.
Whilst you may not need tools like this when you’re using Search Console, they’re useful for checking the indexing status of pages on the go, e.g. on your phone when you can’t or don’t want to log in to Search Console.
- Ultra-simple single or batch URL indexing checker
Small SEO Tools Index Checker
Small SEO’s set of free tools are awesome for SEO beginners and novices. This simple tool allows you to check up to 5 URLs for indexing purposes. It can be re-run again with more URLs.
- Simple lightweight URL indexation checker
Northcutt Index Checker
From another excellent free SEO tool provider, Northcutt. The Northcutt index checker tells you whether a page or domain has been indexed or not – simple. Note that large sites will probably exceed their request limit (but most are fine).
- Simple index checker for single pages or entire domains
Link Indexing Tools
Not everything on the internet gets indexed. Google will only index pages that it deems relevant, valuable, and trustworthy – pages that deserve a spot in its library.
This can pose issues, and links you want to be indexed in Google may not be indexed in some scenarios.
Link indexing tools are tools that help ‘push’ links to Google and other search engines (but mainly Google), forcing (or trying to force) Google to index them.
They’re relatively grey-hat. The general idea is to ‘push’ links that Google would probably not index naturally.
They can be handy when you publish a guest post and don’t really know when it’ll be indexed but are usually used for indexing spammy or low-quality content that is ignored by Google.
There are some legit uses, say the site you publish a guest post on could be very bloated, or poorly optimized for crawl budget, meaning your content won’t be indexed for weeks, months, or at all!
The guest post itself might be very good, though, but is unfortunately lost in the noise, and Search Console can’t help here as the site isn’t your property.
A link indexer can help you in this scenario, though.
Other Legit Uses for Link Indexing Tools Include:
- Getting social profiles indexed – these might contain your link. Some social profiles on lesser-known sites have a larger area for writing bios – this provides you with an opportunity to link your site for some link juice. But you’ll need Google to index that page to pick up that link. Link indexers can help you do that.
- This also might apply to other social site content, for example, some people have used Omega Indexer to get their YouTube playlists indexed. This could yield traffic from people clicking onto your site from that playlist. Another example is Pinterest boards – Google simply might never index them and you can’t find out why. Another legit use for link indexing tools.
- If you create some good content for another site, they publish it, but you find it’s unindexed after some time, you can use a link indexer to push it to Google. Of course, there will likely be legitimate reasons why Google didn’t index naturally and spammy or manual actioned sites could be one reason. But, it’s possible that the post doesn’t get indexed due to crawl budget or other technical SEO issues.
- If you make changes to social profiles (e.g. site address, email, phone numbers, etc), then you can use link indexers to push these to Google. You don’t have control over these sites so Search Console would be of no use.
- If a site goes down and has a lot of links attached to it, link indexers are a quick way to get those links re-indexed. Basically, they just alert Google that this renewed site is back on the internet.
Tier 2 and Tier 3 Backlinks
he kind of links you push to a link indexer are known as tier 2 or tier 3 backlinks.
Tier 2 backlinks are links from comments, social profiles, posts, and other content where your link is placed in relatively thin content.
Tier 1, on the other hand, are your high-grade genuine content-based links that are designed for organic ranking as well.
Tier 3 links are basically spam. They can be posted on any page (that allows link following) with low trust and authority.
Obviously, these links are often not indexed at all, thus providing no link juice.
That’s why you use a link index tool to ‘push’ these links to Google – a common grey-hat or black-hat link-building method.
To do this, these sites likely have some sort of network where your link is placed temporarily until indexed, or somehow hit Google’s server with that link until it indexes it. They might use other ways to guide Google to your link also, like providing it with some (likely fake) social signals.
The following tools should be used with caution:
Omega Indexer is an indexing tool that works for the above use cases.
You create a ‘campaign’ and add your links then request Omega ‘drips feeds’ them to Google. This helps links stay under the Google Penguin radar – which downplays or refuses to index low-quality links.
Omega Indexer works over around 5 days. It’s not 100% successful but testimonials reveal a good rate of success with hard-to-index links.
Omega Indexer works via credits. 1 Credit = 1 Link = $0.02.
- Allegedly uses Google Search Console to ‘drip feed’ Google your links for indexing
- Works in around 5 days
- Cheap to use and experiment with
Elite Link Indexer
Very similar to Omega Indexer, Elite Link Indexer is another paid tool for pushing hard-to-index tier 2 and tier 3 links to Google.
The claim indexation rates of around 64% (high for a service like this). They claim to use white-hat methods to get links noticed by Google.
Elite Link Indexer costs $10 a month for 5,000 links, enough for most users.
- Alleged 64% success rate
- Claims to use white-hat methods
Google and other search engines are choosey with their index, which is fair enough.
Without screening content and its links, Google would be rammed full of spam content.
For sure, the best method of link building is via the creation of quality content and content marketing. Use Search Console and the URL Inspection Tool to request indexing of genuine content you publish on your site (though it will be indexed naturally after some time – usually just days for small sites).
If you’re looking for methods of pushing other links to Google using grey-hat link-building tools, then there are options. These tools do have legitimate uses too, though, so shouldn’t always be frowned upon.
Proceed with caution and good luck!