Although the Backlink Analytics tool is one of the oldest features of SEMrush, however, we have to admit that it may have been the weaker link of our SEO toolkit. We knew we had to step up our game, so about a year and a half ago, we set on changing the status quo.
SEMrush, while being a well-rounded toolkit for digital marketers, has always had a soft spot for SEO. Helping people drive organic traffic from search engines to their content has been one of our most important goals since its inception.
This goal led us to become a world-renowned SEO Suite, allowing us to win multiple industry awards over the years.
SEO is tricky, as it involves complex and intertwined moving parts. To get the top rankings, you have to nail down every single step on-site and off-site. Throughout this entire process, we strive to provide our users with the best solutions.
To stay praiseworthy, we are continuously working on improving our toolkit. Today, we are proud to share with you our latest breakthrough:
A Completely Renovated Backlink Database
We needed a major improvement in the quality of our backlink data. There was no workaround, but a complete overhaul of our data-gathering process. To focus on our end goal, we put on hold the development of all other backlink features and made a huge list of things that would improve our backlink data delivery to clients.
The path was clear, and all we had to do is work on crossing off the items on our list.
Working Out the Overhaul
We won’t bore you with the technical details of our backlink database’s overhaul, but here is a quick rundown of what was done:
Crawler. After carefully examining the drawbacks and boundaries of the existing architecture, we decided to rewrite our crawler from scratch. And so we did, we have designed an entirely new approach to our data gathering.
Crawling queue. The first tests of our new crawler revealed that its request queue was not properly handling the amount of data it was now collecting. We tried solving this by simply increasing hardware capabilities, but it was not good enough, so we developed a more efficient crawling queue.
Seeding. To provide our crawler with a quality initial seeding, we queued up all the URLs from Google’s Top 100 for 450 million keywords from our Organic Research tool; this ensured that our database was relevant from the ground up.
Storage. Increased data collection obviously demands more storage space—we had to quadruple our server size.
Measuring Our Growth
To find out exactly where we are as a backlink provider, we decided to measure ourselves against the best: Majestic, Ahrefs, and Moz.
We will explain the methodology in a second. First, let’s assess our development progress during the past six months. Looking at the relationship between the four top SEO tools, you can see that we have made a giant step forward.
The ratio of SEMrush backlink database to the competitors
How We Conducted the Comparison
It was not easy to figure out a methodology that would be both clear-cut and fair.
You can always find the domains that will show your backlink tool in a good light, that is why we decided to use a random set of 100 domains (out of 100,000) for each month to show us how the contestants performed during the past six months.
We were looking at the number of referring domains and the total number of backlinks each contestant had for the 100 domains.
Next, for each domain in the test sample, we compared the ratio of SEMrush results to the results of our competitors. So, if the ratio is less than 1 — the SEMrush database has less information for the test domain. A ratio greater than 1 shows by how many times the SEMrush result exceeded.
To get the final score, we calculated the median of all results.
We have made a huge leap forward with our backlink database, and it feels great to look at the numbers and pat ourselves on the back, but, obviously, it is not just about the numbers.
The quantity of data does not necessarily convert in quality, and we are making a great effort to ensure that our database stays fresh and useful.
Now that we have a new data gathering process, we will build upon it designing new features and capabilities that will make our tools even stronger. Stay tuned for more exciting news!
Go ahead and try our Backlink Analytics for yourself; we welcome your feedback!