zh
English Español Deutsch Français Italiano Português (Brasil) Русский 中文 日本語
Submit post
Go to Blog

The New SEMrush Backlink Database: Bigger, Better, Faster

72
Wow-Score
The Wow-Score shows how engaging a blog post is. It is calculated based on the correlation between users’ active reading time, their scrolling speed and the article’s length.

The New SEMrush Backlink Database: Bigger, Better, Faster

This post is in English
Elena Terenteva
This post is in English
The New SEMrush Backlink Database: Bigger, Better, Faster

Although the Backlink Analytics tool is one of the oldest features of SEMrush, however, we have to admit that it may have been the weaker link of our SEO toolkit. We knew we had to step up our game, so about a year and a half ago, we set on changing the status quo.

SEMrush, while being a well-rounded toolkit for digital marketers, has always had a soft spot for SEO. Helping people drive organic traffic from search engines to their content has been one of our most important goals since its inception.

This goal led us to become a world-renowned SEO Suite, allowing us to win multiple industry awards over the years.

The New SEMrush Backlink Database: Bigger, Better, Faster. Image 0

SEO is tricky, as it involves complex and intertwined moving parts. To get the top rankings, you have to nail down every single step on-site and off-site. Throughout this entire process, we strive to provide our users with the best solutions.  

To stay praiseworthy, we are continuously working on improving our toolkit. Today, we are proud to share with you our latest breakthrough:

A Completely Renovated Backlink Database

The SEMrush Backlink Database

We needed a major improvement in the quality of our backlink data. There was no workaround, but a complete overhaul of our data-gathering process. To focus on our end goal, we put on hold the development of all other backlink features and made a huge list of things that would improve our backlink data delivery to clients.

The path was clear, and all we had to do is work on crossing off the items on our list.

Working Out the Overhaul

We won’t bore you with the technical details of our backlink database’s overhaul, but here is a quick rundown of what was done:

Crawler. After carefully examining the drawbacks and boundaries of the existing architecture, we decided to rewrite our crawler from scratch. And so we did, we have designed an entirely new approach to our data gathering.

Crawling queue. The first tests of our new crawler revealed that its request queue was not properly handling the amount of data it was now collecting. We tried solving this by simply increasing hardware capabilities, but it was not good enough, so we developed a more efficient crawling queue.  

Seeding. To provide our crawler with a quality initial seeding, we queued up all the URLs from Google’s Top 100 for 450 million keywords from our Organic Research tool; this ensured that our database was relevant from the ground up.  

Storage. Increased data collection obviously demands more storage space—we had to quadruple our server size.

Measuring Our Growth

To find out exactly where we are as a backlink provider, we decided to measure ourselves against the best: Majestic, Ahrefs, and Moz.

We will explain the methodology in a second. First, let’s assess our development progress during the past six months. Looking at the relationship between the four top SEO tools, you can see that we have made a giant step forward.

The ratio of SEMrush backlink database to the competitors

The ratio of SEMrush backlink database to the competitors The ratio of SEMrush backlink database to the competitors

How We Conducted the Comparison

It was not easy to figure out a methodology that would be both clear-cut and fair.

You can always find the domains that will show your backlink tool in a good light, that is why we decided to use a random set of 100 domains (out of 100,000) for each month to show us how the contestants performed during the past six months.  

We were looking at the number of referring domains and the total number of backlinks each contestant had for the 100 domains.

Next, for each domain in the test sample, we compared the ratio of SEMrush results to the results of our competitors. So, if the ratio is less than 1 — the SEMrush database has less information for the test domain. A ratio greater than 1 shows by how many times the SEMrush result exceeded.

To get the final score, we calculated the median of all results.

The ratio of SEMrush backlink database to the Competitors

Conclusion

We have made a huge leap forward with our backlink database, and it feels great to look at the numbers and pat ourselves on the back, but, obviously, it is not just about the numbers.

More on this 9 Ways to Make the Best of SEO Link Building with SEMrush Post Maria Raybould

The quantity of data does not necessarily convert in quality, and we are making a great effort to ensure that our database stays fresh and useful.

Now that we have a new data gathering process, we will build upon it designing new features and capabilities that will make our tools even stronger. Stay tuned for more exciting news!

Go ahead and try our Backlink Analytics for yourself; we welcome your feedback!

Try our backlink database

For your own domain

Please specify a valid domain, e.g., www.example.com

Elena Terenteva
SEMrush

SEMrush employee.

Elena Terenteva, Product Marketing Manager at SEMrush. Elena has eight years public relations and journalism experience, working as a broadcasting journalist, PR/Content manager for IT and finance companies.
Bookworm, poker player, good swimmer.
Share this post
or

Comments

2000
Justin Williams
Enthusiast

Occasionally takes part in conversations.

Looking good there. Great perspective. This is well-done article.
Helper

An experienced member who is always happy to help.

Great to hear! I've been recommending your backlink tool for a while now, so I guess I'm going to have to update that.
Enthusiast

Occasionally takes part in conversations.

This is a great news! Thank you :-)
Paul Lovell
Pro

Asks great questions and provides brilliant answers.

This is great, Even more data to dig into Thanks so much for the continued growth
Newcomer

Either just recently joined or is too shy to say something.

having a bunch of worthless content sites linking to yours does not help SEO, search engines are smart enough now to see what a real site is, a real business at a real location that has real content. so being yourself and not a trickster is the way to get page one. thousands of spam sites linking back to yours, those days are over. so how do you account for that?????
Elena Terenteva
SEMrush

SEMrush employee.

RobotKiller-BotnetAssasin
Hi! You are right, trustworthy links are much better for your website. For discovering toxic and weak links you can use SEMrush Backlink audit tool, which will help you to find potentially dangerous links and send them to the Google Disavow tool right away.
Peter Gerardi
Enthusiast

Occasionally takes part in conversations.

Unfortunatly after being a loyal customer for over 7 years, SEMrush has decided they are so big now that they no longer honor grandfathered in loyal customers who were there from the beginning....they now have a shady Sales department. I was tricked by their sales guy....total scam now. Soooo disappointed. SEMrush used to be amazing to work with, now they are just all about money and greed. What a shame. I remember back in the day....I know his last name was Black....I would deal with him and he was so good to me and my agency....so sad what has become of things....the sales guy was so condescending to me....unreal...be VERY careful with their sales department....I really got screwed over.
Elena Terenteva
SEMrush

SEMrush employee.

Peter Gerardi
Hi Peter! Thanks for highlighting the problem. We are taking this kind of cases very serious because despite our growth we still cherish the same good old principals and our customers' satisfaction is the number one priority. We are investigating this case and if you don't mind our customer success specialists are going to reach you. You can also contact me personally.
Peter Gerardi
Enthusiast

Occasionally takes part in conversations.

Peter Gerardi
I would like to follow up on this comment. SEMrush reached out to me today and we cleared up the issue I was dealing with. I am relieved because we have had an over 7 year relationship and I just couldn't believe what I was experiencing was what they intended. Miscommunication seems to have caused the situation . I respect that SEMrush quickly reached out and made things right!
Elena Terenteva
SEMrush

SEMrush employee.

Peter Gerardi
Appreciate your comment, Peter! Glad we could overcome this miscommunication.
Expert

Provides valuable insights and adds depth to the conversation.

Great to see Semrush continuing to develop and evolve the tools available.
Elena Terenteva
SEMrush

SEMrush employee.

White Rabbit
Thank you! Yes, growing and developing constantly :)
Newcomer

Either just recently joined or is too shy to say something.

I've tested several domains and Moz shows more referring domains, is this tool complete?
Elena Terenteva
SEMrush

SEMrush employee.

Bernard Stark
Hey Bernard, yes, the tool is complete, but data collection is an ongoing process, and we continue to improve our Backlink Analytics tool and collect the data. About the ref.domains and the Moz, different tools can show different results and it all depends on the specific data set of each tool for each domain.
And if you look at our graphs you can notice that soon we are going to catch up Moz.
By the way could you please send me your domains to check them?
Andre Schaefer
Newcomer

Either just recently joined or is too shy to say something.

Glad to see that Semrush updated their database and can now provide more data.

I have to admit that i regularly use other tools to do my competitor researches. But I've been a huge fan of the traffic growth graph as it helps clients to visualize their visitor growth. Might give your tool another shot soon.

Andre
Brooks Manley
Enthusiast

Occasionally takes part in conversations.

Wow I'm stoked. Thanks for staying on top of things. I was literally just beginning to look at adding Ahrefs or Majestic to our toolset.
Elena Terenteva
SEMrush

SEMrush employee.

Brooks Manley
Hi Brooks! Glad you are with us :) What do you think we should also add or improve?
Brooks Manley
Enthusiast

Occasionally takes part in conversations.

Elena Terenteva
I think the tool is great – we just work with pretty small businesses – so having data on each and every backlink makes a big difference and is important to us. So just having a lot more links in the database is huge for us.
Elena Terenteva
SEMrush

SEMrush employee.

Brooks Manley
We appreciate it! And we will continue to do our best. Thank you!
Enthusiast

Occasionally takes part in conversations.

Positive steps, look forward to seeing the results of the update.
Elena Terenteva
SEMrush

SEMrush employee.

Michael James Field
Huge positive steps! Share your feedback with us!
Enthusiast

Occasionally takes part in conversations.

Great content to share with colleagues. We can't afford to buy premium packages of your but still, we love your blog's content. Hope will get the same content in future from you gyuys!
Thanks
TryDigital Team
Elena Terenteva
SEMrush

SEMrush employee.

TryDigital
Oh, thank you so much!
Enthusiast

Occasionally takes part in conversations.

There is no other tool which can give that detailed analysis of the site and manage it automatically in clicks.
About 74% of business in the industry already adapted SEMrush and are relying on it for their growth.
Totally deserving!!
Elena Terenteva
SEMrush

SEMrush employee.

Princy Tripathi
Appreciate your comment, Princy!
Enthusiast

Occasionally takes part in conversations.

Very nice job. Thank you.
Elena Terenteva
SEMrush

SEMrush employee.

Chat
Thank you, Chat!

Send feedback

Your feedback must contain at least 3 words (10 characters).

We will only use this email to respond to you on your feedback. Privacy Policy

Thank you for your feedback!