zh
English Español Deutsch Français Italiano Português (Brasil) Русский 中文 日本語
Submit post
Go to Blog

The New SEMrush Backlink Database: Bigger, Better, Faster

71
Wow-Score
The Wow-Score shows how engaging a blog post is. It is calculated based on the correlation between users’ active reading time, their scrolling speed and the article’s length.

The New SEMrush Backlink Database: Bigger, Better, Faster

This post is in English
Elena Terenteva
This post is in English
The New SEMrush Backlink Database: Bigger, Better, Faster

Although the Backlink Analytics tool is one of the oldest features of SEMrush, however, we have to admit that it may have been the weaker link of our SEO toolkit. We knew we had to step up our game, so about a year and a half ago, we set on changing the status quo.

SEMrush, while being a well-rounded toolkit for digital marketers, has always had a soft spot for SEO. Helping people drive organic traffic from search engines to their content has been one of our most important goals since its inception.

This goal led us to become a world-renowned SEO Suite, allowing us to win multiple industry awards over the years.

The New SEMrush Backlink Database: Bigger, Better, Faster. Image 0

SEO is tricky, as it involves complex and intertwined moving parts. To get the top rankings, you have to nail down every single step on-site and off-site. Throughout this entire process, we strive to provide our users with the best solutions.  

To stay praiseworthy, we are continuously working on improving our toolkit. Today, we are proud to share with you our latest breakthrough:

A Completely Renovated Backlink Database

The SEMrush Backlink Database

We needed a major improvement in the quality of our backlink data. There was no workaround, but a complete overhaul of our data-gathering process. To focus on our end goal, we put on hold the development of all other backlink features and made a huge list of things that would improve our backlink data delivery to clients.

The path was clear, and all we had to do is work on crossing off the items on our list.

Working Out the Overhaul

We won’t bore you with the technical details of our backlink database’s overhaul, but here is a quick rundown of what was done:

Crawler. After carefully examining the drawbacks and boundaries of the existing architecture, we decided to rewrite our crawler from scratch. And so we did, we have designed an entirely new approach to our data gathering.

Crawling queue. The first tests of our new crawler revealed that its request queue was not properly handling the amount of data it was now collecting. We tried solving this by simply increasing hardware capabilities, but it was not good enough, so we developed a more efficient crawling queue.  

Seeding. To provide our crawler with a quality initial seeding, we queued up all the URLs from Google’s Top 100 for 450 million keywords from our Organic Research tool; this ensured that our database was relevant from the ground up.  

Storage. Increased data collection obviously demands more storage space—we had to quadruple our server size.

Measuring Our Growth

To find out exactly where we are as a backlink provider, we decided to measure ourselves against the best: Majestic, Ahrefs, and Moz.

We will explain the methodology in a second. First, let’s assess our development progress during the past six months. Looking at the relationship between the four top SEO tools, you can see that we have made a giant step forward.

The ratio of SEMrush backlink database to the competitors

The ratio of SEMrush backlink database to the competitors The ratio of SEMrush backlink database to the competitors

How We Conducted the Comparison

It was not easy to figure out a methodology that would be both clear-cut and fair.

You can always find the domains that will show your backlink tool in a good light, that is why we decided to use a random set of 100 domains (out of 100,000) for each month to show us how the contestants performed during the past six months.  

We were looking at the number of referring domains and the total number of backlinks each contestant had for the 100 domains.

Next, for each domain in the test sample, we compared the ratio of SEMrush results to the results of our competitors. So, if the ratio is less than 1 — the SEMrush database has less information for the test domain. A ratio greater than 1 shows by how many times the SEMrush result exceeded.

To get the final score, we calculated the median of all results.

The ratio of SEMrush backlink database to the Competitors

[UPD] Community Responses

As expected, we’ve had a lot of feedback on this post, and we would like to thank you all for your responses! 

For the most part, the community was very supportive, and one of the first people to give us kudos was Alyeda Solis.

This was followed by a barrage of positive messages, with Gregg Lee putting the cherry on top—Brian Dean has checked and approved our database growth.

Of course, we’ve also had a good share of criticism. Russ Jones claimed to prove us wrong with his own research.

After a bit of back and forth, he revised his conclusion. But we still have to disagree with it.

To quote Russ: “Comparing link indexes accurately is no easy endeavor.”

That’s completely true. First off, getting a truly random sample of domains is an important and very complex part of a quality backlink index comparison. We are really appreciative of the methodology that Russ presented in his article, it’s quite a helpful piece. Yet we cannot agree with the way he assesses and compares the indexes themselves.

The method he uses only shows the likelihood of one index having more data than the other. It does not reveal how much more data there is, which means the method cannot be used for the real comparison (if your goal is to find out which index has more data).

The following are examples illustrating why. 

Example 1:

Let’s say we have backlink indexes for Contestant 1 (C1) and Contestant 2 (C2). 

The comparison for a sample of 12 domains shows that C2 wins every time:

example1

According to Russ’s approach, C2 is the absolute champion. But in reality, the difference between indexes is 0.1%, which means that the C2 and C1 indexes are basically equal. 

Example 2:

This time, let’s say, the comparison shows that out of a sample of 12 domains, C2 has 9 wins, and C1 has 3 wins:

example2

Once again, according to Russ’s approach, C2 here is 3 times bigger than C1. But by looking at the actual index sizes, you can see that for 75% of sample domains the indexes are almost equal (0,1% difference), and for 25%, C1 has a complete victory (C2 has no data). Overall, C1 in this example has a better backlink index.

These examples are extreme, but they do illustrate the flaws of the approach. Without knowing how much data there actually is, you can not claim that one backlink index is more useful for SEO than another.

Our comparison method acknowledges this, as we were calculating the median using the actual numbers of referring domains and backlinks.

Russ kindly shared the sample used in his research so that we could verify it ourselves. The results turned out to be very similar to those presented in our research.

SEMrush / Moz comparison

In terms of number of backlinks this graph shows a drastically different picture to what Russ has presented in his research.

We also took the first domain (amotherthing.com) from Russ’s sample and ran it through the interfaces of both SEMrush and Moz. 

The numbers proved to be different from those presented by Russ.

Russ’s research:  
SEMrush: 28469 backlinks
Moz: 404078 backlinks

Tool interfaces:
SEMrush: 37.7k backlinks
Moz: 26.8k backlinks

SEMrus/Moz interfaces

Anyway, we wanted to thank Russ for his time and ideas, as we believe that healthy competition is a good incentive for us and the industry as a whole.

Conclusion

We have made a huge leap forward with our backlink database, and it feels great to look at the numbers and pat ourselves on the back, but, obviously, it is not just about the numbers.

More on this 9 Ways to Make the Best of SEO Link Building with SEMrush Post Maria Raybould

The quantity of data does not necessarily convert in quality, and we are making a great effort to ensure that our database stays fresh and useful.

Now that we have a new data gathering process, we will build upon it designing new features and capabilities that will make our tools even stronger. Stay tuned for more exciting news!

Go ahead and try our Backlink Analytics for yourself and as usual we welcome your feedback!

Try our backlink database

For your own domain

Please specify a valid domain, e.g., www.example.com

Elena Terenteva
SEMrush

SEMrush employee.

Elena Terenteva, Product Marketing Manager at SEMrush. Elena has eight years public relations and journalism experience, working as a broadcasting journalist, PR/Content manager for IT and finance companies.
Bookworm, poker player, good swimmer.
Share this post
or

Comments

2000
Newcomer

Either just recently joined or is too shy to say something.

Great SEO All in One tool.
Newcomer

Either just recently joined or is too shy to say something.

great site
Helper

An experienced member who is always happy to help.

Thanks for such a great post. I am using premium version of SEMRush. I added 3 projects and when I analyzed Backlinks of one of my project, I found many links are toxic links and lost. My question is how I can upload file for disavow links in bulk. Currently I have to check all links manually and then press + and then add those links, which is time taking. There are more than 2500 backlinks.
Newcomer

Either just recently joined or is too shy to say something.

So recently, my backlinks jumped from 7.5k to 10.1k (30 days) according to SEMRUSH should I worry? should I do a disavow? My Google console hasn't mentioned anything about shady business. The new links are spammy with odd Pornistic name but are not a porn site. Either it's a coupon site or just a blank site. What do you guys recommend? A lot of them repeat itself...you think google bot will over look it? some of the spammers have a "no follow" link.
Newcomer

Either just recently joined or is too shy to say something.

Thanks for this work!

I agree that we have a different approach. I defend why I think adversarial metrics like mine are more useful in my seminal work on link index measuring here: https://moz.com/blog/big-fast-strong-backlink-index-comparisons

Long story short, in the example you have of 999 vs 1000, let's say that is the case. Let's say it turns out Moz only had 1 more RLD for every domain in our index. What does that mean? Well, you would be right that it means that Moz's index is only marginally larger than SEMRush. But it would also mean that 100% of the customers who signed up for our link index would have a better experience than those who chose SEMRush for links. Sure, it is only one link per site, but objectively speaking every SEMRush customer would have a worst experience given hypothetical presented.

The goal of my metrics is to show the likelihood a customer will get more data using one index or another. Your goal is to show the descriptive statistics of the index. One measures what real users would experience.
Eugene
SEMrush

SEMrush employee.

Russ Jones
Hi Russ, thanks for joining the conversation. First of all since you work for MOZ and I work for SEMrush obviously we are both biased. So whatever our arguments are, eventually we will most likely agree to disagree. That said I don't want to change your mind. I just feel your methodology actually doesn't help to optimize for user experience as you claim. For small domains 10 URLs might be meaningful, for bigger not so much. And 1 URL is not a real difference at all. User experience is not binary. User can see huge or small difference and can say that results are "similar" when they are not exactly the same but close. So conclusion depends on the size of domain and actual difference relative to the total size. Then some domains will be searched more frequently then other in your DB. In random sample you always may have domains that nobody searched in DB this month(or maybe ever). If you win for such domain it has no impact on user experience. I don't know about your users but our users don't search for random domains. So while sample has to be random for statistical study it is not really perfect if you want to talk about user experience. We never had a goal to quantify user experience in such comparison because in doing so we would come up with extremely complicated and subjective methodology. Simple methodology like yours wouldn't be sufficient enough for us to judge user experience. You can do whatever you want with your users, but you can't tell other people how they should measure user experience of their customers and demand them to do it your way. So our study is not about user experience it is about raw size of the index which is way more objective and easier to measure. And for that specific question I believe we have a much more suitable methodology as you seem to agree.
Newcomer

Either just recently joined or is too shy to say something.

with the new database i wonder if there is a way to see both the old data and the new. My thought is if this discovers more links that were already there and it flags it as new? I need to show hard numbers and not have to explain why there was an increase. Sure we can come in saying we are excited about data and all. But we should be able to parse this down.
Nick Samuel
Pro

Asks great questions and provides brilliant answers.

Backlinks have always been the weakest part of the SEMrush suite, so I'm glad to see work has gone into improving this.

As Moz found out when they rebuilt "link explorer" from the ground up, it's certainly no small undertaking and is a seriously complex endeavour.

I must confess I haven't really used the new backlink checker just yet, but if the numbers in this info graphic are an accurate reflection then SEMrush just became even more phenomenally good value for money!
Enthusiast

Occasionally takes part in conversations.

Looks terrific. Eagerly wait for a competitors backfire.
Dejan Mladenovski
Helper

An experienced member who is always happy to help.

Good to see SEMRush stepping up their backlink tool which I found was quite weak against other major providers such as Ahrefs. I will definitely be testing the tool to compare the performance of the backlink tool in terms of extensiveness and freshness in various niches, would be interesting to see the results
Elena Terenteva
SEMrush

SEMrush employee.

Dejan Mladenovski
Hi Dejan! Looking forward to your feedback!
Thomas Minarik
Enthusiast

Occasionally takes part in conversations.

I guess I caught this post just in time. I am thinking of switching to ahrefs, mainly due to their backlinks tool is meant to be far superior (?). I assume this is due to the number of backlinks in their database? Going by the first graph, it seems that Semrush backlinks database is now bigger than ahrefs? Did I get that right?
Elena Terenteva
SEMrush

SEMrush employee.

Thomas Minarik
Hi Thomas! Yes, you’re right, according to our last research we know more backlinks than Ahrefs in general. This And great backlink tool is not just about the number of backlinks in the storage, it's also about management tools - we have Backlink audit tool for it. It would be great if you’ll be able to share your feedback with us on both - the quality of our database and backlink management tools.
Expert

Provides valuable insights and adds depth to the conversation.

Great to hear! I've been recommending your backlink tool for a while now, so I guess I'm going to have to update that.
Enthusiast

Occasionally takes part in conversations.

This is a great news! Thank you :-)
Paul Lovell
Master

A veteran community member.

This is great, Even more data to dig into Thanks so much for the continued growth
Newcomer

Either just recently joined or is too shy to say something.

having a bunch of worthless content sites linking to yours does not help SEO, search engines are smart enough now to see what a real site is, a real business at a real location that has real content. so being yourself and not a trickster is the way to get page one. thousands of spam sites linking back to yours, those days are over. so how do you account for that?????
Elena Terenteva
SEMrush

SEMrush employee.

RobotKiller-BotnetAssasin
Hi! You are right, trustworthy links are much better for your website. For discovering toxic and weak links you can use SEMrush Backlink audit tool, which will help you to find potentially dangerous links and send them to the Google Disavow tool right away.
Peter Gerardi
Enthusiast

Occasionally takes part in conversations.

Unfortunatly after being a loyal customer for over 7 years, SEMrush has decided they are so big now that they no longer honor grandfathered in loyal customers who were there from the beginning....they now have a shady Sales department. I was tricked by their sales guy....total scam now. Soooo disappointed. SEMrush used to be amazing to work with, now they are just all about money and greed. What a shame. I remember back in the day....I know his last name was Black....I would deal with him and he was so good to me and my agency....so sad what has become of things....the sales guy was so condescending to me....unreal...be VERY careful with their sales department....I really got screwed over.
Elena Terenteva
SEMrush

SEMrush employee.

Peter Gerardi
Hi Peter! Thanks for highlighting the problem. We are taking this kind of cases very serious because despite our growth we still cherish the same good old principals and our customers' satisfaction is the number one priority. We are investigating this case and if you don't mind our customer success specialists are going to reach you. You can also contact me personally.
Peter Gerardi
Enthusiast

Occasionally takes part in conversations.

Peter Gerardi
I would like to follow up on this comment. SEMrush reached out to me today and we cleared up the issue I was dealing with. I am relieved because we have had an over 7 year relationship and I just couldn't believe what I was experiencing was what they intended. Miscommunication seems to have caused the situation . I respect that SEMrush quickly reached out and made things right!
Elena Terenteva
SEMrush

SEMrush employee.

Peter Gerardi
Appreciate your comment, Peter! Glad we could overcome this miscommunication.
Ricci M.
Pro

Asks great questions and provides brilliant answers.

Great to see Semrush continuing to develop and evolve the tools available.
Elena Terenteva
SEMrush

SEMrush employee.

Ricci M.
Thank you! Yes, growing and developing constantly :)
Newcomer

Either just recently joined or is too shy to say something.

I've tested several domains and Moz shows more referring domains, is this tool complete?
Elena Terenteva
SEMrush

SEMrush employee.

Bernard Stark
Hey Bernard, yes, the tool is complete, but data collection is an ongoing process, and we continue to improve our Backlink Analytics tool and collect the data. About the ref.domains and the Moz, different tools can show different results and it all depends on the specific data set of each tool for each domain.
And if you look at our graphs you can notice that soon we are going to catch up Moz.
By the way could you please send me your domains to check them?
Andre Schaefer
Newcomer

Either just recently joined or is too shy to say something.

Glad to see that Semrush updated their database and can now provide more data.

I have to admit that i regularly use other tools to do my competitor researches. But I've been a huge fan of the traffic growth graph as it helps clients to visualize their visitor growth. Might give your tool another shot soon.

Andre
Brooks Manley
Helper

An experienced member who is always happy to help.

Wow I'm stoked. Thanks for staying on top of things. I was literally just beginning to look at adding Ahrefs or Majestic to our toolset.
Elena Terenteva
SEMrush

SEMrush employee.

Brooks Manley
Hi Brooks! Glad you are with us :) What do you think we should also add or improve?
Brooks Manley
Helper

An experienced member who is always happy to help.

Elena Terenteva
I think the tool is great – we just work with pretty small businesses – so having data on each and every backlink makes a big difference and is important to us. So just having a lot more links in the database is huge for us.
Elena Terenteva
SEMrush

SEMrush employee.

Brooks Manley
We appreciate it! And we will continue to do our best. Thank you!
Enthusiast

Occasionally takes part in conversations.

Positive steps, look forward to seeing the results of the update.
Elena Terenteva
SEMrush

SEMrush employee.

Michael James Field
Huge positive steps! Share your feedback with us!
Enthusiast

Occasionally takes part in conversations.

Great content to share with colleagues. We can't afford to buy premium packages of your but still, we love your blog's content. Hope will get the same content in future from you gyuys!
Thanks
TryDigital Team
Elena Terenteva
SEMrush

SEMrush employee.

TryDigital
Oh, thank you so much!
Helper

An experienced member who is always happy to help.

There is no other tool which can give that detailed analysis of the site and manage it automatically in clicks.
About 74% of business in the industry already adapted SEMrush and are relying on it for their growth.
Totally deserving!!
Elena Terenteva
SEMrush

SEMrush employee.

Abhishek Jaiswal
Appreciate your comment, Princy!
Enthusiast

Occasionally takes part in conversations.

Very nice job. Thank you.
Elena Terenteva
SEMrush

SEMrush employee.

Chat
Thank you, Chat!

Send feedback

Your feedback must contain at least 3 words (10 characters).

We will only use this email to respond to you on your feedback. Privacy Policy

Thank you for your feedback!