For quite some time, the SEO industry has been arguing over what is commonly referred to as zero-volume keywords. You can read these two articles (one and two) and make up your mind on this topic. I actually have countless examples where writing on zero-volume topics yielded remarkable results, as the one shown below. There are usually no secret recipe: you need to know your audience and/or get lucky to encounter an easy SERP.
That being said, there are situations where search volumes are useful and accurate. It’s easy to get the data for a couple of hundreds with any mainstream tool or the Google Keyword Planner if you have running ads (otherwise you’ll get volume brackets). But how can you retrieve search volumes for thousands or even millions of keywords? In this article, I’ll go through my go-to solution.
Why would you even need data for thousands of keywords?
This is a legit question you may ask yourself. At the end of the day, we usually don’t need to get the search volume for every keyword we have in our Google Search Console, for instance. That’s true!
But imagine for a moment that you work for the French national railway company (SNCF). The French network is composed of more than 3,000 train stations (source), which ends up creating a bit less than 9,000,000 possible routes between them. You’ll obviously want to cover the most searched in your architecture, and you’ll have to base your decision on:
- First-part data such as internal searches and sales
- Third-part data, such as the number of searches on Google and other search engines
And this is where the ability to retrieve search volume at scale will be handy!
Usual approach
If you want to get data for your keywords, you usually have two options.
Google Keyword Planner
This tool from Google can be the easiest option in most cases. Nevertheless, you cannot add more than 700 keywords at once. If you have a couple of thousands, you could add them by batch, but it quickly becomes annoying and time-consuming.
There is still the Google Ads API, but setting it up is complex and requires making your way through Google’s documentation.
Other tools
Some tools have an API to get keyword data, including search volumes. But I don’t often recommend using most of them for many reasons:
- Price: the API is (almost) always only available with the most expansive plans. Moreover, you often have to spend a lot of API credits to get all the information you need.
- Documentation: the documentation (even if Semrush has improved a lot on that) is bad. Have a look at what ScrapingBee does (source) where everything is well-explained, and code snippets can be copied. And then compare it to most SEO providers. I really don’t like spending an hour figuring out how the API works.
- Batch requests: one critical features for our usage is to be able to retrieve data by batch, meaning that one call can return the data of hundreds or thousands of keywords. Most of the API I’ve tried don’t allow it.
How do I do it?
Obviously, I’m not writing this article to let you down and tell you that you can’t retrieve keyword volume at scale. I’ve discovered an API (DataForSEO) that has all the required functionalities you need. DISCLAIMER: I’m not paid by the company to write this article. I just happen to be a delighted customer, but they are not well-known.
The only issue: it’s an API; therefore you need to write code or use a no-code tool to do it for you. But you can get the volume for 1,000,000 keywords in less than an hour for just $50. Amazing, right? Also, you can get data for Google but also Bing. 🎉
I won’t provide the codes you need to make the calls because:
- I work in Python, but you may use another language. If you are also using Python, have a look at this repository to speed your implementation. If you’re not, have a look at the documentation.
- This article is aimed at explaining the steps you have to follow. If you need help with the code, reach out to DataForSeo.
To get the search volume, you need to follow these steps:
- Pick a location: you have a specific endpoint to get the list. These locations are the same available in Google Ads and are usually a country, a region or a city. Once you’ve found yours, write it down. You can also use the location_name, but I prefer to use the latter.
- Divide your keywords in batch of up to 1000: One of the best features of this API is that you just need one call to get the data for 1,000 keywords. If you have 1,600 keywords, you need to send two calls: one with 1,000 keywords and another with 600. Using the POST method (see documentation) you then request the data. The data is not sent immediately unless you use the LIVE method (more expensive). Be careful to check the limitations you have for your keywords (forbidden characters, size) because if one of your keywords doesn’t pass the check, the full batch will fail.
- Wait a moment and check that your data is available: based on the number of batches, you may have to wait <1mn or way more. But there is an endpoint to check if your data can be downloaded.
- Download your data: using the last endpoint, you can download what you wanted. The output (after a bit of data manipulation): the data you need!
Conclusion
Retrieving keyword volume at scale is sometimes required, as it helps you to understand the popularity of specific keywords. The method mentioned in this article is the one I use, and I always recommend. If you don’t know how to code, just ask someone who does and save time and money on the long term!
