Integrating Umbraco CMS with Semrush

Working with Semrush

Semrush is a marketing SaaS platform that provides online visibility management and content marketing through a single platform,  on all key channels. It is a trusted data provider whose data on search volume, correlation, website traffic, and digital trends will provide you with valuable keywords suggestions so that you can write relevant content that people enjoy reading. 

Semrush analyzes the data for you from the world’s largest database of 20 billion keywords, 310 million ads and 17 billion URLs crawled per day and gives you instant recommendations on SEO, content marketing, and advertising that can help you improve your online visibility in days. You can find out more about their mission here.

 

What is Keyword Research

Keywords are the terms that people enter into search engines to find information about. In digital marketing, keywords are the intersection of you and your potential customers. You can target keywords by writing content on your website or through an advertising budget on Google Ads.

When you’re trying to find a product or service search engine, you describe it in words and type them into the search field. The search engine matches those words with the words on various websites that are included in its index and gives you the most relevant results - websites that best match the words you typed. They are the most important ingredient of the successful promotion of any project. 

Adding relevant keywords to your website content will help search engines associate your project with search queries and improve your rankings. Therefore, it is crucial to do keyword research to find the right keywords for your project.

Keyword research is the study of these search terms to find out where the search traffic in your market comes from, to determine which keywords your website can target with marketing. This featured tool will run a full analysis of your keyword and based on the provided results you can decide whether you should enter into competition for it. As it is an integral part of any digital marketing strategy, its importance will never fade away.

Keyword research is the first step in the SEO copywriting process and an essential part of your SEO strategy.

 

Benefits:

• Understand and reach your target audience
• Find new relevant topics to cover on your site
• Get qualified leads
• Increase sales
• Improve ad campaigns, get clicks and impressions
• Build awareness of your brand

 

Why should you do keyword research on a regular basis?

• Keywords change - there will always be multiple ways to say the same thing and trends often change the queries people search for online.
• Your audience grows and evolves, and the way people use search engines changes.
• Search engines evolve too - algorithms get more advanced to better understand the content and also match users’ intent.
• You need to keep upgrading your content for it to be relevant.

 

Security Protocols

Semrush uses the OAuth protocol to secure access to their API. OAuth is an open-standard authorization protocol which enables applications to access data on behalf of users without needing to know their username and password. This provides a better separation of concerns between Umbraco backoffice services.

All the communication is wrapped around by an Umbraco-managed authentication proxy that handles various security scenarios, in this particular case as well as for Hubspot, an OAuth security layer.

The flowchart below provides an insight into the component’s workflows:

In order to gain access to the keyword search tool, you first need to be authorized against Semrush. You can either use a free account (limited to 10 queries per day and 10 keywords search), a free trial, or a subscription-based plan. Read more about this here.

This can be handled from the UI by selecting the Connect button. A Semrush managed static OAuth page will prompt for credentials or account registration. The editor will then be presented with the option to Approve or Deny the authorization process.

If Approved, the system will request the access token from Semrush by routing the pipeline through the OAuth Proxy. The API will respond with an access token, an expiration timestamp, and a refresh token which will be saved serialized into the website’s database.

Query requests to Semrush will use the stored access token to retrieve data. On each content load, the Semrush API is pinged using a mock request to ensure the validity of the access token. If a 403 Unauthorized code is received, then a new request routed through the OAuth proxy server will be triggered but this time using the refresh token as input for retrieving a new access token.

If Denied, the process will come to a halt.

 

Analytics API

We have extended the editing experience when working with content in the Umbraco backoffice by embedding the Semrush API and the keywords research tool into a custom Content App that will enhance your experience working on a specific content item.

Semrush uses its own machine learning algorithms and trusted data providers to present the data from their 142 get databases. Their keyword database is the largest on the market exceeding 20 billion records.

More details on the Semrush API as well as pricing and variations can be found in the documentation right here.

Keyword reports that have been integrated with our custom component:

1. phrase_fullsearch: provides a list of broad matches and alternate search queries, including particular keywords or keyword expressions.
2. phrase_kdi: provides keyword difficulty, an index that helps to estimate how difficult it would be to seize competitors’ positions in organic search within Google’s top 100 with an indicated search term.
3. phrase_organic: lists domains that are ranking in Google’s top 100 organic search results with a requested keyword.
4. phrase_related: provides an extended list of related keywords, synonyms, and variations relevant to a queried term in a chosen database.
5. phrase_these: provides a summary of up to 100 keywords, including its volume, CPC, competition, and the number of results in a chosen regional database.
6. phrase_this: provides a summary of a keyword, including its volume, CPC, competition, and the number of results in a chosen regional database.

 

The reports and metrics used by the component consist of the following:

• Volume: the average number of times per month that a keyword was queried on Google (on the national level).
• Keyword difficulty: an estimation of how easy it would be to rank your website for this keyword with SEO.
• Global Volume: the combined total of the keyword’s monthly search volumes across all of our country databases.
• Intent: the purpose of a user’s query in a search engine. Intents can be read by the search engine algorithms to show the proper results and SERP features.
• Results on SERP (Search Engine Results Page): the number of URLs displayed in organic search results for a given keyword.
• SERP Features: hover over the icons to see which special SERP features are present on the results page for this keyword.
• CPC: cost-per-click add advertiser pays Google for a click on their PPC ad on this keyword’s SERP.
• Com: the density of advertisers bidding on this keyword for their ads on the national level. A score of 1.00 indicates the highest level of competition, and a score of 0.00 indicates no competition.
• PLA: the top Product Listing Ads (Google Shopping) seen on this keyword’s SERP.
• Ads: the top Google Ads PPC ads seen on this keyword’s SERP.
• Trend: how much interest web searchers have shown in a given keyword in the last 12 months.

 

Content App Description

The Semrush interface is available for all types of content items as well as for all user groups. 

The separation of permissions between Administrators and Others is minimal, and covers only the use case managing the connectivity with Semrush:

UI Interface is split into two views:

 

1. Content Properties

1. Contains a paragraph with descriptive text and controls for either selecting a content property or starting a new search.
2. Allowed content properties are fields of type textbox or textarea of the content item, grouped by the tab section:

      c. Once an action is selected, the Keyword Search area will become available.

2. Keyword Search

 

While the authorization process against Semrush has not been completed, the search functionality is disabled.

An Administrator will have to authorize the site with Semrush by selecting the Connect action button. After the registration/login process is completed the Administrator will receive notification of the scopes covered by the application, which he can Approve or Deny.

Umbraco’s backoffice notification service will provide feedback on the result of the user’s actions:

The search tool filters provide helpful information into the expected result of each selected filter, as well as each column of the results table:

In order to provide the best user experience and optimize the interaction with the API, the results are cached for an hour under a custom key formatted as: [query]-[database]-[search_method]. For example, the key lamp-us-phrase_organic will return a cached set of records that will facilitate the pagination of the table, without having to request data from the API.

An administrator will be able to revoke the access to the search tool from the Status editor:

Conclusions

We’ve open-sourced this integration and the current release of this package targets Umbraco version 8. The full source code for the content app is available here on GitHub. In the future, we will align the package with the newest releases of Umbraco version 9 and version 10 featuring .NET 5/6.

The proxy app targeting .NET5 is available here on GitHub - and, using a generic implementation can support different OAuth authorization scenarios. Please feel free to “test drive” the two integrations together or individually, as the demonstrated methods have multiple applications.

 

This is the third installment of a series that covers integrations with providers like CommerceTools and Hubspot.