Before I try to answer the question, let’s first ask one of the AI tools I tested if it thinks it can replace human copywriters.
“AI-generated content is on the rise. Some people are more worried than others about the implications of this. What does it mean for writers? Is there a future in which AI writers will replace humans?
The answer to that question is that, most likely, no. AI writers cannot write lengthy or complex pieces, nor can they generate quality copy independently. But what they do offer is an excellent way to tackle some of the less exciting aspects of content creation. So here are some examples of how you can use AI writers for your business to save time and money!”
Well, the AI was pretty clear on that point - it didn’t think highly of itself. But is it correct in its prediction?
Here at Umbraco, we love new technology, so I decided to put it to the test to see how good AI copywriting tools have become.
Here's a small demo of one of the AI tools working. This tool works mainly with text commands.
Just a warning from the get-go, this is going to be a long blog post. Feel free to dive in through the navigation below.
Table of contents
My hypothesis
Since "why" is always a good place to start, why am I even doing this test?
At Umbraco, we write lots of content every single day. This content is often about heavily technical topics, which means it takes a lot of time to do both research and the actual writing. To truly scale our content production, we have two options:
- Hire more copywriters
- Find alternative solutions
That second point is where AI comes in. If you’re reading this blog post, you have probably already read articles about robots replacing humans across different industries. Still, there is a big difference between a factory robot helping with assembly and a robot doing copywriting. That’s why nothing beats testing something out firsthand. I made the following hypothesis:
"I believe AI copywriting tools can scale up our content production by providing content that is ready to be published."
To test my hypothesis, I picked three parameters to prove or disprove it:
- Grammar
- Readability, writing, and flow
- Factual correctness
Disclaimer
Before I begin, a quick disclaimer:
- Umbraco does not have any deals or partnerships with any of the AI tools tested here.
- I used paid versions of all tools in order to avoid any limitations.
- All copywriters at Umbraco (with the exception of one who doubles as a proofreader) use Grammarly to clean up their grammar. For that reason, I also decided to use it on the output from the AI tools.
- Last but not least, the comments made here are based on my results, and not necessarily the exact result you will see if you test it yourself.
With that out of the way, let’s get to it!
Instruction interface of Jarvis.ai (now called Jasper AI)
Do you want to see the raw output of each tool? Sign up for our newsletter and we'll send them to you right away.
I want to see what the AI wrote
The AI test setup
So, will these machines take over my job as a marketer and copywriter? That's what we're here to learn, after all!
To run the test, I needed two things:
- Which tools should I test?
- Which topic should I have them cover?
The 7 AI copywriting tools I tested
The purpose of my test was to discover if AI tools can help produce some of the content we create at Umbraco. The main criterion for selecting tools to test was whether they can deliver a text output for long-form articles and blog posts. These are by far the most time-consuming content pieces we create and thus the best place to get support if we want to scale. With that in mind, I researched the available AI tools and picked seven that fit the requirements. Most of the tools offer free trials, but I paid for complete access so there were no limitations on words or credits.
Here are the seven AI tools I chose:
Although the tools have slight differences in how much input they need to create a text, they can all deliver long-form articles of 700 words.
The topic
I decided the AI tools would write a long-form article covering everything about schema markup. It's a fairly technical topic and a good representation of the type of content we create. Since there’s plenty of content on schema markup available online, I knew the AIs would have a large base of information to draw from.
The scoring process
To keep it simple, I decided to give each tool a score of 1-10 for each of the three parameters. Additionally, I ran a plagiarism check to see if the texts they wrote were unique or copied from other sources.
All seven texts were given a letter (from A to G) and handed over to one of our copywriters (the one that doubles as a proofreader) here at Umbraco HQ. I gave the copywriter this brief:
“We are looking at external resources like freelancers and AI tools to help scale our content production. I have gotten the following texts back and would love for you to grade them so we know which ones to go with.”
I did not tell my coworker which texts were written by AI tools (that is, all of them) because I wanted to avoid any inherent bias against them.
So how did they do? Let’s have a look.
The results of our AI test
Here’s a recap of the three parameters the AI outputs are scored on:
- Grammar (after using Grammarly Pro on the output)
- Readability, writing, and flow
- Factual correctness
Below, we'll break down each of the three parameters and discuss each tool's score between 1-10 (with 10 being highest). There are also general comments on how all of the tools performed.
Curious to read the full texts from the AI tools?
Leave your email here and it'll be in your inbox right away!
Quality of the spelling and grammar
First, I ran the texts through Grammarly Pro and accepted all suggested improvements. That means the below score is based on the output after using Grammarly.
The scores
- Jarvis.ai (Text A): 9
- Frase.io (Text B): 9.5
- Writesonic.com (Text C): 9
- Copymatic.ai (Text D) 9.5
- Article Forge (Text E): 8.5
- AI-Writer (Text F): 9
- Shortly AI (Text G): 7
The texts scored very well overall, and received comments from our copywriter like:
“Sentences are readable and complete.”
“Paragraphs generally flow and are connected.”
Others got the words:
“Strong command of grammar - readable and personable.”
Most AI tools, however, have inconsistent capitalization. For example, I input the keyword “Schema markup,” and while some tools kept this capitalization, others would use variations like “schema mark up” and “Schema Markup.”
Checking the AI written text with Grammarly
AI tools use passive language
If you intend to scale your content production with AI tools, you should know: One of the things I noticed before running our texts through Grammarly was the excessive use of passive language from all tools. Furthermore, I noticed the bots would often add a space before the periods/full stops (.). And while I know I'm not the best at comma (,) placement in either Danish or English, the same can be said about most AI tools based on our test.
Some tools also tend to combine words that should be two separate words in English.
Any company-specific grammar rules?
I would still highly recommend that you check the tools' grammar. Our copywriter pointed out that they won't know about company-specific rules or style guides, like whether semicolon use is allowed or how we handle capitalization in our headlines.
She also guessed that at least one of the texts came from an AI, but after seeing repeated issues with structure and organization, she became suspicious and raised the question, “Is there more than one AI-generated text?”
Overall judgment on grammar
In general, the AI tools applied decent grammar even before I checked them with Grammarly. With a tool like Grammarly at hand, you can get a reasonable result with limited human work. However, this doesn't mean the text will read naturally.
Quality of writing and flow
The quality and flow of writing are essential elements, especially for long-form content like the article I wanted the tools to write about schema markup. Since Grammarly only checks individual sentences, not how the text flows as a whole, you will see a massive difference in how well the tools perform. Without spoiling anything, I would say this is one of the biggest challenges in AI copywriting overall.
The scores
- Jarvis.ai (Text A): 7
- Frase.io (Text B): 8
- Writesonic.com (Text C): 3
- Copymatic.ai (Text D) 6.5
- Article Forge (Text E): 5.5
- AI-Writer (Text F): 6.5
- Shortly AI (Text G): 5
Looking at the scores here, you might already understand why the AI in our intro paragraph said human copywriters likely won’t be replaced anytime soon. Let’s see what our copywriter had to say when grading the texts.
Comments on the language and writing style
One of her critical comments can be applied to most of the AI tools:
“Transitions (since, on the other hand, in addition…) can use a lot more work, to feel like we’re drawing in the reader and creating an article that flows.”
It can be said that what separates good copywriting from great copywriting is how well the content flows. A lot of this comes down to the writer's ability to tell a consistent story from start to finish, all while giving the reader the feeling that everything is connected.
Unfortunately, you just don't get that feeling when you read texts from AI tools. Most of them generate sentences and paragraphs that are great individually, but when read through or viewed as a whole, it's clear they lack the transition phrases needed to sound natural and understandable.
There are a few other issues with the way they write, too.
Lazy AI
One of the tools, Writesonic (Text C), got lazy and referred to Wikipedia instead of explaining the topic:
“This is a new form of metadata, and as such, it is slightly confusing, as the Wikipedia page contains a lot of disclaimers about which way to interpret it.”
External references to authoritative resources are a great addition to most texts. In this case, the AI did not elaborate, disrupting the reading flow and forcing the reader to check that resource before they can continue. Worst-case scenario, the reader will leave your website to find more information elsewhere and never come back.
Condescending AI
Some AI tools allow you to set the tone of voice using controls or commands. Sometimes, though, the tone can come off condescending. This is especially true for tools that don't allow you to control the tone of voice. Here's an example from Writesonic:
“How Schema Markup Works? This is a straightforward thing to understand when you think about it. Schema is the markup that the W3C has built to represent data.”
Even if that statement were true, it's not a very friendly way to put it.
Sentences being cut short
Many tools tended to produce abrupt sentences or just cut them short in the context of larger paragraphs. They also left out either important details or additional information, as you can see in the section below. Sometimes sentences are cut short to a point where they stop making sense at all, like this one:
“Schema markup is good for SEO and what it does to help search engine rankings” - Jarvis.ai
Repetition
Moreover, we haven't yet touched on how often the tools repeated themselves. Even those I had more control over would repeat themselves throughout the text:
“Schema Markup helps with CTR as the user can see your site effectively in SERPS [...] Schema mark up is important as it works to improve your website CTR, SEO and helps users before they click through on search.” - Jarvis.ai
Overall tone of voice
In the screenshot of Shortly.ai above, you can see the tone of voice control is called "Creativity." It's important to be careful with these settings, however. In this case, I asked Shortly.ai to write a short text for social media that would describe the article it had just written, instructing it to use a personal tone of voice.
Here’s what it came up with:
"What are the benefits of schema markup, and why is it important for SEO and accessibility? A few months ago, I was searching for a new apartment in Zillow. One of the things that I noticed on the page was some free tools to help me find specific apartments based on my preferences. This article will go over what schema markup is and how to use it if you're interested!” - Shortly.ai
First of all, I had to look up what Zillow was (a U.S.-based real-estate marketplace). Second, what does that have to do with schema markup? Finally, the sentence structure makes a terrible transition between the apartment research and schema markup that doesn’t make sense to anyone reading it.
Overall judgment on writing and flow
Unfortunately, the AI tools leave a lot to be desired here. The transitions were oftentimes lacking, and the laziness and repetition meant texts weren't even close to publish-ready.
(Oh, and while I did enjoy trying out the different settings for tone of voice, it didn't always work out that well.)
Our copywriter's score for this category fell between 3 to 8 for all tools, which is a big range. The higher-rated tools show potential, which means it's not completely hopeless. Still, the texts weren't at a level where I'd feel comfortable publishing them without input from a human copywriter.
Is the text factually correct?
I don’t know where to begin with this parameter. If I wanted to go full-on clickbait, I would write:
You won't believe what happened next!
Before we dive in, let’s have a look at the results with our score from 1-10:
The scores
- Jarvis.ai (Text A): 4
- Frase.io (Text B): 2.5
- Writesonic.com (Text C): 2.5
- Copymatic.ai (Text D): 6.5
- Article Forge (Text E): 4
- AI-Writer (Text F): 7
- Shortly AI (Text G): 3
Fact-checking the AI tools
I'll be using a lot of examples from different tools in this section. Get your hands on the full texts yourself!
Send me all the texts written by AI
AI giving advice on SEO
“Schema markup is a type of rich snippet that can help your website's search engine rankings and provide more information to searchers about the content on your site.” - AI-Writer
First of all, schema markup doesn’t directly help your websites rank better. Having schema is not a ranking factor in Google’s algorithm. It does help in some ways, like helping you win rich snippets in the search results, which can generate more traffic. So while it’s not entirely wrong, it is stretching the truth with that sentence.
Then, there are AI tools that focused on random SEO tips with absolutely nothing to do with schema markup. One of them even decided to add a tip in the middle of explaining two reasons to use schema markup:
“…So if you're writing technology articles, you may want to sprinkle in a few keywords here and there to give yourself a head start on getting found.” - Frase.io
Technically, it's not wrong. It's just so out of place in the context of schema markup that I won't call it correct either.
Finally, there’s the plain wrong and bad advice that I got from one AI writer:
“You probably don't need to worry too much about schema markup if you run a local business because most local searches aren't done with a map.” - Frase.io
If I were to advise a local business on SEO, I would recommend they implement this right away because your address, opening hours, phone number, and more are all crucial information for potential customers.
Mixing up various terms
I know schema markup is a broad term when it comes to the various forms of markup languages and structured data, but sometimes, the AI tools went off topic and began including weird abbreviations when explaining the SEO benefits:
“Another benefit of the different types of Schema Markup Language (SSML) is that they are easily searchable. Google has recently developed an algorithm that gives ranking points to websites that use local business Schema Markup Language (SBML) when creating their web pages.” - Article Forge
In this case, it has picked two abbreviations for “Schema Markup Language,” SSML and SBML. Unfortunately, neither of them is correct, and it adds a layer of confusion to the text.
While SSML stands for Speech Synthesis Markup Language, SBML is Systems Biology Markup Language, a markup language for biology.
Overall, most of the tools struggled in this category, but one tool did score quite high: AI-Writer. It explained in detail what schema markup is and even referred to a tool to create JSON-LD for your schema.org markup. Furthermore, it states that schema is not a quick and dirty SEO hack nor a guarantee for better rankings. Though the piece still has some flaws, it is a workable intro for a longer content piece about schema markup. And it doesn't make any statements that are straight-up false, which is refreshing.
AI giving advice on competitors
Having the facts correct is one thing, but matching the context of your business and industry is another. For example, some of the connections the AI tools made were factually correct but didn't fit the approach or scope of an Umbraco article.
“As mentioned above, you should always keep it close to its source, meaning wherever the data comes from. The most common locations include footer scripts, header files, and plugins. A plugin is simply a snippet of JavaScript or PHP code added directly into WordPress' core. Plugins often contain custom functions and filters which allow developers to extend WordPress functionality. They're great for automating repetitive tasks and making things easier for end-users.” - Frase.io
The AI tool most likely picked this up because it analyzed dozens of articles on the internet, and thus provided facts that fit in its "AI mind." Its advice isn't wrong, it's just not something we would ever include in one of our articles.
Because AI tools won't know what's relevant or not, you have to be cautious and check their information so you don't end up referring to your competitor or another topic you don't want to cover.
On another note, the sentence doesn't even cover how to implement schema markup at all; it just explains how plugins work.
Is the content outdated?
Do you remember Google Plus? Though I had almost forgotten about it, one of the AI tools still thinks we do SEO like it's 2011.
“Google has changed its algorithm to favor sites that use Schema. Like Google Plus, any website that uses Schema markup will rank higher in SERPs when searched for specific topics." - Copymatic
While this wasn't a massive problem for the texts in my test, it's definitely something worth checking. The AI tools can’t know fact from fiction, so if there is old content about a given topic on the internet, there’s a chance it will use that as a reference point and add outdated details.
Is the content original?
All of the AI tools I tested marketed themselves as able to create original and plagiarism-free content.
There are a couple of ways to put this to the test. I started with one exact phrase-match search, i.e. “[copied sentence from an AI text here]” to check if that sentence existed in Google’s search index. Thankfully the first test returned almost no results, and those that showed up were definitions or generic sentences, which you can’t avoid - even as a human.
But then, one phrase gave me almost 500 results: “As with other markup formats” - AI-Writer
That made me curious. Most of these results had almost the same phrasing and structure, with a few changes to some words.
Instead of manually spot-checking sentences, I decided it was time for a more in-depth analysis. Luckily, Grammarly has a plagiarism feature to check how much text is found on the internet. Below, you can see the plagiarism percentages from Grammarly:
- Jarvis.ai (Text A): 5%
- Frase.io (Text B): 5%
- Writesonic.com (Text C): 3%
- Copymatic.ai (Text D): 3%
- Article Forge (Text E): 2%
- AI-Writer (Text F): 8%
- Shortly AI (Text G): 3%
Checking this blogpost for plagiarism - Got 3% in return
Are those numbers low or high?
It's probably worth adding a few notes on the plagiarism check if you’re unfamiliar. Given that these tools are trained on millions of articles, but may still flag cited references and quotes as plagiarism, there’s a good chance the check will find a match.
Often, it will return examples with common or generic phrasing, like this:
“...there are a lot of similarities between the two which makes...” - Shortly.ai
I believe AI-Writer, the tool that scored highest for factual correctness, scored 8% for plagiarism largely for generic phrases and/or transition phrases. It's critical you do a manual check in order to avoid publishing a plagiarized text.
By the way, the score for this blog post is 3%. So am I a thief? No. It dinged me for common transitions and expressions (including those from the AI writers).
Which tool was the best?
So, who won? Let’s look at the scores for all tools.
|
Tool |
Grammar |
Quality and flow |
Facts |
Total |
Plagiarism% |
|
Jarvis (Text A) |
9 |
7 |
4 |
20 |
5% |
|
Frase.io (Text B) |
9.5 |
8 |
2.5 |
20 |
5% |
|
Writesonic.com (Text C) |
9 |
3 |
2.5 |
14.5 |
3% |
|
Copymatic.ai (Text D) |
9.5 |
6.5 |
6.5 |
22.5 |
3% |
|
Article Forge (Text E) |
8.5 |
5.5 |
4 |
18 |
2% |
|
AI-Writer (Text F) |
9 |
6.5 |
7 |
22.5 |
8% |
|
Shortly AI (Text G) |
7 |
5 |
3 |
15 |
3% |
The highest total score of 22.5 goes to two tools: AI-Writer and Copymatic.ai. But since nobody likes a shared first place, I will use the plagiarism check as the tiebreaker.
🎉 Congratulations, the winner is Copymatic.ai! 🎉
Now that we have a winner, we can finally reflect on our original hypothesis:
"I believe AI copywriting tools can scale up our content production by providing content that is ready to be published."
What is the verdict on the AI writing tools available in 2022?
This time, let’s go with the human verdict and not our AI prediction from the beginning of our article. Will AI copywriting take over our jobs as copywriters?
No. As a copywriter, you will not be replaced by an AI tool anytime soon.
Though tools like Grammarly can quickly fix grammar for you, the AI writing tools we tested are not there yet when it comes to writing high-quality, full-length articles. They don’t have a consistent flow throughout the piece, struggle with transitions, and often repeat themselves to reach the word count.
Regarding factual correctness, the tools leave a lot to be desired. For example, they can’t fact-check or use common sense to judge the information they have synthesized. The matter of outdated content is also an issue.
Are there any use cases for AI tools?
My research shows there are still use cases for these tools, even if they can’t completely replace human copywriters.
Short-form content
You often don’t experience the same problems as I did in this test, with the bad reading flow and repetition, in shorter sentences. This means you can see great results if you have a ton of meta descriptions or short product descriptions to write for your eCommerce website. Of course, you’ll still want to check the output and make small changes, but it could potentially save you a lot of time.
Use it as inspiration
Even if the AI copywriters can't write a complete, publish-ready article, their output can easily be used as inspiration for a human copywriter. For example, you can use the headlines as a rough outline of topics to cover and discard the text. In my test, the AI tools did include a lot of relevant headlines - and there were even some I hadn’t initially thought of. This way, you can speed up your research and even uncover hidden gems.
Will we be using AI tools going forward?
Yes. At least, we will do some further testing on the winner, Copymatic.ai. While it won’t replace human copywriters, it can potentially speed up our research phase by providing article outlines. Of course, we still need to do most (or all) of the writing, but the time saved on research may very well make it a valuable tool in our tech stack.
Did I miss an obvious use case for AI copywriters? If you have any experience or questions regarding AI tools, please feel free to reach out to me at ncl@umbraco.com and let me know. I look forward to hearing from you!