MIT Study: Do Readers Prefer Human, AI, or Human+AI Content? Answered in 3 Bar Charts

by | Dec 19, 2023

MIT Sloan’s Business School recently conducted an insightful study evaluating various content creation methods. This unique research pitted ten professional content creators from a leading consulting firm against ChatGPT-4. The participants, including these creators and the AI, generated advertising content for products and persuasive content for campaigns.

The findings offer valuable insights for professionals in digital marketing and SEO. Let’s explore the study’s details and its key takeaways.

AI vs Human Content Study Highlights:

So, to understand this study and the charts below, you’ll first want to understand the basics of this study.

  • Researchers:
  • Date Written: May 20, 2023

  • Study Size: 1,212 participants

  • Content Creation Methods:

    • Human: 10 professional content creators from a leading consulting firm

    • AI: Using ChatGPT-4

    • Augmented Human: Human editing AI-generated content

    • Augmented AI: AI refining human-generated content

  • Writer Prompts:

    • Create persuasive campaign content in under 100 words. Encourage actions like stopping racism or starting recycling.

    • Write advertising content for a product in under 100 words. Use provided product pictures and feature descriptions.

  • Participant Asks:

    • Willingness-to-Pay: State the largest amount they would pay for the content.

    • Interest or Persuasion: Rate their interest in the product from the campaign.

  • Conditions:

    • Baseline: Participants evaluated content without knowing its creation method (human or AI).

    • Partially Informed: Participants were aware of the four content creation methods but not which was used for each piece.

    • Informed: Participants knew the exact content creation method used for each piece.

Study Findings

The first chart shows the satisfaction ratings given by the 1,212 participants. In the baseline condition, AI-generated content was the most satisfying, closely followed by Augmented AI. Human-only content scored the lowest.

MIT AI content study

The second chart reveals the maximum amount participants would pay for the advertised product. AI and Augmented AI content were more compelling in creating perceived value.

MIT AI content study

The third chart illustrates interest and persuasion of the content, separated by campaign and product writing. Here, AI was greatly preferred for campaign copy, while Augmented AI was more favored for product copywriting.ai vs human copywriting results chart

Key Takeaways

Participants Prefer AI or Augmented AI Content

This really surprised me.  Given that AI content is relatively new to the mainstream, considerably faster to produce, and significantly less expensive to create, I would expect professional writers to create more compelling writing.  According to the study, participants preferred AI and AI-augmented content due to three reasons:

AI and Augmented AI Perceived as Higher Quality

Participants often viewed content created by AI and augmented AI as superior in quality compared to that created by humans or augmented humans.

Effectiveness of ChatGPT-4

The use of ChatGPT-4, a sophisticated AI model, was effective in generating content that resonated well with the participants, indicating the potential of AI in content creation.

Human Bias in Content Evaluation

There was a noticeable bias towards human-generated content. When participants knew the content was created by humans, they rated it higher, suggesting a preference for human authorship.

No Significant AI Aversion

Contrary to some expectations, the study found no substantial negative bias against AI-generated content among the participants.

Objective Evaluation in Baseline Condition

In the baseline condition, where participants were unaware of the content’s origin, their evaluation was based solely on the content itself, leading to an impartial assessment of AI-generated content.

Collaboration Between Humans and AI

The study highlighted the effectiveness of human-AI collaboration in content creation, suggesting that a combined approach could yield high-quality results.

In Conclusion

This study suggests that AI is a powerful tool in content creation, but it doesn’t signal the end for human content creators. The study’s scope was limited to short prompts ideal for AI, and human writers excel in understanding audiences, storytelling, and adding personality in long-form copy.

Interestingly, Augmented AI scored similarly to human writing. Common practice involves AI drafting first and then human editing, but the study suggests the reverse workflow could be beneficial. In my content creation, I plan to use ChatGPT-4 for final revisions and note that aversion to AI content is rapidly diminishing.

The full study is available online if you’d like to dig deeper and extract even more leanings.  And here’s another idea: download the PDF and upload it to ChatGPT-4, then you can ask it questions for direct answers!

 

Human vs AI Content FAQs

Does AI-generated content have better quality than human-generated content?

The study found that content created by AI and augmented AI was often perceived as higher quality compared to human-only or augmented human content.

How does knowing the source of content (human or AI) affect its evaluation?

Participants showed a bias towards human-generated content. When they knew content was created by humans, they tended to rate it higher, indicating a preference for human authorship.

Is there a negative bias against AI-generated content?

Contrary to some expectations, the study revealed no significant negative bias against AI-generated content among participants.

How was the AI used in the study prompted to generate content?

ChatGPT-4, the AI used in the study, was given specific prompts to create persuasive campaign content or advertising content for products in under 100 words, often including specific campaign goals or product features.

What is the advantage of combining human and AI efforts in content creation?

The study highlights the effectiveness of combining human creativity with AI’s analytical capabilities, suggesting that this collaborative approach can lead to the production of high-quality content.

Can AI replace human content creators?

While AI shows potential in content creation, the study suggests that the most effective approach might be a collaborative one where AI augments human creativity, rather than replacing it entirely.

FAQs

Can AI really write a blog post that people will want to read?

AI has advanced to a point where it can generate informative and engaging content. However, the key to making AI-written posts readable and enjoyable lies in combining AI’s capabilities with human creativity and insight. This blend ensures the content is not only SEO-friendly but also genuinely interesting to readers.

How does AI help in SEO for blog posts?

AI can significantly enhance SEO efforts by identifying trending keywords, suggesting related topics, and even optimizing the structure of the content for search engines. By leveraging AI for these tasks, writers can ensure their blog posts are more likely to rank well on search engine results pages (SERPs).

What are the limitations of using AI for blog post writing?

While AI can produce well-structured and coherent content, it may lack the depth of understanding and personal touch that comes from human experience. AI-generated content can sometimes feel generic or lack nuance, highlighting the importance of human oversight and editing.

How can I ensure my AI-written blog post is unique?

To ensure uniqueness, use AI as a starting point for generating ideas and drafting content. Then, infuse the post with your personal insights, experiences, and style. It’s also important to edit and revise AI-generated content to avoid any potential issues with duplicity or superficiality.

Are there ethical considerations in using AI for writing blog posts?

Yes, transparency and honesty about the use of AI in content creation are important. It’s ethical to use AI as a tool in the creative process, but misleading readers into believing AI-generated content is entirely human-produced can raise ethical questions. Always strive for clarity and honesty in your content creation process.

How can I get started with using AI writing tools for my blog posts?

Begin by experimenting with AI writing tools, such as ChatGPT, to understand their capabilities and limitations. Use these tools for brainstorming, drafting, and even SEO optimization, but remember to add your unique perspective and voice to the content before publishing. But always use an AI writing tool responsibly.

Free Ebooks & Video Training

Download PPC, SEO, and lead generation resources from our library to take your performance to the next level.

Ryder Meehan

Ryder has been on a 16-year journey to master digital marketing from every aspect. His resume includes Razorfish, Slighshot, Fossil, Samsung Mobile and Tatcha before launching Upgrow. Ryder is the acting CEO, heading business development and account services. He has been featured as a digital marketing leader on Forbes, PRNews, Business.com, Workamajig, Databox, Fit Small Biz and other outlets.

Read More from Our Blog

14 Essential ChatGPT Prompts for SEO
14 Essential ChatGPT Prompts for SEO

AI is dramatically changing the way businesses operate and productive capacity, and that is especially true for SEO.  Marketers who embrace AI and leverage it properly will sprint ahead of competitors in the coming years. AI has improved our output volume, increased...

read more