MISSOULA, Mont. — From Shakespeare and Steinbeck to Scorsese and Spielberg, humanity has always had a certain knack for storytelling. New research by a team from the University of Montana suggests, however, we may soon end up handing over our creative reigns to more capable mechanical hands. Researchers report AI can match the top one percent of human thinkers on a standard test for creativity.
More specifically, this project assessed eight responses generated by ChatGPT, the application powered by the GPT-4 artificial intelligence engine. This project was led by Dr. Erik Guzik, an assistant clinical professor in UM’s College of Business. The research team utilized the Torrance Tests of Creative Thinking (TTCT), a well-known tool that’s been used for decades to assess human creativity.
Besides the AI-generated responses, researchers also submitted answers from a control group made up of 24 UM students taking Dr. Guzik’s entrepreneurship and personal finance classes. Those scores were compared with another 2,700 college students on a national scale who had taken the TTCT in 2016. All submissions were scored by the Scholastic Testing Service, which had no idea AI had created many of the responses.
The ensuing results placed ChatGPT in elite company when it comes to creativity. The AI application landed in the top percentile for fluency, or the ability to generate a large volume of ideas – and for originality, or the ability to come up with new ideas. The AI did, however, slip a bit to the 97th percentile when it came to flexibility, or the capacity to generate different types and categories of ideas.
“For ChatGPT and GPT-4, we showed for the first time that it performs in the top 1% for originality,” Dr. Guzik says in a university release. “That was new.”
It’s worth noting that some of Dr. Guzik’s human UM students also placed in the top one percent for creativity. However, ChatGPT outclassed the vast majority of college students nationally. These tests took place during the spring semester, and this work was assisted by Christian Gilde of UM Western and Christian Byrge of Vilnius University. These findings were recently presented in May 2023 at the Southern Oregon University Creativity Conference.
“We were very careful at the conference to not interpret the data very much,” Dr. Guzik explains. “We just presented the results. But we shared strong evidence that AI seems to be developing creative ability on par with or even exceeding human ability.”
Dr. Guzik adds that at one point he asked ChatGPT what it would indicate if it performed well on the TTCT. The AI responded with a strong answer:
“ChatGPT told us we may not fully understand human creativity, which I believe is correct,” the researcher recalls. “It also suggested we may need more sophisticated assessment tools that can differentiate between human and AI-generated ideas.”
TTCT is protected proprietary material, which means ChatGPT couldn’t “cheat” by accessing information on the test via the internet or a public database.
Dr. Guzik has been fascinated by creativity since he was a child. As a seventh grader growing up in the small town of Palmer, Massachusetts, he enrolled in a program for talented-and-gifted students. That program introduced him to the Future Problem Solving process developed by Ellis Paul Torrance, considered a pioneering psychologist who also just happened to create the TTCT. According to Dr. Guzik, he fell in love with brainstorming at that time and how it taps into human imagination, and he remains active to this day with the Future Problem Solving organization. He even met his wife at one of its conferences.
So, Dr. Guzik and his team set out to test the creativity of ChatGPT after playing around with it over the past year.
“We had all been exploring with ChatGPT, and we noticed it had been doing some interesting things that we didn’t expect,” he notes. “Some of the responses were novel and surprising. That’s when we decided to put it to the test to see how creative it really is.”
Dr. Guzik explains the TTCT test uses prompts that mimic real-life creative tasks. For example, can you think of new uses for a product or improve this product?
“Let’s say it’s a basketball,” he specifies. “Think of as many uses of a basketball as you can. You can shoot it in a hoop and use it in a display. If you force yourself to think of new uses, maybe you cut it up and use it as a planter. Or with a brick you can build things, or it can be used as a paperweight. But maybe you grind it up and reform it into something completely new.”
Researchers expected that ChatGPT would be good at creating a lot of ideas (fluency), since that’s generally what generative AI is designed to do. Sure enough, it excelled at responding to the prompt with many ideas that were relevant, useful, and valuable as assessed by the evaluators.
Dr. Guzik and his team were more surprised at how efficient the AI was at creating new original ideas, seen as a hallmark of human imagination. Test evaluators are provided with lists of common responses for a prompt, or answers that are almost expected to be submitted. The AI scored in the top percentile for coming up with fresh responses.
“At the conference, we learned of previous research on GPT-3 that was done a year ago,” Dr. Guzik says. “At that time, ChatGPT did not score as well as humans on tasks that involved original thinking. Now with the more advanced GPT-4, it’s in the top 1% of all human responses.”
As AI technology continues to advance at a rapid rate, Dr. Guzik predicts it will become a key tool for the business world going forward, as well as a significant new driver of both regional and national innovation.
“For me, creativity is about doing things differently,” Dr. Guzik comments. “One of the definitions of entrepreneurship I love is that to be an entrepreneur is to think differently. So AI may help us apply the world of creative thinking to business and the process of innovation, and that’s just fascinating to me.”
“I think we know the future is going to include AI in some fashion,” he concludes. “We have to be careful about how it’s used and consider needed rules and regulations. But businesses already are using it for many creative tasks. In terms of entrepreneurship and regional innovation, this is a game changer.”
You might also be interested in:
- ‘I used ChatGPT to force my landlord to fix my washer and dryer’
- ChatGPT for the mind? Brain activity decoder reads and writes down your thoughts
- Building a team with more diverse personalities leads to greater creativity
