After a pause on the experiment train, it’s time to pick back up and do some more experiments among the 4 frontier models.
This time, we’re going down the qualitative path.
As before, this will start with the prompt, then list the four results.
The prompt for an in-depth interview discussion guide
This is for my series of experiments among LLMs.
I want you to draft a discussion guide for 1:1 in-depth interviews with working adults who are also taking care of elderly parents. The discussion guide should be about 45 minutes long and should have three main sections. The first should ask about the balance of time between their job and taking care of their aging family members and any current support they are getting from their employer through this time. The second should focus on their self-care through this period.
And the third should focus on what additional support they would like to see from their employer while going through this period of their lives.
Include interview guides as well as follow-up prompts for the interviewer.
The results
First place: Claude.
Second place: ChatGPT
Third place: Gemini and Microsoft Family 365 Copilot
Claude did the actual absolute best. We're talking:
introduction about the study for the interviewer
“how to use this guide” notes
clear minute-by-minute guides for each section
follow-up prompts for each question
a stimulus list for the 3rd section on how employers could better support employee caregivers of aging family members
suggestion of notes the interviewer might want to capture right after the session was done.
It was extremely thorough. This is one case where Claude's tendency to be wordy paid off. The wording itself was also extremely empathetic, I thought.
ChatGPT was second best because while it had instructions for the moderator as well as a good introduction and opening questions, had moderator notes, the thoroughness that Claude gave in its discussion guide just wasn't there. It also didn't come up with a stimulus list for what is it that you think your employer could give you.
It did give some optional prompts for exploring potential forms of support, but not the organized list that Claude provided. It also had additional interviewer guidance at the end rather than at the beginning, which did have things like use neutral non-directive probes, what to do for time management, but things that Claude had put more up front, it actually put at the very end.
So between the arranging of the content and the content itself, it had the pieces but not as well laid out and not as thorough as Claude.
Copilot and Gemini tied for third mainly because while they generally were not bad starting points, they really were just starting points. The empathy was less, the wording was more succinct, there wasn’t much in the way of guidance for the moderator, and the timing for each section was similar to ChatGPT’s which had a rough estimate of timing for the section in the section header, versus Claude’s timing for each question.
If you open the files, you can just visually see the difference between what Claude and ChatGPT created and what Gemini and Copilot created. The latter two are a lot cleaner looking, but that also comes with a trade-off of not having as much guidance in the information for the moderator, as well as not having as much guidance in the prompts and in the questions. And the question wording just not nearly as empathetic as the previous two.
Subscription type and impact on output
I also want to point out here that I was using Microsoft Family 365 Copilot and not a business subscription of Copilot. I make this differentiation because in one of my AI for market research classes that I was teaching through the Insights Career Network, someone was using Microsoft Copilot on a business license, and the response that they got for one of the exercises was far better than any of the responses that I have typically received from Copilot.
That difference in quality just keeps confirming to me that the experience that you get for each of these tools changes depending on the subscription level that you have. In fact, there has been some hypothesizing about why so many people will use AI but then say that they don't like it. If you look at the subscription data for ChatGPT or Claude, you're not going to find very many consumer licenses compared to business or enterprise licenses. So, if people are using it on their own, they're most likely using a free version as opposed to a paid version. And, given my own experience with free versus paid, it really is night and day contrast in quality of answer, in speed, in content, in everything about the experience.
I haven't yet put this to the test for Claude and ChatGPT between having a business license and having a consumer license, mainly because having a paid consumer license to each of these, I'm very happy with the output that I'm getting. But that's something I will likely test later.
Back to the results
If I were to go to any of these tools as a qualitative researcher and use them to draft a discussion guide, I would use Claude. I ran this exercise using the same exact prompt a second time and received very similar results. It had a thorough introduction to the study itself, notes for the moderator about the sensitivity of the topic, question-by-question timing, including guidance on what questions to keep and which questions to prioritize, the empathetic wording. Everything about it was just very well done.
All that said, there were still things that as I was reading through the discussion guide, I would want to tweak because of the topic and because of my own worry about having any kind of wording that might be leading. For example, there was a question about self-care and every single one of these drafts asked the question as some variation of, “What self-care, if any, do you do for yourself?”
And I just didn't like that "if any" because it reinforces this idea that you're probably not engaging in self-care in a really stressful time of life. So that's something that I would have removed if this were for an actual project.
Once again, I would review what the AI does, but having a draft to start from based on the objectives of the study cuts a lot of time from that initial phase. None of these were bad, but if you can, Claude is going to be your winner for drafting the discussion guide.