This week we are continuing our expansion on the cookie survey, but this time taking an n=1000 data file and putting it into all four major models and asking them all to draft a set of slides for the report. As usual, here is the prompt and here are the models that were tested side by side.
Create a report with the findings from the attached data. Highlight the cookie favored most by all ages, then the cookie favored by each age bracket. The report should have an executive summary and a recommendation for a new bakery owner on the opening day cookie menu. Use the attached template for the report.
Model Tested | Grade Earned |
|---|---|
F | |
C+ | |
C+ | |
Gemini Pro the standalone version | F- - - |
B | |
B+ | |
A- |
The Results
Microsoft Copilot used 2 ways
There are 2 ways that you can use Copilot for building slides. The first way is to use the Copilot desktop app, in a similar way that you would use ChatGPT or Claude. You open a chat, you upload your data to be analyzed, and then you ask it to create slides. The second is to use Copilot inside of PowerPoint. In this case, you follow these steps:
Sign in to your Microsoft 365 account (you need to be on a subscription that includes having co-pilot within each of the Office apps)
Open Powerpoint
Create a new presentation, either completely blank or from a template
In the Copilot chat on the right side panel, upload the data to be analyzed and enter your prompt.
You might need to tell it to use the template you’ve opened after it’s analyzed the data (at least, I needed to when it asked what style the slides should take).
The Recommendation from Copilot desktop app
Chocolate chip leads, with 30.6% of the votes.
Opening day: “Lead with Chocolate Chip, supported by several complementary flavors and one rotating seasonal offering.”
The first approach through the desktop app, I would give an F. It was absolutely terrible. It analyzed the data and gave me the results in a report in Word, but it didn't create slides despite the prompt telling it to create slides.
When I told it to create slides, I didn't specify to use the template that I had uploaded, which is my fault. So, the slides that were created were absolutely bare bones. It looked like someone took the bullet points from the chat that summarized the findings, pasted them into separate slides, and called it a day. From start to finish, the evidence was really light, the story was non-existent, and the recommendation for what to do on opening day was ridiculously vague except for “make sure chocolate chip is on the menu.”
The Recommendation from Copilot in PPT
Chocolate chip leads with 30.6% of the votes.
Opening day: Start with five: chocolate chip, double chocolate, oatmeal raisin, shortbread, and sugar.
• Allocate roughly 40% of opening inventory to chocolate chip; split the balance across support flavors.
• Target soft/chewy or crisp-edge texture—the preference of 55.2% of respondents.
• Track sell-through by SKU and rebalance batch sizes on day two.
Method note: synthetic n=1,000; no real respondents. Treat findings as directional and validate with local sales.
The second approach using Copilot inside of PowerPoint was much better. However, it followed the same approach as using the Copilot desktop app, which is to say that it analyzed the data first and paid no attention to the fact that I had told it in the prompt that I wanted it to use the template that I had provided.
Instead, it analyzed the data and then asked me what style did I want to have the slides in and offered me a few different styles. I pointed it to the template that I had actually opened in addition to the template that I had uploaded and it actually used the template this time. The result was eight slides that met the prompt objective, but it did seem to have some struggles with the formatting of the text within those slides. As you'll see in the output, one of the slides with bullets has double bullets because it seems like it couldn't figure out how to do single bullets. And in the slides that have three text boxes assigned for splitting text up, it didn't do a good job with having a headline and then having text and separating what was headline versus body text within those boxes.
For the Copilot in ppt version, I’d give it a C+. The recommendation was actually pretty well-constructed, but the design was shoddy.
Gemini
As it turns out, Gemini, like Copilot, has two ways that it can be used to create slides. The first is opening Google Slides and then using Gemini within Google Slides. The second is to use Gemini outside of Google Slides on its own as the app.
The Recommendation from Gemini in Google Slides
Chocolate chip is the leader.
Opening day: offer chocolate chip, double chocolate fudge, and cinnamon sugar snickerdoodle, then expand to oatmeal raisin and buttery shortbread for the 35+ crowd, and bake all core cookies “to achieve a soft and chewy center with crisp edges.”
One call out: there are no percentages given. In Copilot, there was a table showing the selection rate of each cookie was “favorite cookie.” All you get in this version of the report is a summary of the findings, without numbers to back up any of the findings. That lack of numbers gives this a C+.
The Recommendation from Gemini Pro standalone
42% said they loved chocolate chip.
The under-18 crowd who was not included in the dataset chose Frosted Sugar as their favorite (a type not included in the dataset, either).
Opening day: anchor with chocolate chip, include colorful frosted sugar for the youth, oatmeal raisin for the niche play.
I don't know what dataset Gemini used for this, but it's not the one that I uploaded. First of all, the percentage of those who said they love chocolate chip is cited as 42% rather than 30.6%. It included the under 18 crowd who was not represented in the survey, It also listed Frosted Sugar Cookie as an option in the study, which only appeared in the open-end responses. And when I told it to apply the template that I had given it, it just created its own thing with kind of sort of elements of the template applied.
If there was a grade lower than an F, this report would get it.
ChatGPT
There is only one way to have done this with ChatGPT and that is via the chat. Currently, there is no plugin that I know of to use ChatGPT inside Powerpoint.
This report was developed in one shot. I uploaded the files. I gave it the prompt. It worked for 8 minutes and 31 seconds, analyzing the data and creating the report all in one go.
The Recommendation from Sol 5.6
Chocolate chip is the clear leader, with 30.6% of the vote.
Opening day: chocolate chip as the flagship, followed by oatmeal raisin, peanut butter, shortbread, and double-chocolate.
Overall, this presentation is a lot better, in both the way the template was used and in the analysis. That said, there are obvious design issues in this Powerpoint with some text overlapping other text, and some text being too small for the space that it's in. But overall, it did a pretty good job.
I’d give this one a B. It’s still going to require time and effort to go in and clean things up to be presentable.
Claude in 2 ways
And that brings us to Claude. There are two ways to use Claude to build this report. The first is in the desktop app, and the other is using the Claude plugin in Powerpoint. I did this both ways.
The Recommendations from Claude Desktop - Sonnet 5
I should say first that it took three phases of using Claude before the slides were finally built. Thankfully, I didn't have to re-prompt it each time, as it was able to continue the process, but I was surprised to see the fact that it was running into “maximum tool call requests”.
Chocolate chip is the leader with 30.6% of the votes.
Opening day: Start with chocolate chip, oatmeal raisin, shortbread, double chocolate, peanut butter, and sugar. Hold snickerdoodle and ginger molasses for rotation. “Tactical note. Younger customers buy more often, so keep Double Chocolate visible near the register. Older customers respond to nostalgia and coffee pairing, so place Oatmeal Raisin and Shortbread near the coffee station.”
Overall, this report was pretty good, but it did have a couple of design issues with text overflowing the space that I had. I give this a B+.
The Recommendations from Claude in Powerpoint
Similar to Copilot in PowerPoint, I opened a new PowerPoint presentation from the template that I wanted to use, then uploaded both the data file and the template, and included the prompt.
Chocolate chip leads with 30.6% of the vote.
Opening day: start with chocolate chip, oatmeal raisin, peanut butter, shortbread, and double chocolate. This recommendation came with a bar chart that showed how you could potentially increase the share of audience by adding these cookies all together, indicating these are the five cookies that are the favorite of 74% of buyers.
An interesting thing about this one: this is the only draft presentation that called out the source of the data on every slide. This footnote on every slide is a classic market research presentation best practice. I didn't specify this in the prompt; it added it by itself.
None of the other iterations from any of the other models, including Sonnet 5, had that. I tested to see if it was the model itself, Opus 5.5 (the default in Powerpoint) that was the differentiator since I had used Sonnet 5 in the desktop app. However, it turns out that wasn't the case. When I used Opus 5.5 instead of Sonnet 5 in the desktop app to run the same exact exercise, I ended up with almost the exact same presentation that Sonnet 5 had developed, with the same six-cookie list for opening day.
However, when I ran the same exercise with Claude in PowerPoint, it did the exact same thing with the source callouts at the bottom of each slide.
So, for Claude in Powerpoint, I’m giving it an A-. You’ll definitely still want to review, instead of just send this off to a customer, but the level of intensity needed is lower than any of the rest would require.
The Takeaway
After running this experiment as many times as I did across models and across approaches possible for creating slides in Powerpoint, I think the best tool to use is the Claude extension in PowerPoint. Granted, this was a fairly straightforward dataset. It didn’t involve driver's analyses or regression analyses. Those will be coming. But for something as straightforward as this, the Claude extension in PowerPoint was by far the best and would require the least amount of effort to clean up and tweak for your way of talking.
If you don't have access to this particular combination, then my recommendation would be to approach this in steps with the LLM. Break it down into:
Analyze the data (give the business objective it’s analyzing for).
Create the slides.
Apply the template look and feel.
When it comes to the actual recommendations, you'll see that each model seemed to have a different recommendation. And this is where the researcher’s expertise, knowledge of the business, and knowledge of the people you would be presenting this to come into play.
None of these LLMs know your customer or the business the way you would; that’s the difference you will add to the report and to the delivery. That’s the thing no LLM can replace.