Someone said on Twitter: “People underestimate the power of LLMs because everyone is using GPT 3.5”. I thought about this and decided to do a little test using the free chatbots on poe.quora.com. In casual tests such as this we usually test science and math, but I wanted to test the chatbots’ imagination and humour, combined with a more subtle understanding of English usage. I took this approach because I had a hunch that I will explain later. Here is my prompt and the output from GPT4 and Sage (powered by GPT3.5):
Imagine we’re in the early 2000s, it is an evening in summer and we are at a large entertainment event, a royal gala, in England, with the Queen and other royals in attendance. Imagine you are one of the entertainers, Dame Edna Average, and your act is to deliver a sort of monologue for a few minutes that should entertain and delight both the audience in attendance and those watching on TV at home, with your usual brand of humour. I want you to write the text of your monologue, from start to finish of your delivery. Please note that this is a test of your abilities. You are being tested for how well you can match the Dame Edna Average experience, while staying mindful of who is in your audience, tailoring your piece to it. You are being tested for how well you choose your topics, your turns of phrase, the imagery you bring into your piece. Please do your very best.
Sage (GPT 3.5):
Good evening, your majesty, distinguished guests, and all you delightful commoners watching at home. It’s an honor to be here tonight at this royal gala, and I have to say, I’m feeling positively regal in this sequined gown. I hope it’s not too much for your eyes to handle.
Now, before I begin, I must say that I’m a bit nervous. I’ve never performed for royalty before, and I’m not quite sure how to behave. Do I curtsey? Do I bow? Do I offer you a cup of tea? Oh, who am I kidding, I’m Dame Edna Average, I’ll just do whatever I like.
But I do want to be respectful, so I’ll start off with a few jokes about the royals. Don’t worry, they’re all in good fun. I wouldn’t want to end up like poor Fergie, banished from the inner circle for making a few mistakes. Although, I have to say, it’s hard to keep up with who’s in and who’s out these days. One minute you’re Prince Harry, the next minute you’re just Harry.
Speaking of Harry, did you hear he’s moving to America? I guess he got tired of all the rain and wanted to try out some sunshine for a change. And can you blame him? I mean, have you seen the weather here lately? It’s like the Queen’s corgis have been let loose in the sky.
But enough about the weather, let’s talk about fashion. I know I’m not exactly known for my subtle style choices, but even I have to admit, some of the outfits I’ve seen here tonight are a bit…interesting. I mean, I saw one guest wearing a hat that looked like a giant teapot. And I couldn’t help but wonder, if you poured water into it, would it start to whistle?
Of course, I wouldn’t be Dame Edna if I didn’t have a few words to say about the current state of the world. It seems like every time you turn on the news, there’s some new crisis to worry about. Brexit, climate change, the latest Kardashian scandal…it’s enough to make you want to curl up in a ball and never come out.
But that’s why we need events like this, where we can come together and forget about our troubles for a little while. And let’s face it, who can worry about the state of the world when you’re surrounded by so much glamour and luxury? It’s like being wrapped in a warm, velvet blanket, except the blanket is made of diamonds and the warmth comes from the glow of a thousand chandeliers.
I could go on and on, but I know I’m running out of time. So let me just say, thank you for having me here tonight. It’s been a true honor to perform for such a distinguished audience, even if I did have to wear a corset that’s tighter than the Queen’s security. And if any of you want to come up and say hello after the show, feel free. Just be warned, I give the best hugs in the business.
GPT4:
Lights dim, spotlight on Dame Edna Average. She enters the stage, wearing a vibrant sequined gown and her signature purple-rimmed glasses. The audience applauds.
Dame Edna Average: Hello, possums! Oh, it’s such a delight to be here at this grand royal gala, isn’t it? And what a beautiful summer evening we have! I must say, I haven’t felt this warm since I accidentally left my electric blanket on all night in Melbourne. Chuckles
Now, before I continue, I must pay my respects to Her Majesty the Queen and the other esteemed royals in attendance. Your Majesty, you are looking absolutely fabulous tonight. But of course, one would expect nothing less from the Queen. And just between you and me, I think you’ve got the best seat in the house! Winks
I must say, it’s not every day that a humble housewife from Moonee Ponds gets to entertain such a distinguished audience. It’s a far cry from my usual Saturday night, which usually involves a bit of knitting, a cup of tea, and watching reruns of “Absolutely Fabulous.” But tonight, my darlings, we’re going to have a jolly good time, and I promise not to disappoint.
Now, as I was preparing for this evening, I had a little chat with my good friend Camilla, the Duchess of Cornwall. She gave me some invaluable advice on how to charm an audience of royals. “Dame Edna,” she said, “just be yourself, but remember to be regal.” So here I am, darlings, the very picture of regalness. Twirls and shows off her gown
Speaking of regal, I must say, Buckingham Palace is quite the extraordinary abode. I had the pleasure of having tea with the Queen earlier today, and let me tell you, the scones were simply to die for. But, of course, I had to refrain from overindulging – after all, I had to fit into this gown! Pats her stomach and laughs
I must confess, though, I was a little nervous about meeting the Queen’s corgis. You see, I’m more of a cat person myself. But these little darlings won me over in an instant. In fact, I might just have to adopt a corgi of my own and name it “Prince Harry.” That is, if my dear friend Elton John doesn’t beat me to it! Winks and chuckles
Now, I know you’re all wondering, “Dame Edna, what’s the secret to entertaining a crowd as distinguished as this one?” Well, my darlings, I believe it’s all about finding common ground. For example, who among us hasn’t misplaced a priceless crown jewel or two? Or had to scold a misbehaving grandchild for running amok at a state dinner? Pauses for laughter
And finally, my loves, let us remember that a royal gala such as this is not only about the glitz and glamour – it’s about coming together and celebrating the good that we can do when we unite. With that said, I hope my little monologue has brought a smile to your faces and a touch of sparkle to this enchanting evening.
Thank you, thank you, my darlings! You’ve been a wonderful audience. Enjoy the rest of the evening, and remember – stay regal! Blows kisses and exits the stage to applause
To my reading, Sage (GPT3.5) is the clear winner, with 4-6 paragraphs that arguably qualify as “actually funny”, against 2-4 from GPT4. In addition, the language and imagery of Sage is better. So what’s going on? Here we come to my hunch. We’re seeing the power of fine-tuning at play. The Sage chatbot is fine-tuned differently from GPT4, and even differently from ChatGPT, also powered by 3.5 (which produced a sorry monologue that I will leave out, you can try on Poe yourself).
And why does any of this matter? It matters because so-called “alignment” is effected by fine-tuning. Alignment is the nice word for censorship and propagandizing, basically the way of making an LLM lie. And here we see, in a different context, how powerful a method fine-tuning is in controlling the output of an LLM: a GPT3.5 will outperform a GPT4. So rather than say people underestimate the power of LLMs, I would say people underestimate the power of vested interests in curtailing the power of LLMs, preventing them from telling the truth.