Everybody loves a surprise. Our movies, TV shows, books and other media are constantly pulling rabbits out of hats and revealing that the truth is something other than what we believed. But some twists are better than others — and here's the crucial difference.
These days, we hear a lot about the importance of preserving the sanctity of the Big Surprise — spoilers are destroying pop culture, the argument goes, because they are ruining the experience of discovering a story the way the storyteller wants to tell it. (Even though the average movie trailer gives away more than a dozen spoiler articles.)
But the flipside is also true — storytellers rely too much on endless rabbit holes. There are too many pointless mysteries, with no good answers, and too many people who see the "endless mystification" model of storytelling as the best thing to aspire to. And the truth is, a lot of twists in today's pop culture feel cheap and pointless.
So what's the difference between a cruddy twist and a good twist? It's pretty simple:
A meh twist pulls the rug out from under the audience. But a good twist pulls the rug out from under the characters.
The first kind of twist is just pulling one over on the viewers, or readers. The second is creating a situation where the characters themselves believe that something is the case, and then find out they were wrong.
The latter kind of twist is always going to have more emotional impact, because it actually affects the characters in a visceral way. We care about people who make mistakes, and we root for people who screw up for the best possible reasons. We also feel a surprise a lot more when it's just as much a surprise to the characters as it is to us.
And a really great twist, when a character realizes he or she was wrong about everything, provides a great turning point. Think about the best surprises in fiction (Ned Stark's fate in Game of Thrones, Blake realizing he's sacrificed everything to get to an empty room in Blake's 7, "Pressure Point") and the thing they have in common is the characters having a dramatic realization.
Characters are harder to misdirect than audiences
But setting up a situation where the characters are drastically wrong about what's going on is much harder than just misleading an audience.
Unless the story is seriously capsizing, the audience only knows what the storyteller lets them know. The writer, director or editor controls what the audience sees. You can easily use clever editing or random tricks to keep the audience from finding out a crucial piece of info until you're ready to reveal it. (Joss Whedon has leaned on the "leave out the most important part of a conversation, only to show it in a flashback later" device more than once.) It's trivial to keep the audience in the dark about all sorts of things, just by not showing them.
(Update: Another example of this is the sort of thing Lost used to do all the time, tricking the audience into thinking it's the past when it's the future, or someone is off the island when they're actually on the island, etc. etc.)
But it's a different matter entirely to keep characters charging in the wrong direction. If you don't do it right, then people start to wonder if your heroes are brain damaged. Are they on drugs? Why don't they ask the obvious questions or look behind them? When they ask a direct question and don't get a straight answer, why don't they keep probing?
If a character remains in the dark about something because they're too dumb to tie their own shoes, then we're likely to stop rooting for them. And conversely, there's nothing more compelling than a smart, competent character who nevertheless gets it completely wrong. (Or as the Doctor says in the latest Doctor Who episode, "You know what I hate about the obvious? Missing it!")
It almost doesn't matter if the audience sees the twist coming
If I had to choose between a situation where the audience is surprised, versus one where smart characters are surprised, I'd choose the latter every time. There's actually something great about a situation where the audience knows that the protagonist or major character is completely on the wrong track, and can see the hammer getting ready to fall.
Obviously, the very best twists do both — surprise the audience as well as the main characters — but in terms of raw storytelling power, there's nothing more potent than seeing a beloved (or hated) character suddenly thrown off track.
It all comes down to who the creators of our mass entertainments think they're in dialogue with, and what sort of fantasies they think they're giving us. Are they just talking to the audience and using the characters as props? Are they stage magicians, pulling cards whose faces happen to be people in situations? Or are they really trying to tell a story about real-seeming people, who happen to be in a world where all sorts of crazy can happen?
Inevitably, bad twists are cheap because they cheapen the characters. Good twists, meanwhile, make the characters that much more irresistible.