Man, I sure slept alot.
But I digress. Back to failure. What I aim to do in this post is to take a look at how when the PCs fail in your campaign, it's something that can not only drive the game forward, but it can make for more memorable moments in your game. Knowing that they can suffer consequences for failure, but that it won't shut down what they can do, helps to establish the game world as more real for the players.
Before I go much further, however, I am reminded that we are talking about driving the game forward, which means we must avoid things that stop it. So let's take a look at some things to avoid:
The PCs encounter a mystic riddle which they must solve in order to get through to the next room where the guardian of the Staff of Doom awaits to give them their final test. Now say you've come up with a cunning riddle or puzzle and the players just don't have a clue... say you even made it a skill check (or challenge) and that the PCs just don't seem to have the proper skills. So they stare frustratingly at the wall, and get more and more agitated as you keep prompting them to try again. That's boring, agonizing, and nobody likes trying the same thing over and over again (I'll talk about how this applies to skill challenges later.) So what can you do?
It's all about the consequences. Sure, they may not be able to puzzle their way through the mystic ward, but even if they can't do that, they've got to get through to the next room. So instead of making the consequence of failure "you don't get past" why not have it be something else, like, "monsters attack" or "you take a penalty to certain skill checks with the guardian."
This makes it less about whether or not the characters succeed, but more about how much it costs them to do so. This is where the majority of the tension stems from when we're engaged in any other kind of story. We know the hero of the book (probably) won't die, and that she'll ultimately succeed, right? But what keeps us turning the page is finding out how she will use her skills and abilities to ultimately triumph and what it will cost her to do so.
Let's take another example: The PCs are negotiating a peace accord, so that they can fight a greater foe. Now, here if they fail, one consequence could be "peace isn't achieved" which means that the battle with the greater foe can still happen, it's just harder as the PCs don't have the allies they need. Another thing you could do, however, is increase the cost of peace. Perhaps the PCs must end the negotiation in favor of one of the opposing sides, when they'd really just want a balanced or even oppositely weighted decision.
Or suppose they are trying to pick the lock on a door, but fail... they can still get through the door, but they might open it just as guards are coming around the corner.
This option works in combat as well. Suppose the PCs are out to try and stop an evil horde of goblins, and in an encounter with some particularly powerful goblin casters, they end up having to retreat. You can have the goblins become a little more empowered, making for slightly harder fights in the next few encounters, or even create an "against the horde" encounter as they rally behind their mighty warcasters.
The possibilities are limitless, but remember... ultimately, what keeps us going in these things is to find out how the heroes win.