Horror has many themes and spawns many emotions which are generally shades of fear and disgust. The trouble with fear is that one of humanity's greatest fears is that of loss. Loss of loved ones. Loss of status. Loss of control. Loss of self. If the loss actually occurs, than fear can become grief or anger. It is when the loss is anticipated, but yet to occur, and when the threatened object or desire is in jeopardy, that fear is at its strongest point. The greater the risk of the loss, without its guarantee, the better, and that risk is automatically stronger for vulnerable characters who lack the resources to muster against it.
The difficulty is that players generally want to be more awesome than they are in real life. They prefer to play someone who is interesting more often than they want to play someone vulnerable. Awesome people either have nothing left to lose, care not for your simple quandaries of loss, or are simply capable of beating back the jeopardy and winning without much more than a frown and some focused might.
Unless you’re lucky enough to have players who deeply desire a chance to identify with the trembling protagonist from REC rather than Jackie Chan, you’re also likely to have to deal with – at the very least – providing a supernatural or otherwise heroic template / class to keep the players interested.
• Sit down and discuss with the players what horror is, what horror isn't, and try to get them all on the same page with character creation. Pretty much any class / race / concept can be tweaked to fit a horror campaign, from a fearless paladin who underlines the fear of others and the folly of ignoring one's terror, to an arrogant vampire who doesn't see what they have to lose until they lose it. Still, it is harder to mesh certain concepts so it's best to bring it all out into the open before players get their hearts set on that wise-cracking, sleazy P.I. who refuses to take anything seriously (which could work, by the way, if the player is happy to watch their P.I. slowly lose their cool and become as scared as the rest of them).
• Even if their character is larger-than-life, help the player find those identifiable bits to connect with their own character. Sitting down and discussing the character's history can often help with this. Try to get into the character's head. Ask several questions, particularly focusing on motivations and drives. See if you can work out with the player where certain personality traits have come from. Do bear in mind that some absolutely fantastic roleplayers can't describe their character's psychology, especially before the game begins, so this might be something you delay doing until the game has been underway, or in the event that a character doesn't make sense to you.
• Occasionally humble the character and increase the threat level by forcing the protagonist into a situation where their powers don’t work or their technological marvels fails. Don’t overdo it, though, and do sign post it quite vividly. If they’re playing, say, Mage: the Awakening, part of the gaming contract was the ability to use Arcana. If there's a session or two where they can't then that can be a unique and interesting experience so long as you define it as such. Think of television shows (Buffy, X-Files, and Supernaturals) have weakened the protagonists during certain episodes to reduce their options. Those episodes are cool because the plots are vivid, interesting, and with a visible end point.
• Focus on questions the characters can’t answer through the use of their powers to encourage players to use mundane and identifiable means rather than relying on their super powers.
• Force the players into a reactive mode when they investigate too deeply into the unknown forces so that they understand that the enemies aren't just going to wait for them.
• Ensure rash behavior sparks off serious consequences, perhaps not to them but to others. If their brusque treatment of several school employees during an investigation into the school sets off the monster on a murderous rampage early, killing off a few students in the meantime, then they're more likely to step gingerly the next time.
• Be up front with the players. They may take it personally if even their allegedly heroic and larger-than-life protagonist is getting stomped on. Let them know there’s going to be more difficult genre conventions in play.
• Reward them for being ordinary. If someone’s playing a typist and another is a military commando, make sure the typist gets plenty of opportunities to shine. Otherwise, they might as well make a commando the following day!
• Focus on the little details and personal plots such as the land lord who they’re helping through a difficult divorce. It’ll hopefully encourage them to see the world through humbler, more human eyes, and give them another attachment as well.