Laughter is the best medicine for negative emotions. If you can make someone else laugh, you can probably defuse (or at least reduce) their anger, fear, or sadness. Cracking a joke can also alleviate your own negative feelings. So if your horror game tactics are working, you might be annoyed to find your players cracking jokes and making movie references to break the tension. Out-of-Character (OOC) jokes can ruin the immersion and obliterate tension so encourage such tension relief to occur In-Character. Humor provides increased attachment between protagonists (boosting the humanity of the situation) and thus can make later events more frightening as the losses mean more. That’s why the horror genre often still has the characters themselves grimly try to make light of the situation without losing the overall dramatic tension.
Players will naturally attempt to diagnose the enemy. This is both a reaction to fear (throw it in a category to gain power over it) and also a natural human reaction. Humans try to understand everything around them. Pigeon holes aren’t scary. Neither are known stat blocks. This is another reason why it’s best to keep the full on monster reveal to the last scenes.
Try to play up the ambiguity for a time. A werewolf in human form that crosses into the spirit world at a loci can appear to be a ghost. A vampire that only shows its super strength could really be anything at all. You can also turn expectations on their head. Use a vampire bloodline (sort of like a template to those D&D fans out there) so that this particular vampire can be found eating the still living like a zombie.
This trick also works in D&D and Pathfinder a treat. Spend time looking at the monsters in the Monster Manual and consider how you could describe it – especially piecemeal – in a way that leaves its true identity unknown.
If people make the right Knowledge check to identify it, give them information based on anecdotes but no name. Let them know that the lone survivor of a brave group of hunters was a steward who stabbed it to death with a silver letter opener so that they can still know how to defeat its damage reduction (and that it has one) without allowing the players to immediately figure out its stat blocks. This way their skills still come in handy, you enrich the story world’s history, build immersion AND retain a level of ambiguity about your monsters.
Play up the threat level of the enemy so that in a horror game the enemy is (almost) always a credible threat. The protagonists might be out-classed, out-gunned, or out-numbered so that they’ll think of fleeing the situation. Be aware that this will encourage them to think tactically or, perhaps, flee. If so, allow these options. If you keep railroading them into stand up fights with superior foes, you’ll breed frustration not fear.
Encourage lateral thinking. Each time they don’t resort to fisticuffs against an aberration is a time that you can build on the fear. This can be trickier in Pathfinder / D&D as mundane threats like collapsing ceilings often become more easily ignored at higher levels. Consider weighting it so that such creative solutions still have a chance to trap the monster despite its Reflex Save (particularly if you can justify it through the creature’s size versus the tunnel’s size, etc.)
Play up the desire to not get hurt. Descriptions can help with this. Make it brutal. Health levels are an arbitrary term that is hard for anyone to engage with, so instead give them a vivid description they can really latch onto and identify with, such as a broken arm. Then remember that injury afterwards. Even if there is no in-game effect, reinforcing the pain is often enough to encourage the players to back off from combat.
Certain threats might even toy with the characters or appear to be more dangerous than it actually are through property damage, ambushes, or simply stalking the characters in a way they can't prevent (such as through mirror reflections).
Twist things around. Yet not always. Sometimes the church should give them safe from the demon. Sometimes the ominous house should be haunted. But sometimes that werewolf might be there to save them while that poor, lost woman could be the slasher that wishes to end their lives.
Allow the Action Hero to get his jollies by throwing in environmental hazards that they must brave to flick that switch to drop that blast door on the enemy. Or allow the odd car chase or fisticuffs against regular, mundane foes so that everyone gets a chance to feel powerful on occasion. Just don’t let these events overshadow the Big Bads.
So, hopefully all of that will help you wrap your heads around how to scare your players through the monsters themselves. If you have any other advice to give (or if you just really like this article), include it in the comments below.