I found 999: Nine Hours, Nine Persons, Nine Doors to be very unsettling. I played it in bed at night with headphones on and it totally sucked me in. I guess this is a different type of horror to many of the games suggested here, which I personally don’t find scary.
The first game is much creepier than the second, I think due to a combination of the character designs, the writing and the general plot. The second game feels more akin to Danganronpa, in that the characters and setting are a bit surreal. Because it was a 3DS game, it also uses cartoony 3D models that make everything a bit lighter and less gritty than the original game. I haven’t played the third one yet (still need to get around to 100% completing the second game).