Vision by sound? Creating a custom shader

So how do you do something like this?

“A narrative horror adventure featuring a young blind woman who must solve mysteries and escape a deadly presence, all without sight.”

“Perception is a first-person narrative horror adventure that tells the story of Cassie, a blind heroine who uses her extraordinary hearing and razor-sharp wits to unravel the mysteries of an abandoned estate that haunts her dreams.”

I have seen this implementation from a project of a university but seems that it advanced a lot more from then. The game itself seems pretty neat and if you are into this horror style with no weapon to protect yourself, I advise you to back it. As always the gears in my mind started turning. “How do you do such a thing?” . Perception is made with Unreal Engine but as I come from the Unity3D environment I will unravel my thinking with that toolset in mind.

Let’s start from the simple things. The wind is just a simple emmisive particle. With a gradient color from black to blueish and black again. It adds a lot to the aesthetic of the game and is pretty realistic too! I shiver by looking at it.

While the wind is blowing it’s supposed to make a sound, and “light up” the objects that it surrounds. Also seems that Cassie, our protagonist, has a cane that she can extend and hit to the ground to make a noise and “light” the objects around it. Now, that is a bit tricky.

My initial thought was a shader. Every material should be black and should hold the position of the sound that spawned. Then calculate the offset of the sound by measuring how much time has passed. Simple but how is it implemented? I’m a bit of a noob about shaders but the logic fits and this should work. I’m looking forward in hearing your feedback for optimisation.

Here is the sample project.

The scene includes a walled room, and a spaceship . There are 2 files which are useful here. First let’s take a look at the shader. The vertex function calculates the current position of the model and the world position and passes it to the fragment function.

v2f vert(float4 v:POSITION) {
    v2f o;
    o.pos = mul(UNITY_MATRIX_MVP, v);
    o.wp = mul(_Object2World, v);
    return o;

The fragment function calculates the distance of the pixel from the sound emit position. Then it calculates the weight of the two colors (default black and highlighted). I multiplied the sound time offset by 4 to make it faster. Then subtracted from the distance and take the absolute value to find how close the pixel is to the soundwave. Divided by 0.2 to make the soundwave thinner, and clamped the value because the numbers between 0 and 1 is what’s important. Then return the color of the pixel with the lerp function.

fixed4 frag(v2f i) : COLOR {
    float dist = distance(i.wp, _SoundPos);
    float weight = clamp(abs((dist - (_SoundTimeOffset * 4))) / 0.2, 0, 1);
    return lerp(_HighlightColor, _DefaultColor, weight);

Then, in the main.cs script which is attached to my character, I set some shader global variables. First of all the time offset when player has spawned a sound. That is in order to calculate the sound wave offset. And in the Update I save the sound spawn position and save the timestamp which it spawned to calculate later the offset.

There are lots of things one can do to improve it, like add a fade to the end so that the sound wave doesn’t go infinitely. I hope this was informative :)

Posted on 4 June, 2015, in Unity3D and tagged , . Bookmark the permalink. 2 Comments.

  1. Nice!

  2. I’ve been looking up how to do this everywhere and no one could seem to give a straight answer or an example and I’ve finally come across yours! but I was thinking on how I could get certain things to affect the shader the same way as pressing the ‘E key’.
    Here is what I was wondering:
    1. How would I get different audio sources to create that effect in the location those sounds come from (footsteps, something falling over across from the player, low volume wind etc…)
    2. How can I make sure the shader produce from where the player is standing, as if the sound has been hit on the floor below them without having to look down.
    3. How could I get the effect to fade the slightest bit quicker but also be able to produce another one without the previous one disappearing.
    4. And finally, how could I implement a microphone input to be able to trigger the very same effect (either using unity’s built in audio engine or Wwise).
    if you manage to see this soon, could you please get back to me as soon as possible

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: