MSc Final Project Weblog 8: Iterative Audio and VFX Design

<16/10/21 - 31/10/21>

Audio Asset Creation & Iterative Implementation

Started this process by creating a series of themed drum hits for the clean & glitch technology themed biomes, these were originally grouped into kick, snare, hats, miscellaneous percussion, impacts and melodic hits. However, after experimenting with the implementation within the sequencer setup in Unity I decided that the impacts (which contained significantly more baked-in complexity) were a truer representation of my vision of the final project and contributed to a more cinematic feel, which in turn help steer the project away from feeling like a feature-lite video game environment.

Also began creating themed abstract ambiences, using a combination of recorded and synthesised audio material - mostly using the soft-synth Omnisphere - layered and processed in Ableton Live. For the purposes of this project I need the files to seamlessly loop, which I do via some cross-fading magic in Adobe Audition (possible in any DAW - chosen for its batch file export convenience).

I wanted to go quite deep on the sound design for the project, to try create abstract psuedo-musical sound effects (typical of my audio work as Sub Denizen) with a thematic connection to their assigned biome. This is partly an iterative process with audio editing and effects, and partly a creative process in the generation/discovery of suitable source material. As with all my media productions, sonic source material for audio is highly varied and follows a 'whatever works' approach that in creative/technical terms values results over method.

For this project (so far) the following have been used:

1. Synthesis - digital (subtractive, frequency modulation, granular) and analogue (some Eurorack output, home made electronic).

2. Foley - Real world recorded audio samples, edited, layered and manipulated.

3. Raw Data Sonification - Adobe Audition provides the ability to import any file as an audio file, generated some interesting results with a PDF and point cloud sound files, in combination with creative use of convolution reverb and impulse responses.

Using raw data from a PDF as sound design material in Adobe Audition.

Textures and VFX Graph - Useful for FFT and Mesh Data

Continued experimenting with VFX graph, following multiple tutorials and examining the work of Keijiro Takahashi (the same person who created the point cloud importer package for Unity). It was through this that I discovered a feature of said package that I was previously unaware of: the ability to import the point clouds as 2D textures (specifically design for use with VFX graph). Found a few of Keijiro's examples that showed me how to set up the input in the graph editor. I then began experimenting with audio-visual interaction of the VFX graph/point cloud creations I have been making. I did this by converting the FFT output array into a 2D texture.

 private void ProccessFftTexture()
    {
        audioSource.GetSpectrumData(spectrum, 0, FFTWindow);

        if (isTexture)
        {
            Texture2D tex = new Texture2D(spectrum.Length, 1, TextureFormat.RFloat, false);

            tex.SetPixelData<float>(spectrum, 0);
           
            tex.Apply();
            vfx.SetTexture(Shader.PropertyToID("FFT"), tex);
        }

        

    }

   

Currently the issue is using the texture data to create smooth animation, current attempts have been quite jittery - the iteration continues...