16/07/21 - 24/07/21
Project Summary
The nature of my intended final project is multi-faceted, but overall amounts to a practical exploration and culmination of several areas of interest: 3 dimensional (3D) interactive exploratory experiences (as opposed to a video game in the traditional sense of the term), creative/abstract applications of machine learning (ML), and data sonification using artificial life algorithms.
It will build conceptually on 2 previous works of mine (Nodus Tollens (NT ) & Prion), as well as all my previous artistic projects using the Unity game engine (e.g Simulacra & The Hill That Bleeds). Furthermore, it will venture into the realm of creative machine learning, in particular the concept of abstract datasets inspired by the works of Sofia Crespo et al. The final project will be an interactive 3D environment, built in Unity, using point clouds as a visual medium (inspired by Kathrin Hunze's Singularis; also see this blog post), created with generative machine learning techniques and sonified with granular synthesis algorithms.
Nodus Tollens (2020) demonstrative of the point cloud aesthetic I wish to develop and complement with machine learning, sonification and additional animation.
Research into Generative AI Point Clouds
Expanding on the research for the project proposal, the following are neural networks that have potential use in constructing a generative neural network that can output point clouds:
Although I will explore multiple avenues when trying to solve this problem, my initial instinct points me towards a combination of the 3DGAN and PointNet. A Python notebook pertaining to the latter features a function for random sampling of meshes to form point clouds, and thus seems logical to convert outputs of 3DGAN than try to make it function with a point cloud input, as this increases the availability of datasets (that do not require additional processing).
I have also been considering dataset ideas, such as training on a large batch of Mixamo characters, 50% humans, 50% monsters (all humanoid), the outputs of which could be easy to rig and animate, adding model movement that could be coupled with animation of point clouds. Natural environments will also be an important factor as my plan is to continue developing the procedural generation algorithm I started with NT, to this end I am interested in combining trees and bushes with industrial and archaeological items in my proposed training set experiments.
Furthermore, as an aside and a reminder to myself: As I concluded the writing of this post I discovered this useful GitHub which forms a sort of 3D machine learning wiki. At the time of writing I have not been able to explore fully, but this has a good chance of highlighting more potential approaches to solving my particular ML problem.
Considerations on Unity Implementation & Project Aesthetics
Given often lengthy processes of machine learning, I intend to start developing and testing audio-visual systems for use in the project alongside the ML work so as to ensure to have a contextually relevant environment in which to test the generative output. Furthermore, this is also an important step in case the ML process ends in failure; by getting a head start on the implementation it allows me to focus more on aesthetic polish and bug fixing, rather than starting construction of the audio-visual systems
To this end, I purchased some 3rd party assets potentially of use to this project in a recent Unity asset store sale:
Shapes (a vector drawing library with high quality lines that will hopefully let me port the line drawing aspect of Prion over to these new 3D point swarms, note to self: if done with moving line attractor instead of from every point and could add movement to static point clouds at low computational cost)
FM Points (a commercial point cloud importer for Unity, to compare with PCX which was used for NT; has some built mesh effects to study, learn from and expand such as a wind distortion effect. From preliminary tests can see that mesh effects are being applied via the shader rather than using compute shaders, as I had previously attempted, adopting this approach myself does not mean abandoning compute shaders however, as they will be useful for calculating flock movements )
Volumetric Fog & Mist (this one also caught my eye as a way to add movement to static elements of the scene; creative colour uses could also be interesting; basic Unity fog was important to aesthetic of NT and useful in obfuscating the loading of the procedural tiles, having a more powerful fog tool for this project can only be a good thing).