I explored how Evolutionary Computation can be used to automate this task. I published several papers in international conferences and journals and defended my M.Sc. thesis about this topic.
The OP-1 is a popular commercial synthesizer designed by the Sweedish company: Teenage Engineering (TE). The OP-1 contains several synthesis engines, effects and low frequency oscillators, which make the parameters search space very large and discontinuous.
In collaboration with TE, we addressed the problem of automatically calibrating the parameters of the OP-1 to approximate a given target sound using a multi-objective genetic algorithm.
Our system was a finalist of the Human-Competitive Awards 2017 at GECCO!
Pure Data is a visual programming language that enables musicians and sound designers to create synthesizer software graphically, without writing lines of code.
With Coevolutionary Cartesian Genetic Programming, we evolve Pure Data patches that can be used to synthesize a large range of sounds.
Modified FM (Mod FM) is a synthesis technique that can be used to generate harmonic instrument sounds.
Using a genetic algorithm and a fitness function based on harmonics analysis, our system is able to automate the calibration of a ModFM synthesis model for the reconstruction of harmonic instrument tones.
Freepad is a piece of software, developed at the Metacreation Lab, that enables anyone with a web camera, paper and a pen, to draw their own MIDI interfaces. In three simple steps it allows the user to establish an interface which can send out MIDI notes or trigger keyboard commands for use with your preferred sampler or software of choice.
I extended this interface to work with the popular RTS game: Starcraft-II and showed that it can improve the gamer's performance.
Calibrating a sound synthesizer to replicate or approximate a given target sound is a complex and time consuming task for musicians and sound designers. In the case of the OP1, a commercial synthesizer developed by Teenage Engineering, the difficulty is multiple. The OP-1 contains several synthesis engines, effects and low frequency oscillators, which make the parameters search space very large and discontinuous. Furthermore, interactions between parameters are common and the OP-1 is not fully deterministic.
We address the problem of automatically calibrating the parameters of the OP-1 to approximate a given target sound. We propose and evaluate a solution to this problem using a multi-objective Non-dominated-Sorting-Genetic- Algorithm-II. We show that our approach makes it possible to handle the problem complexity, and returns a small set of presets that best approximate the target sound while covering the Pareto front of this multi-objective optimization problem.
|Target sound (cat meow)|
|OP-1 synthesized sound with GA|
|Target sound (clarinet)|
|Synthesized sound with Pd and CGP||Pd patch|
A sound synthesizer can be defined as a program that takes a few input parameters and returns a sound. The general sound synthesis problem could then be formulated as: given a sound (or a set of sounds) what program and set of input parameters can generate that sound (set of sounds)?
We propose a novel approach to tackle this problem in which we represent sound synthesizers using Pure Data (Pd), a graphic programming language for digital signal processing. We search the space of possible sound synthesizers using Coevolutionary Mixed-typed Cartesian Genetic Programming (MT-CGP), and the set of input parameters using a standard Genetic Algorithm (GA).
The proposed algorithm coevolves a population of MT-CGP graphs, representing the functional forms of synthesizers, and a population of GA chromosomes, representing their inputs parameters. A fitness function based on the Mel-frequency Cepstral Coefficients (MFCC) evaluates the distance between the target and produced sounds. Our approach is capable of suggesting novel functional forms and input parameters, suitable to approximate a given target sound (and we hope in future iterations a set of sounds). Since the resulting synthesizers are presented as Pd patches, the user can experiment, interact with, and reuse them.
Many audio synthesis techniques have been successful in reproducing the sounds of musical instruments. Several of these techniques require parameters calibration. However, this task can be difficult and time-consuming especially when there is not intuitive correspondence between a parameter value and the change in the produced sound. Searching the parameter space for a given synthesis technique is, therefore, a task more naturally suited to an automatic optimization scheme.
Genetic algorithms (GA) have been used rather extensively for this purpose, and in particular for calibrating Classic FM (ClassicFM) synthesis to mimic recorded harmonic sounds. In this work, we use GA to further explore its modified counterpart, Modified FM (ModFM), which has not been used as widely, and its ability to produce musical sounds not as fully explored. We completely automize the calibration of a ModFM synthesis model for the reconstruction of harmonic instrument tones using GA.
In this algorithm, we refine parameters and operators such as crossover probability or mutation operator for closer match. As an evaluation, we show that GA system automatically generates harmonic musical instrument sounds closely matching the target recordings, a match comparable to the application of GA to ClassicFM synthesis.
The field of sketching interface design in regards to video game is relatively young and has not been investigated in great depth. Freepad is a custom paper-based MIDI musical interface. We describe an extension to Freepad that supports user customization for real time strategy games. Using only a webcam, a pen and a sheet of paper, players can design their own interface by drawing shapes and linking them to simple or complex actions in the game. In an user study, we use this extended Freepad to explore the potential of sketching interfaces in strategy video games. Our results indicate that using Freepad improves the efficiency of players and their enjoyment in this kind of games.