by: (Bent Bisballe Nyeng, deva, 20001467), (Jonas Suhr Christensen, suhr, 20032491), (Rasmus Berlin, batluder, 20030589)
Contents
Introduction
The main purpose of this project is to play around with various synthesis methods to create instruments. It has never been a goal to create instruments with the same sound as "real" instruments, but more an experience in what sounds different methods results in.
Playing instruments alone is no fun, so we needed some kind of way to use these instruments in combination, such as playing a midi file. For this we made a java application which read the information stored in a midi file and then plays it through a network connection on a host running PD. The neat thing about java is that handling midi files are relatively easy in java since everything is kind of handled by the framework, hence we can put our effort in doing the instruments.
The audio generation are done in PD which receives the information about what should be played from the java application. The PD mixer then routes the information and plays the instrument corresponding to the midi message. By doing this we have isolated the task for PD to mainly generate, mix and play audio, which is what PD are supposed to be good at.
We played different midi files over the system, but in the end we scaled it down to one midi file of which we could play all the instruments. It is of course possible to play other midi files as well, but then one will have to match the midi file instruments with the PD application. A recording of the midi file played with out application and instruments can be obtained by clicking on one of the following links:
output.mp3
output.ogg
output.flac
output.wav
The tools we have used in the project can be downloaded the following places:
The java midi file reader
To read and play a midi file we chose to use a simple java application
which sends the midi file information to PD through a network
connection. The java application consist of two parts; the midi file
player (PlayMidiFile.java) and a implementation of
javax.sound.midi.Receiver (NetReceiver.java). This means that the java
midi file player reads a midi file, creates a sequence, a sequencer
and a synth (with our Receiver implementation) and begins to play the
midi file. This code is basically the code example PlayMidiFile.java
from the first quarter of the course.
Besides setting up the midi system the midi file player also supports
on-line changing of the play tempo (Beats per minute). This is done with
the '+', '-' and '=' followed by an enter keystroke. '+' and '-' each
increase or decrease the default tempo stored in the midi file by 0.1
percent and '=' restores the tempo to the default.
The midi file player are started with the arguments
'playmidifile midi-file host port', where midi-file is the
midi file one wants to play and the host and port are the addressing
of the PD service of which the midi information should be send to.
The NetReceiver.java is given as the Receiver to the transmitter of the sequencer. This means that each time the sequencer receives a midi message, this message is going through our receiver. When constructing a NetReceiver one needs to give it a String representing the host of which it connects to and a port number, which is then used to open a tcp connection to the host. Whether to use a TCP connection or UDP is not critical but it would have been more proper to use a UDP since we process real time data. But we found it more straight forward to set up an TCP connection in java.
When an midi messages are received from the sequencer the method send() is used. Our implementation of the method only handles instances of javax.sound.midi.ShortMessage (since this is all we support on the PD side). If a message type of that instance are received, the information stored in the message are parsed into the four values 'track', 'node', velocity' and 'command'. The first tree values are taken directly from the midi message while the latter are either '0' or '1' depending on the midi command. If its a node off or node on its either '0' or '1' and if its anything else the midi message are discarded. Finally the four tuple are passed on to the netSend() method which writes the values as a string to the TCP connection. Each string (tuple) are required to end with a ';' by PD, so before returning netSend() appends a ';' to the stream.
PlayMidiFile.java
NetReceiver.java
The mixer
The mixer is the main pd patch, receiving the MIDI events from the java application, and filtering the separate tracks from one another, in order to send them to the correct instruments.
The mixer patch (main.pd)
The mixer consists of 3 subpatches: The input patch, the
channel and the instrument patches.
The input.pd patch is the one actually doing
the tcp network receiving. It is made out of the netreceive
patch that does most of the work for us. The network protocol itself
took a little reverse engineering, but turned out to be fairly
simple.
The rest of the patch is simply number output for the eye
candy, and a midi-node to frequency conversion (easier for us to handle
later on).
The input patch (input.pd)
The channel.pd patch is a two
pass patch. The first pass takes care of the splitting of the
instrument data, i.e. filters the instrument it is configured to
handle, and passes the data on to the output.
Second pass is an audio pass where the volume is controlled by the
fader, and the played output is shown in the VU-Meter.
It furthermore has a mute check box for convenience.
The channel patch (channel.pd)
The instruments are crafted with an input taking a tuple of the node
frequency, and the attack velocity between 0 and 1.
The instrument must furthermore have a mono audio output, which can be
connected back into the channel patch.
All the channels are connected to a single channel patch, called the master which ultimately controls the volume of the entire system, prior to sending it to dac~. We did not want to handle stereo signals, so we simply send the mono signal to both channels.
Polyphonic instruments
The synthetic instruments of our project are in general monophonic. But in music we often wants to play combinations of tones from an instrument at the same time to perform chords, hence we need to make the monophonic instruments polyphonic.
For this we have created a pattern which easily can be used on the various instruments of which we need polyphonic performance. The pattern uses the poly object from the PD library. The poly objects take as input a pair containing a tone and a velocity and outputs a triple which consist of the input pair and a instrument number where the instrument number are changed depending on the chord. The triple is then packed so that it can be routed by the instrument number into monophonic instances of the target instrument. Our polyphonic instruments are restricted to play four tones simultaneous but it can easily be changed by reconfiguring the poly object. Further on we use voice stealing which if a fifth tone are supposed to be played stops (steals) the first tone and plays the new fifth tone instead. The abstraction applied on the bell instrument (i_ding.pd) is showed in the figure.
The bell instrument (i_ding.pd) as polyphonic.
ADSR
Most instruments follow the same pattern in the volume over time as you hit a node or a chord. First it will very rapidly reach a peak volume, this is known as the attack. When at it's peak, it Will instantly start to decrease in volume, usually slower then the attack, but still quite fast, this is know as the decay. Many instruments can sustain a tone for quite some time, where the volume is almost constant, this is known as the sustain, and happens right after the decay. Lastly when the is somehow stopped, some instruments will take a while to reach silence, known as the release. As an example, a trumpet will have a very short, while a bell will have a very long release.
The adsr abstraction used in our instruments was stolen from the examples. It creates an attack-decay-sustain-release envelope, and was very useful to us in most of our instruments. It takes 5 arguments: Peak is the multiplier for the volume, attack is the attack time in ms, decay is the decay time in ms, sustain is the volume level of the sustain in percent of the peak, release is the release time in ms. The abstraction also has inlets for all these values, but more important, it has a trigger inlet, where a positive value triggers the attack decay, and a zero triggers the decay.
A pattern used in many of our instruments, can be seen in the right part of i_r-sub-synth.pd. Here the velocity is send only to the adsr if it is zero, meaning note off. But if it is another value, it is send to the adsr to trigger note on, to a multiplier to control the volume of the final sound, and to a bang that triggers a delay to send a note off in case no node off was ever send. This avoids tones hanging, if we stop the playback, and emulates that the instrument can not hold a tone forever.
Instruments
The OSC instrument
The osc instrument is our basis instrument, created as a sort of proof of concept instrument. It illustrates how the inlet and outlet are connected, and how to parse the tuple from the inlet.
The osc instrument with adsr (i_osc.pd)
The architecture is quite simple. The inlet contains a tipple like this (frequency, velocity), and the velocity is 0 to indicate a node off.
We first created the osc instrument using only the osc patch, but later on decided to introduce the adsr to remove the clips whenever changing node.
Sub-synth
This instrument was an attempt to use subtractive synthesis to generate a simple yet nice sounding instrument, using ring modulation and a low-pass filter.
The sub-synth instrument using ring modulation (i_r-sub-synth.pd)
We use a oscillator to generate the tone as a sine wave, and ring modulate it by multiplying it with a one octave lower saw wave generated by a phasor. We use the adsr, to give the instrument a ADSR envelope, as described earlier. Last it is filtered by a low-pass filter, set to filter out all tones over 320Hz, this makes the sound less noisy, and makes the low frequency artifacts more audible.
The instrument makes a nice noisy bass line, with it's low frequency artifacts, that gives it a deep sound. But it also makes a nice sound for chords in higher frequencies, when made into a polyphonic instrument by use of the poly pattern described earlier.
Flute
This instrument was done specifically for the song, that we had chosen to demo our project with. As we had not implemented pitch bend and other control signals in our interface, and still wanted to use the song, we had to come up with a special "western" flute instrument, that mimics the pitch bend in the song.
The "western" flute instrument with custom pitch bend (i_flute.pd)
The pitch bender reads it's values from an array. The values read, is multiplied by 0.833333 to make pitch bend a 5th up and down. The pitch bend is updated every 10ms, is sounds as this update is fast enough to make smooth sliding sound.
Rather than saving the values of the array into the patch, we chose to generate it at load time. It had to slide up and down twice in about 300ms, was what we found by trial and error. We calculate this by using a cos** to generate a smooth curve. But the tone after this sounded to flat, and we needed to make the tone fall slowly after the bends, to make it sound right. We did this last fall by a simple linear function using line**.
At first we generated the tone by a simple oscillator. But we also wanted to try using frequency modulation, and we tried to improve the sound by the means of this modulation technique. The examples in pd about frequency modulation, like E09.FM.spectrum.pd and E10.complex.FM.pd, really show phase modulation. We looked up how to do true frequency modulation, and used this technique. What it does in simply to modulate the frequency of our oscillator by a oscillator centered around a level of 1. We use 6 times the carrier frequency as our modulation frequency, meaning the frequency by which the carrier frequency is altered. And a modulation index of 0.2, meaning by what factor we will change the frequency up and down. We could have used a little more time fine tuning these parameters, for example we would probably have had a more drastic change in the sound by using a lower modulation frequency. And had we used envelopes to control these parameters, we could maybe have gotten a more realistic sound.
Like in the sub-synth patch we also use adsr** to create a volume envelope, and a low-pass filter to filter overtones.
The bell instrument
The bell instruments (i_ding.pd) is an effort to make an synthetic instruments which sounds like a common bell. It is based on the analysis of the sound of a bell found on the Internet (see the bottom of this section for a link). Basically the analysis states that a bell sound consists of a fundamental tone with multiple partials with different amplitude, tone and envelope. The amplitude is lower for the higher tones and the envelopes depends on whether its a high or low frequency tone. The attack of the envelope should be steady for all the tones, while the the decay phase are longer for the deeper frequencies. Therefore doing the lifetime of a sound of the bell converges to the deeper tones (ie. the lower frequencies becomes dominant over time).
The ding abstraction (i_ding.pd) with 10 different
oscillator divided in 3-4 groups with different amplitudes and
envelopes.
To comply the analysis we have used additive synthesis. This means
that we have a lot of oscillators generating at different frequencies
which in the end are added to each other. In our setup we have 10
oscillators, the first nine are specified from the fundamental tone as
described in the bell analysis article. The ratios can be seen in the
figure. The final oscillator is running at a very deep frequency and
was added in an attempt to give the bell a deeper sound. To simplify
the synthesis we have divided the different oscillators into three
groups with different envelopes. The oscillators running at the
highest frequencies are placed together, and the oscillator generating
the lowest frequencies are in a group, and finally the in between
oscillators are placed in a group. The envelope for the first group
have a faster decay and release time, while the group with the lowest
frequencies have a long decay and release time. The amplitude factor
also depends on the group and increases while going to the lower
frequency oscillators. The oscillator running at the lowest frequency
are multiplied by a factor one and a half amplitude. All the values
(both the amplitudes and the envelope values) are based on cutting the
value by half from group to group where the group containing the
lowest tones have the highest value.
We decided to use the abstraction as an instrument to play tunes, so
the originally fundamental frequency of 200 Hz from the article had to
be changed to the natural harmonic tone, which is done by neutralizing
the transformation of the fundamental tone from 440 Hz to 200 Hz (so
that it is 440 Hz). Finally as we wanted a deeper sound we moved all
tones an octave downward.
Drums
The drum kit patch is a wrapper patch that splits the incoming signal
and activates the various drum patches according to the incoming
node.
To split the nodes, they are first converted back into midi node numbers to
be able to easier sort them.
Our test song uses only a small subset of the drums so we have only
implemented the following four instruments: kick drum,
snare drum, hi-hat and some sort of crash cymbal/gong sound.
The drum split patch (drumkit.pd)
Kick drum
The kick drum is implemented using three oscillators and three line
patches. The line patches are controlling the oscillator frequencies
sweeping 60-80Hz in 100 milliseconds, 1000-0Hz in 15 milliseconds and
4000-3000Hz in 80 milliseconds. The first sweep are used to produce the
low frequency of the skin, which is rather low tuned. The second sweep
is used to introduce the resonances produces by the skin being not
exactly in tuning. The final high frequency sweep is used to emulate
the attack of the mallet on the skin.
Ultimately we do a adsr filter controlled by the velocity from the
input, and finally a low pass filter to make it all sound just right.
The kick drum patch (i_kickdrum.pd)
Snare drum
The snare drum is implemented somewhat like the kick drum. It uses two
sweeps, 500-200Hz in 20 milliseconds and 320-300Hz in 30 milliseconds,
combined with an adsr and a low-pass filter.
On top of the oscillators we add some noise to reproduce the sound of
the snares rattling.
The snare drum patch (i_snare.pd)
Hi-hat
The hi-hat is implemented using a noise generator as a basis, and an oscillator to controls its amplitude. On top of these components are a low-pass filter and a high-pass filter to remove the very high and the very low frequencies. It is (as the rest of the drums) controlled by an adsr connected with the velocity from the input.
The hi-hat patch (i_hihat.pd)
Crash Cymbal/Gong
The crash cymbal/gong is based on the bell instrument described
earlier. It is based on a version that is out of tune, so that it has
no distinct tone (or rather a lot of them).
Furthermore we added ring modulation to gain a more varied sound.
To make it more cymbal- or gong-like some noise is added together with
a high-pass filter, to remove the lower frequencies. All in all we get
a very noisy something that sounds a little like something and a lot
like nothing, but in some mysterious way pretty funny.
The crash cymbal/gong patch (i_dong.pd)
Brass
When we wanted to start making the instruments, we thought that using an example synth, like J08.classicsynth.pd, would aid us in understanding how to make nice sounding instruments, and maybe we could even create different instruments by just selecting other parameters for this synth. But the 16x subpatch is very hard to understand and has no explanations. Also this patch uses two cpole~ and one rpole~ filter. The names "complex one-pole (recursive) filter" and "real one-pole (recursive) filter" hints that these filters are very CPU-demanding, and they are. Actually this instrument will take up 50% of the CPU power on the fast dual Pentium 4 3GHz machines. So we started building our instruments from scratch, and just experiment with different techniques, a more fun approach that forces you to read and learn a bit more. We kept a modified version of the synth from that example, and named it i_brass, as it sort of sounds like a brass instrument.
The brass patch (i_brass.pd) is a modified version of the J08.classicsynth.pd
We tried to optimize it a bit by using a bp~ filter instead of a vcf~ filter, but it made no measurable difference. And we added our volume envelope as in the other instruments.
Conclusion
Letting java handle the hassle of reading midi files and thereby isolate the task of PD to handle the audio part of the project resulted very well - in contrast to the last project (project 1 of the digital audio course) where we used PD to do things PD probably wasn't designed to do. The development of the instruments was quite straight forward in PD and it was therefore fairly easy to create each instrument and later fit their parameters to get the correct sound.
It has been interesting to play with the different synthesis methods. Experience the output of the different modulation techniques has been insightful. Also thinking in real instruments and trying to - by more or less - simple means to make a patch with a similar sound has been an exciting progress.
We think that the initial goals of the project and the resulting project are fairly consistent and that we ended up with a proper result. The progress has given us some insights and experiences in digital audio which is basically what it was all about.
