Tonfall – Open Source Audio Framework

Tonfall is an opensource audio framework for Actionscript I have written for the Audio Code Clash Workshop at Flash on the beach to provide an entry point for the attendees on that subject.

Tonfall is not meant to be a fully implemented audio engine but you may learn a lot about signal processing and sequencing in common. It covers most questions I am receiving by email and is not that far away from the audio engine we use in Audiotool.

Features

  • Holding a list of processors and time information
  • SignalProcessor for audio generation taking TimeEvents (like notes) into account
  • Design Pattern to process polyphonic audio generation (e.g. synthesizer)
  • Static Delay as a template for other effects
  • Note to frequency mapping
  • WavFile Encoder
  • Samples of the Casio RZ1 drumcomputer
  • AmenBreak sample
  • Super Mario playing Sequencer
  • Fully launchable Metronome (Hello World of Audio DSP)
  • Simple implementation of ToneMatrix

Tonfall is built with FDT 4.0, which I highly recommended. I checked-in the project files as well to let you compile right away.

I also highly recommend to check out Justin Windle‘s AS3 Particle Node Sequencer, which is based on Tonfall and a very nice implementation of what you can do with it. Thanks again Justin!

Pitch MP3

Pitching MP3 (Not PitchShift!) is possible since Flash10s new Sound API. Today I saw Lee Brimelow’s post about doing so, providing some source code. Cause we spend a lot of time to keep things running more smoothly in the AudioTool, I created another version which has some advantages.

  • No objects are created in runtime (memory usage)
  • The SampleDataEvent is receiving a static blocksize (steady latency)
  • Linear interpolation (sound quality)
  • Speed can go down to zero

{source code}
Continue reading “Pitch MP3”

FFK09 Karplus Strong Sources

As promised here are the sources from the 10 Hot Minutes session, where I presented a very simple Karplus-Strong algorithm implementation.

More sources from the session Boing Bumm Tschak can be downloaded here.

Endless Karplus Strong – Click to feed noise:

Get Adobe Flash player

An enhanced version emulating a guitar can be found here.
It is basically the same code as above!

John Davey was able to make a very nice shot from Mario and me. Great conference indeed.

Andre & Mario

FP10 SoundAPI changes

Without telling anyone, Adobe changed the Sound API on the recent released Flash Player 10. When we released our AudioTool yesterday we encountered a ridiculous long latency of almost one second. The latency time is the time between computation of the sound and the point where you can listen to the actual sound. It is important to keep this value low for live experience. After hours of research some people figure out, that the amount of samples you pass to the sound card is changing the latency dynamically BUT will never decrease. If you once pass a higher number of samples you cannot go back easily.

Here is how we handled sound creation before in the AudioTool
We simply passed 2048 samples (minimum amount) every time the SampleCallbackEvent was fired. The playback was stable – even when scrolling the desktop with a lot of devices. For processes that would block the CPU for some time (e.g. create new plugin instances), we passed 8192 sample (maximum amount) to remain stable playback and execute the processes after. So we dynamically increased the latency once to have more space for heavy computation. Afterward we went back to normal (2048 samples). In fact this worked great!

The Release Player changed all that
With the latest player, every time you increase the number of samples you will increase the overall latency. And there is no way of decreasing it again without stopping the SoundChannel. So now we just pass 3072 samples (~160ms latency) all the time. Meaning that critical processes won’t be caught and will cause gaps in playback.

Again we now have 4 times more latency, but playback is overall less stable.

The worst thing is, that nobody told us and I am still not sure, what that is all about. The new sound feature was great and worked great for all release candidates. And all the sudden they changed it – again – without telling anyone. If we had this information before, it would not have been so hard to accept it. And of course we may have found some workarounds.

Read also Joe Berkovitz post | Joa Ebert post | Kai-Philipp Schöllmann post

What is wrong with the Matrix3D.transformVector?

I admit that I am not an expert on Matrix3D computation, but is it really me or just a bug?

In my understanding an inverted matrix can neutralize an already applied transformation to a Vector3D. So having a Vector3D transformed by a matrix and afterward to the inverted one returns the original Vector coordinates, right?

Check out this simple test.

Can anyone explain this or should I log a bug?

Update: If I use transformVectors instead – the computed values are as expected.

Too late? Very simple but striking feature request for Flash10 3D!

Oh dear! Two simple additional optional parameters for the following methods would allow you to use native performance and save memory.

Thinking of a simple 3d viewing pipeline, you have something like

  • Setup geometrie, textures and lights
  • Transform local coordinates to camera space
  • Clipping against Frustum
  • Project and render

They are a lot of helpers that helps you doing the transformation and projection, especially

  • flash.geom.Matrix3D
  • flash.geom.Utils3D

I assume that Matrix3D.transformVectors and Utils3d.projectVectors are lightning fast, cause you can pass a Vector (sources and targets) to them. A single(!) Vector could contain the entire 3d world geometry as a list of (x,y,z,x,y,z,x,y,z,x,y,z,…). That is why I am thinking that the Flashplayer is able to execute this well formatted list extremely fast! Much faster than calling the methods to transform several Vector3D objects in a AS3 loop.

So far so good. Where is the problem?
The worst thing from the memorys point of view is the clipping part, cause vertices are added and just valid for a single frame (if the Camera or Geometry is moving). A big case for the garbage collection.
Trying to figure out the best data structure for a 3d engine in Flash, I thought of something like this: All 3d world vertices are stored in a single Vector. Furthermore I would also leave some extra space for the expected vertices in the clipping process (increase the Vector length). With this setup, you reserve memory space and there is no reason for the garbage collection to remove used clipping data. You can easily access the clipped vertices with an offset in the Vector.

Vector fixedSized
Note: The Vector would stay fixedSized. It is on you to allocate enough space for the clipping process when you setup.

But it won’t work for now
Imagine you already clipped geometry: Next time you transform the Vector into a new coordinate space, all the clipped data from the last frames will be transformed as well. Two simple optional parameter would change that.

Solution
Matrix3D.transformVectors(vin:Vector, vout:Vector, startIndex: uint, endIndex: uint):void
Utils3D.projectVectors(m:Matrix3D, verts:Vector, projectedVerts:Vector, uvts:Vector, startIndex: uint, endIndex: uint):void

Thats it! That simple!
Now you could force the Flashplayer only to transform the necessary range. You simply overwrite the new clipped data in the Vector every frame, project and draw them. If you know the region, where a 3d-object has allocated its vertices you could use the startIndex and endIndex value to transform it at once.

Too late? I guess so. Too bad I started so late…

Update
I created an official feature request at Adobe bugbase. It would be a great feature!
Please vote. It may not be too late.

Away3D – A 3d demo convinces entirely

We all know the current hype on papervision3d and I really admire the effort the whole team put into it. However I spend a lot of time talking to Fabrice Closier on the last Flash conferences and he pointed me to their latest demo. He almost let me forget, that we all still need to deal with linear texture mapping in Flash. The perspective texture correction is so convincing, I am really impressed. It is actually very difficult to keep control of it, while moving the camera close to polygons. So keep an eye on this engine as well. Great work, Fabrice!

Away3D | Papervision3D

Popforge Updates

Joa & I are very busy at the moment contributing more stuff to our little open source project PopForge. There are lots of new stuff in the line. For now, I recoded the 909 from the scratch – hopefully less CPU intensive and with better sound quality. The sound samples and grains are now stored into a single file, introducing Joa’s compressing FurnaceFormat. Quite elegant!

Source code 909 GUI | Source code Popforge package (including 909 processor)

Furthermore I have updated the AIR version of the 909. You can now stream a Wavefile to disk while playing your session. If AIR will be some day as distributed as the FlashPlayer, we will have good times. I like it. Joa is also preparing FUI, a smart and simple GUI Creationset to build UIs in actually no time. This will be a huge timesaver for creating and debugging synthesizer and effect devices.

Rock’N’Roll!

Air909 – First Air Application

I decided to give AIR a try building my 909 as an AIR application. It comes now with a chromeless, shaped window and save/load functionality to store the whole content (including the parameter values) of the 909 permanently to your harddrive. I have rewritten the 909 engine for making it open source in the next time. Everything should sound as before, even better and less CPU intensive.

Make sure you have the Adobe Integrated Runtime (AIR) installed.

Install Air909 and share your beats!

Please update to the latest version. There was a serious playback bug, that causes clicks.
NEW: Hold SHIFT while clicking START to save a WAV – stream directly to your harddrive!

Note: Make sure you save the file with the *.909 extension.

#

David Rowald captured me while playing around with the 909 and put it on youtube. Damn you ;)

Weird behavior of numbers in as3

My colleague Bram de Jong at splicemusic did some tests with number types in arrays to find out, if we can save memory somehow. However his findings are rather not expected that way. Numbers can loose their type in an irreproducible way.

So lets try:

trace( getQualifiedClassName( uint( 0 ) ) ); // int
trace( '28bits', getQualifiedClassName( uint( 0x0fffffff ) ) ); // int
trace( '29bits', getQualifiedClassName( uint( 0x1fffffff ) ) ); // Number

Above all check this:

			var num: uint;

			for( var i: int = 0 ; i < 64 ; i++ )
			{
				num = 1 << i;

				trace( i, getQualifiedClassName( num ) );
			}

From 0 to 27 it is int
From 28 to 31 it is Number
From 32 to 59 it is int (again)
From 60 to 63 it is Number (again)

The disadvantage of this is that Numbers are slower in computation as well as reading, writing in an Array. And however it doesn't make any sense at all. Bram has already sent a note to adobe. I'll keep you updated.

SimpleMouseEvents (onReleaseOutside, onDragOut,…)

Some of the most confronted issues in AS3 are the missing mouse events as onReleaseOutside, onDragOut, onDragOver and so on. I saw that question over all boards. While coding the TR909, I had the same problem and I tried to figure out a common solution. I wrote a wrapper class, which adds itself as a listener to the passed target object. It takes care of the fact, that the stage can be null (required for releaseOutside) and the stage focus. It also provides (if necessary) the trackAsMenu mode to receive onDragOut, onDragOver, if the target itself doesn’t have the focus.

source code + sample application

Flash sound issues on vista

I installed Windows Vista today and besides having problems getting familiar with the new look and locations of stuff, I found that the onSoundComplete event is completely out of time on any player I tested. So even in the standalone with the classic design it is really glitchy. Damm, this breaks my flow in programming audio stuff for flash. I am really thinking of getting back to XP for the time it isn’t fixed. I found this blog entry, but I do not have the sound card settings as him.

Any change to get some official notes on this?

Flex2 Beta3 disappointments

I cannot couch my disappointment about the current beta3 related to the BitmapData.setPixel method.
A simple test confirms, that the setPixels method is now 30% slower than before. After that test, I converted my 3dengine (beta2) to the current beta3 and the framerate drops from 76 to 55FPS. So, simply put, I will abort the research about-perspective texture-mapping in AS3.

I know, there are many wonderfull things to do with AS3, but hey, it was is still a dream…

AS3 Bresenhams line algorithm

here is an implementation of the bresenhams line algorithm. I tried to avoid the Math.abs, which I found in most source or pseudo codes. I think, this couldn’t be much faster in AS3. I’m developing a custom line algorithm, cause its the first step trying to code an own texture mapper, since the one I used is ported from C.

Here it is:

private function bresenham( x0: int, y0: int, x1: int, y1: int, value: int ): void
{
	var error: int;

	var dx: int = x1 - x0;
	var dy: int = y1 - y0;

	var yi: int = 1;

	if( dx < dy )
	{
		//-- swap end points
		x0 ^= x1; x1 ^= x0; x0 ^= x1;
		y0 ^= y1; y1 ^= y0; y0 ^= y1;
	}

	if( dx < 0 )
	{
		dx = -dx; yi = -yi;
	}

	if( dy < 0 )
	{
		dy = -dy; yi = -yi;
	}

	if( dy > dx )
	{
		error = -( dy >> 1 );

		for( ; y1 < y0 ; y1++ )
		{
			output.setPixel32( x1, y1, value );
			error += dx;
			if( error > 0 )
			{
				x1 += yi;
				error -= dy;
			}
		}
	}
	else
	{
		error = -( dx >> 1 );

		for( ; x0 < x1 ; x0++ )
		{
			output.setPixel32( x0, y0, value );
			error += dy;
			if( error > 0 )
			{
				y0 += yi;
				error -= dx;
			}
		}
	}
}

watch here Flash8.5Player required

Missing AS3 examples

I’m converting my AS3 examples from time to time to the current BETA built. I will update this posting, whenever an example is updated to the current flash8.5player.

Perspective Texture Mapping Thanks to Denis Chait for the 3dMax scene
: Nice adaption with a video, projected onto the wall (speaker: Blixa Bargeld)

200 Circles coliding | source

BitmapCloud 9180 particles | source

Stay tuned.

AS3 Particles and Lines

Particles are really fun to play with. Here is another 2D example with line collisions. Click and drag to draw a line. You can also move the endpoints. Releasing an endpoint below or above the stage will remove a line. The collision detection is done by finding the intersection from the line and the moving path from each particle (672).

Particles and Lines require flash8.5player

Try your own bitmaps -default-size 256 384 -default-frame-rate 120 -optimize

AS3 BitmapCloud 3D

A short study to test, how many 3D particles could be rendered in AS3. Seems a huge. The code scanlines a bitmap and creates 3D coordinates with a random z-value. The computed x,y values result a normal 2D view, when projecting without further transformation. By clicking the mouse button, you can rotate the particles in 3d space. In this case, I’m rotating and projecting 9180 particles.
try your own pictures -default-size 256 256 -default-frame-rate 120 -optimize

BitmapCloud 3D (require flash8.5player)

Hashed Slowmotion Camera

I have no idea, if this could break your computer, since I’m using a lot BitmapData instances, while applying this effect. Be carefull.
But if it’s running fine, you can do thinks with yourself, you didn’t noticed before, hehe. I cannot stop making faces with it.

8ball lab | camera/motion_hash (flash8player & webcam required)

AS3 Raycaster

A fast implementation of a raycaster. You can move forward by pressing the mouse button. This version also provides a camera roll and height, ceil and floor tiles. To boost up the code, I’m using a lookup table for sin, cos, tan and massive bitshifting. The code is accordingly unreadable. What I’m really missing is a typed Array with simple bytes to write faster code, just reading RGB values. I have the feeling, that the getPixel method is too slow. This causes the small stage (240x160px).

Update 2010
The files are missing for some reason. However the Raycaster is still here even though it is very old now.

AS3 Performance

This is an old experiment from flashmx times. After converting to AS3, I’am really exciting. While the optimized AS1 version makes up to 12 circles, the new version can handle 200 very easily. Note that these are 19900 distance calculations each frame plus moving, bounding and resolving the detected collisions. No problem to imagine, that you can write a 9ball billard application which high resolution physics.

200 circles coliding : flashplayer8.5 needed

//-default-size 256 256 -default-frame-rate 120

AS3 Wavemap

I’ve implemented this great tutorial The Water Effect Explained now in AS3. Maybe, its a little small, but it only looks good, when you get at least 50 fps (best 80-90).
I have done this in AS2 before with blendModes and the ConvolutionFilter, trying to emulate the very simple algorithm for performance, but the results are not as the original. So I know, that AS3 is much faster than AS2…, but its too slow in such cases. One solution is to blow the output twice, but you really need a high fps, which is mostly not possible to get in browsers which other things moving around.

Anyway, I have modified the tutorial to compute with Integers (int), which are much faster than floats(Number). The result is rendered in gray to the output. If anyone found possebilities to increase the performance, tell me.

AS3 wavemap (Flash8.5 Player needed) – one step before water simulation (displacement)

Compile this with -default-size 80 80 -default-frame-rate 80

Another Gooify

grant skinner has done this already and I tried it before, but got lost. I wanted a solution where the displacement output is rendered onto the source image. This would be more fluid, cause you could move pixels endless in the bitmap. But after grants post, I tried another solution, which appears to be the same as grants one. here is the source to play with.
Continue reading “Another Gooify”

AS3 Perspective Texturemapping

This is the first step of a long journey. The example parses 3ds (3dmax) files and renders the scene pixel by pixel with a ‘real’ zBuffer for exact z-Sorting and a simple distance shading. There is a lot of space to increase performance. The most problem is drawing the pixels on the screen. The calculation of 236 faces connecting 190 vertices while clipping at the front plane is no performance problem. Commenting out the setPixel row, increases the FPS to 120.

Note: This example requires FlashPlayer 8.5

perspective texturemapping part 1 | based on Mikael Kalms 1997

BlendMode Math

If you wonder, what blendmodes are doing exactly, you can read this article about blendmodes in common. I tried to work with them in a accurate way to emulate complexer algorithms like animated water waves. Generically you can save a lot of performance, if you need to calculate every single pixel colorvalue from two bitmaps. In most cases, there is a simple solution using ColorTransform and Blendmodes. One problem is that the flash results are different in some cases, so here are my findings:

// **BLENDMODES
// based on: http://www.pegtop.net/delphi/blendmodes/
// modified to fit flash output

/*
	[for each color component] RGB
*/

// ADD
// c' = Math.min( 255, Math.max( 0, c0 + c1 ) )

// SUBTRACT
// c' = Math.max( 0, c0 - c1 )

// MULTIPLY
// c' = Math.floor( ( c1 * c0 ) / 0xff )

// SCREEN
// c' = 255 - Math.floor( ( 255 - c0 ) * ( 255 - c1 ) / 255 )

// LIGHTEN
// c' = c0 > c1 ? c0 : c1

// DARKEN
// c' = c0 < c1 ? c0 : c1

// DIFFERENCE
// c' = c0 > c1 ? c0 - c1 : c1 - c0

// INVERT ( no influence from c1 )
// c' = 255 - c0

// OVERLAY
// c' = c0 < 128 ? Math.floor( ( c1 * c0 ) / 127 ) : 255 - Math.ceil( ( 255 - c0 ) * ( 255 - c1 ) / 127 )

// HARDLIGHT
// c' = c1 < 128 ? Math.floor( ( c1 * c0 ) / 127 ) : 255 - Math.ceil( ( 255 - c0 ) * ( 255 - c1 ) / 127 )

For example if you like to compute the average colors from 2 different bitmaps, you can follow like this:

import flash.display.*;
import flash.geom.*;

//-- test here 2 colorvalues
var c0: Number = 234;
var c1: Number = 255;

//-- source bitmaps
var b0: BitmapData = new BitmapData( 1, 1, false, c0 );
var b1: BitmapData = new BitmapData( 1, 1, false, c1 );

//-- resulting bitmap
var be: BitmapData = new BitmapData( 256, 256, false, 0 );

//-- compute average
var half: ColorTransform = new ColorTransform( .5, .5, .5, 1, 0, 0, 0, 0 );
be.draw( b0, new Matrix, half );
be.draw( b1, new Matrix, half, 'add' );

trace( be.getPixel( 0, 0 ) );
trace( Math.floor( ( c0 + c1 ) / 2 ) );

Of course this is a simple one, but you get the idea.

Sphere Texturemapping via DisplacementMapFilter

I thought a long time about it and tried it the second time now. The idea is a freeze a current pixels displacement, as it naturally occur when projecting bitmaps into another enviroment like a sphere coordinate system. I created a gradient table in flash8 where the x and y values are stored as color-components (red/blue). I switch to 3dmax where I build my model (here a sphere) and map the gradient onto the mesh. The next step is to look, where the original positions of the texture was by comparing the source and the projected bitmap and compose a displacement map of it. I think, this is the screwiest workaround, I’ve ever done in flash.

Continue reading “Sphere Texturemapping via DisplacementMapFilter”

Bezier clipping against all orthogonal axis

After I trying some studies with my AS1 solution I found a lot of misbehavior, while clipping. So I started to develope a completely new solution in AS2. The code provides to clip against any orthogonal axis and furthermore it’s easy to extend in 3D. The hardest issue was to solve a division by zero in a few cases. I’ve tested it for a long time and it seems to be solid for me. Anyway, if you found any bug, please leave a comment.
Continue reading “Bezier clipping against all orthogonal axis”

Tight FPS Solution

PercyPea (well known from flashkit – mario kart thread) sent me an email, where he describes a solution to force the browser plugin to hold the FPS very tight.
I tried it for my tileengine games/supermario. It seems to be true and I really would like you to give me your framerate at this example. It should be about 32/33 and actually firefox makes it fine. PercyPea makes also a thread on flashkit, where you can post your comments.

The idea is to adjust 120 FPS and slow down by running a loop untill the necessary milliseconds are passed. Simple as for example java do.

FDT – not just another eclipse plugin !

powerflasher has developed an eclipse plugin for actionscript the last six months and it is currently entering the beta phase. I’m working with fdt for three days now and I can say that this is a killer plugin. Never thought, that any eclipse plugin for actionscript could be that intelligent. It saves a lot of time and writes a lot of code for you automatically. Once you’ve started developing with it, you don’t want to miss it anymore ! Congrats to Nico Zimmermann from powerflasher, who is the man behind fdt.

fdt will be a commercial product, different from asdt, which is still in alpha development and incommensurable. watch the fdt demo videos

Methods on primitives

Theodore Patrick posted a comparison of calling a method on primitives, once by themself and by creating an object before.

In my opinion the second line is faster than the first one, cause the interpreter can find the method faster from the object list. According to Theodore Patrick its about the garbage collection. The flashplayer creates an object, when you are not working with an instanced primitive and delete it after calling the method.

//-- Testing by Theodore Patrick

//primitive string
a = "Hello World";

//object string
a = new String("Hello World");

i=10000;
while(i--){
	a.split( " " );
}
trace( getTimer() );

Anyway, what you have to keep in mind is, that you have to instance any primitive by new when you need to call a method heavily. Simple computation like a*b or expressions like a = !b are much slower if you are using objects.

This is found is a similar way by holger kohnen in 2003!

More confusion can be found, when comparing primitive values

//a = true;
a = new Boolean( true);
trace( a === true); // false
//------------------------
//a = 3;
a = new Number( 3 );
trace( a === 3 ); // false

Keep this in mind…

Sourcecode flashconference stuttgart 05

For everyone who joined the flashconference, especially the rigibody session by mario and me, I’ve uploaded the source code right now. As we said, there is much to do, but its like a short briefing, how rigid body can be simulated for starters. The source code isn’t optimized, so you can follow the instructions much better since we are using a vector class.

It was a very nice session and I hope we can repeat this. thanks to sabine, tibor, david, max and elias to come along with me on this conference. We had a great time ! Greetings to patrick thiel and holger eggert, not to forget frank baumgartner, who was accidentally in stuttgart the same time and joined for some beer in the evening (hehe).

related links:
read also marios blog entry
as2 gamepackage

Compute 3D rotations with localToGlobal

Driving home from the flashconference in stuttgart we had a nice conversation about MovieClip.localToGlobal. I thought it is possible to write a complete 3D camera class with it and it is.

//-- setup
var xClip: MovieClip = createEmptyMovieClip( '0', 0 );
var yClip: MovieClip = createEmptyMovieClip( '1', 1 );

//-- apply angles
	xClip._rotation = xa * 180 / Math.PI;
	yClip._rotation = ya * 180 / Math.PI;

//-- for each point
		pt = points[p];

		pp = { x: pt.y, y: pt.z };

		xClip.localToGlobal( pp );

		tmpy = pp.x;
		pp.x = pt.x;

		yClip.localToGlobal( pp );

		tmpx = pp.x;
		tmpz = pp.y;
//-- end

The trick is to rotate 2 MovieClips representing 2 axis in 3d space. But it has not the performance boost at all, even using the native MovieClip function. A simple sin/cos Matrix is a lot of faster, but the idea was nice :o)

localToGlobal3D.zip (Rotating around 2 Axis)