Syncing beat detection algorithm to multiple Pixelblazes

Hi All!

I’ve been working on a beat-reactive lighting setup for my RGB-reactive fractal art booth and I feel like I’m so close, but I just can’t prevent my pattern from drifting in and out of sync. I was hoping one of you geniuses on here might be able to lend a helping hand?


Here’s what the goal is:

I’ve got five follower PBs that are each driving their own 6ft light bar (55 pixels each). These will be providing the light for my RGB-reactive art and so I’d like the color fade to progress with the beat of the music so the art “dances” along.


The journey so far:

I have zero experience writing beat detection algorithms so I started by standing on the shoulders of @jeff and his beautiful Music Sequencer and trimmed out everything except the bare essentials used for beat detection.

The code I ended up with ran flawlessly on a single leader PB, but when I ran it on the fleet, the color fading would drift in and out of sync between the followers.

I poured over forum posts for inspiration and while reading through the replies to this post, I saw that @wizard and @jeff were talking about how timing code that’s based upon “delta” can have problems syncing, since each board’s delta will depend on the individual frame rate that it’s running at. Instead it was suggested that syncing would be better achieved by using the time() function, since calls to the time() function are ostensibly synced between all the Pixelblazes in a group.

Figuring that I’d finally found the silver bullet for my problem, I wrote a timer function that accumulates time based upon calls to the time() function rather than relying on “delta.”

But alas! The boards still drift in and out of sync and I’m at a loss for ideas of anything further to try.

Here is a video that demonstrates the behavior. You’ll notice that it’s spot on for almost 2.5 minutes before the de-syncing behavior starts acting up. It even drifts back into sync for a while before everything goes haywire again.


Since it works great for a single Pixelblaze and could serve as a useful starting point for beginners for creating more beat-reactive patterns without having to dive all the way in with the Music Sequencer, I went ahead and uploaded it to the patterns library. (It’s called “Sound Reactive Color Fade”)

I’ve commented the code extensively so it should be pretty easy to follow along and understand the purpose behind everything. I’d be super grateful if anyone could give it a look and give me some feedback on what they think.

Thanks!


P.S. Here’s another bonus test video that shows the results when things stay synced for the most part.

3 Likes

Wow! This is awesome! What an interesting problem!

Congratulations on being one of the few who dove into that code and came out alive.

Here’s my theory. I think some of the sensor board’s frequency bin values are occasionally getting dropped (they send values from the selected PB with sensor board to the others over UDP, which as you may know, is an async protocol that doesn’t wait for confirmation ACKs/retries). Therefore, your timers might not be the problem; it might be that each follower is occasionally detecting beats on an incomplete set of sensor board audio samples.

I’m having trouble suggesting a solution without support for network-sync’d variables or a message-passing infrastructure. These are things discussed for a future release but have no near-term milestone.

@wizard can you think of anything clever to allow the leader (or the one with the SB) to be the source of truth for audio event detection?

1 Like

Found your video in the other thread! Awesome!

Mine looks like this: https://youtu.be/M676v9ckS9I

(that’s controlled by the sliders in my Lego pixelblaze cases, but I have also done audio and brainwaves with my bluetooth eeg, which also has an accelerometer so you can make the colours rotate by tilting your head)

2 Likes

Ha that’s why I experienced some delay on my own setup. I thought it was due to the different mapping, but now I’m sure it was actually out of sync, it’s exactly the same problem as you !

Could we not simply use one pixelblaze to push pixels to another one ?

That’s what I thought the problem might be, too. It seems like the code on each follower is being executed on very slightly different audio data.

I was hoping there may be a way to have the leader PB do all of the beforeRender() computations and then push out a set of updated HSV values to all the followers at once for each frame. Given that you mentioned network-synced variables aren’t a reality yet though, it sounds like that strategy might be off the table for now.

Thanks a bunch for the input!

I would experiment with WiFi, ultimately it’s the network latency/retries that would cause sensor board data to arrive at very different times and possibly miss values if the retry overlaps them.

Using a router closer to the PBs or switching channels might help out. Make sure the antennas aren’t near metal things or signal blocking materials, etc.

It’s all running asynchronously even with the best WiFi network possible, there’s no way to have exactly the same number of render cycles for every sensor data sample.

Pushing pixels instead of sensor board data would make this problem much much worse.

It’s also possible to use the analog inputs to send data: you could wire them up to PB GPIO (yes a bit of a hack) and send 5 bits of digital data. Maybe a node variable or beat counter or something.

1 Like

Woah, those are some sweet fractal renders!

I’ve never actually generated a fractal purely with code from a seed myself. What software tools did you use on them?

P.S. I’m totally gonna use that head-tilt controller trick. Thanks for the idea! :smiley:

Thanks for the input!

  • I’m afraid that the same behavior happens even when I have the leader in AP mode with all the boards lined up next to each other, so I can’t imagine that poor WifFi connection is the issue.

  • Is the reason that pushing pixels would be worse because of the larger amount of data contained in a frame of pixels relative to a frame of sensor board data?

  • I’m afraid I’m not able to follow your idea with using the analog inputs, do you mind elaborating?

P.S.
A little about me so you know what kind of knowledge base I’m working with: I’ve got a B.A. in applied mathematics, but I only ever did one introductory Fortran course. Besides that, all my coding knowledge is self-taught with the aid of the almighty Google whilst working on little personal projects, so my coding expertise is probably a bit more patchwork than most.

I think his suggestion, which is clever and hilarious, is to have the PB that has the sensor board send itself an event or value. You’d connect an output pin on the Pixelblaze to one of the analog input pins on the sensor board. The analog input values are then transmitted to all the members of the sync group.

Using five of the GPIO outputs (high or low values for 1 or 0) connected to all five analog inputs would let you broadcast a 5 bit value, 0-31.

Now, there’s still the possibility that the sensor board UDP packet containing these values could be dropped, but you could use it in a way that’s more resilient to one or two drops.

For example, the 0-31 value could represent the step in a state machine. One implementation would be to have the sensor board node be the source of truth, and its calculating the definitive number of beats detected so far (it would need to loop around from 31 back to zero when the 32nd beat is detected). Each node stores its internal concept of how many beats have been detected. If new analog input values are received that advance this beat counter, then that node knows to trigger the actions that should be taken when a beat is detected.

This would likely result in a self-synchronizing behavior where if one node misses a set of sensor board values, it can catch up (and likely within one or two of the 40Hz sensor board update cycles). For example, imagine all nodes but one just noticed that they thought 7 beats had been detected but they now receive “8” in the analog inputs. They fire their onBeat() events (for example switching to the next color or starting an animation that will happen over the next second). The node that missed the initial advance from 7 to 8 will notice in a subsequent beforeRender() frame and can catch up.

An alternate implementation is that the 5-bit value received represents “the number of 25ms (40Hz sensor board update) intervals that have passed since a beat was last detected”. This essentially gives you 31 more chances to get everyone a state that will be perfectly sync’d once everyone’s received / decoded the value.

3 Likes

TL;DR: GIMP and MathMap

Thanks! Most of these I made with MathMap, which is an ageing GIMP plugin which has its own little expression language which is translated to C, compiled, and run once for each pixel (like pixelblaze’s Render2D basically). It’s usually used as a filter, since the expression can read pixels from your “input image”, but I use it to render images from scratch. Also like pixelblaze, variables in the expression can become UI elements so it is easy to experiment. As a plugin, it renders to GIMP tiles, so I can render an image that would not fit in memory, e.g. up to 54092x36632 which I would scale down for printing on A3+ paper at 720ppi.

http://www.complang.tuwien.ac.at/schani/mathmap/

Oooh, I think that makes sense!

Self-referential trickery FTW!

I’m gonna try this out and report back with results.

It worked! I was able to get all the followers to source their ground-truth beat count from the leader and stay synced flawlessly. Here’s the updated script:

/*---------------------------------------------------/< Credits >/----------------------------------------------------\\

Much appreciation to Jeff Vyduna for his very extensive Music Sequencer posted here:
https://forum.electromage.com/t/music-sequencer-choreography/1549

This pattern slowly fades through the color spectrum and jumps to the next nearest color upon detection of a bass beat. 

This version is specifically tailored to facilitate synchronization across a fleet of followers. It accomplishes this by
connecting IO pins 16, 17, and 19 on the leader Pixelblaze to the analog inputs "A0," "A1," and "A2" of its sensor 
board. This allows the leader to handle all the beat detection and broadcast a state variable to all the followers 
to prevent them from drifting out of sync with one another.

>> THIS PATTERN WILL NOT RUN PROPERLY UNLESS YOUR LEADER PIXELBLAZE HAS ITS IO PADS CONNECTED TO THE ANALOG <<
>>      INPUTS ON ITS SENSOR BOARD AND YOU HAVE ASSIGNED IT A NODE NUMBER DIFFERENT THAN ITS FOLLOWERS      <<

-MyMathematicalMind, 2023. MIT License

//---------------------------------------------------\< Credits >\----------------------------------------------------*/

//-----------------------------------------------/< Global Variables >/-----------------------------------------------\\

var leaderNode = 2  //**** MAKE SURE YOUR LEADER IS ASSIGNED A DIFFERENT NODE NUMBER THAN THE FOLLOWERS

//GPIO pin variables
var digitalOutputPins = [16, 17, 19, 21, 22] // IO pins: IO16(pin16), IO17(pin17), IO19(pin19), IO21(pin21), IO22(pin22)
for (i = 0; i < 5; i++) {pinMode(digitalOutputPins[i], OUTPUT)}
export var analogInputs

// Sensor board variables
export var light = -1 // If this remains at the impossible value of -1, a sensor board is not connected.
export var frequencyData = array(32)  //Get all the frequencies samples in an array

//Color hue variables
var red = 0, yellow = 0.1, green = 0.33, cyan = 0.5, blue = 0.66, magenta = 0.9
export var colors = [red, yellow, green, cyan, blue, magenta]
export var currentColor = 0

//Timing/State Machine variables
export var BPM = 130 // Debounce timer is set to enable algorithm to catch 16th notes at this BPM
export var globalDelta = 0
export var elapsedTime = 0, currentTime = 0, prevTime = 0
export var beatCount = 0, trueBeatCount = 0

//Debounce variables
var minBeatRetrigger = .2 /* How much of a currently defined quarter note beat must pass before a detected instrument 
                              will retrigger? E.g. use .2 to allow .25 retrigger (e.g. to catch sixteenth note drums)*/
var debounceTimer = beatsToMs(minBeatRetrigger)

//Bass variables
var bass, maxBass, bassOn    /* Bass and beats. (bassOn==true) is what triggers the debounce calculation and ultimately 
                                whether or not beatDetected() is called*/
var bassSlowEMA = .001, bassFastEMA = .001 // Exponential moving averages to compare to each other
var bassThreshold = .02      // Raise this if very soft music with no beats is still triggering the beat detector
var maxBass = bassThreshold  // Maximum bass detected recently (while any bass above threshold was present)
var bassVelocitiesSize = 2*pow(beatSensitivity, 2) + 11*beatSensitivity + 2 /* bassVelocitiesSize=5 seems right for 
most. Up to 15 for infrequent bass beats (slower reaction, longer decay), down to 2 for very fast triggering on doubled 
kicks like in drum n bass */
var bassVelocities = array(bassVelocitiesSize) /* Circular buffer to store the last 5 first derivatives of the 
                                                `fast exponential avg/MaxSample`, used to calculate a running average */
var lastBassFastEMA = .5, bassVelocitiesAvg = .5
var bassVelocitiesPointer = 0 // Pointer for circular buffer

//Tempo inference variables
// Store the last 8 intervals between beats. Longest = 50BPM on beat 1 (4800 ms = 60 / 50BPM * 1000 * 4)
var beatIntervalSamples = 8, beatIntervalPtr = 0, beatIntervalTimer = 0
var beatIntervals = array(beatIntervalSamples)

//UI controlled variables
export var beatSensitivity = 0.5
export var fadeSpeed = 8, displayedFadeSpeed = 0.5

//-----------------------------------------------\< Global Variables >\-----------------------------------------------\\

//---------------------------------------------/< UI Control Functions >/---------------------------------------------\\

export function sliderBeatSensitivity(_v) { beatSensitivity = _v }
export function showNumberBeatSensitivity() { return beatSensitivity }

export function sliderFadeSpeed(_v) { 
  displayedFadeSpeed = _v
  fadeSpeed = 4 + 8*_v
}
export function showNumberFadeSpeed() { return displayedFadeSpeed }

//---------------------------------------------\< UI Control Functions >\---------------------------------------------\\

//-------------------------------------------/< Beat Detection Functions >/-------------------------------------------\\

function processSound(delt) {
  processInstruments(delt)
  inferTempo(delt)
}

// Debounce detector
function debounce(trigger, fn, duration, elapsed) {
  if (trigger && debounceTimer <= 0) { 
    fn()
    debounceTimer = duration
  } else { 
    debounceTimer = max(-3e4, debounceTimer - elapsed)
  }
}

function processInstruments(delt) {
  // Assume Sensor Board updates at 40Hz (25ms); Max BPM 180 = 333ms or 13 samples; Typical BPM 500ms, 20 samples
  // Kickdrum fundamental 40-80Hz. https://www.bhencke.com/pixelblaze-sensor-expansion
  bass = frequencyData[1] + frequencyData[2] + frequencyData[3]
  maxBass = max(maxBass, bass)
  if (maxBass > 10 * bassSlowEMA && maxBass > bassThreshold) maxBass *= .99 // AGC - Auto gain control
  
  bassSlowEMA = (bassSlowEMA * 999 + bass) / 1000
  bassFastEMA = (bassFastEMA * 9 + bass) / 10
}

function inferTempo(delt) {
  bassVelocities[bassVelocitiesPointer] = (bassFastEMA - lastBassFastEMA) / maxBass /* Normalized first derivative of 
                                                                                        fast moving expo avg */
  bassVelocitiesAvg += bassVelocities[bassVelocitiesPointer] / bassVelocitiesSize
  bassVelocitiesPointer = (bassVelocitiesPointer + 1) % bassVelocitiesSize
  bassVelocitiesAvg -= bassVelocities[bassVelocitiesPointer] / bassVelocitiesSize
  bassOn = bassVelocitiesAvg > .51 // `bassOn` is true when bass is rising
  
  debounce(bassOn, beatDetectedWrapper, beatsToMs(minBeatRetrigger), delt)
  beatIntervalTimer += delt
  // Longest = 50BPM on beat 1 (4800 ms = 60 / 50BPM * 1000 * 4)
  if (beatIntervalTimer > 5000) beatIntervalTimer = 5000 // No-beat ms threshold to reset beat detection 
  
  lastBassFastEMA = bassFastEMA
}

function beatDetectedWrapper() {
  if (beatIntervalTimer >= 5000) { // Clear beat intervals, it's been too long since a beat
    beatIntervals.mutate(() => 0)
    beatIntervalTimer = beatIntervalPtr = 0
  }
  beatIntervals[beatIntervalPtr] = beatIntervalTimer
  beatIntervalTimer = 0
  beatIntervalPtr = (beatIntervalPtr + 1) % beatIntervalSamples

  beatDetected() // Calls a user-customized function that happens whenever a beat is detected
}

// Do this whenever a beat is detected
function beatDetected() {
  if (nodeId() == leaderNode) {
    if (currentColor < colors[1]) {
      beatCount = 1
    } else if (currentColor < colors[2]) {
      beatCount = 2
    } else if (currentColor < colors[3]) {
      beatCount = 3
    } else if (currentColor < colors[4]) {
      beatCount = 4
    } else if (currentColor < colors[5]) {
      beatCount = 5
    } else {
      beatCount = 0
    }
    trueBeatCount = beatCount
    setBeatColor()
  }
  
}

//-------------------------------------------\< Beat Detection Functions >\-------------------------------------------\\

//----------------------------------------/< Timing/State Machine Functions >/----------------------------------------\\

function updateDeltaTimer() {
  currentTime = 1000*time(1/65.536)
  if (currentTime >= prevTime) {
    elapsedTime += currentTime-prevTime
  } else {
    elapsedTime += 1 - (prevTime - currentTime)
  }
  prevTime = currentTime
  globalDelta = elapsedTime
  elapsedTime = 0
}


function txTrueBeatCnt() {
  if (trueBeatCount == 0) {
    digitalWrite(digitalOutputPins[0], 0)
    digitalWrite(digitalOutputPins[1], 0)
    digitalWrite(digitalOutputPins[2], 0)
  } else if (trueBeatCount == 1) {
    digitalWrite(digitalOutputPins[0], 1)
    digitalWrite(digitalOutputPins[1], 0)
    digitalWrite(digitalOutputPins[2], 0)
  } else if (trueBeatCount == 2) {
    digitalWrite(digitalOutputPins[0], 0)
    digitalWrite(digitalOutputPins[1], 1)
    digitalWrite(digitalOutputPins[2], 0)
  } else if (trueBeatCount == 3) {
    digitalWrite(digitalOutputPins[0], 1)
    digitalWrite(digitalOutputPins[1], 1)
    digitalWrite(digitalOutputPins[2], 0)
  } else if (trueBeatCount == 4) {
    digitalWrite(digitalOutputPins[0], 0)
    digitalWrite(digitalOutputPins[1], 0)
    digitalWrite(digitalOutputPins[2], 1)
  } else {
    digitalWrite(digitalOutputPins[0], 1)
    digitalWrite(digitalOutputPins[1], 0)
    digitalWrite(digitalOutputPins[2], 1)
  }
}

function rxTrueBeatCnt() {
  trueBeatCount = round(analogInputs[0]) + round(analogInputs[1])*2 + round(analogInputs[2])*4
}

//----------------------------------------\< Timing/State Machine Functions >\----------------------------------------\\

//------------------------------------------------/< Misc Functions >/------------------------------------------------\\

function SB() { return light != -1 }  //Function to check if sensor board is connected: if(SB())==true if detected

function beatsToMs(_beats) { return (1000 / BPM * 60 * _beats) }

function progressColor() { 
  currentColor += deltaColor(beatCount, 0.001)
}

function setBeatColor() {currentColor = colors[beatCount]}


// Calculate color increment based upon beatCount and timeStep
function deltaColor(i, dt) {
  if (i != 5) {
    return (
      dt*(colors[i + 1] - colors[i])*fadeSpeed
    )
  } else {
    return (
      dt*(1 - colors[i])*fadeSpeed
    )
  }
}

function round(_v) {
  return ceil(_v - 0.5)
}

//------------------------------------------------\< Misc Functions >\------------------------------------------------\\

//----------------------------------------------/< Runtime Functions >/-----------------------------------------------\\

export function beforeRender(globalDelta) {
  updateDeltaTimer()
  if (SB()) processSound(globalDelta)  //Update all sound related variables to be used by the patterns
  if (time(0.001/65.536) == 0) progressColor()
  if (nodeId() == leaderNode) {
    txTrueBeatCnt()
  } else {
    rxTrueBeatCnt()
    if (beatCount != trueBeatCount) {
      beatCount = trueBeatCount
      setBeatColor()
    }
  }
  
}

export function render2D(index, x, y) {
  hsv(currentColor, 1, 1)
}

//----------------------------------------------\< Runtime Functions >\-----------------------------------------------\\

Only needed 8 of the 32 bits to accomplish it too, lol.


However, in my excitement to share my success with my friends, I brought a couple of the LED bars out to an event last night and when I went to fire up the first bar (the one with the leader attached) I ran into some problems. As it turns out, without any strain relief on the connections soldered to the LED output, the ground connector had finally had enough and it snapped off as I was pulling the PB out of its enclosure.

Right before this happened, I had been connecting to the board’s WiFi and it threw out a message that Pixelblaze had recovered from a fail-state and had re-booted into fail-safe mode. After getting back home, re-soldering the ground connection, and trying to fire it up again, now the code won’t run properly :confused:.

I’m thinking maybe I need to do a full factory-firmware reset, but I’m not sure how I go about flashing the board and I can’t find instructions on how to do so anywhere.

Besides that, I suppose it’s possible that when the ground connection snapped it came into contact with the data connection and shorted, but there’s no visible indicators on the board that would point to that.

Any further advice you guys (@wizard @jeff) have for me would be greatly appreciated.

1 Like

No need for factory reset, and there isn’t an option. There’s WiFi reset but that just configures WiFi.

My guess is that some setting got reverted, this can happen if there are brown outs and the failsafe config kicks in. That’s what the ui warnings about. Changing anything in the ui can cause this setting to become permanent. Better to power off completely, then power back on if it was a temporary issue.

This fail safe system is what PB uses instead of factory resets to keep a PB from becoming permanently bricked. For each reboot, it will load less and less of the original configuration allowing you to fix a problem (like a corrupt file causing reboots) without losing all your data.

I would check your led settings, those would be more likely to get reset accidentally.

Patterns would remain unchanged and should still work otherwise.

If a loose wire shorted something bad, there may be hardware damage. That could indeed cause the led output to stop working but I would check settings first!

If the sensor board was damaged it would mean the gpio trick and data isn’t going to work either. Check the dropdown menu on the top right and see if the SB1.0 still shows up. You can also use vars watch to see the SB data. If it’s not updating continuously with new data there might be an intermittent connection.

If you want, you could reinstall/repair the firmware using the /recovery tool. I suspect this isn’t your problem though if you can connect and the app loads.

Once you have it working again, I’d get a backup!

You can back up the patterns and settings from the settings tab, then restore them later. The xl sized pbs can keep a local copy and restore too. This might be handy if you want to snapshot a production working setup and have a “panic button” available. This setting is hidden under advanced settings only available if you add ?setup to the url. Once a snapshot is made, you can start a restore by holding the button during power on.

1 Like

Thanks so much for the info! Everything’s back up and running and working flawlessly!

2 Likes

So … wait, the leader/follower stuff transmits all of the sensor board data? Nice!

1 Like

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.