Pixelblaze keeps restarting

[yomh]

Today I upgraded one of my Pixelblaze V3 standards to the latest firmware 3.51.

After that I was working on a new pattern and think I made a mistake somewhere in the code. At some point a message about Fail-safe mode came up and I didn’t know what that meant so I kept working on the pattern for a couple minutes.

I then power cycled the board.

Now I can’t get the Pixelblaze interface to come up. It starts loading, if I try with curl I can see that it outputs part of the bootstrap CSS in the HTML head, but then stops outputting.

Looking at my router’s log, I can see it keeps requesting a DHCP address over and over so it seems like the board must keep restarting.

Since I still have some brief network connectivity, is there a way to send a factory reset command or something using curl? I do have a backup I could load.

The board is in a shelf so it would be a pain to physically get to it. I can do it if necessary but I’d rather do it over the network if there is a way. Thanks.

[yomh]

Also I see this post about fail-safe mode Pixelblaze V3 Fail-safe: forgets configuration or resets to WiFi setup mode and it doesn’t seem like that is working in my case as I can see in the router log that it has power cycled many times more than 5 and it is still connecting to my network so it must not be going to wifi setup mode, and assuming the problem is with my pattern it must not be entering fail-safe mode at all as the first thing it would do is disable the pattern.

[wizard]

Are the LEDs on at all? It sounds like a power supply cutting out. A full power loss won’t trip the reboot detection fail safe stuff. I’ve seen cases were the LEDs come on bright enough to trip an overload on the power supply, it cuts power, then restarts.

[yomh]

No, the LEDs aren’t on at all and don’t appear to come on at all when it restarts. The Pixelblaze is connected to 2 output expanders, and one power supply powers one half of the strips and one powers the other.

My “normal” pattern is solid white on these shelves so they would always have pretty much the maximum power draw (which is quite a bit below the power available from the power supplies). I usually have them going like that most of the year.

Today I was trying to come up with a Christmas pattern so it would be quite a bit less power draw than normal since it was different colors, not just white. That’s when I ran into the problem.

So I think it is unlikely that it is the power supply unless the supply just happened to fail when I was updating the pattern. Certainly possible! But unlikely considering the timing.

[wizard]

5v? Might be worth checking that the power isn’t doing something weird, and if it is the PB seeing the led will help identify what’s going on.

If voltage is too high the large regulator will overheat and go into thermal throttling, which also cuts power to the controller.

[yomh]

Actually 12V. The 12V supply is feeding the output expander (which is set to 12V) and that feeds the Pixelblaze. I was able to measure the voltage going to the Pixelblaze from the output expander and it is holding at pretty close to exactly 5V. No dips in voltage seen. The main orange lights are on steady on both the output expander and the Pixelblaze.

[yomh]

OK, I got it working. I was able to reach in there and push the button on the Pixelblaze so that it switched to a different pattern. Once that happened, I was able to access the web interface.

I will note that while the Pixelblaze was in its “stuck” phase, I was able to ping it and, like I said, trying to access the web interface would get part of the HTML downloaded before it stopped. I could get the “/recovery.html” page to load, although I never tried doing anything on it. I tried using the pixelblaze-client code and some websocket code to talk to it, and I could establish communications, but never enough to switch the pattern.

Without knowing the code I can’t speculate too much, but it seems like there is something with my pattern that caused the board to get in a bad state and, for some reason, the fail-safe didn’t operate as one would expect.

Once I switched the pattern I downloaded a backup and also downloaded the bad pattern and then restored to an earlier backup.

I’ve attached the bad pattern here so you can see if it is something bad with the pattern itself or if my board had a glitch for another reason. I was playing with seeing if an AI knew how to write Pixelblaze code so that’s where this code came from.

NICK - Christmas Shelves Fade.epe (7.2 KB)

[wizard]

Oh wow! Thats exactly one of the things the failsafe is designed to protect against. Thanks for grabbing the bad pattern first, will check this out and see if I can reproduce it.

Can you dm the backup as well? The epe file doesn’t contain the binary, just source, and a backup would have the binary. If something got corrupted it might only show up in the backup.

[yomh]

OK sent you a message! Curious what you’ll find. Thanks!

[zranger1]

The pattern is locking up because it repeatedly allocates memory in render(). Somewhere around line 46, it does this:

 var color = [lerp(color1[0], color2[0], t), lerp(color1[1], color2[1], t), lerp(color1[2], color2[2], t)];

which allocates a new 3-element array for every pixel.

It’ll wind up using all the Pixelblaze’s memory within just a few frames. To fix it, allocate the color array outside of any function with something like var color = array(3), then in render, just set the elements for each pixel:

color[0] = lerp(color1[0], color2[0], t);
eolor[1] = ... etc.

(Edit: There’s more – each use of color1 = [1,0,0] and so forth allocates new arrays. Since the Pixelblaze doesn’t have garbage collection, this will eventually cause a crash too. You’ll need to allocate any arrays you want to use at initialization time. )

A side note - this has been my experience w/using the various LLMs for Pixelblaze patterns too. You can use it to help nudge you toward a solution or help with boilerplate, but it almost never produces usable, idiomatic code without a whole lot of prompting. Sometimes more prompt text than it would’ve been to just write the program. It’s definitely not ready to take over our programming jobs yet!

[zranger1]

(FWIW, here’s a short pattern that uses palettes for a more Pixelblaze-idiomatic way of doing this job.)

var palette = [
  0.0, 1, 0, 0,
  0.33, 1, 1, 1,
  0.666, 0, 1, 0,
  1, 1, 1, 1
]
  
  setPalette(palette);

export function render(index) {
  paint(index/pixelCount)
}

[yomh]

Thanks! Using the LLM was more of an experiment, I was curious how much it knew of a more “obscure” language like Pixelblaze’s. I admit this is the first time I’ve tried writing a Pixelblaze pattern more than “everything white” and I didn’t spend much time reading the language documentation first. What you say about memory management makes sense. I just didn’t anticipate the issue of running out of memory causing the board to get in a hard to recover state.

[wizard]

Yep, looks like many small arrays can run the system out of resources. It wasn’t counting some of the overhead properly. I’ll get that fixed in the next update.