V3 Standard: every flash write crawls, but the chip dumps perfectly. SPIFFS with zero free blocks?

Hi Ben. I’ve got a V3 Standard that went bad on 2026-07-20. Before touching anything I pulled a full flash dump, and I think I’ve ruled out the flash chip itself. I’d love a sanity check before I erase the filesystem partition, mostly because I can’t tell what an erased FS boots into.

Quick disclosure since this is my first post here: I leaned on Claude pretty heavily for both the diagnosis and this writeup. The bench work and the numbers are mine and real, but if this reads suspiciously organized for a newb, that’s why.

The symptom was odd. The board renders patterns fine and joins WiFi fine, but anything that touches flash crawls: pattern loads, config writes, POST /wifisave, even small HTTP responses get retried a few times and then finish agonizingly slowly, or time out. Every kind of write was affected about equally, and plenty of hard power cycles changed nothing. In AP mode HTTP would eventually answer after about 90 seconds; in station mode I never got an answer inside three minutes.

So I read the whole 4 MB out twice over a CP2102 on the expansion header, at 115200, back to back:

dump 1 dump 2
bytes 4,194,304 4,194,304
time 385.4 s 385.2 s
rate 87.1 kbit/s 87.1 kbit/s
sha256 b8819e24…ade3 b8819e24…ade3

Byte-identical, 0.2 s apart, and about 94% of the theoretical ceiling for 115200 8N1, so the serial link was the limit, not the flash. No retries or stalls during either read. A few supporting reads:

  • Chip: ESP32-D0WD-V3 rev 3.0. Flash mfr 0x20, device 0x4016, 4 MB, 3.3 V.
  • Flash status register 0x0200: QE set, no block-protection bits, no lock bits, no stuck WEL.
  • FLASH_CRYPT_CNT = 0.

All of which looks like a healthy chip, and is why I stopped believing my own “bad flash chip” theory.

Here’s what the image shows instead. Partition table:

nvs      data 0x02  0x009000   20K
otadata  data 0x00  0x00e000    8K
app0     app  0x10  0x010000 1280K
app1     app  0x11  0x150000 1280K
spiffs   data 0x82  0x290000 1472K

Measuring the SPIFFS region out of the dump:

used 74.0%  /  free (0xFF) 26.0%
4K blocks: 0 fully-used, 368 partial, 0 fully-erased  (of 368)

Every one of the 368 blocks is partially written and not one is fully erased. The free space is real, but it’s smeared across every block instead of concentrated anywhere usable. My read: SPIFFS has to garbage-collect on every single write (copy live pages out, erase the block, write back) with no clean block to work in, which would explain why reads stayed fine while writes fell off a cliff, and why bigger writes hurt more than small ones. There are also around 30 stale config.json copies scattered through the image that were never reclaimed. I should say the GC story is me reasoning backwards from the block layout, not something I watched happen; the block distribution is the measurement, the rest is my theory about it.

Some possibly relevant history: a few days before this started, I had an ESP8266 button box hanging off the Pixelblaze’s 5 V rail on alligator clips, a temporary lash-up so my kid could turn the lights off from bed. I now think that connection was intermittent and the board ate unclean power events for days before it gave up. I’ve seen you describe brownouts as a source of flash corruption, and the timing fits better than anything else I’ve got. That connection is gone for good; the bedside control is a battery-powered remote now.

One more thing I’m curious about: this board has the dual-app OTA scheme above, leaving 1472K for the filesystem. The replacement V3 Standard I’m running now reports 2,884,241 bytes of storage, nearly twice as much, and that number was identical before and after I updated it from v3.51 to v3.67, so it looks like a partition-scheme difference rather than a firmware-version thing. I couldn’t pull the dead board’s firmware version out of app0. If it was on an older scheme with half the filesystem, that would explain it filling up and fragmenting a lot faster.

The actual questions:

  1. If I erase just the SPIFFS partition, does the firmware reformat and rebuild it on next boot? The thing I’m worried about is that the web app lives on that FS and gets installed by the updater rather than regenerated, in which case I’d be trading “slow FS” for “no web UI.”
  2. Do /recovery and POST /update still work with an empty filesystem? If they do, this is low risk. If they don’t, I’d want the .stfu in hand first.
  3. Does this profile match what you’d expect from repeated brownouts? Perfect raw flash reads, zero free SPIFFS blocks, uniform write slowness. And is a board in this state worth reviving, or is a wipe and reflash the realistic path?

No rush on any of this. The wall is already back up on a replacement board, so nothing’s blocked. I’ve got the dumps, partition table and block statistics saved and can share any of the numbers, and I’m happy to run whatever else you’d like while the old board is still on the bench.

Thanks for building something that survived being diagnosed this thoroughly.

cheers, Tarek.

Yeah, sounds like the filesystem is getting too full to be functional. SPIFFS can do that when pushed near capacity with fragmentation. Basically it ends up thrashing trying to compact all that half used space by copying it around to make fully erased blocks, which are then immediately used. And the compaction process itself can kind of chain-react, and it grinds and can take tens of seconds to write tiny files. I would remove some patterns (get a backup first!) to free up space.

One of the things that has bugged me about SPIFFS and filesystems for microcontrollers in general! Something I’ve been working on. However this will be a Pixelblaze V4 thing, or at the least an XL-only thing, since those have some extra space for a second filesystem.

If you have a V3 XL and a regular V3, they have different sized flash and different partitions. Technically a non-XL could be flashed onto an XL, but yeah, you lose space. If you put an XL image on a non-XL, then I don’t imagine the filesystem that is half off the end of the flash would work out very well.

/recovery does not work without a filesystem, since it loads from recovery.html.gz. /update will though, and uploading an .stfu file can put the main web app (and recovery) back in place. HOWEVER, there’s no easy way to configure wifi without the main or recovery app, and with no filesystem it’s going to default to setup mode. You’d have to connect in setup mode, move the capture portal window out of the way, and use /upload in a browser.

Closing the loop: the board is fully recovered, and your diagnosis held up all the way down.

I tried the pattern-delete route first, but this board turned out to be past it. In station mode it never returned a single byte: I gave a plain HTTP GET nine and a half minutes and got nothing after the TCP connect, and a websocket upgrade got no 101 in five. Setup mode was livelier, but /list, /config.json and /delete?path= all 404 there, so I think the file API just isn’t routed in setup mode. The interesting detail is that even those 404s took 13 to 67 seconds each, which says the SPIFFS grind was dragging down the whole firmware loop, not just filesystem calls.

So I took the erase path with your answers in hand. esptool erase_region 0x290000 0x170000 over the expansion header, exactly the spiffs partition and nothing else, with the full dump as the undo button. It booted into setup mode as you predicted.

One correction that might help the next person who finds this thread: on this board’s firmware, /upload 404s in setup mode. What worked was POST /update with the multipart field named update (I pulled recovery.html out of the .stfu and read the form to get the field name). The captive portal’s setup page also 404s in this state, since pbWiFiSetup.html lives on the filesystem that’s now empty, so the browser route may not exist on older firmware at all. A scripted POST of v3.67.pb32.stfu went through fine, and after the install the web app and captive portal came back exactly as you described.

WiFi setup, then restored my backup. Same chip, fresh filesystem: config.json now answers in 0.26 seconds where it previously never answered at all, all 28 patterns are back, and packet loss went from 33 to 100 percent down to zero. Every symptom traced to the filesystem state and nothing else.

And the partition mystery from my first post: solved, by you, in your first reply :man_facepalming: Your XL paragraph made me go dig up my order confirmation, and the replacement is the XL variant, not a second Standard like I said. So 2,884,241 bytes versus 1472K is just XL flash versus Standard flash, no anomaly at all. The .stfu understandably doesn’t repartition (I verified the table on the recovered board is byte-identical before and after), so the old board keeps its 1472K and I’ll keep its pattern load lean.

Thanks again. The tens-of-seconds-per-tiny-write description matched what I saw so precisely that I stopped second-guessing the diagnosis, and the FASTFFS tease is noted for whenever V4 lands.

Thanks again for building such a sweet piece of hardware - it’s a ton of fun to make patterns on!

cheers,
Tarek.

2 Likes