Save/persist variables

I was thinking of implementing it so that the save slot values were held in memory until the pattern gets unloaded, so you could write to them as often as you like without worrying. Similar to the load/save handler functions but API driven instead of magic exports.

I’m thinking it would be suitable for perhaps a dozen or so scalar values. The example use case is for storing a few modes/settings that change from button presses.

These should be items that you want to be persistent, and ideally based on user input, not intended as a way to capture the intermediate pattern state or pixel buffer so that it seamlessly resumes when switching patterns. Doing that could cause excessive flash wear e.g. if used in a repeating playlist.

Which reminds me, in general all of the persistence in PB today originates from a user interaction. There’s nothing automated that can wear flash out intentionally.

Perhaps this use case can be solved in another way entirely.

What if you could define a pin as a hardware button control that supported a stepped mode selection. That would be a whole lot less button handling code, and the persistence could happen automatically as currently done with UI controls. It’s not quite as flexible, but this seems like a common use for a button.

pin = 26
numModes = 4
pinMode(pin, INPUT_PULLUP)
persistentModeButton(pin, FALLING, numModes, modeChangeHandler)

function modeChangeHandler(mode) {
  // called after init with saved values, or any time button is pressed
  // mode ranges from 0 to numModes-1 before wrapping back to 0
}

It’s like registering an on change callback, but the system can do all the button debouncing, state incrementing, and persistence logic. Setting numModes to 2 would give you a simple toggle button.

If I go this route, I would add a non-persistent modeButton() version of the API as well.

The caveat here is that this would allocate some resources, so it wouldn’t be suitable for reconfiguring on the fly unless it was changed to a register / unregister kind of API pair.

1 Like

I like persistentModeButton()/modeButton() a lot! Bet you’d see a lot more button-controlled builds.

Let me switch sides here, and give my use case for the saveSlot() approach too. It’s pretty simple – it would make the Multisegment pattern (and similar things) actually workable without an external controller to save the per-segment parameters.

Right now, it uses one controller to select a segment, and at least one other to set parameters for that segment. When you leave the pattern, the control settings are saved, but the accumulated segment settings are not. With even the ability to save a few scalar values, they could be.

1 Like

I’m not sure what you’re proposing here, as my use case wasn’t pin or button based.

I tweaked a handful of settings (sliders and pickers) and I leave the pattern and come back, and it’s restored. The use case here is to have a way to save a NON-slider or picker value, and have exactly the same behavior. Perhaps the code wants to record a state, but my example use case above is also good: play counter, or rotating between modes, sans user interaction

So here’s a concrete example using current playlist as the driver:

I build a playlist of 3 patterns.
Pattern one is normal, call it P.
Pattern two has a mode flip… each time it runs, it either does A or B
Pattern three has a mode flip between 3 states, D, E & F

When I play the playlist, the behavior would be
PADPBEPAFPBDPAEPBF (it would then repeat)
This is a pretty simple example but I could come up with more complex (so please don’t ask me to just build the above sequence as the playlist…) My point is, doing a playlist of this level is entirely depending on a way for a pattern to record a value that sticks thru load and is updated on unload by the code. And yes, I’m aware that running this playlist would mean a flash write likely happens on 2/3s of the pattern unloads.
It could be argued that isn’t much, or is too much.

Thought I might finally weight in. I haven’t so far because I generally come around to eventually appreciating that whatever Wizard decides was actually the most thoughtful, even when my initial opinion differed.

While UI control values are scoped to the current pattern (by the current filesystem schema), I’m with zranger1 in that the most exciting use cases are across-patterns. One Pixelblaze is a trust zone similar to a domain, each pattern is like a page, so it kind of feels like I’m looking for the equivalent of Window.localStorage.

Common use cases:

  1. Setting a color palette that other patterns on the board will consume (e.g. in a playlist).
  2. Setting a speed, or fade-in/fade-out time for all patterns
  3. Setting a transform API argument (e.g the aspect ratio for rectangular matrices, or a current rotation angle)
  4. Receiving a value once from websockets and being able to store it for future patterns in a playlist.

This means I’d really like to support string-like names or keys so that accessing them across patterns is semantic. I imagine saveSlot(1, x) and readSlot(1, x) is not going to work well across my own patterns, but especially poorly if people are importing other people’s patterns from the library (unless scoped to the current pattern). I don’t necessarily want to add a string type to the lang, but maybe this is appropriate ES6-compatible interpreter magic; under the hood you could store them in the filesystem using the same string-ish methods Wizard uses to store named patterns and named controls (just not under each pattern’s directory).

You might consider namespacing weird board-specific concepts like a decaying flash ram, and using scary function names for things that eventually die:

Board.flash.burnItem(key, value)

Perhaps the board can store a global write count and disable burnItem() with an editor error when a threshold has been passed? Just throwing out ideas here.

I believe most flash on ESPs can handle 10K-100K writes before problems might show up, so preventing per-frame or per-render writing is really important.

Therefore, after following this line of reasoning to its conclusion, I think I’m in favor of a namespaced KV store that only writes values to flash on pattern-unload. Yes, you lose the values if you’ve changed them and the board loses power. I think this also makes the decorator or export persist var approach attractive. I do love memcached and Redis, and I generally capitalize singletons, so if not using an additional keyword or decorator, consider the less-magic:

BoardVars.set(key, value) // Only written to flash on unload

Hmm.


[quote="wizard, post:16, topic:1735"] If your code pushed a new state for a control, it would make sense to reflect that in the UI so that they were consistent. [/quote]

I actually asked for this a few times, as it’s been lovely in some other paradigms that supported 2-way data binding, state<->UI control. A common example of how this could be useful is when you import a pattern that has a default value for a variable controlled by a slider or picker: We’ve all experienced how the initial slider or picker state is commonly wrong on first import. Some sliders should start life in the middle of their range.

2 Likes

I like the key/value idea… Works on singlet patterns or global usage.

Since all flash consumption is currently user driven, I could certainly see @wizard’s hesitancy to add anything that could eat flashes, by accident or on purpose. It would suck to have to say “oh sorry, your PB is worn out flash wise, no way to fix it.”

Any alternative ideas? An external API that could push a value (and pull a value back) could be interesting for example. “Cloud Storage”?

@jeff, good stuff, thank you for your thoughts. I always like hearing your ideas, and I borrow a lot from them :grin:

My plan for aspect ratio is to handle that on the mapper tab so you don’t have to code around it. Perhaps a dropdown with these options:

  • fill - This is default. If necessary, the map will be stretched or squished to fit every dimension.
  • contain - The map keeps its aspect ratio, but is resized to fit the largest dimension.
  • cover - The map keeps its aspect ratio, but is resized to fit the smallest dimension. Some of the map may go outside one world unit. Centered around 0.5.

Implementation-wise this is an initial transformation matrix that is applied. The map dimensions would still be normalized to take advantage of the full resolution. The code has this now, but it’s set to an identity matrix (same as fill). I think contain would be transparent to patterns written for world unit coordinates between 0-1, they just wouldn’t be asked to draw the entire world. With cover you might see coordinates out of that range, and most existing patterns would handle that OK, but not all.

I like the local storage analogy BoardVars, and string keys would be much easier to use and keep straight than save slots, if a bit harder to implement. I still have concerns if it is used in a playlist and persists each time, I wouldn’t want to write to flash every time the pattern cycles automatically.

@Scruffynerf, right my main concern is doing something that would create a bunch of returns, frustrate customers, and/or shorten PB’s useful lifetime.


I’m starting to see these as potentially 2 different, not necessarily unrelated things:

  • A system for sharing state across patterns, or within a single pattern between runs.
  • A system for saving state between resets, in flash.

These could work together or independently.

I kicked off this new thread because I found @Petros’s use case for physical controls interesting, plus it’s slightly different from previous discussions. Looking back, I didn’t see an explicit request to save those values on flash permanently, just between pattern switching, but that is where my ideas tended toward.

I could certainly see persistent state useful for the multi segment pattern, are there other asks for persistence specifically?

If these are separable features, I’m happy to focus on the shared state first.


@zranger1, I’m coming around to see what you meant with system/library/palettes.

We’ve previously talked about global code, and I think there’s ideas there that blend with some of the asks here. Global code with its own state could solve much of the non-flash use cases if it persists between patterns.

Imagine that global code could be done in an extension or plugin kind of way. Like patterns, you could install one or more of these extensions. Extensions are a module (just like a pattern) and could export variables. Unlike library code, extensions have a lifecycle outside of the pattern itself.

On the pattern side, we’d use the import directive to consume exports from an extensions. Import can fail if the module doesn’t exist, and the importer doesn’t define the values/defaults. Import accepts a string/path, can specify which identifiers are of interest, and rename them for local use.

An extension could share an array (an in-memory saveSlot if you will), or expose an API:

export var sharedArray = array(100)
var counter = 0
export function callCount() { return ++counter }
var mode
export function getMode() {return mode}
export function setMode(newMode) {mode = clamp(newMode, 0, 10)}

And used in a pattern, renaming as needed:

import {sharedArray as foo, 
        callCount as bar, 
        getMode, 
        setMode
    } from 'extensions/myExtension'

Note that import would copy the exported var to a local var, so you can’t assign a new value to an imported variable and have it change the extension’s state, but an array is a reference and could be mutated, and functions could be called which can handle pretty much anything else.

You could implement getters/setters for a set of names instead of exposing an array for a bit more structure to keep things clear. Not quite the same as a key value store.

I think extensions might have a slightly different API / convention, but I could see how shared UI controls, access to gpio, etc. would let you write an extension to handle a lot of that stuff and share it across patterns.

2 Likes

I’ve been following this thread for a while and I don’t remember if this has been suggested, so:

The Global section could be like the mapper: another code tab where any variable introduced is persistent between patterns, and for burning to flash there’s a Save Code button and a Save Values button.

I do like the idea of enabling multiple such scripts at a time, since you might have a shared color value between all of your patterns, but want to swap out an easing or timing function separately.

Both the mapper and this global code section would benefit from storage of multiple scripts. :relaxed:

3 Likes

@sorceror There is a simple elegance to this that’s very appealing. I think it’s very Pixelblaze.

With zero desire to rebuild NPM, I wonder if this could be simple enough to encapsulate in a .EPE, though backwards compatibility needs a lot of thought. Now that upgrading is one-click and pretty reliable, I’d also advocate for less backwards compatibility to minimize complexity.

I’d support the notion that EPEs include a minimum required firmware version, and the UI tells you if you need to upgrade because this EPE uses global code/values.

1 Like

My case was I believe a bit different to what is being discussed here. I was looking for a way to “pass” the new hue value to the picker/slider when it changed because of the press of a button. I’ll try to describe it a bit better: In a pattern, I might have both a hue slider, and a button that will advance the hue value, giving the ability to change it without having to use the phone. Either can be used at any time, but in the end it’s the new hue value that has to be saved, regardless jf it was altered via button or slider.

2 Likes

Ah, that does help!

Idea for posting data back to a control, I think an API call with the name would be the way to go (and would need some basic string support).

updateControl('MySlider', 0.5)
updateControl('MyRGBColor', 0.5, 0.2, 0)

I’m thinking this would then trigger control handler functions when called via code.

If called inside the same control handler it shouldn’t trigger another update (no infinite loops), but would be useful if it still kept the updated value.

If called from other control handlers, it might set off a chain-reaction, but I think to avoid an infinite loop I would keep a flag and only fire any given control handler once per original call.

This way you could make some controls dependent on others with feedback in the UI, but won’t shoot yourself in the foot too easily. I don’t exactly have a use case for that in mind, but it might open up some interesting applications.

@jeff, is this close to what you were looking for with 2-way data binding? Too complex, unforseen gotchas?

2 Likes