I guess I still don’t understand how sliders and pickers work then? What decides to persist now?
The UI? Whatever the UI talks to?
Yes, the UI sends down new slider values. After a while, it sends them with a save command.
When a pattern loads, the engine reads saved controls and calls the appropriate functions.
1 Like
Ok, so given my new understanding:
The sliderMySlider function, does two things:
- Tells the UI to add a slider
- Acts as a function
The UI sends updates to PB saying:
- sliderMySlider value is v
- Save v persistently as sliderMySlider
There is currently NO way to do the following:
- Push a value to a slider (no PB> UI commands)
- Push a value into persistence (no PB > flash commands)
Those seem the minimal requirements going forward. Nobody is asking (that I know of) for PB>UI commands, though a bunch of us want more UI options like buttons/switches (on/off) which is really a binary slider (so UI change but no functionality change).
So some way to push a value into persistence is the real ask here. Given concerns of flash wearout… How does slider persistence (which is basically move the slider, when you stop, single byte is saved in flash on a request from UI) aboid wearout? Solely by not having save constantly?
So the key to avoiding accidentally wearing out the flash is to make persistent variables avoid constant saving.
In my mind, this absolutely means NOT using a var approach. People tend to use vars for changing variables and accidentally using a rapidly changing variables would cause rapid flash writes, unless some sort of throttle existed.
Function still makes more sense to me:
We can control how often we call a function, like on a button push, hopefully with a key debounce.
Abuse is possible, but it would be easy to add protections since we can write code to avoid rapid writes. We could try to control frequency of write, or time out (delta?)
A Save argument to make it explicitly push to flash, seems pointless, I can write a normal function otherwise to do the above, and within it call the Persistence function, rather than make the persistence function not always persist.
Simple is better. Warn people that this function should be called infrequently to avoid wear.
So I think we’re back to my original request:
We’re asking the PB to persist Name as value.
You don’t need to worry about myVariable
And we never get access to Name, just can send a byte (0…1) or a 16.16 to it, knowing it’ll last between runs of the pattern.
This could give, as an example of other uses, a pattern run counter. First time thru your PB, the counter is zero. Not as part of the looping code but as part of setup code, the counter is incremented, So each run of the pattern is one higher (32k runs). This could be used as a seed value so every run of pattern is unique (mod 32k). Or every 5 runs, do something, as an example.
Absolutely still abusable, so maybe you need to add a way to disable persist saves? global Setting?
Oooh ooh ooh…
Where is the slider/picker value stored? Its not in the .epe, right? (Is a loaded pattern still in flash as .epe, or unpacked, or what?)
It’s connected to the pattern ID? So sliders don’t collide with similarly named slider of another pattern?
What if there was a wildcard storage? Where name collision could happen? Then global persistent sliders could happen? So…
export function gsliderGlobalSpeed(v)
would store the slider value in the wild card/global storage instead of the pattern ID storage bucket… And all patterns would share that value.
Actually - the var approach - using a decorator or keyword near the variable declaration, then dealing with persistence only at pattern load/unload time - makes it so you can save users from doing catastrophically clueless things with the persist function, like calling it from render(). They will do that. You know they will.
So, ideally, the eventual solution wouldn’t require the user to know or care much about the inner life of an ESP32.
Also, we might want a way, besides deleting the pattern, to get rid of stored variables. Possibly just letting a pattern call clearPersistCache() or something would be enough.
2 Likes
The epe is generated on the fly when you want to export a pattern and is in a portable JSON format that is easier to parse. It includes most of the important portable bits from the pattern like the source code, preview, name, etc, but doesn’t include the compiled binary of the pattern which isn’t necessarily designed to be portable between PB variants (e.g. v2 and v3).
The actual patterns on PB are stored at /p/<id> in an efficient packed binary structure and controls data is stored at /p/<id>.c in JSON in a separate file. Controls data are much smaller than the pattern and almost always fit inside one filesystem block and cause a minimal amount of flash wear.
Perhaps, but it is easier to think of the value as a single thing, and useful in a network scenario. I have heard that when the brightness setting changes (from any client), that a websocket message should be emitted so that other clients can update their UI state. If your code pushed a new state for a control, it would make sense to reflect that in the UI so that they were consistent.
That still doesn’t give the engine a way to get the value if you call that from inside your code, unless it does something sneaky in the compiler/VM. It would have to inject code with API calls into your function definition, or set traps to intercept the function call itself. Exporting a var/function just exposes the name and variable (in this case, the var holds a function), and for controls that is enough to work out the UI control type and name, and call that function with some data.
It is possible with a single exported function, but to do this with a single function would be complicated. First lets look at how a pair of functions could work:
var foo = 32 // default value when the pattern initializes
export function loadPersistentMyName(valueFromStorage) {
foo = valueFromStorage // this happens IF there was a previously saved value
}
export function savePersistentMyName() {
return foo // when this function is called, we return the value we want to save
}
The trick/downside here is that the MyName has to be the same between both functions or it will mysteriously break. There’s no link between the foo variable and the MyName persistent control outside of the function implementation – the engine is unaware of foo, only a persistent thing called “MyName” and the 2 load/save functions for it.
The engine would call loadPersistentMyName() after the main code initialized with a previously saved value, or it won’t call it at all, just like a control.
At some point or points in the future, the engine could call savePersistentMyName() and save the return value for a persistent value called “MyName”, probably along with controls data.
To combine these into a single function, things get more complicated. This function has to handle both cases:
var foo = 32 // default value when the pattern initializes
export function loadOrSavePersistentMyName(action, maybeValueFromStorage) {
if (LOAD_ACTION == action) {
foo = maybeValueFromStorage // a loading action is called IF there was a previously saved value
} else if (SAVE_ACTION == action) {
return foo // when this action is called, we return the value we want to save
}
}
To my eye, this is much more complex and requires more boilerplate code, with the only benefit being that you don’t have to worry that “MyName” is consistent across a pair of functions. This handler function is doing too may things, and doesn’t behave like a proper function anymore (conditionally returning a value, or having no return at all).
A single var approach could be done, and thinking about it a bit more, should be a naming convention + export so that PB code could continue to be compatible with JS, but might have a bit too much or too little magic.
The engine could change the value just after initialization code ran, and read it when necessary, and would have the default value during pattern init. The code wouldn’t have a way of knowing if/when the value changed and potentially invalid values could be injected, and would have to ensure the variable wasn’t holding intermediate values when it could be read for saving. The pattern code would have to be more defensive about how it used that variable everywhere.
3 Likes
I liked the saveSlot / readSlot idea, or another storage api like function as it solves the problem, allows greatest flexibility, and doesn’t add some maybe unexpected behavior.
1 Like
Another interesting set of issues: What can be stored and how much? Single values? (That would be my preference for an initial version) Arrays? (What happens in the case of nested arrays?)
Are there limits on per-pattern storage, both in terms of size and number of variables? Is there a limit on overall variable storage space?
I’m still not fond of giving users the opportunity to beat their flash ram to death at 100fps via an unrestricted API. It’ll result in a ton of service calls.
Sure, those of us who are already experienced programmers can be trusted to restrict the rate, but as we’ve seen from the latest tutorial poll, there are a lot of people who are more into the LED art construction side of things. They’re really important to Pixelblaze’s future. Probably more so than we are. It helps them out if we can keep the “friction” of coding - and the possibility of bricking your Pixelblaze - as low as possible.
2 Likes
I actually agree this is the best approach to avoid unintended flash wear by rogue programmers who use any user initiated method. If @wizard can make it seamlessly behind the scenes, that would help. Not perfect, as if you ran a pattern and the PB rebooted/powercycled, you’d lose the current value but at least any intentional change away would record the current state.
Assuming this hypothetical storage api saveSlot could protect the flash by committing when necessary or at an interval - however I don’t think it’s a huge cause for concern it’s not like the beginner first time pattern is going to be looking to this api.
I was thinking of implementing it so that the save slot values were held in memory until the pattern gets unloaded, so you could write to them as often as you like without worrying. Similar to the load/save handler functions but API driven instead of magic exports.
I’m thinking it would be suitable for perhaps a dozen or so scalar values. The example use case is for storing a few modes/settings that change from button presses.
These should be items that you want to be persistent, and ideally based on user input, not intended as a way to capture the intermediate pattern state or pixel buffer so that it seamlessly resumes when switching patterns. Doing that could cause excessive flash wear e.g. if used in a repeating playlist.
Which reminds me, in general all of the persistence in PB today originates from a user interaction. There’s nothing automated that can wear flash out intentionally.
Perhaps this use case can be solved in another way entirely.
What if you could define a pin as a hardware button control that supported a stepped mode selection. That would be a whole lot less button handling code, and the persistence could happen automatically as currently done with UI controls. It’s not quite as flexible, but this seems like a common use for a button.
pin = 26
numModes = 4
pinMode(pin, INPUT_PULLUP)
persistentModeButton(pin, FALLING, numModes, modeChangeHandler)
function modeChangeHandler(mode) {
// called after init with saved values, or any time button is pressed
// mode ranges from 0 to numModes-1 before wrapping back to 0
}
It’s like registering an on change callback, but the system can do all the button debouncing, state incrementing, and persistence logic. Setting numModes to 2 would give you a simple toggle button.
If I go this route, I would add a non-persistent modeButton() version of the API as well.
The caveat here is that this would allocate some resources, so it wouldn’t be suitable for reconfiguring on the fly unless it was changed to a register / unregister kind of API pair.
1 Like
I like persistentModeButton()/modeButton() a lot! Bet you’d see a lot more button-controlled builds.
Let me switch sides here, and give my use case for the saveSlot() approach too. It’s pretty simple – it would make the Multisegment pattern (and similar things) actually workable without an external controller to save the per-segment parameters.
Right now, it uses one controller to select a segment, and at least one other to set parameters for that segment. When you leave the pattern, the control settings are saved, but the accumulated segment settings are not. With even the ability to save a few scalar values, they could be.
1 Like
I’m not sure what you’re proposing here, as my use case wasn’t pin or button based.
I tweaked a handful of settings (sliders and pickers) and I leave the pattern and come back, and it’s restored. The use case here is to have a way to save a NON-slider or picker value, and have exactly the same behavior. Perhaps the code wants to record a state, but my example use case above is also good: play counter, or rotating between modes, sans user interaction
So here’s a concrete example using current playlist as the driver:
I build a playlist of 3 patterns.
Pattern one is normal, call it P.
Pattern two has a mode flip… each time it runs, it either does A or B
Pattern three has a mode flip between 3 states, D, E & F
When I play the playlist, the behavior would be
PADPBEPAFPBDPAEPBF (it would then repeat)
This is a pretty simple example but I could come up with more complex (so please don’t ask me to just build the above sequence as the playlist…) My point is, doing a playlist of this level is entirely depending on a way for a pattern to record a value that sticks thru load and is updated on unload by the code. And yes, I’m aware that running this playlist would mean a flash write likely happens on 2/3s of the pattern unloads.
It could be argued that isn’t much, or is too much.
Thought I might finally weight in. I haven’t so far because I generally come around to eventually appreciating that whatever Wizard decides was actually the most thoughtful, even when my initial opinion differed.
While UI control values are scoped to the current pattern (by the current filesystem schema), I’m with zranger1 in that the most exciting use cases are across-patterns. One Pixelblaze is a trust zone similar to a domain, each pattern is like a page, so it kind of feels like I’m looking for the equivalent of Window.localStorage.
Common use cases:
- Setting a color palette that other patterns on the board will consume (e.g. in a playlist).
- Setting a speed, or fade-in/fade-out time for all patterns
- Setting a transform API argument (e.g the aspect ratio for rectangular matrices, or a current rotation angle)
- Receiving a value once from websockets and being able to store it for future patterns in a playlist.
This means I’d really like to support string-like names or keys so that accessing them across patterns is semantic. I imagine saveSlot(1, x) and readSlot(1, x) is not going to work well across my own patterns, but especially poorly if people are importing other people’s patterns from the library (unless scoped to the current pattern). I don’t necessarily want to add a string type to the lang, but maybe this is appropriate ES6-compatible interpreter magic; under the hood you could store them in the filesystem using the same string-ish methods Wizard uses to store named patterns and named controls (just not under each pattern’s directory).
You might consider namespacing weird board-specific concepts like a decaying flash ram, and using scary function names for things that eventually die:
Board.flash.burnItem(key, value)
Perhaps the board can store a global write count and disable burnItem() with an editor error when a threshold has been passed? Just throwing out ideas here.
I believe most flash on ESPs can handle 10K-100K writes before problems might show up, so preventing per-frame or per-render writing is really important.
Therefore, after following this line of reasoning to its conclusion, I think I’m in favor of a namespaced KV store that only writes values to flash on pattern-unload. Yes, you lose the values if you’ve changed them and the board loses power. I think this also makes the decorator or export persist var approach attractive. I do love memcached and Redis, and I generally capitalize singletons, so if not using an additional keyword or decorator, consider the less-magic:
BoardVars.set(key, value) // Only written to flash on unload
Hmm.
[quote="wizard, post:16, topic:1735"] If your code pushed a new state for a control, it would make sense to reflect that in the UI so that they were consistent. [/quote]
I actually asked for this a few times, as it’s been lovely in some other paradigms that supported 2-way data binding, state<->UI control. A common example of how this could be useful is when you import a pattern that has a default value for a variable controlled by a slider or picker: We’ve all experienced how the initial slider or picker state is commonly wrong on first import. Some sliders should start life in the middle of their range.
2 Likes
I like the key/value idea… Works on singlet patterns or global usage.
Since all flash consumption is currently user driven, I could certainly see @wizard’s hesitancy to add anything that could eat flashes, by accident or on purpose. It would suck to have to say “oh sorry, your PB is worn out flash wise, no way to fix it.”
Any alternative ideas? An external API that could push a value (and pull a value back) could be interesting for example. “Cloud Storage”?
@jeff, good stuff, thank you for your thoughts. I always like hearing your ideas, and I borrow a lot from them 
My plan for aspect ratio is to handle that on the mapper tab so you don’t have to code around it. Perhaps a dropdown with these options:
-
fill- This is default. If necessary, the map will be stretched or squished to fit every dimension. -
contain- The map keeps its aspect ratio, but is resized to fit the largest dimension. -
cover- The map keeps its aspect ratio, but is resized to fit the smallest dimension. Some of the map may go outside one world unit. Centered around 0.5.
Implementation-wise this is an initial transformation matrix that is applied. The map dimensions would still be normalized to take advantage of the full resolution. The code has this now, but it’s set to an identity matrix (same as fill). I think contain would be transparent to patterns written for world unit coordinates between 0-1, they just wouldn’t be asked to draw the entire world. With cover you might see coordinates out of that range, and most existing patterns would handle that OK, but not all.
I like the local storage analogy BoardVars, and string keys would be much easier to use and keep straight than save slots, if a bit harder to implement. I still have concerns if it is used in a playlist and persists each time, I wouldn’t want to write to flash every time the pattern cycles automatically.
@Scruffynerf, right my main concern is doing something that would create a bunch of returns, frustrate customers, and/or shorten PB’s useful lifetime.
I’m starting to see these as potentially 2 different, not necessarily unrelated things:
- A system for sharing state across patterns, or within a single pattern between runs.
- A system for saving state between resets, in flash.
These could work together or independently.
I kicked off this new thread because I found @Petros’s use case for physical controls interesting, plus it’s slightly different from previous discussions. Looking back, I didn’t see an explicit request to save those values on flash permanently, just between pattern switching, but that is where my ideas tended toward.
I could certainly see persistent state useful for the multi segment pattern, are there other asks for persistence specifically?
If these are separable features, I’m happy to focus on the shared state first.
@zranger1, I’m coming around to see what you meant with system/library/palettes.
We’ve previously talked about global code, and I think there’s ideas there that blend with some of the asks here. Global code with its own state could solve much of the non-flash use cases if it persists between patterns.
Imagine that global code could be done in an extension or plugin kind of way. Like patterns, you could install one or more of these extensions. Extensions are a module (just like a pattern) and could export variables. Unlike library code, extensions have a lifecycle outside of the pattern itself.
On the pattern side, we’d use the import directive to consume exports from an extensions. Import can fail if the module doesn’t exist, and the importer doesn’t define the values/defaults. Import accepts a string/path, can specify which identifiers are of interest, and rename them for local use.
An extension could share an array (an in-memory saveSlot if you will), or expose an API:
export var sharedArray = array(100)
var counter = 0
export function callCount() { return ++counter }
var mode
export function getMode() {return mode}
export function setMode(newMode) {mode = clamp(newMode, 0, 10)}
And used in a pattern, renaming as needed:
import {sharedArray as foo,
callCount as bar,
getMode,
setMode
} from 'extensions/myExtension'
Note that import would copy the exported var to a local var, so you can’t assign a new value to an imported variable and have it change the extension’s state, but an array is a reference and could be mutated, and functions could be called which can handle pretty much anything else.
You could implement getters/setters for a set of names instead of exposing an array for a bit more structure to keep things clear. Not quite the same as a key value store.
I think extensions might have a slightly different API / convention, but I could see how shared UI controls, access to gpio, etc. would let you write an extension to handle a lot of that stuff and share it across patterns.
2 Likes
I’ve been following this thread for a while and I don’t remember if this has been suggested, so:
The Global section could be like the mapper: another code tab where any variable introduced is persistent between patterns, and for burning to flash there’s a Save Code button and a Save Values button.
I do like the idea of enabling multiple such scripts at a time, since you might have a shared color value between all of your patterns, but want to swap out an easing or timing function separately.
Both the mapper and this global code section would benefit from storage of multiple scripts. 
3 Likes
@sorceror There is a simple elegance to this that’s very appealing. I think it’s very Pixelblaze.
With zero desire to rebuild NPM, I wonder if this could be simple enough to encapsulate in a .EPE, though backwards compatibility needs a lot of thought. Now that upgrading is one-click and pretty reliable, I’d also advocate for less backwards compatibility to minimize complexity.
I’d support the notion that EPEs include a minimum required firmware version, and the UI tells you if you need to upgrade because this EPE uses global code/values.
1 Like
My case was I believe a bit different to what is being discussed here. I was looking for a way to “pass” the new hue value to the picker/slider when it changed because of the press of a button. I’ll try to describe it a bit better: In a pattern, I might have both a hue slider, and a button that will advance the hue value, giving the ability to change it without having to use the phone. Either can be used at any time, but in the end it’s the new hue value that has to be saved, regardless jf it was altered via button or slider.
2 Likes
Ah, that does help!
Idea for posting data back to a control, I think an API call with the name would be the way to go (and would need some basic string support).
updateControl('MySlider', 0.5)
updateControl('MyRGBColor', 0.5, 0.2, 0)
I’m thinking this would then trigger control handler functions when called via code.
If called inside the same control handler it shouldn’t trigger another update (no infinite loops), but would be useful if it still kept the updated value.
If called from other control handlers, it might set off a chain-reaction, but I think to avoid an infinite loop I would keep a flag and only fire any given control handler once per original call.
This way you could make some controls dependent on others with feedback in the UI, but won’t shoot yourself in the foot too easily. I don’t exactly have a use case for that in mind, but it might open up some interesting applications.
@jeff, is this close to what you were looking for with 2-way data binding? Too complex, unforseen gotchas?
2 Likes