[Storage] Add support for storing some data in External EEPROM

[tonhuisman]

Resolves #1011
Forum requests: here here here here here here

Features:

• [Storage] Add settings for External EEPROM, supported models: AT24Cxx (xx: 32, 64, 128, 256, 512, 1024 and possibly AT24C2048), and also FRAM models: MB85RCxx (xx: 64, 128, 256, 512, 1M).
• Save Task values that are stored in RTC also in EEPROM, when enabled per value in Device page
• Save numeric values in a 'slot' (float resolution) using WriteEE,<slot>,<value>, limited to max. 1024 slots (less on smaller EEPROMs)
• Erase entire 'slots' range using command WriteEE,erase,erase
• Read numeric values from a 'slot' using [ReadEE#<slot>] generic Rules variables syntax
• Fetch max. number of available slots using [ReadEE#max]
• View current used 'slots' in the External EEPROM values page (/eepromvars endpoint), available from the Tools page, hiding currently stored Task Values by adding ?tasks=0 and showing also disabled tasks by adding &enabled=1 to the page-url (advanced feature 😉) For 0-values no hex value display is included.
• [CRC] Calculate checksum reading from a function
• Add settings on the Hardware page for restoring settings from EEPROM at Cold and/or Warm boot
• Add check for write-ability of EEPROM, and set to ReadOnly mode if we can't write (WP (write-protect) can be enabled by a jumper on some boards)
• Add command WriteEE,Check,WP to re-evaluate the write-ability of the EEPROM
• Get Write-protected status using [ReadEE#WP], 1 = write-protected
• Show Write-protected state on Hardware page
• Show the available nr. of slots on the Hardware page
• Separate EEPROM (AT24Cxxx) and FRAM (MB85RCxxx) models in the type selector (general term is still EEPROM)
• Add a checkbox per Task value to enable/disable storing the value in EEPROM, EnabledDisabled by default, only visible if the EEPROM (or FRAM) is enabled and available.
• Store and check some versioning and system parameters in EEPROM, avoiding incompatible data to be restored.
• All EEPROM related code moved to new ESPEasy::eeprom namespace for better structure and maintainability.
• Saving of UserVars to EEPROM is now handled in the 'background', to avoid a large time-penalty when a variable is saved in EEPROM. A save interval in seconds can be set to spread the load a bit more (see screenshot)

TODO:

• External EEPROM data layout to be determined
• Implement actual storing of data
  • UserVar (not cleared on cold boot)
  • GPIO Pinstate buffer
  • Rules variables via command (WriteEE command)
  • String variables via command
  • [C016] Cache elements
  • ? Other ?
• Implement loading/restoring from EEPROM
  • UserVar (if checksum is correct) user configurable on Cold and/or Warm boot
  • GPIO Pinstate buffer
  • Rules variables via variable expansion ([ReadEE#<slot>])
  • String variables via variable expansion
  • [C016] Cache elements
  • ? Export values to json ?
  • ? Other ?
• Testing
• Update documentation

Hardware page configuration: (updated: 2025-08-31)

[chromoxdor]

@tonhuisman I got my EEPROMs today. Tell me when and what needs to be tested or how I can be of assistance.

[tonhuisman]

Ah, great 😃

You can of course test if the Task Values and Rules variables (writeEE,n,value / [readEE#n]) (NB: n = range 0..MAX-1 !) are restored correctly once you have enabled the corresponding settings on the Hardware page (see screenshot above). Restore should be successful after a power cycle 😄 to simulate the Cold boot situation.
And once you have some Rules values stored, you can view the /eepromvars endpoint, and even add ?tasks=1 to the url to also see the stored values per task.
For Cache values (C016) there's no 'restore' or 'view' yet.

NB: I'm thinking of limiting the Rules vars to f.e. 1024 (less on smaller EEPROM sizes!), so more space will be available for Cache values,

[chromoxdor]

Ok... i am already in testing mode (I didn't wait for your reply :P )

Everything seems to work fine. (I have everything covered you mentioned)
But two things for now.

1. Taskvalues are written to the EEPROM when they generate an Event. But often (at least in my case), I have multiple tasks that have an interval of one second. This would be a little bit too many write cycles for an EEPROM over time (not for an FRAM, of course). Maybe it would be better to choose which tasks will be written to the EEPROM. Just keep the layout for all tasks on the EEPROM but only write when enabled in the task? What do you think?
2. Why not showing zero values on /eepromvars? It shows, that this slot has been already used. (0 vs. NaN)

[chromoxdor]

NB: I'm thinking of limiting the Rules vars to f.e. 1024 (less on smaller EEPROM sizes!), so more space will be available for Cache values,

I see no reason not to do it that way unless somebody really wants to write a lot of vars :)

[tonhuisman]

1. Values are only written if the data is different from what's already stored, to avoid wearing out the EEPROM
2. Showing all values would generate a page of about 'a mile long', so when using value 1, 128 and 256 you would really be searching for your value..., especially on a small device (phone?) On FRAM, the values are initially 0, on EEPROM they are initially NaN, so both are ignored. Managing the Rules variables is the users' task, not ESPEasy, IMHO

[chromoxdor]

Would it be complicated to make the number of Rules var slots user selectable and put them at the end of the EEPROM and use the rest for C16? The ammount of user selectable slots could be also written to the EEPROM insted of storing it on the flash....

2. On FRAM, the values are initially 0, on EEPROM they are initially NaN, so both are ignored.

I get it. Would make a user-selectable amount of slots even more useful :)
But at least for the EEPROM, it would be OK to differentiate between 0 and NaN for convenience. (at least for the user.. not for the code size ;) )

[chromoxdor]

1. Values are only written if the data is different from what's already stored, to avoid wearing out the EEPROM

But for an analog reading of a LDR it still could mean 1 write per second

[tonhuisman]

I'll see where too put that extra checkbox per value 😅 I have already found a bit per value so I can store this setting 😎

Btw, I think it should be Enabled by default, and user can decide to turn it off

Ah, and there's a particular reason not to skip values, as I'm also storing the Checksum for this data, and validating that against the calculated checksum, so when skipping a value the checksum won't be correct anymore 😱

[chromoxdor]

I'll see where too put that extra checkbox per value 😅 I have already found a bit per value so I can store this setting 😎

Maybe it could also be stored on the EEPROM/FRAM?

[tonhuisman]

I'll see where too put that extra checkbox per value 😅 I have already found a bit per value so I can store this setting 😎

Maybe it could also be stored on the EEPROM/FRAM?

Hmm, not sure if that's such a good plan

[chromoxdor]

Hmm, not sure if that's such a good plan

Me neither :) Just thinking out loud...

[tonhuisman]

But for an analog reading of a LDR it still could mean 1 write per second

What to do for Taskvalues that are not stored, store 0 or NaN (once)? So the storage will be 'clean' for disabled values?

[chromoxdor]

What to do for Taskvalues that are not stored, store 0 or NaN (once)? So the storage will be 'clean' for disabled values?

Hmmm.. very good question. I would make sense to use NaN. But that can not be represented (only when choosing a specific other value for it i guess?)
The only thing I could think of would then be to store en/disabled in a bitmap on the first bytes to have a flag... but my knowledge is limited. 🤷‍♂️

Edit: ahhh since it is float 0xFFFFFFFF for NaN would be it right?

[chromoxdor]

The GPIO state could be saved in a bitmap and another bitmap for "en/disabled saving state".

[chromoxdor]

I am probably stating the obvious by the memory layout could be like:

Offset (bytes)	Size (bytes)	Purpose	Notes
0	8	GPIO enabled/disabled state	64 bits → 64 GPIOs max
8	8	GPIO state	64 bits → 64 GPIOs max
16	128	Task values	Fixed block
144	N × 4	Rules variables (slots)	User-selectable count of 4-byte values
...	Remaining	C016	Uses rest of EEPROM

[tonhuisman]

For GPIO pins we want to provide a way to store the states for MCP and PCF GPIOs, so 64 bits isn't going to cut it. And maybe we should store some more data, like PWM data and/or input/output, haven't coded anything for that yet. It's still slow-cooking in a background thread in my head 🤓

And the same is the case for String type variables; rewriting the entire data blob when a string length changes isn't very efficient, and neither is leaving 'growing space' between entries, as that'll often prove to be too small, causing another blob-rewrite, or leave much unused space 🫨

[chromoxdor]

It seems a value gets written no matter what, or there is an issue with the log.
The eeprom page is showing "-" for all values, as I would expect.

Log:

476918: Dummy: value 1: 95.00
476919: Dummy: value 2: 0.00
476920: Dummy: value 3: 0.00
476921: Dummy: value 4: 0.00
477285: EEPROM: UserVar: 4 bytes (359.99 ms) written to AT24C256

[chromoxdor]

or there is an issue with the log.

Since I can see the I2C working because SDA shares the GPIO with the internal LED of the ESP, I am confident something gets written for real.

[tonhuisman]

Yes, the checksum still gets written, I will fix that

[chromoxdor]

I could immediately make use of this for a new coffeemachine2ESPEasy conversion project. And pressing the buttons, it is also a good indicator for continuous writing to the EEPROM as it starts lagging as hell.

[tonhuisman]

477285: EEPROM: UserVar: 4 bytes (359.99 ms) written to AT24C256

Hmm, that's not looking nice, 360 msec added on a PLUGIN_READ 😞
I'll take care of that by moving the EEPROM writing to a separate RTOS task (ESP32 only), as this kind of delay on every value update isn't acceptable.

[tonhuisman]

Saving of the UserVars to EEPROM is now handled in 'the background', though not in a RTOS task as using the I2C bus for a few-hundred milliseconds isn't easily made thread-safe (and quite undesirable too).

[chromoxdor]

Yes, the checksum still gets written, I will fix that

This doesn't seem to be fixed 😳
EEPROM: Write UserVars success 0 in 46 msec.

But it is much faster :)

[tonhuisman]

The 0 is the number of bytes written, so...

[chromoxdor]

The 0 is the number of bytes written, so...

Ahhh... And I was wondering why there is zero success 😁
Actually i thought it would be something like that but the I saw the internal LED blinking (the one that shares the GPIO with SDA) when this in the logs appeared. Why is there I2C communication at all when there is nothing to write?

[tonhuisman]

It's reading to compare data before writing, so there will always be activity on I2C.

[chromoxdor]

It's reading to compare data before writing

But still... if there is nothing to write, then there is no need to read and no need for I2C activity which then, even if it is only around 40ms, doesn't occupy the I2C bus.

[tonhuisman]

It's reading to compare data before writing

But still... if there is nothing to write, then there is no need to read and no need for I2C activity which then, even if it is only around 40ms, doesn't occupy the I2C bus.

To be able to compare data, the current EEPROM-data must be read to compare to the new data. We don't want to (read: can't) keep a buffer in memory that reflects the EEPROM content.

[chromoxdor]

To be able to compare data, the current EEPROM-data must be read to compare to the new data.

Sorry, I don't want to be a pain in the ass 😁 but it seems i don't get it.
Assume I don't need the ability to write task values and disable them all. (only maybe the var slots in rules) why would i need a comparison of anything for a task? Every active task running causes a read that is unnecessary then...

[tonhuisman]

First thing that's done is a validity-check of the current EEPROM: https://github.com/letscontrolit/ESPEasy/pull/5382/files#diff-87033636f7a2095187789ef49c13c2e2767c2e1d8eb19618aa17d932dd58ab53R175-R182

That involves a 'couple of bytes' being read, but those are not counted, and even if the params are not OK, the writes to update the params are not counted.
I could add a static or global bool to store the first params state, and never read/check it again (similar to the read-only test I do at startup), but you can in fact replace the EEPROM while the unit is running, and it could write (valuable) UserVar data in a clean EEPROM without correct params data, and then it won't restore that as that params-state is required to be OK before doing a restore on Cold or Warm boot 😨

NB: AFAIK, reading from an EEPROM is harmless to the device, as opposed to writing, that will eventually wear out the chip. (Most EEPROMs are specced at ~10⁶ writes per byte, though I haven't validated that.)

[tonhuisman]

Every active task running causes a read that is unnecessary then...

There currently is scheduled only a single write action per task update, if a new update comes while the data is still being processed, the new write action is ignored, as it will be handled on the next 'run'.

[chromoxdor]

NB: AFAIK, reading from an EEPROM is harmless to the device, as opposed to writing, that will eventually wear out the chip. (Most EEPROMs are specced at ~106 writes per byte, though I haven't validated that.)

I know. It’s not so much about the cycles but about the time that the read occupies the I2C bus. If you have multiple tasks with a 1-sec interval and let this be multiple I2C devices, then there is a lot of stuff going on on the bus. I even have some tasks with a 100-ms (or was it 50ms...) interval getting the analog data of a light sensor (LDR)... I would think that theoretically could lead to issues.
I would maybe, for swapping between EEPROMs, do an initialization option to prepare it for the new device....

[chromoxdor]

and even if the params are not OK, the writes to update the params are not counted.

to get a better understanding.. what params are we talking about?

[tonhuisman]

what params are we talking about?

See: https://github.com/letscontrolit/ESPEasy/pull/5382/files#diff-fa9213718305321296c98a66921379e1f7d2b6e78f02155044ff828739556a8cR77-R84

• Version (EEPROM content-version, currently 1)
• Max. tasks
• Vars per tasks
• RTC Cache address (in EEPROM, when included in the build)
• Pinstate address (in EEPROM)

[chromoxdor]

Would it be a problem to treat a task, where all "Eeprom" fields are disabled as an disabled task?
From the EEPRM point of view it would make no difference right....

[chromoxdor]

BTW: 0 is still not being displayed as a slot value in /eepromvars.

[tonhuisman]

Would it be a problem to treat a task, where all "Eeprom" fields are disabled as an disabled task? From the EEPRM point of view it would make no difference right....

Well, yes, that's the intention, but I now see that some improvements can be applied in the code. There's also the feature that I write a NaN value if the taskvalue is not enabled, but currently that's re-checked every time for an enabled task. Will fix that soon-ish.

[tonhuisman]

@chromoxdor and others that are building this PR locally:
⚠️
There has been a breaking change in the settings for enabling the storage of task values in EEPROM (there was overlap with another setting I added elsewhere), so please check your configurations! (this is of course the risk of working with PR code: Things may change or break 🫣)
⚠️

[chromoxdor]

Thanks for the heads up! What’s the other setting?

[tonhuisman]

What’s the other setting?

MQTT State Class per Value