File write using SPIFFS under Sloeber 4.4 sometimes puts incorrect characters in file

[danny-levinson]

Environment

Using Sloeber 4.4 product on Macos 10.13.6; compiling for Wemos D1 mini board, using esp8266 platform 3.0.2 (SloeberProduct-V4.4.0_mac64.2021-07-04_06-48-31).

Problem

Some specific combinations of writes & seeks put DEL characters (0xff) into files, usually in place of CRLF (0x0d+0x0a). This problem only presents on Sloeber with SPIFFS. Compiling on the Arduino IDE or using LittleFS instead of SPIFFS on Sloeber eliminates the problem.

Here is an example program demonstrating the problem behaviour:

#include "Arduino.h"

#include <FS.h>

#define USE_SPIFFS X
#ifdef USE_SPIFFS
#define FileSys SPIFFS
#else
#include <LittleFS.h>
#define FileSys LittleFS
#endif

void setup()
{
	/*
	 * write out 80 x's, then go back 6 characters and write 4 y's and a CR-LF
	 */
	FileSys.begin();
	File f = FileSys.open("/writetest.txt", "w");
	f.print("xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx");
	f.seek(-6, SeekCur);
	f.print("yyyy\r\n");
	f.close();

	/*
	 * dump the last 10 characters as bytes in hex
	 */
	unsigned char buff[120] = "";
	char cbuff[120] = "";
	f = FileSys.open("/writetest.txt", "r");
	f.read(buff, 80);
	for (int i=70; i<80; i++) {
		sprintf(cbuff+(i-70)*3, "%02x ", buff[i]);
	}
	f.close();
	/*
	 * print the dump out to the serial monitor
	 */
	Serial.begin(115200);
	Serial.printf("output: |%s|\n", cbuff);
}

void loop()
{
	delay(100);
}

Incorrect output using SPIFFS

SDK:2.2.2-dev(38a443e)/Core:3.0.2=30002000/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-48-g7421258/BearSSL:6105635
output: |78 78 78 78 79 79 79 79 ff ff |

Expected output produced using LittleFS

SDK:2.2.2-dev(38a443e)/Core:3.0.2=30002000/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-48-g7421258/BearSSL:6105635
output: |78 78 78 78 79 79 79 79 0d 0a |

Notes

My apologies if this is the wrong forum for this problem, but I can only reproduce the bad behaviour using Sloeber. I have tried it out with a second Wemos board in case the first had a fault.

Deleting or adding an 'x' or a 'y' from the example above makes the problem go away. Also, it is possible the problem crops up without any seeks but I have not yet isolated an example.

I have now switched to using LittleFS, but I thought I should report this anyway. I would appreciate knowing if someone else can reproduce the problem.

I first encountered the problem using Log4Esp and it may be related to this report: hunsalz/log4Esp#9; however I was able to reduce the code to the short example above that does not use the logging library.

Project properties

[jantje]

Thanks for a extensive research and problem description.
If the Sloeber compiled program's behaviour is different from the arduino IDE's compiled program's behaviour there must be a difference in the command line instructions.
Can you compare these commands for differences?

[danny-levinson]

Here are the compile logs from the two systems. I'm not including the library compile. My Arduino installation has not been updated for quite a while so there could be a lot of differences.

Sloeber log:
compile-log-sloeber.txt

Arduino log:
compile-log-arduino.txt

I'm guessing the difference is either in the SPIFFS implementation or in the core functions that they reference. I tracked down what I think are the files involved and diff'd them, after first copying the arduino library and changing the names of the .c files to .cpp. Nominally both versions say they are 0.3.7 in the README.md file, but clearly there are lots of differences. I think these are mostly cosmetic, i.e. in casts, deprecated declarations, etc., but I can't say for sure.

spiffs-diffs.txt

[jantje]

I see that on Sloeber you use esp8266 platform version 3.0.2 and on Arduino IDE you use platform version 2.5.0
Given that you are likely to hit differences

[danny-levinson]

Your are right. I've been putting off updating because it often breaks my old code, but now I don't use the Arduino IDE very much. I updated all the boards and libraries and now the Arduino IDE also produces the bad behaviour. In that case, this is the wrong forum. I'll look at posting the issue on the SPIFFS site instead.

Thank you for your help, and thank you for an excellent development system. I was previously working with Doug Schaeffer's Arduinocdt, but I'm making much more headway now that I've switched.

[danny-levinson]

Issue opened on SPIFFS repository: pellepl/spiffs#281 (comment)

[jantje]

I was previously working with Doug Schaeffer's Arduinocdt, but I'm making much more headway now that I've switched.

I never liked Doug Schaeffer's Arduinocdt (and even less his attitude towards Sloeber) but I always keep in mind ...

1. Part of the code of Sloeber started from a copy of Doug Schaeffer's Arduinocdt (part of the boardsmanager)
2. Part of the code of Sloeber started from a copy of Doug Schaeffer's cdt (the gnumakefile generator)
3. Sloeber runs on Doug Schaeffer's cdt

So without Doug Schaeffer no Sloeber

[danny-levinson]

Arduinocdt was far, far better than the original Arduino IDE and yours is better still. I really like the way you've made the libraries & core visible.