Arduino-Pico and SdFat

[XFer012]

Hello,
I'm using Arduino 1.8.19 + Arduino-Pico core v2.4.1 and have SdFat v2.2.0 installed (needed for various other boards).

I'd like to try a few SdFat examples (notably "bench") with my Raspberry Pico, but SdFat v2.2.0 does not build (error below).
By the way I opened an issue on SdFat github.

I seem to understand you ship a fork of SdFat with Arduino-Pico, but it's not picked up it seems (and it's not listed under "Examples for Raspberry Pi Pico").

Any suggestion? How can I "force" Arduino IDE to pick up the bundled SdFat library instead of the standard one?

Build error (for "bench", but it's the same with every SdFat example sketch):

In file included from F:\Arduino\libraries\SdFat\examples\bench\bench.ino:4:
F:\Arduino\libraries\SdFat\src/SdFat.h:452:2: warning: #warning File not defined because __has_include(FS.h) [-Wcpp]
  452 | #warning File not defined because __has_include(FS.h)
      |  ^~~~~~~
bench:69:1: error: 'File' does not name a type; did you mean 'SdFile'?
   69 | File file;
      | ^~~~
      | SdFile
F:\Arduino\libraries\SdFat\examples\bench\bench.ino: In function 'void loop()':
bench:170:8: error: 'file' was not declared in this scope
  170 |   if (!file.open("bench.dat", O_RDWR | O_CREAT | O_TRUNC)) {
      |        ^~~~
bench:193:5: error: 'file' was not declared in this scope
  193 |     file.truncate(0);
      |     ^~~~
bench:236:5: error: 'file' was not declared in this scope
  236 |     file.rewind();
      |     ^~~~
bench:272:3: error: 'file' was not declared in this scope
  272 |   file.close();
      |   ^~~~

[earlephilhower]

You need to use the examples/etc. from the Files->Examples->ESP8266SDFAT, not the SdFat library you've got installed. Those do build on the Pico. You can also use the LittleFS->SpeedTest example (there is a SD mode) or the examples in the SD library.

[XFer012]

Hello, thanks for the quick reply!

I tried the example "ESP8266SDFAT->bench", and get this build error:

In file included from C:\Users\Fernando\AppData\Local\Arduino15\packages\rp2040\hardware\rp2040\2.4.1\libraries\ESP8266SdFat\examples\bench\bench.ino:4:
L:\Progetti\Arduino\libraries\SdFat\src/SdFat.h:452:2: warning: #warning File not defined because __has_include(FS.h) [-Wcpp]
  452 | #warning File not defined because __has_include(FS.h)
      |  ^~~~~~~
C:\Users\Fernando\AppData\Local\Arduino15\packages\rp2040\hardware\rp2040\2.4.1\libraries\ESP8266SdFat\examples\bench\bench.ino: In function 'void cidDmp()':
bench:103:19: error: 'cid_t' {aka 'struct CID'} has no member named 'prv_n'; did you mean 'prv'?
  103 |   cout << int(cid.prv_n) << '.' << int(cid.prv_m) << endl;
      |                   ^~~~~
      |                   prv
bench:103:44: error: 'cid_t' {aka 'struct CID'} has no member named 'prv_m'; did you mean 'prv'?
  103 |   cout << int(cid.prv_n) << '.' << int(cid.prv_m) << endl;
      |                                            ^~~~~
      |                                            prv
bench:104:39: error: invalid use of non-static member function 'uint32_t CID::psn() const'
  104 |   cout << F("Serial number: ") << hex << cid.psn << dec << endl;
      |   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
In file included from L:\Progetti\Arduino\libraries\SdFat\src/SdCard/SdCardInterface.h:28,
                 from L:\Progetti\Arduino\libraries\SdFat\src/SdCard/SdioCard.h:28,
                 from L:\Progetti\Arduino\libraries\SdFat\src/SdCard/SdCard.h:27,
                 from L:\Progetti\Arduino\libraries\SdFat\src/SdFat.h:32,
                 from C:\Users\Fernando\AppData\Local\Arduino15\packages\rp2040\hardware\rp2040\2.4.1\libraries\ESP8266SdFat\examples\bench\bench.ino:4:
L:\Progetti\Arduino\libraries\SdFat\src/SdCard/SdCardInfo.h:298:12: note: declared here
  298 |   uint32_t psn() const {
      |            ^~~
bench:106:19: error: 'cid_t' {aka 'struct CID'} has no member named 'mdt_month'; did you mean 'mdtMonth'?
  106 |   cout << int(cid.mdt_month) << '/';
      |                   ^~~~~~~~~
      |                   mdtMonth
bench:107:23: error: 'cid_t' {aka 'struct CID'} has no member named 'mdt_year_low'
  107 |   cout << (2000 + cid.mdt_year_low + 10 * cid.mdt_year_high) << endl;
      |                       ^~~~~~~~~~~~
bench:107:47: error: 'cid_t' {aka 'struct CID'} has no member named 'mdt_year_high'
  107 |   cout << (2000 + cid.mdt_year_low + 10 * cid.mdt_year_high) << endl;
      |                                               ^~~~~~~~~~~~~
Multiple libraries were found for "SdFat.h"
 Used: L:\Progetti\Arduino\libraries\SdFat
 Not used: C:\Users\Fernando\AppData\Local\Arduino15\packages\rp2040\hardware\rp2040\2.4.1\libraries\ESP8266SdFat
exit status 1
'cid_t' {aka 'struct CID'} has no member named 'prv_n'; did you mean 'prv'?

[earlephilhower]

Sorry, you're still building using the unsupported SDFat library:

L:\Progetti\Arduino\libraries\SdFat\....

You could try removing it completely and building again.

[XFer012]

Thank you, I did that and now it builds OK (is there a way to force Arduino IDE to pick up your library? Or should I remove SDFat any time I build for Pico? I need it for other boards).

But the benchmark does not run (does not find the SDcard. Same reader works ok on Arduino Zero so I rule out a faulty reader/card).

So I tried your suggestion to use LittleFS Speed Test, but when I uncomment
#define TESTFS SDFS

I get a build error:

c:\temp\arduino_modified_sketch_849710\SpeedTest.ino: In function 'void setup()':
SpeedTest:11:16: error: 'SDFS' was not declared in this scope
   11 | #define TESTFS SDFS
      |                ^~~~
c:\temp\arduino_modified_sketch_849710\SpeedTest.ino:141:11: note: in expansion of macro 'TESTFS'
  141 |   DoTest(&TESTFS);
      |           ^~~~~~
exit status 1
'SDFS' was not declared in this scope

[earlephilhower]

You need to add #include <SDFS.h> in the sketch when moving the SD cards. I think it's there, just needs to be uncommented.

As for the IDE picking up other libraries, that is very weird. I have never run into that (un)fortunately. Sorry!

[XFer012]

Just out of curiosity, I tested the original SDFat (from Bill Greiman) with the Pico. I used the latest SDFat 2.2.0

It works, and peaks at 920 KB/s with the Pico overclocked to 250 MHz.

Discussing the performance figures with Bill Greiman, he wrote a test patch for the library which should allow 2.5x the speed (!)

Unfortunately it does not build with Arduino-Pico core, because of an incompatibility with the definition of SPIClass.

Please, if/when you have the time/interest, could it be possible for you to have a look at this (short) discussion?

greiman/SdFat#397 (comment)

Many thanks for your work on Arduino-Pico. Without this core, I would never had a use for the Pico!

[earlephilhower]

This looks like an almost trivial change.

@greiman's problem stems from the Arduino API limitations, unfortunately. I tried to not change any of the upstream ArduinoAPI definitions to keep maximum compatibility, and they define SPIClass === HardwareSPI and fixed API. With no 3-param transfer calls.

Unfortunately to support the Pico's GPIO strength control, I had to modify the main pinMode API. Then to make IPv6 work I had to replace IPAddress with something capable of supporting either mode w/LWIP.

Looks like I need to add a 3-operand transfer call. It's not a big deal (spi_read_write_blocking) and I can implement a simple fallback implementation allowing things like 3rd praty SW SPI to work with it, too.

[earlephilhower]

@XFer012 can you try #819 which implements the bulk SPI transfer change? I don't have a SD card hooked up right now to test.

[XFer012]

It works!!

2.5x speedup:

Manufacturer ID: 0X2
OEM ID: TM
Product: UC0C5
Revision: 5.0
Serial number: 0XD74489B2
Manufacturing date: 3/2017

FILE_SIZE = 5000000
BUF_SIZE = 512 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
2326.66,15641,213,219
2345.22,15578,213,217

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
2418.96,225,210,211
2420.14,244,210,211

Done

Thank you so much, this is a very welcome improvement!!

[greiman]

It would be great if the API would be consistent with Adafruit, Teensy, STM32, and other board packages. Adafruit uses:

void transfer(const void* txbuf, void* rxbuf, size_t count, bool block = true);

Others omit the block argument.

Having this as a member of SPIClass so I can use a pointer allows a second SPI port to be used in a dedicated way. This is a major advantage in handling internal operations in cards.

SD cards now have flash pages sizes of 16KB or more with large RAM buffers to emulate 512 byte sectors. Every time you raise CS, the card write a page. The data then must be copied to a new erased page for the next sector. This can increase flash writes by a factor of 32 or more.

For high end cards internal data copy and write can be 100 or more MB/sec

[guidol70]

I seem to understand you ship a fork of SdFat with Arduino-Pico, but it's not picked up it seems (and it's not listed under "Examples for Raspberry Pi Pico").

Any suggestion? How can I "force" Arduino IDE to pick up the bundled SdFat library instead of the standard one?

I do use an "unofficial" solution to force the use of the SdFat.h from ESP8266SdFat:

Create the file ESP8266SdFat.h which has the following code
#include "SdFat.h"
in the Library Folder
C:\Users\[user]\AppData\Local\Arduino15\packages\rp2040\hardware\rp2040\2.5.2\libraries\ESP8266SdFat\src

and the do not use
#include <SdFat.h>
but
#include <ESP8266SdFat.h>
in your .ino