Suggest: Interlisp.github.io release should auto-update data/bibliography.json

[MattHeffron]

The file data/bibliography.json is just a file in the repository and it must be updated as part of a git commit.
I suggest that the script to update that file, using API access to the Zotero database, be invoked as part of a release "build" of the website. (I.e., scripts/update_bibliography.sh).
Is there some specific value to keeping versions of the bibliography.json file in the repository?

[masinter]

Zotero offers no version management (does it? Asking @AbeJellinek ) so having backups of at least the bibliographic material seems like a good idea. The json file doesn't keep attachments, so not ideal.

There might be a credential problem running update from a script.

[stumbo]

There's probably little value in keeping old versions, if we assume Zotero will be around and we can continue to pull a copy of the bibliography. I have written a GItHub Action that uses our existing script. update_bibliography.sh to pull the bibliography.json file. I set it up to also push the latest version of the file to the GitHub repo. I haven't tested storing the file, but stripping out storing the file and just using it would be fine.

I also have a script written in R that takes a copy of the bibliography downloaded as Better BibTex and translates each entry into a .md file we could include within a bibliography section. It's one of the things I've been playing with to improve the bibliography.

@masinter Zotero does provide the ability to figure out if the bibliography has been updated. The REST interface command:

curl --location 'https://api.zotero.org/groups/2914042'

returns a bunch of information on the Interlisp bibliography, the meta entry,

   "meta": {
        "created": "2021-04-15T22:41:59Z",
        "lastModified": "2023-09-01T19:18:22Z",
        "numItems": 576
    },

holds the key information, the lastModified date of the bibliography. Right now, it's September 1, 2023. If we cache that value, we could determine whether the library has changed and whether we need to pull a new json file.

[stumbo]

The website at presents checks Zotero daily and updates the bibliography entries if the information on Zotero has changed. The Github Action, .github/workflows/gh-pages.yml checks to see if Zotero has changed, and if so, runs the script scripts/updateBibliography.sh to rebuild the bibliographic information.

In addition, the results are cached and on subsequent builds, if there are no changes to the bibliography, we do not rebuild that portion of the system.