Skip to content

v1.1

Choose a tag to compare

View all tags
@DianaMDB DianaMDB released this 07 May 18:28
· 14 commits to main since this release
v1.1
8a33b12
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Enhancement: Write metadata.json as the first file

Enhancement: Write (Patch and Push) a "_TEMP" file and then rename it to original file, instead of directly writing the original file

Enhancement: RTS time interval check : changed logic from Push of change events to push every 20 secs, thus even if no events occur, the changes accumulated will be pushed after TIME interval lapsed

Enhancement: Better datetime handling: changed conversion function from to_datetime() instead of pd.Timestamp() with Timezone also handled

Enhancement: Columns added/ deleted - causes issue if MSFT picks two such files together, for now sufficient delay added

Enhancement : Write a new file (_conversion.log) to LZ if any column was changed to Null based on conversion error

BUG: Decimal 128 was not handled properly - Use MongoDB use MongoDB datatype to identify Bson.Decimal128 and force convert to float64

BUG: Contention Issues: Remove write back to local after reading from LZ in get_file_from_lz

BUG: The loop to get first NOT NULL first record wasn't handled properly and thus initial schema wasn't correct

Enhancement: Integer with NaN is defaulted to Float datatype - use MongoDB datatype to identify it as Int

Enhancement: Timestamp dtype initially was object because of NaN, was force corrected to datetime(ns)

Assets 2
Loading