[FLOW-42] add ingest endpoint #137
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds professor data ingestion and fuzzy matching Implements endpoints and logic to ingest and process professor-course data, including JSON flattening and fuzzy matching. Introduces middleware for admin authentication and updates routing for new admin functionality. Adds support for environment variable to manage admin secret. Fixes PDF parsing build flags for compatibility. Relates to data synchronization and matching improvements.
…ofessors Implement migration to replace materialized view with new table for professor-course associations This migration introduces a new table `public.parsed_prof_taught_course` and replaces the materialized view `materialized.prof_teaches_course` with data seeded from reviews and the new table. It also updates the `course_search_index` and `prof_search_index` materialized views to utilize the new table structure. Additionally, it includes functions for searching courses and professors, along with triggers for refreshing related views.
Introduces normalized processing of professor-course data with JSON staging and SQL-based fuzzy matching for accuracy. Adds categorization logic to distinguish between new, ambiguous, and existing entries. Implements a new SQL table and enum for handling insertion and similarity scoring. Cleans up redundant code and simplifies data flow. Relates to #1745912089107
jerryzhou196
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Mysteriously convenient endpoint if we had professor data...
(about 50% done) - still need to run it and also test on staging