Skip to content

PoC using SQLite as storage - DO NOT MERGE #1294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 97 commits into
base: main
Choose a base branch
from
Draft

PoC using SQLite as storage - DO NOT MERGE #1294

wants to merge 97 commits into from

Conversation

mrodm
Copy link
Contributor

@mrodm mrodm commented Apr 16, 2025

No description provided.

mrodm added 27 commits April 11, 2025 16:10
@mrodm
Load database from file system indexers
9bb0507
@mrodm
Return always all packages in file system indexers
11d59b0
@mrodm
Add database to storage indexer
1db7e48
@mrodm
Fix tests
bc08287
@mrodm
Fix packages served from file system indexers
ef06d36
@mrodm
Update storage indexer
87d54c8
@mrodm
Add context for db repository
45ea5b9
@mrodm
Create memory sqlite database for tests
ba6984c
@mrodm
Use free-cgo sqlite driver
90d94ce
@mrodm
Create independent databases
5712e71
@mrodm
Delete previous database conditionally
abf1e28
@mrodm
Create new databases when updating indexer storage
57a8d1d
@mrodm
Update test get all packages
0c22f5f
@mrodm
Add new function to process packages from queries directly
43318e8
@mrodm
Try to reduce the number of packages in the list
5d7e0ba
@mrodm
Update as pointers
a5ad0bd
@mrodm
Update benchmarks
466dc59
@mrodm
Use same file in benchmark
d6c5a79
@mrodm
Use same files and filter
6d9fe39
@mrodm
Add test case
a92cada
@mrodm
Run migrate inside lock
ffe7f00
@mrodm
Use pointers
51f0104
@mrodm
Add error management
4162103
@mrodm
Close databases
4a49799
@mrodm
Add missing set remote resolver
556127b
@mrodm
Fix linting errors
91bc65c
@mrodm
Add close functions and contexts
850e342
@mrodm mrodm self-assigned this Apr 16, 2025
mrodm added 29 commits May 22, 2025 18:19
@mrodm
Add fields to struct
14a0bbd
@mrodm
Add maxBulkAdd field into indexer struct
c0b0f2d
@mrodm
Add insert in batches
7fa65d5
@mrodm
Restore previous indexers and move new SQL indexer to their own modules
5613412
@mrodm
Update CONTRIBUTING files
2a8be6f
@mrodm
Extract function to create database package
addc754
@mrodm
Duplicate test for sql indexer
4fc2ca4
@mrodm
Remove secondary table
d658720
@mrodm
Add where clause for categories
f8f10f7
@mrodm
Set primary keys name and version
d07565d
@mrodm
Create SQL columns dynamically
2b9ee65
@mrodm
Reuse function
f32ec37
@mrodm
Fix categories filter using parent categories
3fc5d7c
@mrodm
Add commas to make easier search for fields
5f8098a
@mrodm
Fixes related to indices in database
248a0e2
@mrodm
Return data or baseData column from DB depending on the query
01f8f94
@mrodm
Remove unused context
d897fa9
@mrodm
Replace encoding/json by github.com/goccy/go-json
cd7e7f5
@mrodm
Avoid unnecessary checks to join packages if the first package list i…
06586db 
…s empty
@mrodm
Set kibana constraint
259dd44
@mrodm
Set private version fields using setRuntimeFields
89b4dd5
@mrodm
Allow to run local development for storage indexer
7a09d5e
@mrodm
Update go.mod
cc3543b
@mrodm
Run just one Filter.Apply in SQLIndexer Get operation
97f5ae5
@mrodm
Prepare code to create CPU and Memory profiles - to be removed
67fdf28
@mrodm
Fix condition to init storage indexers
8a58ade
@mrodm
Update modernc.org/sqlite lib
c3397b0
@mrodm
Remove unused code
20372ca
@mrodm
Update go.sum
9b68be7
@elasticmachine
Copy link

💚 Build Succeeded

History

cc @mrodm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@mrodm mrodm

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mrodm @prodsecmachine @elasticmachine