Skip to content

Migrate Importer to Advisory v2 & Collect Existing Fix Commits for Project KB #1987

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
ziadhany wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Migrate Importer to Advisory v2 & Collect Existing Fix Commits for Project KB #1987

ziadhany wants to merge 5 commits into from

Conversation

@ziadhany
Copy link
Collaborator

@ziadhany ziadhany commented Aug 25, 2025 •
edited
Loading

@ziadhany ziadhany requested review from TG1999 and keshav-space and removed request for keshav-space October 15, 2025 14:50
keshav-space
keshav-space requested changes Oct 17, 2025
View reviewed changes
Copy link
Member

@keshav-space keshav-space left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ziadhany, see some suggestions.

@ziadhany
Add initial support for collecting ProjectKBP old fix commits.
f3d45ca 
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
@ziadhany
Add ProjectKBv2 importer
80b43fe 
Add a test for the ProjectKB importer and  collect fix commits pipeline.

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
@ziadhany ziadhany mentioned this pull request Dec 17, 2025
Open
@ziadhany ziadhany changed the title Collect existing fix commits for project-kb Migrate Importer to Advisory v2 & Collect Existing Fix Commits for Project KB Dec 17, 2025
@ziadhany
Drop project_kb_msr2019 V1 importer
666e774 
Fix CI falling test
Resolve merge conflict and Update migration file
Remove duplication and create append_patch_classifications function
Update the project-kb test

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
@ziadhany
Copy link
Collaborator Author

ziadhany commented Dec 19, 2025 •
edited
Loading

ProjectKBPipeline logs:

Importing data using project-kb-statements_v2
INFO 2025-12-25 02:21:45.063619 UTC Pipeline [ProjectKBStatementsPipeline] starting
INFO 2025-12-25 02:21:45.063816 UTC Step [clone_repo] starting
INFO 2025-12-25 02:21:45.063888 UTC Cloning ProjectKB Statements advisory data...
INFO 2025-12-25 02:29:23.108803 UTC Step [clone_repo] completed in 458 seconds (7.6 minutes)
INFO 2025-12-25 02:29:23.108968 UTC Step [collect_and_store_advisories] starting
INFO 2025-12-25 02:29:23.156389 UTC Estimated advisories to process: 1297
INFO 2025-12-25 02:29:23.156549 UTC Collecting 1,297 advisories
INFO 2025-12-25 02:29:23.156626 UTC Collecting fix commits from YAML statements under /statements....
INFO 2025-12-25 02:30:58.572925 UTC Progress: 10% (130/1297) ETA: 859 seconds (14.3 minutes)
INFO 2025-12-25 02:34:36.904962 UTC Progress: 20% (260/1297) ETA: 1255 seconds (20.9 minutes)
INFO 2025-12-25 02:37:31.443693 UTC Progress: 30% (390/1297) ETA: 1139 seconds (19.0 minutes)
INFO 2025-12-25 02:40:43.242454 UTC Progress: 40% (519/1297) ETA: 1020 seconds (17.0 minutes)
INFO 2025-12-25 02:42:17.009689 UTC Progress: 50% (649/1297) ETA: 774 seconds (12.9 minutes)
INFO 2025-12-25 02:44:53.889326 UTC Progress: 60% (779/1297) ETA: 620 seconds (10.3 minutes)
INFO 2025-12-25 02:52:08.754026 UTC Progress: 70% (908/1297) ETA: 585 seconds (9.8 minutes)
INFO 2025-12-25 02:54:43.731080 UTC Progress: 80% (1038/1297) ETA: 380 seconds (6.3 minutes)
INFO 2025-12-25 02:56:48.053895 UTC Progress: 90% (1168/1297) ETA: 183 seconds (3.0 minutes)
INFO 2025-12-25 02:59:45.395339 UTC Progress: 100% (1297/1297)
INFO 2025-12-25 02:59:45.400431 UTC Successfully collected 1,297 advisories
INFO 2025-12-25 02:59:45.400619 UTC Step [collect_and_store_advisories] completed in 1822 seconds (30.4 minutes)
INFO 2025-12-25 02:59:45.400702 UTC Step [clean_downloads] starting
INFO 2025-12-25 02:59:45.400781 UTC Removing cloned repository...
INFO 2025-12-25 02:59:45.484518 UTC Step [clean_downloads] completed in 0 seconds
INFO 2025-12-25 02:59:45.484694 UTC Pipeline completed in 2280 seconds (38.0 minutes)
Importing data using project-kb-MSR-2019_v2
INFO 2025-12-25 02:59:45.484808 UTC Pipeline [ProjectKBMSR2019Pipeline] starting
INFO 2025-12-25 02:59:45.484881 UTC Step [clone_repo] starting
INFO 2025-12-25 02:59:45.484942 UTC Cloning ProjectKB advisory data...
INFO 2025-12-25 03:00:26.664989 UTC Step [clone_repo] completed in 41 seconds
INFO 2025-12-25 03:00:26.665172 UTC Step [collect_and_store_advisories] starting
INFO 2025-12-25 03:00:26.666615 UTC Estimated advisories to process: 1281
INFO 2025-12-25 03:00:26.666717 UTC Collecting 1,281 advisories
INFO 2025-12-25 03:00:26.666813 UTC Collecting fix commits from ProjectKB ( vulas_db_msr2019_release )...
INFO 2025-12-25 03:00:27.218416 UTC Progress: 10% (129/1281) ETA: 5 seconds
INFO 2025-12-25 03:00:27.789298 UTC Progress: 20% (257/1281) ETA: 4 seconds
INFO 2025-12-25 03:00:28.228160 UTC Progress: 30% (385/1281) ETA: 4 seconds
INFO 2025-12-25 03:00:28.647814 UTC Progress: 40% (513/1281) ETA: 3 seconds
INFO 2025-12-25 03:00:29.053623 UTC Progress: 50% (641/1281) ETA: 2 seconds
INFO 2025-12-25 03:00:29.462826 UTC Progress: 60% (769/1281) ETA: 2 seconds
INFO 2025-12-25 03:00:29.863025 UTC Progress: 70% (897/1281) ETA: 1 seconds
INFO 2025-12-25 03:00:30.284528 UTC Progress: 80% (1025/1281) ETA: 1 seconds
INFO 2025-12-25 03:00:30.613854 UTC Progress: 90% (1153/1281)
INFO 2025-12-25 03:00:31.077993 UTC Progress: 100% (1281/1281)
INFO 2025-12-25 03:00:31.080928 UTC Successfully collected 1,281 advisories
INFO 2025-12-25 03:00:31.081057 UTC Step [collect_and_store_advisories] completed in 4 seconds
INFO 2025-12-25 03:00:31.081108 UTC Step [clean_downloads] starting
INFO 2025-12-25 03:00:31.081143 UTC Removing cloned repository...
INFO 2025-12-25 03:00:31.087360 UTC Step [clean_downloads] completed in 0 seconds
INFO 2025-12-25 03:00:31.087480 UTC Pipeline completed in 46 seconds

TG1999
TG1999 approved these changes Dec 19, 2025
View reviewed changes
Copy link
Contributor

@TG1999 TG1999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

ziadhany
ziadhany commented Dec 21, 2025
View reviewed changes
keshav-space
keshav-space requested changes Dec 23, 2025
View reviewed changes
Copy link
Member

@keshav-space keshav-space left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ziadhany, see comments below.

vulnerabilities/pipelines/v2_importers/project_kb_statements_importer.py

note_texts = []
for note_entry in yaml_data.get("notes", []):
text_content = note_entry.get("text")
Copy link
Member

@keshav-space keshav-space Dec 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notes section may also contain list of links which should be collected and stored as references see example https://github.com/SAP/project-kb/blob/730d2192bfa9b909246e02bc051e4fad2958a6d9/statements/CVE-2018-16406/statement.yaml

@ziadhany
Split the project-kb into two separate pipelines
b825f02 
Update Project-KB importer

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
@ziadhany
Refactor CSV processing to avoid duplicate iteration over items
caab13b 
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
@ziadhany
Copy link
Collaborator Author

ziadhany commented Dec 25, 2025 •
edited
Loading

@keshav-space I updated the logs for this Project KB and fixed the logic for handling the affected packages. Please have a look when you have time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@TG1999 TG1999 TG1999 approved these changes

@keshav-space keshav-space Awaiting requested review from keshav-space

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ziadhany @TG1999 @keshav-space