Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added openjdk-importer to pipeline #1635

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

harmonicfunc
Copy link

#1496
revision to previous openjdk-importer with new pr

Signed-off-by: Alok Kumar Singh <[email protected]>
@harmonicfunc
Copy link
Author

@keshav-space @pombredanne @ambuj-1211
pls review this pr

@Rishi-source
Copy link

Hi @harmonicfunc , When you add a importer pipeline you have to certainly make test_importer with some test data in the test directory. you can read the tutorial_add_importer_pipeline in docs/source directory. As well as you can also read the contributing guidelines https://aboutcode.readthedocs.io/en/latest/contributing.html . You will also find the slack channel link there which you can join.

@harmonicfunc
Copy link
Author

the logs for importer run are here:
(venv) (base) alok@alok-IdeaPad-5-15ITL05:~/GSOC/vulnerablecode$ ./manage.py import openjdk_importer
Importing data using openjdk_importer
INFO 2024-11-04 10:36:13.527 Pipeline [OpenJDKImporterPipeline] starting
INFO 2024-11-04 10:36:13.527 Step [collect_and_store_advisories] starting
INFO 2024-11-04 10:36:24.174 Collecting 23 advisories
INFO 2024-11-04 10:40:27.413 Progress: 13% (3/23) ETA: 1628 seconds (27.1 minutes)
INFO 2024-11-04 10:40:27.415 Progress: 26% (6/23) ETA: 692 seconds (11.5 minutes)
INFO 2024-11-04 10:40:27.434 Progress: 39% (9/23) ETA: 380 seconds (6.3 minutes)
INFO 2024-11-04 10:40:27.445 Progress: 52% (12/23) ETA: 225 seconds (3.8 minutes)
INFO 2024-11-04 10:40:27.471 Progress: 65% (15/23) ETA: 131 seconds (2.2 minutes)
INFO 2024-11-04 10:40:27.482 Progress: 78% (18/23) ETA: 69 seconds (1.1 minutes)
INFO 2024-11-04 10:40:27.492 Progress: 91% (21/23) ETA: 24 seconds
INFO 2024-11-04 10:40:27.517 Progress: 104% (24/23) ETA: -9 seconds
INFO 2024-11-04 10:40:27.529 Progress: 117% (27/23) ETA: -35 seconds
INFO 2024-11-04 10:40:27.541 Progress: 130% (30/23) ETA: -56 seconds
INFO 2024-11-04 10:40:27.558 Progress: 143% (33/23) ETA: -73 seconds
INFO 2024-11-04 10:40:27.578 Progress: 156% (36/23) ETA: -87 seconds
INFO 2024-11-04 10:40:27.636 Progress: 169% (39/23) ETA: -99 seconds
INFO 2024-11-04 10:40:27.653 Progress: 182% (42/23) ETA: -110 seconds
INFO 2024-11-04 10:40:27.682 Progress: 195% (45/23) ETA: -119 seconds
INFO 2024-11-04 10:40:27.693 Progress: 208% (48/23) ETA: -126 seconds
INFO 2024-11-04 10:40:27.709 Progress: 221% (51/23) ETA: -133 seconds
INFO 2024-11-04 10:40:27.727 Progress: 234% (54/23) ETA: -139 seconds
INFO 2024-11-04 10:40:27.740 Progress: 247% (57/23) ETA: -145 seconds
INFO 2024-11-04 10:40:27.757 Progress: 260% (60/23) ETA: -150 seconds
INFO 2024-11-04 10:40:27.769 Progress: 273% (63/23) ETA: -154 seconds
INFO 2024-11-04 10:40:27.790 Progress: 286% (66/23) ETA: -158 seconds
INFO 2024-11-04 10:40:27.802 Progress: 300% (69/23) ETA: -162 seconds
INFO 2024-11-04 10:40:27.864 Progress: 313% (72/23) ETA: -166 seconds
INFO 2024-11-04 10:40:27.873 Progress: 326% (75/23) ETA: -169 seconds
INFO 2024-11-04 10:40:27.883 Progress: 339% (78/23) ETA: -172 seconds
INFO 2024-11-04 10:40:27.894 Progress: 352% (81/23) ETA: -174 seconds
INFO 2024-11-04 10:40:27.905 Progress: 365% (84/23) ETA: -177 seconds
INFO 2024-11-04 10:40:27.927 Progress: 378% (87/23) ETA: -179 seconds
INFO 2024-11-04 10:40:27.939 Progress: 391% (90/23) ETA: -181 seconds
INFO 2024-11-04 10:40:27.950 Progress: 404% (93/23) ETA: -183 seconds
INFO 2024-11-04 10:40:27.959 Progress: 417% (96/23) ETA: -185 seconds
INFO 2024-11-04 10:40:27.975 Progress: 430% (99/23) ETA: -187 seconds
INFO 2024-11-04 10:40:27.989 Progress: 443% (102/23) ETA: -189 seconds
INFO 2024-11-04 10:40:28.014 Progress: 456% (105/23) ETA: -190 seconds
INFO 2024-11-04 10:40:28.032 Progress: 469% (108/23) ETA: -192 seconds
INFO 2024-11-04 10:40:28.045 Progress: 482% (111/23) ETA: -193 seconds
INFO 2024-11-04 10:40:28.071 Progress: 495% (114/23) ETA: -195 seconds
INFO 2024-11-04 10:40:28.085 Progress: 508% (117/23) ETA: -196 seconds
INFO 2024-11-04 10:40:28.098 Progress: 521% (120/23) ETA: -197 seconds
INFO 2024-11-04 10:40:28.109 Progress: 534% (123/23) ETA: -198 seconds
INFO 2024-11-04 10:40:28.134 Progress: 547% (126/23) ETA: -199 seconds
INFO 2024-11-04 10:40:28.144 Progress: 560% (129/23) ETA: -200 seconds
INFO 2024-11-04 10:40:28.156 Progress: 573% (132/23) ETA: -201 seconds
INFO 2024-11-04 10:40:28.166 Progress: 586% (135/23) ETA: -202 seconds
INFO 2024-11-04 10:40:28.176 Progress: 600% (138/23) ETA: -203 seconds
INFO 2024-11-04 10:40:28.196 Progress: 613% (141/23) ETA: -204 seconds
INFO 2024-11-04 10:40:28.207 Progress: 626% (144/23) ETA: -205 seconds
INFO 2024-11-04 10:40:28.229 Progress: 639% (147/23) ETA: -206 seconds
INFO 2024-11-04 10:40:28.243 Progress: 652% (150/23) ETA: -207 seconds
INFO 2024-11-04 10:40:28.254 Progress: 665% (153/23) ETA: -207 seconds
INFO 2024-11-04 10:40:28.264 Progress: 678% (156/23) ETA: -208 seconds
INFO 2024-11-04 10:40:28.274 Progress: 691% (159/23) ETA: -209 seconds
INFO 2024-11-04 10:40:28.284 Progress: 704% (162/23) ETA: -209 seconds
INFO 2024-11-04 10:40:28.303 Progress: 717% (165/23) ETA: -210 seconds
INFO 2024-11-04 10:40:28.316 Progress: 730% (168/23) ETA: -211 seconds
INFO 2024-11-04 10:40:29.350 Progress: 743% (171/23) ETA: -212 seconds
INFO 2024-11-04 10:40:29.370 Progress: 756% (174/23) ETA: -213 seconds
INFO 2024-11-04 10:40:29.427 Progress: 769% (177/23) ETA: -213 seconds
INFO 2024-11-04 10:40:29.440 Successfully collected 178 advisories
INFO 2024-11-04 10:40:29.440 Step [collect_and_store_advisories] completed in 256 seconds (4.3 minutes)
INFO 2024-11-04 10:40:29.440 Step [import_new_advisories] starting
INFO 2024-11-04 10:40:29.442 Importing 184 new advisories
INFO 2024-11-04 10:40:29.948 Progress: 10% (19/184) ETA: 5 seconds
INFO 2024-11-04 10:40:30.468 Progress: 20% (37/184) ETA: 4 seconds
INFO 2024-11-04 10:40:31.043 Progress: 30% (56/184) ETA: 4 seconds
INFO 2024-11-04 10:40:31.621 Progress: 40% (74/184) ETA: 3 seconds
INFO 2024-11-04 10:40:32.198 Progress: 50% (92/184) ETA: 3 seconds
INFO 2024-11-04 10:40:32.909 Progress: 60% (111/184) ETA: 2 seconds
INFO 2024-11-04 10:40:33.523 Progress: 70% (129/184) ETA: 2 seconds
INFO 2024-11-04 10:40:34.087 Progress: 80% (148/184) ETA: 1 seconds
INFO 2024-11-04 10:40:34.654 Progress: 90% (166/184) ETA: 1 seconds
INFO 2024-11-04 10:40:35.278 Progress: 100% (184/184)
INFO 2024-11-04 10:40:35.308 Successfully imported 184 new advisories
INFO 2024-11-04 10:40:35.308 Step [import_new_advisories] completed in 6 seconds
INFO 2024-11-04 10:40:35.308 Pipeline completed in 262 seconds (4.4 minutes)

pls review the pr!!

@pombredanne
Copy link
Member

The progress logging looks weird:

 INFO 2024-11-04 10:40:29.427 Progress: 769% (177/23) ETA: -213 seconds

769%? Do you know where this is from?

Copy link
Member

@pombredanne pombredanne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pipeline_id = "openjdk_importer"
root_url = "https://openjdk.org/groups/vulnerability/advisories/"
license_url = "https://openjdk.org/legal/"
spdx_license_expression = "CC-BY-4.0"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure this is the license? I cannot see anything on the legal page about this.

We should reach out to [email protected] to ask them about the license of thesei advisories and report here. Or ask at https://mail.openjdk.org/mailman/listinfo/vuln-announce ...

This may be under https://openjdk.org/legal/tou/terms and this may be problematic?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@seanjmullan @andreasst we are trying to integrate the OpenJDK advisories data in VulnerableCode.

  1. Would you know what is the license of the advisory data published at https://openjdk.org/groups/vulnerability/advisories/
  2. Is there a structured data format better suited for data integration than web page scraping that could be available somewhere?

Thanks!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@seanjmullan @andreasst Sorry for the noise. I meant to tag someone in OpenJDK vulnerability group, not the security group! Would you know someone there who could help?

Copy link
Author

@harmonicfunc harmonicfunc Nov 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry for that, will make the changes didnt knew about that, i tried to search for the license but coudnt get it

@harmonicfunc
Copy link
Author

The progress logging looks weird:

 INFO 2024-11-04 10:40:29.427 Progress: 769% (177/23) ETA: -213 seconds

769%? Do you know where this is from?

yeah thats because i took the advisory count from the main page of url:https://openjdk.org/groups/vulnerability/advisories/ , but that might be wrong

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants