Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Checkov Does not Output Results for Files with Specific Symbols #5643

Open
SachithKasthuriarachchi opened this issue Oct 13, 2023 · 20 comments · May be fixed by #6922
Open

Checkov Does not Output Results for Files with Specific Symbols #5643

SachithKasthuriarachchi opened this issue Oct 13, 2023 · 20 comments · May be fixed by #6922
Labels
k8s Kubernetes Parser Changes related to checkov parsing logic

Comments

@SachithKasthuriarachchi
Copy link

SachithKasthuriarachchi commented Oct 13, 2023

Describe the issue

Checkov (latest) does not output the results for very large files (around 50000 lines)

Update to the Description

Checkov does not analyze files with some specific symbols like {{. Check #5643 (comment) for explanation

Additional context

Command:

LOG_LEVEL=DEBUG checkov -f "my-rendered.yaml" \
                --framework kubernetes \
                --skip-check CKV_K8S_14,CKV_K8S_15,CKV_K8S_35,CKV_K8S_43 \
                --external-checks-dir ../../cloud-sre-common/security/tools/static-scanning/iac/checkov/kubernetes

Debug Logs:

2023-10-13T12:22:32.2804737Z [INFO] Checkov scanning: base
2023-10-13T12:22:33.6449598Z 2023-10-13 12:22:33,644 [MainThread  ] [DEBUG]  Leveraging the bundled IAM Definition.
2023-10-13T12:22:33.6452057Z 2023-10-13 12:22:33,644 [MainThread  ] [DEBUG]  Leveraging the IAM definition at /usr/local/lib/python3.10/dist-packages/policy_sentry/shared/data/iam-definition.json
2023-10-13T12:22:34.6490871Z 2023-10-13 12:22:34,648 [MainThread  ] [DEBUG]  Adding the IntegrationFeatureRegistry <checkov.common.bridgecrew.integration_features.features.policy_metadata_integration.PolicyMetadataIntegration object at 0x7fcbb37bf550> with order 0
2023-10-13T12:22:34.6493427Z 2023-10-13 12:22:34,648 [MainThread  ] [DEBUG]  self.features after the sort:
2023-10-13T12:22:34.6495711Z 2023-10-13 12:22:34,648 [MainThread  ] [DEBUG]  [<checkov.common.bridgecrew.integration_features.features.policy_metadata_integration.PolicyMetadataIntegration object at 0x7fcbb37bf550>]
2023-10-13T12:22:34.6498197Z 2023-10-13 12:22:34,648 [MainThread  ] [DEBUG]  Adding the IntegrationFeatureRegistry <checkov.common.bridgecrew.integration_features.features.fixes_integration.FixesIntegration object at 0x7fcbb37bf520> with order 10
2023-10-13T12:22:34.6499841Z 2023-10-13 12:22:34,648 [MainThread  ] [DEBUG]  self.features after the sort:
2023-10-13T12:22:34.6502212Z 2023-10-13 12:22:34,648 [MainThread  ] [DEBUG]  [<checkov.common.bridgecrew.integration_features.features.policy_metadata_integration.PolicyMetadataIntegration object at 0x7fcbb37bf550>, <checkov.common.bridgecrew.integration_features.features.fixes_integration.FixesIntegration object at 0x7fcbb37bf520>]
2023-10-13T12:22:34.6505669Z 2023-10-13 12:22:34,648 [MainThread  ] [DEBUG]  Adding the IntegrationFeatureRegistry <checkov.common.bridgecrew.integration_features.features.repo_config_integration.RepoConfigIntegration object at 0x7fcbab9bfd60> with order 0
2023-10-13T12:22:34.6507371Z 2023-10-13 12:22:34,648 [MainThread  ] [DEBUG]  self.features after the sort:
2023-10-13T12:22:34.6510546Z 2023-10-13 12:22:34,648 [MainThread  ] [DEBUG]  [<checkov.common.bridgecrew.integration_features.features.policy_metadata_integration.PolicyMetadataIntegration object at 0x7fcbb37bf550>, <checkov.common.bridgecrew.integration_features.features.repo_config_integration.RepoConfigIntegration object at 0x7fcbab9bfd60>, <checkov.common.bridgecrew.integration_features.features.fixes_integration.FixesIntegration object at 0x7fcbb37bf520>]
2023-10-13T12:22:34.6524780Z 2023-10-13 12:22:34,651 [MainThread  ] [DEBUG]  Adding the IntegrationFeatureRegistry <checkov.common.bridgecrew.integration_features.features.licensing_integration.LicensingIntegration object at 0x7fcbab9bc0a0> with order 6
2023-10-13T12:22:34.6526851Z 2023-10-13 12:22:34,651 [MainThread  ] [DEBUG]  self.features after the sort:
2023-10-13T12:22:34.6530895Z 2023-10-13 12:22:34,652 [MainThread  ] [DEBUG]  [<checkov.common.bridgecrew.integration_features.features.policy_metadata_integration.PolicyMetadataIntegration object at 0x7fcbb37bf550>, <checkov.common.bridgecrew.integration_features.features.repo_config_integration.RepoConfigIntegration object at 0x7fcbab9bfd60>, <checkov.common.bridgecrew.integration_features.features.licensing_integration.LicensingIntegration object at 0x7fcbab9bc0a0>, <checkov.common.bridgecrew.integration_features.features.fixes_integration.FixesIntegration object at 0x7fcbb37bf520>]
2023-10-13T12:22:34.6541661Z 2023-10-13 12:22:34,653 [MainThread  ] [DEBUG]  Adding the IntegrationFeatureRegistry <checkov.common.bridgecrew.integration_features.features.suppressions_integration.SuppressionsIntegration object at 0x7fcbab9bdfc0> with order 2
2023-10-13T12:22:34.6543488Z 2023-10-13 12:22:34,653 [MainThread  ] [DEBUG]  self.features after the sort:
2023-10-13T12:22:34.6548354Z 2023-10-13 12:22:34,653 [MainThread  ] [DEBUG]  [<checkov.common.bridgecrew.integration_features.features.policy_metadata_integration.PolicyMetadataIntegration object at 0x7fcbb37bf550>, <checkov.common.bridgecrew.integration_features.features.repo_config_integration.RepoConfigIntegration object at 0x7fcbab9bfd60>, <checkov.common.bridgecrew.integration_features.features.suppressions_integration.SuppressionsIntegration object at 0x7fcbab9bdfc0>, <checkov.common.bridgecrew.integration_features.features.licensing_integration.LicensingIntegration object at 0x7fcbab9bc0a0>, <checkov.common.bridgecrew.integration_features.features.fixes_integration.FixesIntegration object at 0x7fcbb37bf520>]
2023-10-13T12:22:34.6562940Z 2023-10-13 12:22:34,655 [MainThread  ] [DEBUG]  Adding the IntegrationFeatureRegistry <checkov.common.bridgecrew.integration_features.features.custom_policies_integration.CustomPoliciesIntegration object at 0x7fcbab9be560> with order 1
2023-10-13T12:22:34.6564785Z 2023-10-13 12:22:34,655 [MainThread  ] [DEBUG]  self.features after the sort:
2023-10-13T12:22:34.6570587Z 2023-10-13 12:22:34,655 [MainThread  ] [DEBUG]  [<checkov.common.bridgecrew.integration_features.features.policy_metadata_integration.PolicyMetadataIntegration object at 0x7fcbb37bf550>, <checkov.common.bridgecrew.integration_features.features.repo_config_integration.RepoConfigIntegration object at 0x7fcbab9bfd60>, <checkov.common.bridgecrew.integration_features.features.custom_policies_integration.CustomPoliciesIntegration object at 0x7fcbab9be560>, <checkov.common.bridgecrew.integration_features.features.suppressions_integration.SuppressionsIntegration object at 0x7fcbab9bdfc0>, <checkov.common.bridgecrew.integration_features.features.licensing_integration.LicensingIntegration object at 0x7fcbab9bc0a0>, <checkov.common.bridgecrew.integration_features.features.fixes_integration.FixesIntegration object at 0x7fcbb37bf520>]
2023-10-13T12:22:34.6655128Z 2023-10-13 12:22:34,659 [MainThread  ] [DEBUG]  Adding the IntegrationFeatureRegistry <checkov.common.bridgecrew.integration_features.features.policies_3d_integration.Policies3DIntegration object at 0x7fcbab9bc580> with order 11
2023-10-13T12:22:34.6656940Z 2023-10-13 12:22:34,659 [MainThread  ] [DEBUG]  self.features after the sort:
2023-10-13T12:22:34.6663599Z 2023-10-13 12:22:34,659 [MainThread  ] [DEBUG]  [<checkov.common.bridgecrew.integration_features.features.policy_metadata_integration.PolicyMetadataIntegration object at 0x7fcbb37bf550>, <checkov.common.bridgecrew.integration_features.features.repo_config_integration.RepoConfigIntegration object at 0x7fcbab9bfd60>, <checkov.common.bridgecrew.integration_features.features.custom_policies_integration.CustomPoliciesIntegration object at 0x7fcbab9be560>, <checkov.common.bridgecrew.integration_features.features.suppressions_integration.SuppressionsIntegration object at 0x7fcbab9bdfc0>, <checkov.common.bridgecrew.integration_features.features.licensing_integration.LicensingIntegration object at 0x7fcbab9bc0a0>, <checkov.common.bridgecrew.integration_features.features.fixes_integration.FixesIntegration object at 0x7fcbb37bf520>, <checkov.common.bridgecrew.integration_features.features.policies_3d_integration.Policies3DIntegration object at 0x7fcbab9bc580>]
2023-10-13T12:22:34.6895478Z 2023-10-13 12:22:34,689 [MainThread  ] [DEBUG]  Loading external checks from /usr/local/lib/python3.10/dist-packages/checkov/bicep/checks/graph_checks
2023-10-13T12:22:34.6899423Z 2023-10-13 12:22:34,689 [MainThread  ] [DEBUG]  Searching through ['__pycache__'] and ['SQLServerAuditingEnabled.json', '__init__.py']
2023-10-13T12:22:34.6907001Z 2023-10-13 12:22:34,690 [MainThread  ] [DEBUG]  Searching through [] and ['__init__.cpython-310.pyc']
2023-10-13T12:22:34.8282623Z 2023-10-13 12:22:34,827 [MainThread  ] [DEBUG]  Popen(['git', 'version'], cwd=/home/vsts/work/1/s/choreo-control-plane/kustomize, universal_newlines=False, shell=None, istream=None)
2023-10-13T12:22:34.8313546Z 2023-10-13 12:22:34,830 [MainThread  ] [DEBUG]  Popen(['git', 'version'], cwd=/home/vsts/work/1/s/choreo-control-plane/kustomize, universal_newlines=False, shell=None, istream=None)
2023-10-13T12:22:35.5685747Z 2023-10-13 12:22:35,567 [MainThread  ] [DEBUG]  No API key present; setting include_all_checkov_policies to True
2023-10-13T12:22:35.5717654Z 2023-10-13 12:22:35,570 [MainThread  ] [DEBUG]  Run metadata: {
2023-10-13T12:22:35.5718684Z   "checkov_version": "2.5.8",
2023-10-13T12:22:35.5719538Z   "python_executable": "/usr/bin/python",
2023-10-13T12:22:35.5720165Z   "python_version": "3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]",
2023-10-13T12:22:35.5720740Z   "checkov_executable": "/usr/local/bin/checkov",
2023-10-13T12:22:35.5721179Z   "args": [
2023-10-13T12:22:35.5722786Z     "Command Line Args:   -f base-rendered.yaml --framework kubernetes --skip-check CKV_K8S_14,CKV_K8S_15,CKV_K8S_35,CKV_K8S_43 --external-checks-dir ../../cloud-sre-common/security/tools/static-scanning/iac/checkov/kubernetes",
2023-10-13T12:22:35.5724226Z     "Defaults:",
2023-10-13T12:22:35.5724505Z     "  --branch:          master",
2023-10-13T12:22:35.5724911Z     "  --download-external-modules:False",
2023-10-13T12:22:35.5725423Z     "  --external-modules-download-path:.external_modules",
2023-10-13T12:22:35.5725915Z     "  --evaluate-variables:True",
2023-10-13T12:22:35.5726303Z     "  --secrets-scan-file-type:[]",
2023-10-13T12:22:35.5726679Z     "  --block-list-secret-scan:[]",
2023-10-13T12:22:35.5727050Z     "  --summary-position:top",
2023-10-13T12:22:35.5727404Z     "  --mask:            []",
2023-10-13T12:22:35.5727754Z     "  --secrets-history-timeout:12h",
2023-10-13T12:22:35.5728074Z     ""
2023-10-13T12:22:35.5728259Z   ],
2023-10-13T12:22:35.5728660Z   "OS_system_info": "Linux-6.2.0-1014-azure-x86_64-with-glibc2.35",
2023-10-13T12:22:35.5729146Z   "CPU_architecture": "x86_64",
2023-10-13T12:22:35.5729453Z   "Python_implementation": "CPython"
2023-10-13T12:22:35.5730066Z }
2023-10-13T12:22:35.5730504Z 2023-10-13 12:22:35,570 [MainThread  ] [DEBUG]  Using cert_reqs None
2023-10-13T12:22:35.5731317Z 2023-10-13 12:22:35,570 [MainThread  ] [DEBUG]  Successfully set up HTTP manager
2023-10-13T12:22:35.5732429Z 2023-10-13 12:22:35,570 [MainThread  ] [DEBUG]  Resultant set of frameworks (removing skipped frameworks): kubernetes
2023-10-13T12:22:35.5733780Z 2023-10-13 12:22:35,570 [MainThread  ] [DEBUG]  BC_SOURCE = cli, version = 2.5.8
2023-10-13T12:22:35.5734859Z 2023-10-13 12:22:35,570 [MainThread  ] [DEBUG]  kubernetes_runner declares no system dependency checks required.
2023-10-13T12:22:35.5735937Z 2023-10-13 12:22:35,570 [MainThread  ] [DEBUG]  No API key found. Scanning locally only.
2023-10-13T12:22:35.7881804Z 2023-10-13 12:22:35,787 [MainThread  ] [DEBUG]  Got checkov mappings and guidelines from Bridgecrew platform
2023-10-13T12:22:35.7898302Z 2023-10-13 12:22:35,789 [MainThread  ] [DEBUG]  Loading external checks from /usr/local/lib/python3.10/dist-packages/checkov/kubernetes/checks/graph_checks
2023-10-13T12:22:35.7902276Z 2023-10-13 12:22:35,789 [MainThread  ] [DEBUG]  Searching through ['__pycache__'] and ['NoCreateNodesProxyOrPodsExec.json', 'ImpersonatePermissions.json', 'RoleBindingPE.json', 'ModifyServicesStatus.json', 'RequireAllPodsToHaveNetworkPolicy.json', 'ReadAllSecrets.json', '__init__.py']
2023-10-13T12:22:35.7913232Z 2023-10-13 12:22:35,791 [MainThread  ] [DEBUG]  Searching through [] and ['__init__.cpython-310.pyc']
2023-10-13T12:22:35.7952409Z 2023-10-13 12:22:35,794 [MainThread  ] [DEBUG]  Running without API key, so only open source runners will be enabled
2023-10-13T12:22:35.7954247Z 2023-10-13 12:22:35,794 [MainThread  ] [DEBUG]  Filtered list of policies: []
2023-10-13T12:22:35.7955598Z 2023-10-13 12:22:35,794 [MainThread  ] [DEBUG]  Received the following policy-level suppressions, that will be skipped from running: []
2023-10-13T12:22:35.7957098Z 2023-10-13 12:22:35,795 [MainThread  ] [DEBUG]  Filtered runners based on file type(s). Result: ['kubernetes']
2023-10-13T12:22:35.7958309Z 2023-10-13 12:22:35,795 [MainThread  ] [DEBUG]  Checking if kubernetes is valid for license
2023-10-13T12:22:35.7959621Z 2023-10-13 12:22:35,795 [MainThread  ] [DEBUG]  Open source mode - the runner is enabled
2023-10-13T12:22:35.8178065Z 2023-10-13 12:22:35,816 [MainThread  ] [DEBUG]  template 0 from file base-rendered.yaml is not a valid k8s template, reason: the key apiVersion does not exist in template structure
2023-10-13T12:22:35.8191404Z 2023-10-13 12:22:35,818 [MainThread  ] [DEBUG]  Loading external checks from ../../cloud-sre-common/security/tools/static-scanning/iac/checkov/kubernetes
2023-10-13T12:22:35.8198565Z 2023-10-13 12:22:35,819 [MainThread  ] [DEBUG]  Importing external check 'CheckLinkerDIntegrationStatus'
2023-10-13T12:22:35.8221959Z 2023-10-13 12:22:35,821 [MainThread  ] [DEBUG]  Loading external checks from ../../cloud-sre-common/security/tools/static-scanning/iac/checkov/kubernetes
2023-10-13T12:22:35.8224512Z 2023-10-13 12:22:35,821 [MainThread  ] [DEBUG]  Searching through ['__pycache__'] and ['CheckLinkerDIntegrationStatus.py', '__init__.py']
2023-10-13T12:22:35.8226702Z 2023-10-13 12:22:35,822 [MainThread  ] [DEBUG]  Searching through [] and ['CheckLinkerDIntegrationStatus.cpython-310.pyc']
2023-10-13T12:22:35.8227892Z 2023-10-13 12:22:35,822 [MainThread  ] [INFO ]  creating Kubernetes graph
2023-10-13T12:22:35.8228885Z 2023-10-13 12:22:35,822 [MainThread  ] [INFO ]  Successfully created Kubernetes graph
2023-10-13T12:22:35.8238200Z 2023-10-13 12:22:35,823 [MainThread  ] [DEBUG]  skip_severity = None, explicit_skip = False, regex_match = False, suppressed_policies: []
2023-10-13T12:22:35.8240596Z 2023-10-13 12:22:35,823 [MainThread  ] [DEBUG]  bc_check_id = BC_K8S_116, include_all_checkov_policies = True, is_external = False, explicit_run: []
2023-10-13T12:22:35.8243947Z 2023-10-13 12:22:35,823 [MainThread  ] [DEBUG]  should_run_check CKV2_K8S_2: True
2023-10-13T12:22:35.8245922Z 2023-10-13 12:22:35,823 [MainThread  ] [DEBUG]  skip_severity = None, explicit_skip = False, regex_match = False, suppressed_policies: []
2023-10-13T12:22:35.8247880Z 2023-10-13 12:22:35,823 [MainThread  ] [DEBUG]  bc_check_id = BC_K8S_117, include_all_checkov_policies = True, is_external = False, explicit_run: []
2023-10-13T12:22:35.8249194Z 2023-10-13 12:22:35,823 [MainThread  ] [DEBUG]  should_run_check CKV2_K8S_3: True
2023-10-13T12:22:35.8250514Z 2023-10-13 12:22:35,823 [MainThread  ] [DEBUG]  skip_severity = None, explicit_skip = False, regex_match = False, suppressed_policies: []
2023-10-13T12:22:35.8252139Z 2023-10-13 12:22:35,823 [MainThread  ] [DEBUG]  bc_check_id = BC_K8S_115, include_all_checkov_policies = True, is_external = False, explicit_run: []
2023-10-13T12:22:35.8253683Z 2023-10-13 12:22:35,823 [MainThread  ] [DEBUG]  should_run_check CKV2_K8S_1: True
2023-10-13T12:22:35.8255622Z 2023-10-13 12:22:35,824 [MainThread  ] [DEBUG]  skip_severity = None, explicit_skip = False, regex_match = False, suppressed_policies: []
2023-10-13T12:22:35.8257162Z 2023-10-13 12:22:35,824 [MainThread  ] [DEBUG]  bc_check_id = BC_K8S_118, include_all_checkov_policies = True, is_external = False, explicit_run: []
2023-10-13T12:22:35.8258519Z 2023-10-13 12:22:35,824 [MainThread  ] [DEBUG]  should_run_check CKV2_K8S_4: True
2023-10-13T12:22:35.8259718Z 2023-10-13 12:22:35,824 [MainThread  ] [DEBUG]  skip_severity = None, explicit_skip = False, regex_match = False, suppressed_policies: []
2023-10-13T12:22:35.8261236Z 2023-10-13 12:22:35,824 [MainThread  ] [DEBUG]  bc_check_id = None, include_all_checkov_policies = True, is_external = False, explicit_run: []
2023-10-13T12:22:35.8262423Z 2023-10-13 12:22:35,824 [MainThread  ] [DEBUG]  should_run_check CKV2_K8S_6: True
2023-10-13T12:22:35.8263615Z 2023-10-13 12:22:35,824 [MainThread  ] [DEBUG]  skip_severity = None, explicit_skip = False, regex_match = False, suppressed_policies: []
2023-10-13T12:22:35.8265156Z 2023-10-13 12:22:35,824 [MainThread  ] [DEBUG]  bc_check_id = BC_K8S_119, include_all_checkov_policies = True, is_external = False, explicit_run: []
2023-10-13T12:22:35.8266359Z 2023-10-13 12:22:35,824 [MainThread  ] [DEBUG]  should_run_check CKV2_K8S_5: True
2023-10-13T12:22:35.8267866Z 2023-10-13 12:22:35,824 [ThreadPoolEx] [DEBUG]  Running graph check: CKV2_K8S_2
2023-10-13T12:22:35.8268744Z 2023-10-13 12:22:35,825 [ThreadPoolEx] [DEBUG]  Running graph check: CKV2_K8S_3
2023-10-13T12:22:35.8269605Z 2023-10-13 12:22:35,825 [ThreadPoolEx] [DEBUG]  Running graph check: CKV2_K8S_1
2023-10-13T12:22:35.8270471Z 2023-10-13 12:22:35,826 [ThreadPoolEx] [DEBUG]  Running graph check: CKV2_K8S_4
2023-10-13T12:22:35.8271322Z 2023-10-13 12:22:35,826 [ThreadPoolEx] [DEBUG]  Running graph check: CKV2_K8S_6
2023-10-13T12:22:35.8272438Z 2023-10-13 12:22:35,826 [ThreadPoolEx] [DEBUG]  Running graph check: CKV2_K8S_5
2023-10-13T12:22:35.8280370Z 2023-10-13 12:22:35,827 [MainThread  ] [DEBUG]  Should run contributor metrics report: None
2023-10-13T12:22:35.8281372Z 2023-10-13 12:22:35,827 [MainThread  ] [DEBUG]  Getting exit code for report kubernetes
2023-10-13T12:22:35.8282309Z 2023-10-13 12:22:35,827 [MainThread  ] [DEBUG]  Soft fail severity threshold: None
2023-10-13T12:22:35.8283148Z 2023-10-13 12:22:35,827 [MainThread  ] [DEBUG]  Soft fail checks: []
2023-10-13T12:22:35.8283963Z 2023-10-13 12:22:35,827 [MainThread  ] [DEBUG]  Hard fail severity threshold: None
2023-10-13T12:22:35.8284786Z 2023-10-13 12:22:35,827 [MainThread  ] [DEBUG]  Hard fail checks: []
2023-10-13T12:22:35.8285578Z 2023-10-13 12:22:35,827 [MainThread  ] [DEBUG]  Use enforcement rules is FALSE
2023-10-13T12:22:35.8287395Z 2023-10-13 12:22:35,827 [MainThread  ] [DEBUG]  In get_exit_code; exit code thresholds: {'soft_fail': False, 'soft_fail_checks': [], 'soft_fail_threshold': None, 'hard_fail_checks': [], 'hard_fail_threshold': None}, hard_fail_on_parsing_errors: False
2023-10-13T12:22:35.8289170Z 2023-10-13 12:22:35,827 [MainThread  ] [DEBUG]  No failed checks in this report - returning 0
2023-10-13T12:22:35.8289677Z 
2023-10-13T12:22:35.8290057Z        _               _              
2023-10-13T12:22:35.8290381Z    ___| |__   ___  ___| | _______   __
2023-10-13T12:22:35.8290761Z   / __| '_ \ / _ \/ __| |/ / _ \ \ / /
2023-10-13T12:22:35.8291078Z  | (__| | | |  __/ (__|   < (_) \ V / 
2023-10-13T12:22:35.8291407Z   \___|_| |_|\___|\___|_|\_\___/ \_/  
2023-10-13T12:22:35.8291716Z                                       
2023-10-13T12:22:35.8292026Z By bridgecrew.io | version: 2.5.8 
@gruebel
Copy link
Contributor

gruebel commented Oct 13, 2023

Hey @SachithKasthuriarachchi thanks for reaching out.

Not sure how the rendered file looks like, but we identified it as not being a valid Kubernetes manifest

[DEBUG]  template 0 from file base-rendered.yaml is not a valid k8s template, reason: the key apiVersion does not exist in template structure

@gruebel gruebel added Parser Changes related to checkov parsing logic k8s Kubernetes and removed outputs labels Oct 13, 2023
@SachithKasthuriarachchi
Copy link
Author

oh thanks @gruebel for pointing that out. Would it be more easy to debug if we change the above log from DEBUG to WARN? Also, is there any clue on identifying where the erroneous resource is?

@SachithKasthuriarachchi
Copy link
Author

also please note that the same file perfectly works on Checkov 2.0.740

@SachithKasthuriarachchi
Copy link
Author

checked with 2.2.150 and was able to reproduce the error. Was there any breaking changes introduced after 2.0.740?

@gruebel
Copy link
Contributor

gruebel commented Oct 16, 2023

not really breaking change, but we check generic files like JSON/YAML for specific keywords and if they are not included that specific framework is skipped. I'm still curious, how your Kubernetes manifest can be valid without an apiVersion field https://kubernetes.io/docs/concepts/overview/working-with-objects/#required-fields

A higher log level doesn't make any sense here, because it will spam the output.Let's take a JSON file, it could be a template for ARM, CloudFormation, Kubernetes, OpenAPI, Terraform or just some generic file. We would then log a warning for each of them for each file.

@SachithKasthuriarachchi
Copy link
Author

@gruebel does checkov skip running checks if the {{ symbol is present anywhere? Following are my observations.

  • Created the following configMap manifest (invalid.yaml)
apiVersion: v1
data:
  pod-utilization.json: |-
    {
          "annotations": {
            "list": [
              {
                "builtIn": 1,
                "datasource": {
                  "type": "datasource",
                  "uid": "grafana"
                },
                "enable": true,
                "hide": true,
                "iconColor": "rgba(0, 211, 255, 1)",
                "name": "Annotations & Alerts",
                "legendFormat": "{{cluster_id}} {{pod}}/{{container}} (usage max)"
                "target": {
                  "limit": 100,
                  "matchAny": false,
                  "tags": [],
                  "type": "dashboard"
                },
                "type": "dashboard"
              }
            ]
          }
kind: ConfigMap
metadata:
  annotations:
    kapp.k14s.io/disable-default-label-scoping-rules: ""
  labels:
    app: cost-analyzer
    app.kubernetes.io/instance: kubecost
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: cost-analyzer
    grafana_dashboard: "1"
    helm.sh/chart: cost-analyzer-1.106.0
  name: prom-benchmark-dashboard
  namespace: kubecost
  • Executed checkov against the above file. This resulted no checkov output similar to above case
checkov -f invalid.yaml --framework kubernetes
  • After removing "{{cluster_id}} {{pod}}/{{container}} (usage max)" line in above file, Checkov worked perfectly

@SachithKasthuriarachchi
Copy link
Author

not really breaking change, but we check generic files like JSON/YAML for specific keywords and if they are not included that specific framework is skipped. I'm still curious, how your Kubernetes manifest can be valid without an apiVersion field https://kubernetes.io/docs/concepts/overview/working-with-objects/#required-fields

A higher log level doesn't make any sense here, because it will spam the output.Let's take a JSON file, it could be a template for ARM, CloudFormation, Kubernetes, OpenAPI, Terraform or just some generic file. We would then log a warning for each of them for each file.

@gruebel I checked the generated yaml and apiVersion is present in all the resources.

@SachithKasthuriarachchi SachithKasthuriarachchi changed the title Checkov Does not Output Results for Very Large Files Checkov Does not Output Results for Files with Specific Symbols Oct 17, 2023
@gruebel
Copy link
Contributor

gruebel commented Oct 20, 2023

interesting we don't skip it on purpose, probably there is an issue with parsing the file.

@gruebel
Copy link
Contributor

gruebel commented Oct 20, 2023

ok, I checked the code and we filter out content, which has {{ inside the yaml file, because we assume it will be a Helm chart.

@SachithKasthuriarachchi
Copy link
Author

yes just because a file contain {{ we can't always assume it as a helm chart. The best counter-example is the rendered manifests of kubecost helm chart.

@SachithKasthuriarachchi
Copy link
Author

@gruebel any plans on assigning someone from Checkov team to fix this?

@SachithKasthuriarachchi
Copy link
Author

any update on this?

@karthickmuthuraj
Copy link

Hi Team,

Could you provide some alternate way or suggest some ideas ?

Copy link

stale bot commented Aug 4, 2024

Thanks for contributing to Checkov! We've automatically marked this issue as stale to keep our issues list tidy, because it has not had any activity for 6 months. It will be closed in 14 days if no further activity occurs. Commenting on this issue will remove the stale tag. If you want to talk through the issue or help us understand the priority and context, feel free to add a comment or join us in the Checkov slack channel at codifiedsecurity.slack.com
Thanks!

@stale stale bot added the stale label Aug 4, 2024
@samavasi
Copy link

is there any workaround for this?

@ryanbasiltrickett
Copy link

Any updates on this issue? I have had to revert to 2.0.740 which for now is a work around but not a long term solution.

@stale stale bot removed the stale label Oct 16, 2024
@SachithKasthuriarachchi
Copy link
Author

any update on this? why does the checkov team keeps ignoring this?

@ryanbasiltrickett
Copy link

ryanbasiltrickett commented Nov 22, 2024

I found my issue being if you create the yaml file using helm in windows it will create a CRLF file and checkov doesn't work with this, you will need to force it to a LF file

@ChananM
Copy link
Contributor

ChananM commented Dec 8, 2024

Hi @SachithKasthuriarachchi, thank you for reaching out.
Unfortunately we won't be able to address this issue in the foreseeable future.
However, we are accepting contributions, and would be happy to review a solution suggested by the community.

@eirmich
Copy link

eirmich commented Dec 24, 2024

Hi @ChananM, I've created a PR to tackle this. #6922 Would you mind reviewing it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
k8s Kubernetes Parser Changes related to checkov parsing logic
Projects
None yet
7 participants