nuclei v3 bug fixes #4176

tarunKoyalwar · 2023-09-21T20:54:17Z

Proposed changes

generate ecdsa key-pair for signing templates if none exist
default key-pair are saved / retrieved from nuclei config folder
fixed trailing whitespace in code protocol response exports

Checklist

Pull request is created against the dev branch
All checks passed (lint, unit/integration/regression tests etc.) with my changes
I have added tests that prove my fix is effective or that my feature works
I have added necessary documentation (if appropriate)

tarunKoyalwar · 2023-09-21T20:55:05Z

example multiprotocol template

id: multi-proto-example-code

info:
  name: multi-proto-example-code
  author: pdteam
  severity: info
  description: |
    multi protocol example using code and http protocol


code:
  - engine: 
      - sh
      - bash
    source: |
      echo "hello from code" | base64 


http:
  - raw:
      - |
        GET /?param={{code_response}} HTTP/1.1
        Host: {{Hostname}}
        Origin: {{BaseURL}}
    
    matchers:
      - type: status
        status:
          - 200
# digest: 4b0a00483046022100c060a867912857a78e261df8fec20d879bc06a00b72305823e136dc0f3de8470022100b001a80a51081cb8c097097d8ba28b1fdc4c76e1ab745375e55677cf0f337b84

tarunKoyalwar · 2023-09-22T16:56:00Z

TODO

review should we keep stdout when response already exists
add missing docs related to args etc
File read sandbox
include file content of flow and source fields while signing templates
~~disable template resign in code protocol~~(TBD)
verify sign keys as env variable (Already implemented just need to verify )
Placeholder + Logic for PD Master Public-Private Key
start index from 1 in multiprotocol and flow(??)
missing template-id in code protocol

[VER] [] Executed Code on target https://scanme.sh/

id: code-template


info:
  name: example code template
  author: pdteam
  severity: info

self-contained: false
code:
  - engine:
      - sh
      - bash
    source: |
      echo output1

http:
  - method: GET
    path:
      - "{{BaseURL}}"
# digest: 4a0a00473045022100addd38e85d219e8916a52893ba8ea3944347b121e9643249c1506f156764cab7022022826f03b26f909f0488d8e5e32ac3428f64dd9a6b8e64c2f769b843140370d2

tarunKoyalwar · 2023-10-01T16:39:36Z

Signing templates by setting public/private keys in enviornment variable (POC)

# set env variables
export NUCLEI_SIGNATURE_PRIVATE_KEY=$(cat $HOME/.config/nuclei/nuclei-template-signer)
export NUCLEI_SIGNATURE_PUBLIC_KEY=$(cat $HOME/.config/nuclei/nuclei-template-signer.pub)
export NUCLEI_SIGNATURE_ALGORITHM=$(cat $HOME/.config/nuclei/nuclei-template-signer.algo)

# move backup keys
cd  $HOME/.config/nuclei && mkdir backup && mv nuclei-template-* backup

# verify keys are not present in default dir
la $HOME/.config/nuclei | grep "nuclei"

# sign example template
./nuclei -t code.yaml -v -sign
[INF] All templates signatures were elaborated success=1 failed=0

tarunKoyalwar · 2023-10-04T16:18:17Z

User experience using signed and unsigned templates

$ ./nuclei -t ~/nuclei-templates/http -sign -v
[INF] Generating new key-pair for signing templates
[*] Enter User/Organization Name (exit to abort) : ProjectDiscovery.io
[*] Enter passphrase (exit to abort): 
[*] Enter same passphrase again: 
[INF] Successfully generated new key-pair for signing templates


$ ./nuclei -t ~/nuclei-templates/http -sign -v
[INF] All templates signatures were elaborated success=6464 failed=0


$  ./nuclei         
                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.0.0-dev

		projectdiscovery.io

[WRN] Found 1 templates loaded with deprecated protocol syntax, update before v3 for continued support.
[INF] Current nuclei version: v3.0.0-dev (development)
[INF] Current nuclei-templates version: v9.6.4 (latest)
[INF] New templates added in latest release: 121
[INF] Templates loaded for current scan: 6895
[INF] Executing 6219 signed templates from 'ProjectDiscovery.io'
[WRN] Executing 687 unsigned templates. Use with caution
[INF] No results found. Better luck next time!

using encrypted private key with passphrase

$ ./nuclei -t ~/nuclei-templates/http -sign -v
[INF] Generating new key-pair for signing templates
[*] Enter User/Organization Name (exit to abort) : tarun
[*] Enter passphrase (exit to abort): 
[*] Enter same passphrase again: 
[INF] Successfully generated new key-pair for signing templates

$  ./nuclei -t ~/nuclei-templates/http -sign -v
[INF] Private Key is encrypted with passphrase
[*] Enter passphrase (exit to abort): 
[INF] All templates signatures were elaborated success=6464 failed=0

tarunKoyalwar · 2023-10-04T19:05:26Z

New Changes

nuclei now only supports ECDSA algorithm . as it is faster and secure than RSA-3048 bit key and also many benefits including less memory usage.
signing utility is dissolved at code references need to be resolved and included while signing and verifying and nuclei already supports that . and it might create inconsistencies in furthur updates

detailed docs about signing templates and related changes are available at

nuclei/docs/template-guide/code.mdx

Lines 23 to 143 in b90f629

    
           ### What does signing a template mean ? 
        
           Template signing is a mechanism to ensure the integrity and authenticity of templates. The primary goal is to provide template writers/consumers a way to trust crowdsource/custom templates ensuring that they are not tampered 
        
           All Official nuclei templates include a digital signature in them and are verified by nuclei while loading templates using ProjectDiscovery's public key shipped with nuclei binary itself. 
        
           Individuals / Organizations running nuclei in their work environment can generate their own key-pair with `nuclei` and sign their custom templates with their private key, thus ensuring that only authorized templates are being used in their environment. 
        
           This also allows entities to fully utilize the power of new protocols like `code` and `javascript` without worrying about malicious custom templates being used in their environment. 
        
           **Points to note** 
        
           - Template signing is optional for all protocols except `code`. 
        
           - Code File References (ex: `source: protocols/code/pyfile.py`) are allowed and content of these files is included in the template digest 
        
           - Payload File References (ex: `payloads: protocols/http/params.txt`) are not included in the template digest as it is treated as a payload/helper and not actual code that is being executed 
        
           - Template Signing is deterministic while both signing and verifying a template i.e if a code file is referenced in a template that is present outside of templates directory with `-lfa` flag then verification will fail if same template is used without `-lfa` flag. (Note this only applies to `-lfa` i.e local file access flag only) 
        
           ## How to sign custom templates 
        
           Simplest and recommended way to generate key-pair and signing/verfifying templates is to use `nuclei` itself. 
        
           #### When Signing a template if key-pair does not exist then nuclei will prompt user to generate a new key-pair with options  
        
           ```console 
        
           $ ./nuclei -t my-template.yaml -sign -v 
        
           [INF] Generating new key-pair for signing templates 
        
           [*] Enter User/Organization Name (exit to abort) : ProjectDiscovery.io 
        
           [*] Enter passphrase (exit to abort):  
        
           [*] Enter same passphrase again:  
        
           [INF] Successfully generated new key-pair for signing templates 
        
           ``` 
        
           > Note: Passphrase is optional and can be left blank when used private key is encrypted with passphrase using PEMCipherAES256 Algo 
        
           #### Signing a template with existing key-pair 
        
           ```console 
        
           $ ./nuclei -t ~/nuclei-templates/http -sign -v 
        
           [INF] All templates signatures were elaborated success=6464 failed=0 
        
           ``` 
        
           ### Template Digest 
        
           When a template is signed, a digest is generated and added to the template. This digest is a hash of the template content and is used to verify the integrity of the template. If the template is modified after signing, the digest will change, and the signature verification will fail which happens during template loading. 
        
           ```yaml 
        
           # digest: 4a0a00473045022023beecb1c4ef5b3b3a4d936a689d0fa5fea35524d23bbc12001fa0b21ca2500b02210082484d006ee0663ba1c8450ff0d10eb053308137af25cde223406c3423c4e5d1 
        
           ``` 
        
           ### Where are keys stored ? 
        
           key-pair generated by nuclei are stored in 2 files in $config/nuclei/keys directory where $config is system specific config directory 
        
           ``` 
        
           $ la ~/.config/nuclei/keys  
        
           total 16 
        
           -rw-------  1 tarun  staff   251B Oct  4 21:45 nuclei-user-private-key.pem # encrypted private key with passphrase 
        
           -rw-------  1 tarun  staff   572B Oct  4 21:45 nuclei-user.crt # self signed certificate which includes public key and identifier (i.e user/org name) 
        
           ``` 
        
           ### Sharing and Using Public Key 
        
           Public key is stored in $config/nuclei/keys/nuclei-user.crt and can be shared with other users / organizations to verify templates signed by you. 
        
           #### Using Public Key 
        
           - A simple way to use public key is to copy it to $config/nuclei/keys directory of other user's machine 
        
           - Another way is to use environment variable `NUCLEI_USER_CERTIFICATE=xxx` to specify path of public key or content of public key directly 
        
           ```console 
        
           $ export NUCLEI_USER_CERTIFICATE=path/to/nuclei-user.crt 
        
           ``` 
        
           or  
        
           ```console 
        
           $ export NUCLEI_USER_CERTIFICATE=$(cat path/to/nuclei-user.crt) 
        
           ``` 
        
           #### Verifying Templates 
        
           Everytime `nuclei` is run, it loads user certificate (aka public key) from above locations and uses it to verify templates. 
        
           `nuclei` also prints identifier of public key being used and warns user of unsigned custom templates 
        
           ``` 
        
           [INF] Executing 6219 signed templates from 'ProjectDiscovery.io' 
        
           [WRN] Executing 687 unsigned templates. Use with caution 
        
           ``` 
        
           ### Managing Private Key 
        
           Private key is stored in $config/nuclei/keys/nuclei-user-private-key.pem and is encrypted with passphrase if provided while generating key-pair. 
        
           It is not used/loaded by default by nuclei and is only used on demand i.e when signing templates using `-sign` flag 
        
           Some Users might want to store / backup or move private key to different location or machine and `nuclei` doesn't enforce any restrictions on that. 
        
           #### Using Private Key 
        
           - A simple way to use private key is to copy it to $config/nuclei/keys directory of other user's machine 
        
           - Another way is to use environment variable `NUCLEI_USER_PRIVATE_KEY=xxx` to specify path of private key or content of private key directly 
        
           ```console 
        
           $ export NUCLEI_USER_PRIVATE_KEY=path/to/nuclei-user-private-key.pem 
        
           ``` 
        
           or 
        
           ```console 
        
           $ export NUCLEI_USER_PRIVATE_KEY=$(cat path/to/nuclei-user-private-key.pem) 
        
           ``` 
        
           > Note: You are responsible for securing and managing private key and nuclei has no accountability for any loss of private key

TBD

discussion and user experience of re-signing of code protocol
hardning mass signing of code protocol templates

tarunKoyalwar · 2023-10-08T16:20:37Z

example code template

id: simple-code

info:
  name: simple-code
  author: pdteam
  severity: info
  tags: code
  description: |
    simple-code

code:
  - engine:
      - py
      - python3
    source: |
      import sys
      print("hello from code s" + sys.stdin.read())
    
    matchers:
      - type: word
        words:
          - "hello from code"

when template does not have any digest/signature

./nuclei -u scanme.sh -t simple-code.yaml 

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.0.0-dev

		projectdiscovery.io

[WRN] Found 1 unverified templates (carefully examine the template before using it and use -sign flag to sign them)
[INF] Current nuclei version: v3.0.0-dev (development)
[INF] Current nuclei-templates version: v9.6.4 (latest)
[WRN] Executing 1 unsigned templates. Use with caution
[INF] Targets loaded for current scan: 1
[INF] No results found. Better luck next time!
[FTL] Could not run nuclei: no templates provided for scan

signing a template

$ ./nuclei -u scanme.sh -t simple-code.yaml -sign                  
[INF] All templates signatures were elaborated success=1 failed=0

auto-verification of signed template before executing

./nuclei -u scanme.sh -t simple-code.yaml      

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.0.0-dev

		projectdiscovery.io

[INF] Current nuclei version: v3.0.0-dev (development)
[INF] Current nuclei-templates version: v9.6.4 (latest)
[INF] New templates added in latest release: 121
[INF] Templates loaded for current scan: 1
[INF] Executing 1 signed templates from 'tarun'
[INF] Targets loaded for current scan: 1
[simple-code] [code] [info] scanme.sh

when template content is modified template is considered as not verified template

./nuclei -u scanme.sh -t simple-code.yaml 

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.0.0-dev

		projectdiscovery.io

[WRN] Found 1 unverified templates (carefully examine the template before using it and use -sign flag to sign them)
[INF] Current nuclei version: v3.0.0-dev (development)
[INF] Current nuclei-templates version: v9.6.4 (latest)
[WRN] Executing 1 unsigned templates. Use with caution
[INF] Targets loaded for current scan: 1
[INF] No results found. Better luck next time!
[FTL] Could not run nuclei: no templates provided for scan

re-signing code template if fragment matches with current user/signer

$ ./nuclei -u scanme.sh -t simple-code.yaml -sign                   
[INF] All templates signatures were elaborated success=1 failed=0

re-signing of code templates when fragment is not present or does not belong to current signer/user

example template

id: simple-code

info:
  name: simple-code
  author: pdteam
  severity: info
  tags: code
  description: |
    simple-code

code:
  - engine:
      - py
      - python3
    source: |
      import sys
      print("hello from code" + sys.stdin.read())
    
    matchers:
      - type: word
        words:
          - "hello from code"
# digest: 4a0a0047304502207cc624edaa99107f1a4b68ad246e3e48e888735cfde4f8882c4d72a5a98009d40221009ef1b4590ebeaee24d60d8485758e3711cba88c2a3eada7c2a7e18f82afb4393:abacus

in this case even if user tries to re-sign code protocol template then it is not allowed as fragment does not match
this means user is trying to sign a code protocol template that does not belong to user without viewing which is discouraged
this can be considered as a security bump to discourage mass signing of templates with code protocol without actually verfiying them visually .

$ ./nuclei -u scanme.sh -t simple-code.yaml -v -sign                
[ERR] could not sign 'simple-code.yaml': [signer:RUNTIME] fragment mismatch: expected '73812c4e0e52692225979bd2d5f05a3c' got 'abacus'
[INF] All templates signatures were elaborated success=0 failed=1

to resolve this situation user needs to manually remove digest from template

ehsandeep

printing issues when running workflows
printing issue with non-yaml files (.yaml) when used with -sign option
printing issue with counter display with workflows for unsigned templates

tarunKoyalwar · 2023-10-12T19:53:44Z

$ ./nuclei                                                          

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.0.0-dev

		projectdiscovery.io

[INF] Current nuclei version: v3.0.0-dev (development)
[INF] Current nuclei-templates version: v9.6.5 (latest)
[INF] New templates added in latest release: 75
[INF] Templates loaded for current scan: 6967
[INF] Executing 6978 signed templates from tarunKoyalwar
[INF] No results found. Better luck next time!

tarunKoyalwar · 2023-10-12T19:57:45Z

@ehsandeep it will show 1 unsigned template because in nuclei file extension filtering is case-sensitive and looks like there is a template in nuclei-templates repo (only 1) that has .YAML as extension instead of usual .yaml (ref: https://github.com/projectdiscovery/nuclei-templates/blob/main/file/nodejs/xss-disable-mustache-escape.YAML)

when renamed to .yaml there will be no unsigned templates as shown in above example

tarunKoyalwar added 2 commits September 22, 2023 02:13

store and generate signer keys

35d6df9

fix trailing newline in code_response

a543655

tarunKoyalwar requested a review from ehsandeep September 21, 2023 20:54

tarunKoyalwar marked this pull request as draft September 21, 2023 20:55

fix formatting and update error string

239cb71

tarunKoyalwar linked an issue Sep 21, 2023 that may be closed by this pull request

Nuclei v3 Minor inconsitencies and enchancements #4173

Closed

tarunKoyalwar self-assigned this Sep 21, 2023

tarunKoyalwar added 3 commits September 22, 2023 03:26

fix integration test

4ef5367

fix rsaSigned code integration test

2560d6b

bug fixes , docs and more

acd78f0

tarunKoyalwar added 5 commits September 22, 2023 23:07

bump go -> 1.21

b015f06

use 'response' as default part in code templates

94e6ad0

disable sourcemaps for all js runtimes

a0167d4

disable eval function

2f222ff

rewrite file validation in sandbox mode

2782b47

ehsandeep added the Type: Maintenance Updating phrasing or wording to make things clearer or removing ambiguity. label Sep 28, 2023

tarunKoyalwar added 5 commits September 29, 2023 19:25

Merge branch 'v3-beta' into issue-4173-v3-bugfixes

0aeefa5

sandbox file read improvements + minor refactor

27382cc

refactor sign and verify logic

91fb250

fix panic and missing id in code protocol

894879c

disable re-signing code protocol templates

c1dbb8e

tarunKoyalwar added 6 commits October 1, 2023 22:14

fix code resigning in tests

b2df497

allow -lfa in test for signing templates

a7eb230

start index from 1 in flow and multiproto

3a5f9d6

remove testfiles

fdd255e

add python in integration test

88a6de9

update code protocol docs

bcb0a26

tarunKoyalwar added 3 commits October 4, 2023 01:28

fix integration test and more

d8cb12e

reworked template signer

bf7b5af

fix lint error

54baca5

tarunKoyalwar added 2 commits October 4, 2023 21:50

display signature stats

04475e2

update docs

b90f629

tarunKoyalwar marked this pull request as ready for review October 4, 2023 19:05

tarunKoyalwar requested a review from Mzack9999 October 4, 2023 19:08

Mzack9999 approved these changes Oct 5, 2023

View reviewed changes

add user fragment to signature

3a5e9b4

tarunKoyalwar and others added 4 commits October 8, 2023 21:51

use md5 to generate fragment

e5092b9

update docs with code re-sign

e34be57

misc updates

751494a

public crt update

d87cd2c

ehsandeep requested changes Oct 11, 2023

View reviewed changes

tarunKoyalwar added 4 commits October 12, 2023 23:51

remove workflow info statement

e2e8a86

fix printing issues

224c1b5

refactor preprocessor logic

d56867f

remove debug statement

1dd64a2

fix failing example test

f6c0bd5

tarunKoyalwar requested a review from ehsandeep October 12, 2023 20:19

ehsandeep and others added 2 commits October 13, 2023 12:01

Merge branch 'v3-beta' into issue-4173-v3-bugfixes

1cc94a1

go mod tidy

253b889

ehsandeep approved these changes Oct 13, 2023

View reviewed changes

ehsandeep merged commit c35162c into v3-beta Oct 13, 2023
8 of 11 checks passed

ehsandeep deleted the issue-4173-v3-bugfixes branch October 13, 2023 07:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nuclei v3 bug fixes #4176

nuclei v3 bug fixes #4176

tarunKoyalwar commented Sep 21, 2023 •

edited

Loading

tarunKoyalwar commented Sep 21, 2023

tarunKoyalwar commented Sep 22, 2023 •

edited

Loading

tarunKoyalwar commented Oct 1, 2023

tarunKoyalwar commented Oct 4, 2023 •

edited

Loading

tarunKoyalwar commented Oct 4, 2023 •

edited

Loading

tarunKoyalwar commented Oct 8, 2023

ehsandeep left a comment •

edited by tarunKoyalwar

Loading

tarunKoyalwar commented Oct 12, 2023

tarunKoyalwar commented Oct 12, 2023

nuclei v3 bug fixes #4176

nuclei v3 bug fixes #4176

Conversation

tarunKoyalwar commented Sep 21, 2023 • edited Loading

Proposed changes

Checklist

tarunKoyalwar commented Sep 21, 2023

example multiprotocol template

tarunKoyalwar commented Sep 22, 2023 • edited Loading

TODO

tarunKoyalwar commented Oct 1, 2023

Signing templates by setting public/private keys in enviornment variable (POC)

tarunKoyalwar commented Oct 4, 2023 • edited Loading

User experience using signed and unsigned templates

using encrypted private key with passphrase

tarunKoyalwar commented Oct 4, 2023 • edited Loading

New Changes

TBD

tarunKoyalwar commented Oct 8, 2023

example code template

when template does not have any digest/signature

signing a template

auto-verification of signed template before executing

when template content is modified template is considered as not verified template

re-signing code template if fragment matches with current user/signer

re-signing of code templates when fragment is not present or does not belong to current signer/user

ehsandeep left a comment • edited by tarunKoyalwar Loading

Choose a reason for hiding this comment

tarunKoyalwar commented Oct 12, 2023

tarunKoyalwar commented Oct 12, 2023

tarunKoyalwar commented Sep 21, 2023 •

edited

Loading

tarunKoyalwar commented Sep 22, 2023 •

edited

Loading

tarunKoyalwar commented Oct 4, 2023 •

edited

Loading

tarunKoyalwar commented Oct 4, 2023 •

edited

Loading

ehsandeep left a comment •

edited by tarunKoyalwar

Loading