Update regex.go

[yuzhe-Mortal]

Increase crawling content by optimizing regular expressions

katana-main/katana-main/pkg/utils/regex.go Connections in web pages are extracted by regular expressions
It's implemented by pageBodyRegex or relativeEndpointsRegex
But you will miss some endpoints

for example

http://www.google.com:8080
https://www.google.com/images/%E4%BA%A7%E5%93%81%E4%B8%AD%E5%BF%83/
/1.php

Then the following regular expressions will help you

    ((?:"|'|\s)                              # Start newline delimiter
  (
    ((?:[a-zA-Z]{1,10}://|//)           # Match a scheme [a-Z]*1-10 or //
    [^"'/]{1,}\.                        # Match a domainname (any character + dot)
    [a-zA-Z]{2,}[^"']{0,})              # The domainextension and/or path
    |
    ((?:/|\.\./|\./)                    # Start with /,../,./
    [^"'><,;| *()(%%$^/\\\[\]]          # Next character can't be...
    [^"'><,;|()]{1,})                   # Rest of the characters can't be
    |
    ([a-zA-Z0-9_\-/]{1,}/               # Relative endpoint with /
    [a-zA-Z0-9_\-/\.]{1,}                 # Resource name
    \.(?:[a-zA-Z]{1,4}|action)          # Rest + extension (length 1-4 or action)
    (?:[\?|#][^"|']{0,}|))              # ? mark with parameters
    |
    ([a-zA-Z0-9_\-/]{1,}/
    [a-zA-Z0-9_\-/]{3,}
    (?:[\?|#][^"|']{0,}|))
    |
    ([a-zA-Z0-9_\-\.]{1,}                 # filename
    \.(?:php|asp|aspx|jsp|json|
         action|html|js|txt|xml|do)             # . + extension
    (?:[\?|#][^"|']{0,}|))                  # ? mark with parameters
  )
  (?:"|'|\s)

reference linking: https://github.com/yuzhe-Mortal/tool/blob/main/Reptile.py

[yuzhe-Mortal]

update Regex #239

[gabriel-schneider-vtex]

+1 for this PR to get accepted, relatives URLs aren't being captured properly as of now.

[tarunKoyalwar]

@yuzhe-Mortal , this seems like a good regex but since this directly affects number of results returned by katana , we have to add more unit tests to validate that we aren't missing out any potential endpoints .

[yuzhe-Mortal]

@tarunKoyalwar I need specific URL examples to modify the regex

[gabriel-schneider-vtex]

@yuzhe-Mortal

@tarunKoyalwar I need specific URL examples to modify the regex

Hi, I've hosted here a fragment of a JS file that has relative URLs in it. LinkFinder finds the relative URLs, where katana does not.

~ ❯ python3 ~/LinkFinder/linkfinder.py -i  https://gbrls.space/katana.js -o cli 
management/commerce/orders/
/generic/
management/commerce/orders/paymentmethods/
/installments/
/payment/
~ ❯ katana -u https://gbrls.space/katana.js

   __        __
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.3

		projectdiscovery.io

~ ❯  

[yuzhe-Mortal]

@gabriel-schneider-vtex @tarunKoyalwar
new Regex

var (
	BodyA0 = `(?:`
	BodyB0 = `(`
	BodyC0 = `(?:[\.]{1,2}/[A-Za-z0-9-_/\\?&@\.?=%]+)`
	BodyC1 = `|(https?://[A-Za-z0-9_\-\.]+([\.]{0,2})?\/[A-Za-z0-9-_/\\?&@\.?=%]+)`
	BodyC2 = `|(/[A-Za-z0-9-_/\\?&@\.%]+\.(aspx?|action|cfm|cgi|do|pl|css|x?html?|js(p|on)?|pdf|php5?|py|rss))`
	BodyB1 = `)`
	BodyA1 = `)`
	// pageBodyRegex extracts endpoints from page body
	pageBodyRegex = regexp.MustCompile(BodyA0 + BodyB0 + BodyC0 + BodyC1 + BodyC2 + BodyB1 + BodyA1)

	JsA0 = `(?:"|'|\s)`
	JsB0 = `(`
	JsC0 = `((https?://[A-Za-z0-9_\-\.]+(:\d{1,5})?)+([\.]{1,2})?/[A-Za-z0-9/\-_\.\\%]+([\?|#][^"']+)?)`
	JsC1 = `|((\.{1,2}/)?[a-zA-Z0-9\-_/\\%]+\.(aspx?|js(on|p)?|html|php5?|html|action|do)([\?|#][^"']+)?)`
	JsC2 = `|((\.{0,2}/)[a-zA-Z0-9\-_/\\%]+(/|\\)[a-zA-Z0-9\-_]{3,}([\?|#][^"|']+)?)`
	JsC3 = `|((\.{0,2})[a-zA-Z0-9\-_/\\%]{3,}/)`
	JsB1 = `)`
	JsA1 = `(?:"|'|\s)`
	// relativeEndpointsRegex is the regex to find endpoints in js files.
	relativeEndpointsRegex = regexp.MustCompile(JsA0 + JsB0 + JsC0 + JsC1 + JsC2 + JsC3 + JsB1 + JsA1)
)

katana.exe -u https://gbrls.space/katana.js -o D:\Desktop\1.json -v -d 10

   __        __
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.2

                projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions.
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
start Crawl,input： https://gbrls.space/katana.js
[js] [404 Not Found] https://gbrls.space/management/commerce/orders/ [bodylen:555] [Source:https://gbrls.space/katana.js]
[js] [404 Not Found] https://gbrls.space/generic/ [bodylen:555] [Source:https://gbrls.space/katana.js]
[js] [404 Not Found] https://gbrls.space/management/commerce/orders/paymentmethods/ [bodylen:555] [Source:https://gbrls.space/katana.js]
[js] [404 Not Found] https://gbrls.space/installments/ [bodylen:555] [Source:https://gbrls.space/katana.js]
[js] [404 Not Found] https://gbrls.space/payment/ [bodylen:555] [Source:https://gbrls.space/katana.js]

[yuzhe-Mortal]

I've also made the code more readable, please let me know if there are other cases

[tarunKoyalwar]

lgtm !

[tarunKoyalwar]

• Follow up issue Improve url regex #337