Increase crawling content by optimizing regular expressions

[yuzhe-Mortal]

Increase crawling content by optimizing regular expressions

katana-main/katana-main/pkg/utils/regex.go Connections in web pages are extracted by regular expressions
It's implemented by pageBodyRegex or relativeEndpointsRegex
But you will miss some endpoints

for example

http://www.google.com:8080
https://www.google.com/images/%E4%BA%A7%E5%93%81%E4%B8%AD%E5%BF%83/
/1.php

Then the following regular expressions will help you

    ((?:"|'|\s)                              # Start newline delimiter
  (
    ((?:[a-zA-Z]{1,10}://|//)           # Match a scheme [a-Z]*1-10 or //
    [^"'/]{1,}\.                        # Match a domainname (any character + dot)
    [a-zA-Z]{2,}[^"']{0,})              # The domainextension and/or path
    |
    ((?:/|\.\./|\./)                    # Start with /,../,./
    [^"'><,;| *()(%%$^/\\\[\]]          # Next character can't be...
    [^"'><,;|()]{1,})                   # Rest of the characters can't be
    |
    ([a-zA-Z0-9_\-/]{1,}/               # Relative endpoint with /
    [a-zA-Z0-9_\-/\.]{1,}                 # Resource name
    \.(?:[a-zA-Z]{1,4}|action)          # Rest + extension (length 1-4 or action)
    (?:[\?|#][^"|']{0,}|))              # ? mark with parameters
    |
    ([a-zA-Z0-9_\-/]{1,}/
    [a-zA-Z0-9_\-/]{3,}
    (?:[\?|#][^"|']{0,}|))
    |
    ([a-zA-Z0-9_\-\.]{1,}                 # filename
    \.(?:php|asp|aspx|jsp|json|
         action|html|js|txt|xml|do)             # . + extension
    (?:[\?|#][^"|']{0,}|))                  # ? mark with parameters
  )
  (?:"|'|\s)

reference linking: https://github.com/yuzhe-Mortal/tool/blob/main/Reptile.py

[yuzhe-Mortal]

Why can't we combine pageBodyRegex and relativeEndpointsRegex
In the absence of Single quote,double quote, and space, endpoints are extracted using html tags