Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

replace net/url with urlutil #399

Merged
merged 8 commits into from
Apr 8, 2023
Merged

replace net/url with urlutil #399

merged 8 commits into from
Apr 8, 2023

Conversation

iamargus95
Copy link
Contributor

  • Replaced net/url with urlutil:
    • url.Parse -> urlutil.Parse where url is given as input or contains some complex exploit.
  • Variables of type *urlutil.URL with URL in the suffix renamed i.e :
    • parsedURL -> parsed

    To avoid it being used as parsedURL.URL.

  • Used urlutil's .Update() method when referencing http.Request.URL as mentioned in the notes here.

@ShubhamRasal ShubhamRasal linked an issue Apr 6, 2023 that may be closed by this pull request
2 tasks
@tarunKoyalwar
Copy link
Member

New changes

  • earlier katana input required scheme i.e http , https (katana -u hackerone.com would fail silently)
  • ^ is fixed now and if input does not have a scheme katana tries to add one and runs
  • adds -debug flag for debug data / verbose logging
  • improved error logging

Copy link
Member

@tarunKoyalwar tarunKoyalwar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm !

$ ./katana -u hackerone.com  | head -n 10

   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/							 

		projectdiscovery.io

[INF] Current katana version v1.0.0 (latest)
[INF] Started Crawling Target: https://hackerone.com
https://www.hackerone.com/
https://hackerone.com
https://www.hackerone.com/sites/default/files/css/css_0tloVjMoPsh3cM5bG6CU7uN0_ka2kUclBNpTcih8VVM.css
https://www.hackerone.com/sites/default/files/js/js_Ikd9nsZ0AFAesOLgcgjc7F6CRoODbeqOn7SVbsXgALQ.js
https://docs.hackerone.com/
https://www.hackerone.com/sites/default/files/js/js_C-5Xm0bH3IRZtqPDWPr8Ga4sby1ARHgF6iBlpL4UHao.js
https://www.hackerone.com/sites/default/files/css/css_Od8DNnS6eVnRJdunvzGK7xDiHeWiMXyL5pWL3odLDbw.css
https://www.hackerone.com/sites/default/files/js/js_4FuDbOJrjJz7g2Uu2GQ6ZFtnbdPymNgBpNtoRkgooH8.js
https://www.hackerone.com/policies
https://www.hackerone.com/security

@tarunKoyalwar
Copy link
Member

@iamargus95 , thanks for PR and helping with refactor

@ehsandeep ehsandeep linked an issue Apr 8, 2023 that may be closed by this pull request
@ehsandeep ehsandeep merged commit 028dd19 into projectdiscovery:dev Apr 8, 2023
@iamargus95 iamargus95 deleted the issue-376-replace-neturl-with-urlutil branch April 8, 2023 10:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support automatic probing when url scheme is not provided Replace net.Url with urlutil wrapper
3 participants