Core Refactor #312

Mzack9999 · 2023-02-14T20:01:10Z

This PR implements the following significant changes (some of them breaking compatibility):

Native correlation between HTTP request/response
Turned nested circular callbacks into a linear sequential flow
Merged stdout + export into a unique helper function
Multiple internal changes to ease metadata availability (no more carrier fields to return info to the caller)

Closes #303

ehsandeep

Merge conflict

ehsandeep

@Mzack9999

Run from dev/main branch

echo https://www.hackerone.com | katana | wc

   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.3							 

		projectdiscovery.io

     669

Run from refactor branch -

echo https://www.hackerone.com | ./katana | wc

   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.3							 

		projectdiscovery.io

     117

tarunKoyalwar

Implementation looks great ! I just noticed a couple of things

we still use url.Parse() and later on .Hostname() so if katana input does not have scheme in url and has port included or other cases it will return empty hostname and etc .
Hence I think it would be great if we could stick to urlutil.Parse()
I am getting opposite results to what @ehsandeep mentioned in above comment not sure why . but I think we can improve/add some verbose and debug logging more specifically
print/log crawl depth for each printed url

// Ex:
[INF] https://hackerone.com/hacktivity [depth:1]

print/log unique navigation requests from source or at least their count

// Ex:
[INF] Found 3 endpoints/paths from bodyparseregex for `scanme.sh`

tarunKoyalwar · 2023-02-23T19:11:51Z

@Mzack9999 also I think we need to fix this . in this PR verbose mode returns less results than without verbose

$ ./katana -u https://hackerone.com -v

   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.3							 

		projectdiscovery.io

[header] [GET] https://www.hackerone.com/

$  ./katana -u https://hackerone.com   

   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.3							 

		projectdiscovery.io

https://hackerone.com
https://hackerone.com/users/sign_in
https://www.hackerone.com/attack-resistance-assessment
https://www.hackerone.com/security-incident
https://www.hackerone.com/sites/default/files/js/js_Ikd9nsZ0AFAesOLgcgjc7F6CRoODbeqOn7SVbsXgALQ.js
https://www.hackerone.com/sites/default/files/js/js_C-5Xm0bH3IRZtqPDWPr8Ga4sby1ARHgF6iBlpL4UHao.js
https://www.hackerone.com/libraries/hackeronesocials/svg/instagram.svg
https://www.hackerone.com/contact
https://www.hackerone.com/libraries/hackeronesocials/svg/linkedin.svg
https://www.hackerone.com/libraries/hackeronesocials/svg/twitter.svg
https://www.hackerone.com/libraries/hackeronesocials/svg/facebook.svg
https://www.hackerone.com/themes/hacker_one/images/h1-short.svg
https://www.hackerone.com/product/attack-surface-management
https://www.hackerone.com/sites/default/files/styles/carousel_spotlight/public/pete-yaworski.png.webp?itok=tERRF7Hc
https://www.hackerone.com/sites/default/files/styles/testimonial_person/public/Benjamin_Vaughn_Hyatt2X.png.webp?itok=qEivB0sA
https://www.hackerone.com/solutions/vulnerability-management-system
https://www.hackerone.com/sites/default/files/styles/testimonial_company_2x/public/Hyatt2X.png.webp?itok=00nbbZjl
https://www.hackerone.com/sites/default/files/styles/testimonial_person/public/Abishek_Gupta_Hired2X.png.webp?itok=eBTNtSuq
https://www.hackerone.com/sites/default/files/styles/testimonial_company_2x/public/Hired2X_0.png.webp?itok=qBm6wfuq
https://www.hackerone.com/sites/default/files/styles/card_default/public/HAC_General_Blog_8_Main_Featured_560x494_L1R1.png?itok=qnbFouxA
https://www.hackerone.com/sites/default/files/styles/card_default/public/HAC_ARM%20Report_Image%20Glitch_L1R1_0.png?itok=g5WJ8lJn
https://www.hackerone.com/sites/default/files/styles/testimonial_person/public/Kevin_Pawloski_GoodRX2X.png.webp?itok=2DT_nE29
https://www.hackerone.com/sites/default/files/styles/card_default/public/BlogHeader_General_2.png?itok=oG0Tv8ju
https://www.hackerone.com/sites/default/files/styles/testimonial_company_2x/public/GoodRx2X.png.webp?itok=gTPC1ATy
https://www.hackerone.com/events/rsa-conference-2023
https://www.hackerone.com/sites/default/files/styles/carousel_tabbed/public/design-scope-image.png.webp?itok=fZocPqUb
https://www.hackerone.com/sites/default/files/styles/carousel_tabbed/public/Carosel_PickSolution%20%281%29%202.png.webp?itok=9RY31jLo
https://www.hackerone.com/sites/default/files/styles/logo_band_white/public/AT_T2X.png.webp?itok=XUFDa0FN
https://www.hackerone.com/sites/default/files/styles/carousel_tabbed/public/testing-platform-image.png.webp?itok=9ym4wmbm
https://www.hackerone.com/sites/default/files/styles/logo_band_white/public/Hyatt2X.png.webp?itok=mwUmuMWm
https://www.hackerone.com/sites/default/files/styles/logo_band_white/public/GM2X_0.png.webp?itok=_mTFOxRw
https://www.hackerone.com/sites/default/files/styles/logo_band_white/public/PayPal2X.png.webp?itok=UJfM8O9a
https://www.hackerone.com/sites/default/files/styles/logo_band_white/public/Nintendo2X_0.png.webp?itok=h2H_a61-
https://www.hackerone.com/sites/default/files/styles/carousel_tabbed/public/HAC-ARM-Product-1-L1R1%402x.png.webp?itok=QulSSLb0
https://www.hackerone.com/themes/hacker_one/images/logo-hackerone-light.svg
https://www.hackerone.com/themes/hacker_one/images/logo-hackerone.svg
https://www.hackerone.com/solutions/attack-resistance-management
https://www.hackerone.com/sites/default/files/css/css_0tloVjMoPsh3cM5bG6CU7uN0_ka2kUclBNpTcih8VVM.css
https://www.hackerone.com/sites/default/files/styles/hero_main/public/HERO_%20%281%29%202.png.webp?itok=eWJYJ0v_
https://www.hackerone.com/6th-annual-hacker-powered-security-report
https://www.hackerone.com/security-at-beyond
https://www.hackerone.com/themes/hacker_one/favicon.ico
https://www.hackerone.com/themes/hacker_one/fonts/Crimson_Pro/static/CrimsonPro-Regular.ttf
https://www.hackerone.com/themes/hacker_one/fonts/Ubuntu_Mono/UbuntuMono-Bold.ttf
https://www.hackerone.com/themes/hacker_one/fonts/Poppins/Poppins-Bold.ttf
https://www.hackerone.com/themes/hacker_one/fonts/Poppins/Poppins-Medium.ttf
https://www.hackerone.com/themes/hacker_one/fonts/Poppins/Poppins-SemiBold.ttf
https://www.hackerone.com/themes/hacker_one/fonts/Poppins/Poppins-Regular.ttf
https://www.hackerone.com/sites/default/files/js/js_NRe-hZ07eiFRt1gRJK2zue5BfPQe92f5H_XATmL_1Ag.js
https://www.hackerone.com/sites/default/files/css/css_ewaZPLL5fah2rnAxlk1Z2WtffKbc3HBNHpfNu2RElsA.css
https://www.hackerone.com/privacy
https://hackerone.com/directory/programs?order_direction=DESC&order_field=resolved_report_count
https://www.hackerone.com/terms
https://www.hackerone.com/policies
https://www.hackerone.com/security
https://docs.hackerone.com/
https://www.hackerone.com/services-2
https://www.hackerone.com/customer-stories/how-wix-improves-their-security-posture-ethical-hackers
https://www.hackerone.com/company-news/announcing-hackerone-2022-attack-resistance-report-security-survey-how-close-your
https://hackerone.com/leaderboard
https://www.hackerone.com/penetration-testing/bug-bounty-vs-penetration-testing-differences-explained
https://www.hackerone.com/hackerone-community-blog
https://www.hackerone.com/vulnerability-management
https://www.hackerone.com/customer-hub/ATT
https://www.hackerone.com/customer-hub/Hyatt
https://www.hackerone.com/customer-hub/GM
https://www.hackerone.com/customer-hub/Nintendo
https://www.hackerone.com/penetration-testing
https://www.hackerone.com/security-compliance
https://www.hackerone.com/resources/customer-story/how-hired-builds-customer-trust-with-hackerone-pentest
https://www.hackerone.com/customer-hub/Paypal
https://www.hackerone.com/knowledge-center/what-vulnerability-assessment-benefits-tools-and-process
https://www.hackerone.com/security-at/2021
https://www.hackerone.com/knowledge-center/what-penetration-testing-how-does-it-work-step-step
https://www.hackerone.com/vulnerability-and-security-testing-blog
https://www.hackerone.com/application-security
https://www.hackerone.com/knowledge-center/security-compliance-ten-regulations-and-four-tips-success
https://www.hackerone.com/knowledge-center/devsecops-quick-guide-process-tools-and-best-practices
https://www.hackerone.com/knowledge-center/beyond-owasp-top-ten-13-resources-boost-your-security
https://www.hackerone.com/knowledge-center/16-types-cybersecurity-attacks-and-how-prevent-them
https://www.hackerone.com/knowledge-center/cloud-security-challenges-solutions-and-best-practices
https://www.hackerone.com/knowledge-center/attack-surface-and-how-analyze-manage-and-reduce-it
https://www.hackerone.com/knowledge-center/what-application-security-concepts-tools-best-practices
https://www.hackerone.com/knowledge-center
https://www.hackerone.com/ethical-hacker
https://www.hackerone.com/hacktivitycon
https://hackerone.com/opportunities/all/search
https://www.hackerone.com/hackers/hacker101
https://www.hackerone.com/events
https://www.hackerone.com/company-news
https://www.hackerone.com/resources
https://www.hackerone.com/press
https://www.hackerone.com/security-at-2022
https://www.hackerone.com/hackers
https://hackerone.com/leaderboard/all-time
https://www.hackerone.com/press-archive
https://www.hackerone.com/customer-stories
https://www.hackerone.com/trust
https://www.hackerone.com/careers
https://www.hackerone.com/leadership
https://www.hackerone.com/partners/aws
https://www.hackerone.com/company
https://www.hackerone.com/partners
https://www.hackerone.com/product/security-assessments
https://hackerone.com/hacktivity
https://www.hackerone.com/partners/integrations
https://www.hackerone.com/product/pentest
https://www.hackerone.com/product/insights
https://www.hackerone.com/product/response-vulnerability-disclosure-program
https://www.hackerone.com/product/bug-bounty-platform
https://www.hackerone.com/services
https://www.hackerone.com/product/overview
https://www.hackerone.com/solutions/government
https://www.hackerone.com/solutions/united-states-federal
https://www.hackerone.com/solutions/financial-services
https://www.hackerone.com/solutions/application-security-testing-software
https://www.hackerone.com/solutions/cloud-security-solution

Mzack9999 · 2023-02-24T16:57:18Z

@ehsandeep This is expected behavior. Katana outputs all discovered and not yet visited endpoints. In contrast, in this branch, the behavior is to output discovered and visited endpoints, so you need to add one level of depth to obtain the same number of results:

echo https://www.hackerone.com | go run . -d 3 | wc
610     610   53619

@tarunKoyalwar I will create follow up tickets both for logs and urlutil as tests are failing and unfortunately it doesn't seem a ready drop-in replacement. I believe that the -v failure is probably just temporary (cdn filtering?):

$ go run . -u https://hackerone.com -v

   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.3                                                    

                projectdiscovery.io

[GET] https://hackerone.com
[a] [GET] https://hackerone.com/users/sign_in
[a] [GET] https://www.hackerone.com/attack-resistance-assessment
[a] [GET] https://www.hackerone.com/events/rsa-conference-2023
[script] [GET] https://www.hackerone.com/sites/default/files/js/js_Ikd9nsZ0AFAesOLgcgjc7F6CRoODbeqOn7SVbsXgALQ.js
[img] [GET] https://www.hackerone.com/libraries/hackeronesocials/svg/instagram.svg
[img] [GET] https://www.hackerone.com/libraries/hackeronesocials/svg/linkedin.svg

Requesting a new review as the reported problems seem either intended behavior vs temporary failure

tarunKoyalwar

Not sure the reason but katana randomly returns different results

$ go run . -u https://hackerone.com -v -jc                       1 ↵

   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.3							 

		projectdiscovery.io

[GET] https://hackerone.com
[a] [GET] https://hackerone.com/users/sign_in
[a] [GET] https://www.hackerone.com/events/rsa-conference-2023
[a] [GET] https://www.hackerone.com/security-at-beyond
[a] [GET] https://www.hackerone.com/attack-resistance-assessment
[html] [GET] https://www.hackerone.com/core%5C/modules%5C/statistics%5C/statistics.php
[a] [GET] https://www.hackerone.com/security-incident
[a] [GET] https://www.hackerone.com/product/attack-surface-management
[a] [GET] https://www.hackerone.com/contact
[html] [GET] https://hackerone.com/directory/programs?order_direction=DESC&amp
[a] [GET] https://www.hackerone.com/solutions/attack-resistance-management
[script] [GET] https://www.hackerone.com/sites/default/files/js/js_Ikd9nsZ0AFAesOLgcgjc7F6CRoODbeqOn7SVbsXgALQ.js
[html] [GET] https://www.hackerone.com/sites/default/files/HAC_ARM_Organic
[img] [GET] https://www.hackerone.com/libraries/hackeronesocials/svg/instagram.svg
[script] [GET] https://www.hackerone.com/gtm.js
[img] [GET] https://www.hackerone.com/libraries/hackeronesocials/svg/linkedin.svg
[img] [GET] https://www.hackerone.com/libraries/hackeronesocials/svg/twitter.svg
[a] [GET] https://www.hackerone.com/6th-annual-hacker-powered-security-report
[img] [GET] https://www.hackerone.com/themes/hacker_one/images/h1-short.svg
[img] [GET] https://www.hackerone.com/libraries/hackeronesocials/svg/facebook.svg
[script] [GET] https://www.hackerone.com/angular.js
[img] [GET] https://www.hackerone.com/sites/default/files/styles/testimonial_company_2x/public/Hyatt2X.png.webp?itok=00nbbZjl
[img] [GET] https://www.hackerone.com/sites/default/files/styles/testimonial_person/public/Benjamin_Vaughn_Hyatt2X.png.webp?itok=qEivB0sA
[img] [GET] https://www.hackerone.com/sites/default/files/styles/carousel_spotlight/public/pete-yaworski.png.webp?itok=tERRF7Hc
[img] [GET] https://www.hackerone.com/sites/default/files/styles/testimonial_person/public/Abishek_Gupta_Hired2X.png.webp?itok=eBTNtSuq
[img] [GET] https://www.hackerone.com/sites/default/files/styles/testimonial_company_2x/public/Hired2X_0.png.webp?itok=qBm6wfuq
[img] [GET] https://www.hackerone.com/sites/default/files/styles/testimonial_person/public/Kevin_Pawloski_GoodRX2X.png.webp?itok=2DT_nE29
[img] [GET] https://www.hackerone.com/sites/default/files/styles/testimonial_company_2x/public/GoodRx2X.png.webp?itok=gTPC1ATy
[img] [GET] https://www.hackerone.com/sites/default/files/styles/carousel_tabbed/public/design-scope-image.png.webp?itok=fZocPqUb
[img] [GET] https://www.hackerone.com/sites/default/files/styles/carousel_tabbed/public/Carosel_PickSolution%20%281%29%202.png.webp?itok=9RY31jLo
[a] [GET] https://www.hackerone.com/solutions/cloud-security-solution
[img] [GET] https://www.hackerone.com/sites/default/files/styles/logo_band_white/public/AT_T2X.png.webp?itok=XUFDa0FN
[img] [GET] https://www.hackerone.com/sites/default/files/styles/carousel_tabbed/public/testing-platform-image.png.webp?itok=9ym4wmbm
[img] [GET] https://www.hackerone.com/sites/default/files/styles/card_default/public/HAC_ARM%20Report_Image%20Glitch_L1R1_0.png?itok=g5WJ8lJn
[img] [GET] https://www.hackerone.com/sites/default/files/styles/logo_band_white/public/Hyatt2X.png.webp?itok=mwUmuMWm
[img] [GET] https://www.hackerone.com/sites/default/files/styles/logo_band_white/public/GM2X_0.png.webp?itok=_mTFOxRw
[img] [GET] https://www.hackerone.com/sites/default/files/styles/logo_band_white/public/PayPal2X.png.webp?itok=UJfM8O9a
[img] [GET] https://www.hackerone.com/sites/default/files/styles/carousel_tabbed/public/HAC-ARM-Product-1-L1R1%402x.png.webp?itok=QulSSLb0
[img] [GET] https://www.hackerone.com/sites/default/files/styles/logo_band_white/public/Nintendo2X_0.png.webp?itok=h2H_a61-
[img] [GET] https://www.hackerone.com/sites/default/files/styles/card_default/public/HAC_General_Blog_8_Main_Featured_560x494_L1R1.png?itok=qnbFouxA
[img] [GET] https://www.hackerone.com/themes/hacker_one/images/logo-hackerone-light.svg
[img] [GET] https://www.hackerone.com/themes/hacker_one/images/logo-hackerone.svg
[img] [GET] https://www.hackerone.com/sites/default/files/styles/card_default/public/BlogHeader_General_2.png?itok=oG0Tv8ju
[link] [GET] https://www.hackerone.com/sites/default/files/css/css_0tloVjMoPsh3cM5bG6CU7uN0_ka2kUclBNpTcih8VVM.css
[img] [GET] https://www.hackerone.com/sites/default/files/styles/hero_main/public/HERO_%20%281%29%202.png.webp?itok=eWJYJ0v_
[link] [GET] https://www.hackerone.com/themes/hacker_one/favicon.ico
[link] [GET] https://www.hackerone.com/themes/hacker_one/fonts/Crimson_Pro/static/CrimsonPro-Regular.ttf
[link] [GET] https://www.hackerone.com/themes/hacker_one/fonts/Poppins/Poppins-Bold.ttf
[link] [GET] https://www.hackerone.com/themes/hacker_one/fonts/Poppins/Poppins-Medium.ttf
[link] [GET] https://www.hackerone.com/themes/hacker_one/fonts/Poppins/Poppins-Regular.ttf
[script] [GET] https://www.hackerone.com/sites/default/files/js/js_NRe-hZ07eiFRt1gRJK2zue5BfPQe92f5H_XATmL_1Ag.js
[link] [GET] https://www.hackerone.com/themes/hacker_one/fonts/Poppins/Poppins-SemiBold.ttf
[link] [GET] https://www.hackerone.com/themes/hacker_one/fonts/Ubuntu_Mono/UbuntuMono-Bold.ttf
[a] [GET] https://www.hackerone.com/privacy
[link] [GET] https://www.hackerone.com/sites/default/files/css/css_ewaZPLL5fah2rnAxlk1Z2WtffKbc3HBNHpfNu2RElsA.css
[a] [GET] https://www.hackerone.com/terms
[a] [GET] https://www.hackerone.com/policies
[a] [GET] https://docs.hackerone.com/
[a] [GET] https://hackerone.com/directory/programs?order_direction=DESC&order_field=resolved_report_count
[a] [GET] https://www.hackerone.com/security
[script] [GET] https://www.hackerone.com/sites/default/files/js/js_C-5Xm0bH3IRZtqPDWPr8Ga4sby1ARHgF6iBlpL4UHao.js
[a] [GET] https://www.hackerone.com/services-2
[a] [GET] https://www.hackerone.com/customer-stories/how-wix-improves-their-security-posture-ethical-hackers
[a] [GET] https://www.hackerone.com/company-news/announcing-hackerone-2022-attack-resistance-report-security-survey-how-close-your
[a] [GET] https://hackerone.com/leaderboard
[a] [GET] https://www.hackerone.com/penetration-testing/bug-bounty-vs-penetration-testing-differences-explained
[a] [GET] https://www.hackerone.com/customer-hub/Hyatt
[a] [GET] https://www.hackerone.com/customer-hub/ATT
[a] [GET] https://www.hackerone.com/customer-hub/GM
[a] [GET] https://www.hackerone.com/resources/customer-story/how-hired-builds-customer-trust-with-hackerone-pentest
[a] [GET] https://www.hackerone.com/customer-hub/Paypal
[a] [GET] https://www.hackerone.com/customer-hub/Nintendo
[a] [GET] https://www.hackerone.com/security-at/2021
[a] [GET] https://www.hackerone.com/hackerone-community-blog
[a] [GET] https://www.hackerone.com/vulnerability-management
[a] [GET] https://www.hackerone.com/security-compliance
[a] [GET] https://www.hackerone.com/company-news
[a] [GET] https://www.hackerone.com/penetration-testing
[a] [GET] https://www.hackerone.com/application-security
[a] [GET] https://www.hackerone.com/knowledge-center/what-vulnerability-assessment-benefits-tools-and-process
[a] [GET] https://www.hackerone.com/ethical-hacker
[a] [GET] https://www.hackerone.com/knowledge-center/security-compliance-ten-regulations-and-four-tips-success
[a] [GET] https://www.hackerone.com/vulnerability-and-security-testing-blog
[a] [GET] https://www.hackerone.com/knowledge-center/beyond-owasp-top-ten-13-resources-boost-your-security
[a] [GET] https://www.hackerone.com/knowledge-center/devsecops-quick-guide-process-tools-and-best-practices
[a] [GET] https://www.hackerone.com/knowledge-center/16-types-cybersecurity-attacks-and-how-prevent-them
[a] [GET] https://www.hackerone.com/knowledge-center/attack-surface-and-how-analyze-manage-and-reduce-it
[a] [GET] https://www.hackerone.com/knowledge-center/cloud-security-challenges-solutions-and-best-practices
[a] [GET] https://www.hackerone.com/knowledge-center/what-application-security-concepts-tools-best-practices
[a] [GET] https://www.hackerone.com/knowledge-center
[a] [GET] https://hackerone.com/leaderboard/all-time
[a] [GET] https://www.hackerone.com/knowledge-center/what-penetration-testing-how-does-it-work-step-step
[a] [GET] https://hackerone.com/hacktivity
[a] [GET] https://hackerone.com/opportunities/all/search
[a] [GET] https://www.hackerone.com/events
[a] [GET] https://www.hackerone.com/security-at-2022
[a] [GET] https://www.hackerone.com/hacktivitycon
[a] [GET] https://www.hackerone.com/hackers/hacker101
[a] [GET] https://www.hackerone.com/hackers
[a] [GET] https://www.hackerone.com/press-archive
[a] [GET] https://www.hackerone.com/leadership
[a] [GET] https://www.hackerone.com/resources
[a] [GET] https://www.hackerone.com/press
[a] [GET] https://www.hackerone.com/company
[a] [GET] https://www.hackerone.com/trust
[a] [GET] https://www.hackerone.com/partners/aws
[a] [GET] https://www.hackerone.com/customer-stories
[a] [GET] https://www.hackerone.com/partners
[a] [GET] https://www.hackerone.com/careers
[a] [GET] https://www.hackerone.com/product/insights
[a] [GET] https://www.hackerone.com/product/pentest
[a] [GET] https://www.hackerone.com/partners/integrations
[a] [GET] https://www.hackerone.com/product/security-assessments
[a] [GET] https://www.hackerone.com/services
[a] [GET] https://www.hackerone.com/product/response-vulnerability-disclosure-program
[a] [GET] https://www.hackerone.com/product/bug-bounty-platform
[a] [GET] https://www.hackerone.com/solutions/united-states-federal
[a] [GET] https://www.hackerone.com/solutions/government
[a] [GET] https://www.hackerone.com/solutions/financial-services
[a] [GET] https://www.hackerone.com/product/overview
[a] [GET] https://www.hackerone.com/solutions/application-security-testing-software
[a] [GET] https://www.hackerone.com/solutions/vulnerability-management-system

$ go run . -u https://hackerone.com -v -jc

   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.3							 

		projectdiscovery.io

[header] [GET] https://www.hackerone.com/

this only happens with this PR tried it with latest release but it returns expected output

tarunKoyalwar · 2023-02-26T06:29:58Z

@Mzack9999 error log file is empty , it looks like this was caused by output writer

$ go run . -u https://hackerone.com -v -jc -elog logg2.txt

   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.3							 

		projectdiscovery.io

[header] [GET] https://www.hackerone.com/
tarun@Taruns-MacBook-Pro:~/Codebase2/katana/cmd/katana(issue-303-core-refactor○) » cat logg2.txt 
tarun@Taruns-MacBook-Pro:~/Codebase2/katana/cmd/katana(issue-303-core-refactor○) »

tarunKoyalwar · 2023-02-26T06:31:22Z

@Mzack9999 total count remains same even if depth is changed

go run . -u https://hackerone.com -d 3 | wc

   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.3							 

		projectdiscovery.io

     117     117    8157

Mzack9999 · 2023-02-26T17:43:05Z

@tarunKoyalwar, hopefully, this should be fixed. The issue was related to the fact that redirects (hackerone.com => www.hackerone.com) set internally a depth of 2 for all subsequent requests, which happened to be the maximum default one via cli options. It was fine for not yet visited nodes but required to be incremented by one unit to mimic the previous behavior, indirectly fixing this issue.

tarunKoyalwar

lgtm!
we probably need to update README.md since we added new flag

tarunKoyalwar · 2023-03-01T09:13:56Z

Result Count

adding some points that decide number of results

Depth is incremented for every redirect and if a website has lot of redirects it will crawl less endpoints if depth is not properly adjusted
https://hackerone.com redirects to https://www.hackerone.com hence number of results differ depending on input url

katana -u https://hackerone.com // returns 1xx results
katana -u https://www.hackerone.com // returns 6xx results

@ehsandeep

outdated

yhy0 · 2023-03-14T02:28:53Z

There are no results

./katana -u https://brutelogic.com.br/knoxss.html -hl

ehsandeep

Few updates:

support for -sr option in headless mode (closes store-response to support in headless mode #217)
issues in headless mode with no headers and status code
including non-active endpoints as part of the output (CLI/JSON)
uniform headers format in json output between headless/non headless mode
Partial body dump in standard/headless mode with -sr
Default extensions deny list is not respected
Empty Host header in http.Request

begin of work

c6a43fb

Mzack9999 added the Type: Enhancement Most issues will probably ask for additions or changes. label Feb 14, 2023

Mzack9999 self-assigned this Feb 14, 2023

Mzack9999 linked an issue Feb 14, 2023 that may be closed by this pull request

Core rewrite for syncronous and easier request/response correlation #303

Closed

Mzack9999 marked this pull request as draft February 14, 2023 20:01

Mzack9999 and others added 4 commits February 15, 2023 13:28

removing enqueue callbacks

2077d10

removing more nested references

ee21b74

unifying output handling

1188eba

adding more response metadata

30b9a3c

Mzack9999 marked this pull request as ready for review February 16, 2023 19:45

ehsandeep previously requested changes Feb 17, 2023

View reviewed changes

ehsandeep requested a review from tarunKoyalwar February 17, 2023 14:20

Mzack9999 added 3 commits February 17, 2023 20:24

Merge branch 'dev' into issue-303-core-refactor

3f17e6d

fixing proxy url parsing

463b8af

typo

f1c96fe

Mzack9999 requested a review from ehsandeep February 17, 2023 19:39

ehsandeep reviewed Feb 20, 2023

View reviewed changes

tarunKoyalwar reviewed Feb 23, 2023

View reviewed changes

Merge branch 'dev' into issue-303-core-refactor

0176886

improving output filtering for empty tags

13747bb

Mzack9999 requested review from tarunKoyalwar and ehsandeep February 24, 2023 17:36

tarunKoyalwar requested changes Feb 26, 2023

View reviewed changes

Mzack9999 and others added 3 commits February 26, 2023 17:11

converting queue to channel like behavior

83045cd

fixing test

c1d367b

increasing default depth to handle 1 level redirect

7faaf14

Mzack9999 requested a review from tarunKoyalwar February 26, 2023 17:43

tarunKoyalwar approved these changes Mar 1, 2023

View reviewed changes

added new flags

4c6c61f

Mzack9999 added 2 commits March 10, 2023 12:35

Merge branch 'dev' into issue-303-core-refactor

c850cc8

fixing syntax

f9a2e88

Mzack9999 and others added 3 commits March 14, 2023 11:06

fixing output

a106fe2

removing already filtered content check

81510fc

attempting to match request w response in headless

c57c984

ehsandeep linked an issue Mar 15, 2023 that may be closed by this pull request

store-response to support in headless mode #217

Closed

ehsandeep requested changes Mar 15, 2023

View reviewed changes

Mzack9999 added 7 commits March 16, 2023 08:42

Merge branch 'dev' into issue-303-core-refactor

8e77204

improving req/resp matching + headless -sr

24fe4ee

custom cli-json friendly marshaling for headers

dc77702

always print result + error output

18a4642

using std functions to dump req/resp

4a48bd6

reworking extension exclusion

d3e994d

adding scope + extension

e2ed5bb

ehsandeep mentioned this pull request Mar 16, 2023

Include url response body in the callback function OnResult. #360

Closed

ehsandeep approved these changes Mar 16, 2023

View reviewed changes

This was linked to issues Mar 17, 2023

Include url response body in the callback function OnResult. #360

Closed

add -probe feature #289

Closed

Expose returned status code in verbose / Rate Limit inference engine feature #145

Closed

Mzack9999 merged commit 36c8530 into dev Mar 17, 2023

Mzack9999 deleted the issue-303-core-refactor branch March 17, 2023 09:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Core Refactor #312

Core Refactor #312

Mzack9999 commented Feb 14, 2023 •

edited

Loading

ehsandeep left a comment

ehsandeep left a comment

tarunKoyalwar left a comment

tarunKoyalwar commented Feb 23, 2023

Mzack9999 commented Feb 24, 2023 •

edited

Loading

tarunKoyalwar left a comment

tarunKoyalwar commented Feb 26, 2023

tarunKoyalwar commented Feb 26, 2023

Mzack9999 commented Feb 26, 2023

tarunKoyalwar left a comment

tarunKoyalwar commented Mar 1, 2023

yhy0 commented Mar 14, 2023 •

edited

Loading

ehsandeep left a comment •

edited by Mzack9999

Loading

Core Refactor #312

Core Refactor #312

Conversation

Mzack9999 commented Feb 14, 2023 • edited Loading

ehsandeep left a comment

Choose a reason for hiding this comment

ehsandeep left a comment

Choose a reason for hiding this comment

tarunKoyalwar left a comment

Choose a reason for hiding this comment

tarunKoyalwar commented Feb 23, 2023

Mzack9999 commented Feb 24, 2023 • edited Loading

tarunKoyalwar left a comment

Choose a reason for hiding this comment

tarunKoyalwar commented Feb 26, 2023

tarunKoyalwar commented Feb 26, 2023

Mzack9999 commented Feb 26, 2023

tarunKoyalwar left a comment

Choose a reason for hiding this comment

tarunKoyalwar commented Mar 1, 2023

Result Count

yhy0 commented Mar 14, 2023 • edited Loading

ehsandeep left a comment • edited by Mzack9999 Loading

Choose a reason for hiding this comment

Mzack9999 commented Feb 14, 2023 •

edited

Loading

Mzack9999 commented Feb 24, 2023 •

edited

Loading

yhy0 commented Mar 14, 2023 •

edited

Loading

ehsandeep left a comment •

edited by Mzack9999

Loading