Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Patch: save correct headers for XHR requests #631

Merged
merged 3 commits into from
Oct 29, 2023

Conversation

ErikOwen
Copy link
Contributor

None of the headers of XHR requests are outputted when performing headless crawls with the -xhr flag set:

> echo "https://projectdiscovery.io" | katana -silent -d 2 -jsonl -headless -xhr -cs "https://projectdiscovery.io" -ob -or -timeout 30 | jq ".response.xhr_requests"
...
[
  {
    "method": "OPTIONS",
    "endpoint": "https://events.framer.com/anonymous"
  },
  {
    "method": "POST",
    "endpoint": "https://events.framer.com/anonymous",
    "body": "[{\"source\":\"framer.site\",\"timestamp\":1697650690430,\"data\":{\"type\":\"track\",\"uuid\":\"a7dd37e7-62ca-459e-fb2a-0df5891948de\",\"event\":\"published_site_pageview\",\"referrer\":null,\"url\":\"https://projectdiscovery.io/requestdemo\",\"hostname\":\"projectdiscovery.io\",\"pathname\":\"/requestdemo\",\"hash\":null,\"search\":null,\"framerSiteId\":\"70c3acc11c8a46451735095dca225b4de6121e4fabcd9bc18c7f46602f2e8e93\",\"context\":{\"framerSiteId\":\"70c3acc11c8a46451735095dca225b4de6121e4fabcd9bc18c7f46602f2e8e93\",\"origin\":\"https://projectdiscovery.io\",\"pathname\":\"/requestdemo\",\"search\":\"\"}}}]"
  }
]
...

The headers will always be null because httpreq.Header is used (see here), which uses the crawler's headers. Instead, the headers from the fetch request should be used.

Here is the output from running the same command above, but with the changes incorporated in this pull request:

> echo "https://projectdiscovery.io" | ./katana -silent -d 2 -jsonl -headless -xhr -cs "https://projectdiscovery.io" -ob -or -timeout 30 | jq ".response.xhr_requests" 
...
[
  {
    "method": "OPTIONS",
    "endpoint": "https://events.framer.com/anonymous",
    "headers": {
      "Access-Control-Request-Method": "POST",
      "Origin": "https://projectdiscovery.io",
      "Referer": "https://projectdiscovery.io/",
      "Sec-Fetch-Mode": "cors",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36",
      "Accept": "*/*",
      "Access-Control-Request-Headers": "content-type"
    }
  },
  {
    "method": "POST",
    "endpoint": "https://events.framer.com/anonymous",
    "body": "[{\"source\":\"framer.site\",\"timestamp\":1697651222844,\"data\":{\"type\":\"track\",\"uuid\":\"a12c91e3-5b8a-6143-84a2-33352d507261\",\"event\":\"published_site_pageview\",\"referrer\":null,\"url\":\"https://projectdiscovery.io/requestdemo\",\"hostname\":\"projectdiscovery.io\",\"pathname\":\"/requestdemo\",\"hash\":null,\"search\":null,\"framerSiteId\":\"70c3acc11c8a46451735095dca225b4de6121e4fabcd9bc18c7f46602f2e8e93\",\"context\":{\"framerSiteId\":\"70c3acc11c8a46451735095dca225b4de6121e4fabcd9bc18c7f46602f2e8e93\",\"origin\":\"https://projectdiscovery.io\",\"pathname\":\"/requestdemo\",\"search\":\"\"}}}]",
    "headers": {
      "Referer": "https://projectdiscovery.io/",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36",
      "Accept": "*/*",
      "Accept-Language": "en",
      "Content-Type": "application/json",
      "Origin": "https://projectdiscovery.io"
    }
  }
]
...

@ErikOwen
Copy link
Contributor Author

Can one of the maintainers review this pull request please 🙏?

@Mzack9999 Mzack9999 self-requested a review October 26, 2023 10:11
@Mzack9999 Mzack9999 added the Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors. label Oct 26, 2023
@Mzack9999 Mzack9999 linked an issue Oct 26, 2023 that may be closed by this pull request
@ehsandeep ehsandeep merged commit e71baf0 into projectdiscovery:dev Oct 29, 2023
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Empty headers with hybrid crawling in XHR requests
4 participants