revise security program for higher signal

[chadwhitacre]

Originally posted by @Nashe at #773 (comment) ...

---

We should get the 300th report this week, time to make statistics and suggestions!

Since the bounty started in June 2015, we got 12.4% of "Resolved" reports (reports leading to a commit or more in our codebase), 15.5% of "N/A" (unrelated to our infrastructure or code, third-party products... we don't use this category anymore), 34.6% of "Informative" (not a vulnerability issue, risk to low to be a really risk…) and… 37.5% of "Duplicates" (some of the informative ones are even in fact duplicates). The reports categorized as "low-quality" ("Informative" + "N/A", even if we should count the duplicates of already public reports) increased since your (very good) post about the creation of Gratipay's bounty program.

First point: I find this amount of duplicates incredibly high. It's definitely related to our high delay to properly resolve issues and to the lack of information given in response of the reports we refuse and make publicly available. We need to address this by trying to fix it as soon the report is triaged. Handling reports takes time: don't close any hazardously formulated but valid reports, try to do the same steps, validate the find or not, reply personally to the researcher… enough reasons to avoid getting duplicates. We'll win time, make quicker fixes, lead to less duplicates, etc.

We get a lot of copied and pasted reports related to "Best practices" that have been reported (and not always awarded…) elsewhere. This kind of "Informative" reports belong more to our Github queue than HackerOne.

We don't attract the right researchers: it's rare to get a report originating from somebody with a positive Signal value. This mechanism is already a good indicator of the serious of the researcher, in addition the followings points:

• the presence (or not) of copied/pasted part of the report from other public reports,
• presence of steps to reproduce/demonstrate the vulnerability,
• possible consequences of the report (exploitation scenario, even if it requires another vulnerability to happen).

Some researchers are often trying to make "easy" money by sending a lot of (often low-quality) reports, "just in case, because it was awarded elsewhere", even if it's not applicable. It's not what we want to receive, but that's up to us give them the opportunity to improve their reports and help them expand their skill set by giving detailed answers and not sending generic messages.

My suggestions are the following:

• I will now try to systematically write summaries on publicly disclosed reports, in order to make the information more easily available. Regarding our CSRF mitigation method, often misunderstood, I went ahead and tried to explain the full reasoning behind this choice and offered one additional bounty as proof of good faith (only for this one).
• Regarding the categories:
  • Start using again "N/A" for irrelevant reports (third-party, not a risk nor a bug). Since, everybody can make mistakes, it would be good to apply it only when the researcher already sent at least one irrelevant report in the last weeks or signs of "bad will" (not sure if this expression really exists in English…).
  • Keep "Informative" for best practices and things not seen as a risk, create a Github ticket, link the GH ticket in the report and close the HackerOne report.
• Award bounty very quickly, but take time to coordinate a fix with the researcher. Involving him in the process will be beneficent for everybody. At least to ensure that the fix is correct and no bypass subsists. Taking time ≠ forgetting the report.
• Offer to tweet a "thank you" message from @gratipay for important issues / good reports,
• Offer to co-write a post on https://gratipay.news/ regarding the vulnerability finding (him) and fix (us) for most severe ones,
• Offer a variable bonus in addition of the bounty to great reports (with PoC, active participation of the researcher in the resolution…),
• Offer a bounty per vulnerability type rather than per "risk" (it's too generic). Twitter's policy is a good example (even if we can't afford the same amount ;-D). I'll make a PR for this.
• Add all the domains (being discussed in bring all domains into scope for security program #511).
• Find a way to include software components managed by Gratipay (postgres.py?) that doesn't already have a separate bug bounty (aspen should get one soon?). Create a specific reward range for these reports.
• Continue to focus the "0 New / 0 Triaged" on HackerOne. I still think it's possible!

Thoughts on theses (draft) ideas?

[chadwhitacre]

And from #782 (comment) ...

After taking a look at ownCloud's policy, I found:

Q: Where should I report bugs without security implication or hardening / best practise guidance?
A: Please report all non-security bugs as well as general hardening advice at https://github.com/owncloud/core. Rule of thumb: if it on it's own is not directly exploitable it is likely to be hardening.

I guess it's worth to tell the users to do it for our case too. In addition of my last week's suggestion,

Keep "Informative" for best practices and things not seen as a risk, create a Github ticket, link the GH ticket in the report and close the HackerOne report.

[chadwhitacre]

I've discovered Google's list of non-qualifying reports. A quick skim suggests some overlap with our own backlog. One way to focus our security program would be to from google import nonvuln.

[chadwhitacre]

@EdOverflow What's the status of this ticket relative to recent updates to our program? Anything we want to take from here or can we close?