testing strategy: add policy for non-blocking jobs by path

This was motivated in part by
kubernetes/test-infra#33463 (comment) and
is part of an effort to document best practices.

contributors/devel/sig-testing/testing-strategy.md

@@ -26,6 +26,21 @@ The Kubernetes job uses [prow](https://prow.k8s.io) to implement the CI system.
 - **Postsubmit:** Runs after code is merged. Useful for building artifacts.
 - **Periodic:** Runs at scheduled intervals. Ideal for monitoring trends and catching regressions.
 
+#### Non-blocking triggered by path
+
+Usually, blocking pre-submit jobs run by default and non-blocking jobs don't. The `/test` command
+has to be used explicitly for such non-blocking jobs. It is possible to configure such
+jobs so that they [run automatically when certain paths are modified](https://github.com/kubernetes/test-infra/blob/ee70308f09c10f7cd933c26c98acc7ebf785d436/config/jobs/kubernetes/sig-node/sig-node-presubmit.yaml#L3201-L3202).
+
+Use this with care! A non-blocking job that fails confuses other contributors
+who are not familiar with the job or the failures. If it runs too often, it
+wastes CI resources. The guidelines for setting up such a job are:
+
+* The job owners are responsive and react to problems with the job.
+* The job must have a low failure rate to avoid confusion in drive-by PRs.
+* The importance of the feature must justify the extra CI resources (depends
+  on how often it gets triggered).
+
 #### SIG Release Blocking and Informing jobs
 
 SIG Release maintains two sets of jobs that decide whether the release is