[Search Perf] Improve throughput of backendRequests

[joe-elliott]

For larger queries backendRequests has a large negative impact on performance.

By placing a series of simple log messages in the RoundTrip function we can see that queries that generate a large number of backend requests are negatively impacted.

For this query we created 300k jobs, but only 3000 of them finished because the limit was satisfied. The total time was 16s but you can see that only ~2s was spent searching while ~14s was spent building the backendRequests()

Query Range: 24 hours
Jobs Created: 306917
Jobs Completed: 3102
Total request duration: 16s 

Timings:
2023-05-12 13:40:06.377:  STARTING REQUEST - q= { resource.service.name = "tempo-query-frontend" && duration > 1s} |  
2023-05-12 13:40:19.880: SENDING JOBS - q= { resource.service.name = "tempo-query-frontend" && duration > 1s} |  
2023-05-12 13:40:21.837: REQUEST SATISFIED - q= { resource.service.name = "tempo-query-frontend" && duration > 1s} |  
2023-05-12 13:40:22.219: DONE - q= { resource.service.name = "tempo-query-frontend" && duration > 1s}

Taking the same query but reducing the time range. In this case roughly the exact same number of jobs were completed successfully, but the query took significantly less time simply b/c it created fewer jobs.

Query Range: 3 hours
Jobs Created: 42730
Jobs Completed: 3451
Total request duration: 3.5s 

Timings:
2023-05-12 13:39:42.235 : STARTING REQUEST - q= { resource.service.name = "tempo-query-frontend" && duration > 1s}
2023-05-12 13:39:43.344 : SENDING JOBS - q= { resource.service.name = "tempo-query-frontend" && duration > 1s}
2023-05-12 13:39:45.126 : REQUEST SATISFIED - q= { resource.service.name = "tempo-query-frontend" && duration > 1s}
2023-05-12 13:39:45.136 : DONE - q= { resource.service.name = "tempo-query-frontend" && duration > 1s}

Log lines locations:
STARTING REQUEST
SENDING JOBS
REQUEST SATISIFED
DONE

The most obvious fix (to me) is for backendRequests() to return a channel that the main function consumes. Then we can generate the jobs in parallel.

[mdisibio]

Related: #2464