Faraday Adapter Parallel Requests On Multiple Domains / Connections

[smythey21]

Looking at Faraday's wiki page on parallel requests it looks like we can easily handle parallelization for a single connection:

conn = Faraday.new(:url => "http://coolness.com") do |faraday|
  faraday.adapter :typhoeus
end

conn.in_parallel do
  response1 = conn.get('/one')
  response2 = conn.get('/two')
end

But if I wanted to have parallel requests between multiple domains / connections, it seems like the Typhoeus Faraday adapter does not support this. To be more clear, I'm expecting something like this:

conn1 = Faraday.new(:url => "http://coolness.com") do |faraday|
  faraday.adapter :typhoeus
end

conn2 = Faraday.new(:url => "http://hotness.com") do |faraday|
  faraday.adapter :typhoeus
end

manager = Faraday::ParallelManager.new # this does not exist of course...
manager.in_parallel do
  response1 = conn1.get('/one')
  response2 = conn2.get('/two')
end

Is this a planned feature for the future? Or maybe with advanced usage is it possible to pass around the Typhoeus managers into the different connections? Like:

manager = Typhoeus::Hydra.new

conn1.in_parallel(manager) do 
  response1 = conn1.get('/one')
end

conn2.in_parallel(manager) do 
  response2 = conn2.get('/two')
end

Thanks in advance!

[iMacTia]

Hi @smythey21 and thanks for opening this!

This is not a planned feature, I'm afraid.
I'm personally not a user of the parallel manager so I wouldn't even be able to estimate the amount of work to get this done, but I'm open to proposal.

One thing I'd suggest though, is to try using absolute paths against a single connection:

conn = Faraday.new do |faraday| # Note url is not mandatory
  faraday.adapter :typhoeus
end

manager = Faraday::ParallelManager.new # this does not exist of course...
manager.in_parallel do
  response1 = conn.get('http://coolness.com/one')
  response2 = conn.get('http://hotness.com/two')
end

Unfortunately I don't have the time to try this up, but could you please give it a shot and let me know if it works?

[smythey21]

Interesting idea! Not sure that will work for my use case though. We have connection objects for 5 different domains each with different configurations (timeouts, failures, network caching, etc.). Using a single connection would mean having to share all of that 😢

[iMacTia]

Sorry to hear that, sounds like this need some development to work fo your case then 😞
As I said there's nothing planned at the moment but I'm happy to review ideas and PRs on this as it sounds like a nice feature 👍
Personally, I like the idea of having a common manager to handle this particular case as you suggested above:

manager = Faraday::ParallelManager.new(...)
manager.in_parallel do
  response1 = conn1.get('/one')
  response2 = conn2.get('/two')
end

Of course this won't replace the conn.in_parallel method for backwards compatibility, but can be used in situations like yours to share the same manager.

[damau]

Hey,

Maybe I have misread something, but this works fine for me:

Also one key gotcha in other bits of parallelism is including the library. You will need to require 'typhoeus'.

Typhoeus Parrallel manager: #Typhoeus::Hydra:0x00007ff5c918fd50

FYI URLS can be any domain.

require 'faraday'
require 'typhoeus'

urls = %[https:// set of urls]

# Uses libcurl and libcurl-multi behind the scenes
# Added this example as it was in the Faraday Docs
conn = Faraday::Connection.new do |builder|
  builder.adapter :typhoeus
end

begin
  conn.in_parallel do
    puts "Parrallel manager: #{conn.parallel_manager}"
    @responses = urls.map do |url|
      conn.get(url)
    end
  end
end

# Gather responses outside of block
puts @responses.map(&:status).join(', ')
puts @responses.map(&:status).compact.count

[smythey21]

Hey @damau that's a nice little trick 😄 but because this approach uses the same Faraday::Connection object it doesn't look like you can configure different properties on the connection (timeouts, failures, network caching, etc.) for the different domains 😞