Datadog AWS PrivateLink Issue on v0.12.1

[awagneratzendesk]

Describe what happened:
I am engaged in setting up Datadog AWS PrivateLink endpoints for logs and metrics in our environment. On version 0.12.1 using a host endpoint of http-intake.logs.datadoghq.com, logs flow and the plugin appears to work perfectly.

After following the PrivateLink guide at https://docs.datadoghq.com/agent/guide/private-link/?tab=logs, I have setup a logs endpoint for api-pvtlink.logs.datadoghq.com using the indicated PL endpoint of com.amazonaws.vpce.us-east-1.vpce-svc-06394d10ccaf6fb97for custom forwarders.

Security groups are setup to allow all inbound/outbound traffic for debugging purposes and the endpoints are reachable and accepting connections from the host.

When I switch the host in the plugin to api-pvtlink.logs.datadoghq.com, I start to receive the below errors.

{"time":"2020-07-09 21:29:39 +0000","level":"warn","message":"[out_send_to_datadog] got unrecoverable error in primary and no secondary error_class=NoMethodError error=\"undefined method `error' for nil:NilClass\"","worker_id":0}
{"time":"2020-07-09 21:29:39 +0000","level":"warn","worker_id":0,"message":"/usr/bundle/ruby/2.6.0/gems/fluent-plugin-datadog-0.12.1/lib/fluent/plugin/out_datadog.rb:140:in `rescue in write'\n/usr/bundle/ruby/2.6.0/gems/fluent-plugin-datadog-0.12.1/lib/fluent/plugin/out_datadog.rb:124:in `write'\n/usr/bundle/ruby/2.6.0/gems/fluentd-1.6.3/lib/fluent/plugin/output.rb:1128:in `try_flush'\n/usr/bundle/ruby/2.6.0/gems/fluentd-1.6.3/lib/fluent/plugin/output.rb:1434:in `flush_thread_run'\n/usr/bundle/ruby/2.6.0/gems/fluentd-1.6.3/lib/fluent/plugin/output.rb:457:in `block (2 levels) in start'\n/usr/bundle/ruby/2.6.0/gems/fluentd-1.6.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'"}

Logs do not flow after the switch.

Describe what you expected:
I expected that using api-pvtlink.logs.datadoghq.com would work just like http-intake.logs.datadoghq.com works.

Steps to reproduce the issue:

1. Setup a DD Private Link endpoint for Custom Forwarders as defined at https://docs.datadoghq.com/agent/guide/private-link/?tab=logs.
2. Point the Datadog fluent output plugin at the local endpoint using plugin version 0.12.1 and FluentD 1.6.3

Additional environment details (Operating System, Cloud provider, etc):
We are using AWS PrivateLink endpoints for Datadog.

[awagneratzendesk]

It was DNS! Closing this issue - after digging into the nitty gritty of our configuration we discovered that DNS was working differently in our K8s containers than we expected.