[RMP] Create a separate Merlin package for the dataloaders

[karlhigley]

Problem:

A number of customers only want to use our dataloaders. They're a thin wedge that we can use to get Merlin adoption amongst teams.

The PyTorch recommendations framework TorchRec would like to make the Merlin dataloader their default without depending on all of Merlin Models. They'd like to publish blog posts about the framework, which creates an opportunity to co-promote one part of the Merlin ecosystem.

The Spark team want to use our dataloaders to accelerate their workflows in TensorFlow and have coordinated with the horovod team to make it an optional dataloader that's natively included with horovod.

Goal:

• Publish a Merlin package specifically for the dataloaders separate from Merlin Models and NVT that both MM and TorchRec can then depend on.
• Integrate the dataloader into as many upstream packages as possible.

Scope:

Publish Merlin dataloaders under a new 'dataloader' package

Constraints:

• TorchRec is using the old (deprecated) version of the dataloaders from NVT
• TorchRec isn't set up to use the new version of the dataloaders in Merlin Models
• Merlin Models still needs to maintain access to the dataloaders too
• Horovod container size was a concern. They're interested in Pip install of RAPIDS. (coming in 22.10)

Blockers

Create a new repo from the Merlin repo template
Note: Julio is blocked on this. Ben has to create a new repo 'dataloader' repo created

Starting Point:

v22.08

• Create a new repo from the Merlin repo template
• Set up CI builds for the new repo
• Migrate dataloader code from NVT or Merlin Models dataloader#13
• Move the orchestration for each python GPU context on to the target GPU
• [BUG?] Imbalanced TorchAsyncItr DataLoader Cuda Memory Usage #375
• Publish new PyPi/Conda packages dataloader#10

v22.09

• Update Merlin Models to use the new dataloader package dataloader#14
• Update nvtabular to use the new dataloader package dataloader#15
• Update Transformers4Rec to use new dataloader package dataloader#16
• Add example for dataloaders for new dataloader repo dataloader#12
• Dataloaders-Comparison to petastorm, tensorflow, default pytorch dataloader dataloader#17
• Dataloaders-Readme dataloader#18
• Dataloaders-Docs update dataloader#20
• Dataloaders-CI infrastructure dataloader#19

[EvenOldridge]

I've created the repo under the name 'dataloader' primarily for seo and to make sure its got a narrow focus but we can rename later.

[viswa-nvidia]

@benfred @jperez999 , who is doing which part of this work ?

Create a new repo from the Merlin repo template
Dataloaders-Set up CI builds for the new repo
Dataloaders-Migrate dataloader code from NVT or Merlin Models(?)
Move the orchestration for each python GPU context on to the target GPU
#375
Dataloaders-Publish new PyPi/Conda packages
Dataloaders-Update Merlin Models to use the new package
NVIDIA-Merlin/dataloader#12
Dataloaders-Comparison to petastorm, tensorflow, default pytorch dataloader
Dataloaders-Readme
Dataloaders-Docs update
Dataloaders-CI infrastructure

[viswa-nvidia]

Pushing it to prioritization to check if we will continue with examples in 22.09

[EvenOldridge]

@benfred to create issues from the bullet point.