Investigate listDatasetFileDetails DBSClient API

[vkuznet]

During testing of WMCore DBS3Reader I found that both Py/Go server takes significant amount of time on the following DBS3Reader API:

     dataset="/VBF1Parked/Run2012D-v1/RAW"
     res = reader.listDatasetFileDetails(dataset)

This API call by itself has several for (nested loops) and received data from

       fileDetails = self.getFileListByDataset(dataset=datasetPath, validFileOnly=validFileOnly, det    ail=True
        blocks = set()  # the set of blocks of the dataset
        # Iterate over the files and prepare the set of blocks and a dict where the keys are the files
        files = {}
        for f in fileDetails:
            blocks.add(f['block_name'])
            files[f['logical_file_name']] = remapDBS3Keys(f, stringify=True)
            files[f['logical_file_name']]['ValidFile'] = f['is_file_valid']
            files[f['logical_file_name']]['Lumis'] = {}
            files[f['logical_file_name']]['Parents'] = []
        # Iterate over the blocks and get parents and lumis
        for blockName in blocks:
            # get the parents
            if getParents:
                parents = self.dbs.listFileParents(block_name=blockName)
                for p in parents:
                    if p['logical_file_name'] in files:  # invalid files are not there if validFileOnly=1
                        files[p['logical_file_name']]['Parents'].extend(p['parent_logical_file_name'])

            if getLumis:
                # get the lumis
                file_lumis = self.dbs.listFileLumis(block_name=blockName)
                for f in file_lumis:
                    if f['logical_file_name'] in files:  # invalid files are not there if validFileOnly=1
                        if f['run_num'] in files[f['logical_file_name']]['Lumis']:
                            files[f['logical_file_name']]['Lumis'][f['run_num']].extend(f['lumi_section_num'])
                        else:
                            files[f['logical_file_name']]['Lumis'][f['run_num']] = f['lumi_section_num']

        return files

I suggest to provide individual APIs to better understand timing spend in them and within internal (nested) for loops of DBS3Reader API call.

[vkuznet]

this is done now, closing the ticket.