performance-data-structure.md

Performance data structure

• The performance data is stored in a MongoDB database given by the environment variables WMARCHIVE_PERF_DB and WMARCHIVE_PERF_COLL, defaulting to aggregated.performance.

• The data is generated by the Aggregation procedure.

• Each document has the following structure, where the number in brackets for the scope filters gives an estimate of the number of distinct values for one day and the comment shows a sample value:

  {
      "scope" : {
          "start_date" : Datetime,
          "end_date" : Datetime,
          "timeframe_precision" : String("hour", "day", "week", "month"), // Matches time difference between `start_date` and `end_date`
          "workflow" : String(~500), // "pdmvserv_EXO-RunIISummer15wmLHEGS-00800_00087_v0__160622_190033_1539"
          "task" : String(~1k), // "EXO-RunIISpring16DR80-01801_1"
          "host" : String(~5), // "vocms0311.cern.ch"
          "site" : String(~50), // "T2_DE_DESY"
          "jobstate" : String(3), // "success"
          "jobtype" : String(~5), // "Processing"
          "acquisitionEra" : String(~25) // "RunIISpring16DR80"
          "exitCode": String(~28) // "99999"
          "exitStep" : String(~10), // "cmsRun1"
      },
      "count": Integer,
      "events" : Integer,
      "performance" : {
          "multicore" : {},
          "cpu" : {
              "TotalJobTime" : Float(seconds),
              "TotalJobCPU" : Float,
              // ...
          },
          "storage" : {
              "writeTotalMB" : Float(MB),
              "readTotalMB" : Float(MB),
              // ...
          },
          "memory" : {
              "PeakValueRss" : Float(MB),
              // ...
          }
      }
  }
  

• The performance dictionary includes all metrics available in the original FWJRs. Refer to the Aggregation procedure documentation for details.

• Also refer to Report 005 and Report 007 for details.