Feature/1569 dataset properties def

[dk1844]

Pilot implementation of #1569

• UI implementation is not part of this PR; properties & property definitions (PDs) are now Rest API-managed only.

• Rest API exists for PD management and for Dataset properties management.

• properties whose PDs have putIntoInfoFile=true are added into _INFO file's metadata with the key being std_ds_ prefix in the Standardization phase, Conformance phase is not affected.

• Pilot limitations: only String and EnumString type properties are supported now.

Dataset API

View dataset with properties:

GET /api/dataset/{datasetName} or GET /api/dataset/{datasetName}/{datasetVersion}-> e.g.

{
    "name": "MyDataset",
    "version": 6,
    ...
    "properties": {
        "requiredKey1": "valueOfRequiredKey1"
    },
    ...
}

View properties of a dataset:

GET /api/dataset/{datasetName}/properties -> e.g.

{
    "requiredKey1": "valueOfRequiredKey1"
}

Replace properties of a dataset:

PUT /api/dataset/{datasetName}/properties with payload

{
    "newKey1": "valueOfNewKey1"
}

-> e.g.

{
    "name": "MyDataset",
    "version": 7,
    ...
    "properties": {
         "newKey1": "valueOfNewKey1"
    },
    "propertiesValidation": null,
    ...
}

(201 Created with header Location:/api/dataset/MyDataset/7)

Dataset properties validation

GET /api/dataset/{datasetName}/{datasetVersion}/properties/valid-> e.g.

{
    "errors": {}
}

or

{
    "errors": {
        "keyWithoutDefinition": [
            "There is no property definition for key 'keyWithoutDefinition'."
        ],
        "enumAbc1": [
             "Value 'valueNotFromEnum' is not one of the allowed values (optionA, optionB, optionC)."
        ]
    }
}

Dataset with validated properties (added in Review)

GET /api/dataset/{datasetName}/{datasetVersion}?validateProperties=true-> e.g.

{
    "name": "MyDataset",
    "version": 6,
    ...
    "properties": {
        "requiredKey1": "valueOfRequiredKey1",
        "keyWithoutDefinition": "someValue",
        "enumAbc1": "valueNotFromEnum"
    },
    "propertiesValidation" : {
        "errors": {
            "keyWithoutDefinition": [
                "There is no property definition for key 'keyWithoutDefinition'."
            ],
            "enumAbc1": [
                 "Value 'valueNotFromEnum' is not one of the allowed values (optionA, optionB, optionC)."
            ]
        }
    }
    ...
}

(if validateProperties is false/missing/invalid, properties are not validated)

Dataset Properties Definition API

View all property definitions at-once (latest version)

GET /api/properties/datasets-> e.g.

[
    {
        "name": "enumAbc1",
        "version": 1,
        "description": null,
        "propertyType": {
            "_t": "StringEnumPropertyType",
            "allowedValues": [
                "optionA",
                "optionB",
                "optionC"
            ],
            "suggestedValue": "optionB"
        },
        "putIntoInfoFile": false,
        "essentiality": {
            "_t": "Optional"
        },
        "disabled": false,
        "dateCreated": "2020-11-09T08:33:33.129Z",
        "userCreated": "user",
        "lastUpdated": "2020-11-09T08:33:33.129Z",
        "userUpdated": "user",
        "dateDisabled": null,
        "userDisabled": null,
        "parent": null,
        "isRequired": false,
        "isOptional": true,
        "createdMessage": {
            "menasRef": {
                "collection": null,
                "name": "enumAbc1",
                "version": 1
            },
            "updatedBy": "user",
            "updated": "2020-11-09T08:33:33.129Z",
            "changes": [
                {
                    "field": "",
                    "oldValue": null,
                    "newValue": null,
                    "message": "PropertyDefinition enumAbc1 created."
                }
            ]
        }
    },
    {
        "name": "requiredKey1",
        "version": 1,
        "description": null,
        "propertyType": {
            "_t": "StringPropertyType",
            "suggestedValue": ""
        },
        "putIntoInfoFile": true,
        "essentiality": {
            "_t": "Mandatory"
        },
        ...
        }
    }
]

View specific property

GET /api/properties/datasets/{pdName} (latest-version) or GET /api/properties/datasets/{pdName}/{pdVerison} -> e.g.

{
    "name": "requiredKey1",
    "version": 1,
    "description": null,
    "propertyType": {
        "_t": "StringPropertyType",
        "suggestedValue": ""
    },
    ...
}

Create a new Property Definition

POST /api/properties/datasets with payload

{
    "name": "myPdKey2",
    "description": "myDesc",
    "propertyType": {
        "_t": "StringPropertyType",
        "suggestedValue": "sortOfDefault"
    }
}

-> e.g.

    "name": "myPdKey2",
    "version": 1,
    "description": "myDesc",
    "propertyType": {
        "_t": "StringPropertyType",
        "suggestedValue": "sortOfDefault"
    },
    "putIntoInfoFile": false,
    "essentiality": {
        "_t": "Optional"
    },
   ...
}

(201 Created with header Location:/api/properties/datasets/myPdKey2/1)

Test-runs:

Spark-job run is guarded with having valid properties:

(tested with std job run):

Required property missing:

Exception in thread "main" java.lang.IllegalStateException: Dataset validation failed, errors found in fields:
 - 'requiredKey1': List(Dataset property 'requiredKey1' is mandatory, but does not exist!)

Property value conformance fails:

Exception in thread "main" java.lang.IllegalStateException: Dataset validation failed, errors found in fields:
 - 'enumAbc1': List(Value bogusEnumValue of key 'enumAbc1' does not conform to the property type of StringEnumPropertyType(Set(optionA, optionB, optionC),optionB).)

Undefined property is present:

 Exception in thread "main" java.lang.IllegalStateException: Dataset validation failed, errors found in fields:
 - 'unwantedExtraProperty': List(There is no property definition for key 'unwantedExtraProperty'.)

A property key/value is written into the _INFO file:

• This very behavior is covered with an integration test, too.
  A property definition with putIntoInfoFile = true exists:
  GET http://localhost:8080/menas_exploded/api/properties/datasets/requiredKey1 ->

{
    "name": "requiredKey1",
    "version": 1,
    "description": null,
    "propertyType": {
        "_t": "StringPropertyType",
        "suggestedValue": ""
    },
    "putIntoInfoFile": true,
    "essentiality": {
        "_t": "Mandatory"
    },
    "disabled": false,
    "dateCreated": "2020-11-06T15:25:30.153Z",
    "userCreated": "user",
    "lastUpdated": "2020-11-06T15:25:30.153Z",
    "userUpdated": "user",
    "dateDisabled": null,
    "userDisabled": null,
    "parent": null,
    "isRequired": true,
    "isOptional": false,
    "createdMessage": {
        "menasRef": {
            "collection": null,
            "name": "requiredKey1",
            "version": 1
        },
        "updatedBy": "user",
        "updated": "2020-11-06T15:25:30.153Z",
        "changes": [
            {
                "field": "",
                "oldValue": null,
                "newValue": null,
                "message": "PropertyDefinition requiredKey1 created."
            }
        ]
    }
}

The dataset has such a field defined:
GET http://localhost:8080/menas_exploded/api/dataset/Cobol1/properties ->

{
    "requiredKey1": "valueOfRequiredKey1"
}

A key being written into the _INFO file:

$ cat _INFO | jq "."
{
  "metadata": {
   ...
    "version": 1,
    "informationDate": "25-02-2019",
    "additionalInfo": {
      ...
      "std_records_failed": "0",
      "std_ds_requiredKey": "valueOfRequiredKey1",
      ...

Known issues

• listing dataset properties via /api/dataset/{name}/{version}/properties will include properties that do not have an existing property definition in the system (because PD are not retrieved at this time ("orphaned properties"). (Should they be?)). When filtering is used, e.g. /api/datase/{name}/properties?putIntoInfoFile=false, such orphaned properties are dropped. However, even if filtering is used with an invalid filter (/api/datase/{name}/properties?putIntoInfoFile=bogus), this fallbacks to only PD-backed properties being returned. Thus, the orphaned properties are dropped, too.

[Zejnilovic]

This approach returns to the user. Maybe we could catch it sooner? Return Validation somewhere? We could also keep it after the REST API Audit as this is not the only one.

Typedefinitionerror: [
  simpletype,
  classza.co.absa.enceladus.model.properties.propertyType.package$EnumPropertyType
];nestedexceptioniscom.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannotconstructinstanceof`za.co.absa.enceladus.model.properties.propertyType.package$EnumPropertyType`,
problem: ThesuggestedvalueoptionDcannotbeused: Value'optionD'isnotoneoftheallowedvalues(optionA,
optionB,
optionC).at[
  Source: (PushbackInputStream);line: 7,
  column: 2
](throughreferencechain: za.co.absa.enceladus.model.properties.PropertyDefinition[
  "propertyType"
])

[dk1844]

I have adjusted the ExceptionHandler to include the message from our PropertyTypeValidationException if found as a L2 underlying cause, otherwise, the "generic" L0 message is used. The relevant integTest has been changed to reflect this. Hope you like it 🥇

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
15 Code Smells

No Coverage information
0.0% Duplication