Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ingestion) ldap: make ldap attrs keys configurable #4682

Merged
merged 31 commits into from
Jun 21, 2022
Merged

feat(ingestion) ldap: make ldap attrs keys configurable #4682

Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
cab247d
feat(ingestion) ldap: make ldap atttrs keys configurable (#4599)
atulsaurav Apr 17, 2022
5e50969
test(ingestion) ldap: Add test case for configurable ldap attrs
atulsaurav Apr 17, 2022
6f5fbab
Add missing coma
atulsaurav Apr 17, 2022
d0099e5
fix: ldap attrs_mapping defaults & test case
atulsaurav Apr 17, 2022
69dfbb3
test(ingestion): fix attrs_mapping in test case
atulsaurav Apr 17, 2022
3b5c051
fix membership test
atulsaurav Apr 17, 2022
0e0946a
Merge branch 'master' into configurable-ldap-ingestion
atulsaurav Jun 6, 2022
7680041
merge upstream changes in prior changes
atulsaurav Jun 7, 2022
bfcc13b
Merge branch 'master' into configurable-ldap-ingestion
atulsaurav Jun 7, 2022
ff8b3c3
Merge branch 'master' into configurable-ldap-ingestion
atulsaurav Jun 7, 2022
015a517
Merge branch 'master' into configurable-ldap-ingestion
atulsaurav Jun 8, 2022
b736547
Doc changes for attrs_mapping between LDAP and DH concepts
atulsaurav Jun 9, 2022
5132e75
Merge branch 'master' into configurable-ldap-ingestion
atulsaurav Jun 9, 2022
7560c03
Update ldap.md
jjoyce0510 Jun 10, 2022
f485349
Changes based on review comments
atulsaurav Jun 20, 2022
3955951
Merge branch 'master' of github.com:datahub-project/datahub into data…
atulsaurav Jun 20, 2022
e077971
Merge branch 'datahub-project-master' into configurable-ldap-ingestion
atulsaurav Jun 20, 2022
474ac7c
Merge branch 'configurable-ldap-ingestion' of github.com:atulsaurav/d…
atulsaurav Jun 20, 2022
4df5568
fix f-string linting error
atulsaurav Jun 20, 2022
74175e0
Fix guess_person_ldap changes from upstream
atulsaurav Jun 20, 2022
68add91
Fix group membership test
atulsaurav Jun 20, 2022
44eebfe
Update docs
atulsaurav Jun 20, 2022
e9aa404
fix default for group description field
atulsaurav Jun 20, 2022
72fad69
Remove breaking change related to Department info
atulsaurav Jun 20, 2022
0fbb294
fix handling of departmentId
atulsaurav Jun 21, 2022
49f14ee
Merge branch 'datahub-project:master' into configurable-ldap-ingestion
atulsaurav Jun 21, 2022
26aa509
Merge branch 'master' into configurable-ldap-ingestion
atulsaurav Jun 21, 2022
d9de05c
Split `attrs_mapping` into `user_attrs_map` & `group_attrs_map`
atulsaurav Jun 21, 2022
9ffb3af
add missing group email attribute
atulsaurav Jun 21, 2022
ec943a4
Merge branch 'configurable-ldap-ingestion' of github.com:atulsaurav/d…
atulsaurav Jun 21, 2022
d3c69b5
Update ldap.md
jjoyce0510 Jun 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 56 additions & 9 deletions metadata-ingestion/archived/source_docs/ldap.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,26 @@ source:
# Options
base_dn: "dc=example,dc=org"

# Optional attribute mapping to allow ldap config differences across orgs
attrs_mapping:
urn: sAMAccountName

# user related attrs
fullName: cn
lastName: sn
firstName: givenName
displayName: displayName
manager: manager
mail: mail
departmentNumber: departmentNumber
title: title

# group related attrs
group_urn: cn
admins: owner
members: uniqueMember
displayName: name
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't this overwrite the display name attribute defined above?

This makes me think we do need 2 distinct mappings:

userMappings
groupMappings

I should have called this out earlier - my apologies for that. What do you think?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No worries, done!


sink:
# sink configs
```
Expand All @@ -42,20 +62,47 @@ sink:

Note that a `.` is used to denote nested fields in the YAML recipe.

| Field | Required | Default | Description |
| ------------------------------ | -------- | ------------------- | ----------------------------------------------------------------------- |
| `ldap_server` | ✅ | | LDAP server URL. |
| `ldap_user` | ✅ | | LDAP user. |
| `ldap_password` | ✅ | | LDAP password. |
| `base_dn` | ✅ | | LDAP DN. |
| `filter` | | `"(objectClass=*)"` | LDAP extractor filter. |
| `drop_missing_first_last_name` | | `True` | If set to true, any users without first and last names will be dropped. |
| `page_size` | | `20` | Size of each page to fetch when extracting metadata. |
| Field | Required | Default | Description |
| ------------------------------ | -------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ldap_server` | ✅ | | LDAP server URL. |
| `ldap_user` | ✅ | | LDAP user. |
| `ldap_password` | ✅ | | LDAP password. |
| `base_dn` | ✅ | | LDAP DN. |
| `filter` | | `"(objectClass=*)"` | LDAP extractor filter. |
| `drop_missing_first_last_name` | | `True` | If set to true, any users without first and last names will be dropped. |
| `page_size` | | `20` | Size of each page to fetch when extracting metadata. |
| `attrs_mapping.urn` | | `sAMAccountName` | An attribute to use in constructing the DataHub User urn. This should be something that uniquely identifies the user and is stable over time. |
| `attrs_mapping.managerUrn` | | `manager` | Alternate attrs key representing same information as manager in the organization. |
| `attrs_mapping.firstName` | | `givenName` | Alternate attrs key representing same information as givenName in the organization. |
| `attrs_mapping.lastName` | | `sn` | Alternate attrs key representing same information as sn in the organization. |
| `attrs_mapping.fullName` | | `cn` | Alternate attrs key representing same information as cn in the organization. |
| `attrs_mapping.email` | | `mail` | Alternate attrs key representing same information as mail in the organization. |
| `attrs_mapping.displayName` | | `displayName` | Alternate attrs key representing same information as displayName in the organization. |
| `attrs_mapping.departmentId` | | `departmentNumber` | Alternate attrs key representing same information as departmentNumber in the organization. |
| `attrs_mapping.departmentName` | | `departmentNumber` | Alternate attrs key representing same information as departmentName in the organization. It is defaulted to `departmentNumber` to not impact existing users. New users are recommended to use descriptive attributes like `department` or `departmantName` that may exist. |
| `attrs_mapping.title` | | `title` | Alternate attrs key representing same information as title in the organization. |
| `attrs_mapping.countryCode` | | `countryCode` | Alternate attrs key representing same information as title in the organization. |
| `attrs_mapping.group_urn` | | `cn` | Alternate attrs key representing same information as the cn for the LDAP group. |
| `attrs_mapping.admins` | | `owner` | Alternate attrs key representing same information as owner in the organization. |
| `attrs_mapping.members` | | `uniqueMember` | Alternate attrs key representing same information as group members in the organization. |
| `attrs_mapping.displayName` | | `name` | Alternate attrs key representing same information as group display name in the organization. |
| `attrs_mapping.description` | | `info` | Alternate attrs key representing same information as group description in the organization. |

The `drop_missing_first_last_name` should be set to true if you've got many "headless" user LDAP accounts
for devices or services should be excluded when they do not contain a first and last name. This will only
impact the ingestion of LDAP users, while LDAP groups will be unaffected by this config option.

### Configurable LDAP

Every organization may implement LDAP slightly differently based on their needs. The makes a standard LDAP recipe ineffective due to missing data during LDAP ingestion. For instance, LDAP recipe assumes department information for a CorpUser would be present in the `departmentNumber` attribute. If an organization chose not to implement that attribute or rather capture similar imformation in the `department` attribute, that information can be missed during LDAP ingestion (even though the information may be present in LDAP in a slightly different form). LDAP source provides flexibility to provide optional mapping for such variations to be reperesented under attrs_mapping. So if an organization represented `departmentNumber` as `department` and `mail` as `email`, the recipe can be adapted to customize that mapping based on need. An example is show below. If `attrs_mapping` section is not provided, the default mapping will apply.

```yaml
# in config section
attrs_mapping:
departmentNumber: department
mail: email
```

## Compatibility

Coming soon!
Expand Down
Loading