Add support of Elasticsearch Token Filters #16843

denispetrische · 2024-10-07T12:51:12Z

Also:

Added displaying of "settings" segment of elastic search, to make sure, that analyzers with token filters were created.
Analyzers that contain filters will be created with "default" name

Now it is available to create analyzers and filters like that:

"OrchardCore_Elasticsearch": {
      "ConnectionType": "SingleNodeConnectionPool",
      "Url": "http://localhost",
      "Ports": [ 9200 ],
      //"Username": "admin",
      //"Password": "admin",
      //"CloudId": "Orchard_Core_deployment:ZWFzdHVzMi5henVyZS5lbGFzdGljLWNsb3VkLmNvbTo0NDMkNmMxZGQ4YzAzN2=",
      //"CertificateFingerprint": "75:21:E7:92:8F:D5:7A:27:06:38:8E:A4:35:FE:F5:17:D7:37:F4:DF:F0:9A:D2:C0:C4:B6:FF:EE:D1:EA:2B:A7",
      //"EnableApiVersioningHeader": false,
      "Analyzers": {
        "default": {
          "type": "standard",
          "filter": [
            "lowercase",
            "english_stop",
            "english_stemmer"
          ]
        },
        "custom_analyzer": {
          "type": "standard",
          "filter": [
            "lowercase",
            "english_stop",
            "english_stemmer"
          ]
        }
      },
      "Filter": {
        "english_stop": {
          "type": "stop",
          "stopwords": "_english_"
        },
        "english_stemmer": {
          "type": "stemmer",
          "language": "english"
        }
      }
    }

github-actions · 2024-10-07T12:51:39Z

Thank you for submitting your first pull request, awesome! 🚀 If you haven't already, please take a moment to review our contribution guide. This guide provides helpful information to ensure your contribution aligns with our standards. A core team member will review your pull request.

If you like Orchard Core, please star our repo and join our community channels.

denispetrische · 2024-10-07T13:03:24Z

@dotnet-policy-service agree

…dCore into issue16384

Piedone · 2024-10-07T19:05:58Z

Could you please check this, @Skrypt?

Piedone · 2024-10-14T17:06:18Z

@Skrypt what do you think? No problem if not, just please reply.

Skrypt · 2024-10-15T01:01:27Z

I will take look.

Piedone · 2024-10-15T12:03:00Z

Great, thank you!

Skrypt · 2024-10-15T22:29:54Z

src/OrchardCore.Modules/OrchardCore.Search.Elasticsearch/Controllers/AdminController.cs

@@ -432,14 +432,15 @@ public async Task<ActionResult> ForceDelete(ElasticIndexSettingsViewModel model)
        return RedirectToAction(nameof(Index));
    }

-    public async Task<IActionResult> Mappings(string indexName)
+    public async Task<IActionResult> IndexInfo(string indexName)


So this was named Mappings because that's how ElasticSearch calls it in its terminology.

https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

So, need to see what is the difference between GetIndexMappings and GetIndexInfo.

Can you send me a screenshot of the resulting textfield in the Admin UI so that I can compare?

Sure, here it is. The whole idea is to see that analyzers and filters were applied.

{ "default_newind": { "aliases": {}, "mappings": { "_meta": { "last_task_id": 0 }, "_source": { "excludes": [ "Content.ContentItem.DisplayText.Analyzed" ] }, "dynamic_templates": [ { "*.Inherited": { "path_match": "*.Inherited", "match_mapping_type": "string", "mapping": { "type": "keyword" } } }, { "*.Ids": { "path_match": "*.Ids", "match_mapping_type": "string", "mapping": { "type": "keyword" } } } ], "properties": { "Content": { "properties": { "ContentItem": { "properties": { "ContainedPart": { "properties": { "Ids": { "type": "keyword" }, "Order": { "type": "float" } } }, "ContentType": { "type": "keyword" }, "DisplayText": { "properties": { "Analyzed": { "type": "text" }, "Normalized": { "type": "keyword" }, "keyword": { "type": "keyword" } } }, "FullText": { "type": "text" }, "Owner": { "type": "keyword" } } } } }, "ContentItemId": { "type": "keyword" }, "ContentItemVersionId": { "type": "keyword" } } }, "settings": { "index": { "routing": { "allocation": { "include": { "_tier_preference": "data_content" } } }, "number_of_shards": "1", "provided_name": "default_newind", "creation_date": "1728302642111", "analysis": { "filter": { "russian_stemmer": { "type": "stemmer", "language": "russian" }, "english_stemmer": { "type": "stemmer", "language": "russian" }, "russian_stop": { "type": "stop", "stopwords": "_russian_" }, "russian_synonyms": { "type": "synonym_graph", "synonyms": [ "\u0437\u043E\u043B\u043E\u0442\u043E, AU, XAU", "\u043F\u043E\u0437\u0438\u0446\u0438\u0438, \u0438\u043D\u0442\u0435\u0440\u0435\u0441" ] }, "english_stop": { "type": "stop", "stopwords": "_russian_" } }, "analyzer": { "default": { "filter": [ "lowercase", "russian_stop", "russian_synonyms", "russian_stemmer", "english_stop", "english_stemmer" ], "type": "custom", "tokenizer": "standard" } } }, "number_of_replicas": "1", "uuid": "V-oB0lgQRmWMiKQCJLLLTg", "version": { "created": "7160299" } } } } }

So, need to see what is the difference between GetIndexMappings and GetIndexInfo.

The difference now, is that shows not only "mappings" section but full index information

Ok for IndexInfo then

Skrypt · 2024-10-15T23:01:07Z

src/OrchardCore/OrchardCore.Search.Elasticsearch.Abstractions/ElasticsearchOptions.cs

@@ -7,4 +7,6 @@ public class ElasticsearchOptions
    public string IndexPrefix { get; set; }

    public Dictionary<string, JsonObject> Analyzers { get; } = [];
+
+    public Dictionary<string, JsonObject> Filter { get; } = [];


I think we should leave the name as "Filter", because it is how this section called in elasticsearch terminology.

https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-custom-analyzer.html
Settings.Analysis.FIlter and there listed custom token filters

hmm, ok, that's weird but I guess that we can live with it.

Skrypt · 2024-10-15T23:04:21Z

...rdCore.Modules/OrchardCore.Search.Elasticsearch/Extensions/ElasticsearchOptionsExtensions.cs

@@ -0,0 +1,81 @@
+using System.Text.Json;


Ok I like this code. It is cleaner.

Skrypt · 2024-10-15T23:07:23Z

Looking good from first review. Did not test yet.

Skrypt

Approved but needs to be tested.

...rdCore.Modules/OrchardCore.Search.Elasticsearch/Extensions/ElasticsearchOptionsExtensions.cs

src/OrchardCore.Modules/OrchardCore.Search.Elasticsearch/Startup.cs

…sions/ElasticsearchOptionsExtensions.cs Co-authored-by: Hisham Bin Ateya <[email protected]>

…up.cs Co-authored-by: Hisham Bin Ateya <[email protected]>

github-actions · 2024-10-17T15:15:41Z

This pull request has merge conflicts. Please resolve those before requesting a review.

hishamco · 2024-10-17T15:18:20Z

@denispetrische could you please fix the conflict to react to my recent changes in #16896

denispetrische · 2024-10-17T17:34:29Z

@denispetrische could you please fix the conflict to react to my recent changes in #16896

Done

github-actions · 2024-10-17T21:57:17Z

Congratulations on your first PR merge! 🎉 Thank you for your contribution! We're looking forward to welcoming other contributions of yours in the future. @all-contributors please add @denispetrische for code.

If you like Orchard Core, please star our repo and join our community channels.

allcontributors · 2024-10-17T21:57:28Z

@github-actions[bot]

I've put up a pull request to add @denispetrische! 🎉

hishamco · 2024-10-17T22:00:18Z

Thanks for your contribution @denispetrische

denispetrische added 2 commits October 4, 2024 16:43

AddedFiltersSupport

6f0527b

Create default analyzers

b81ff24

denispetrische and others added 3 commits October 7, 2024 16:04

Merge branch 'main' into issue16384

6500770

Mappings renamed to Index info

478002d

Merge branch 'issue16384' of https://github.com/denispetrische/Orchar…

0aaabb5

…dCore into issue16384

denispetrische changed the title ~~Issue16384~~ Issue16384 Added support of ElasticSearch Token Filters Oct 7, 2024

TokenFilterBuildingInfo sealed

8ccbf7e

Piedone changed the title ~~Issue16384 Added support of ElasticSearch Token Filters~~ Add support of Elasticsearch Token Filters Oct 9, 2024

Skrypt reviewed Oct 15, 2024

View reviewed changes

Skrypt approved these changes Oct 16, 2024

View reviewed changes

hishamco requested changes Oct 17, 2024

View reviewed changes

denispetrische and others added 9 commits October 17, 2024 08:12

Update src/OrchardCore.Modules/OrchardCore.Search.Elasticsearch/Exten…

31e63c5

…sions/ElasticsearchOptionsExtensions.cs Co-authored-by: Hisham Bin Ateya <[email protected]>

Update src/OrchardCore.Modules/OrchardCore.Search.Elasticsearch/Exten…

fa95b48

…sions/ElasticsearchOptionsExtensions.cs Co-authored-by: Hisham Bin Ateya <[email protected]>

Update src/OrchardCore.Modules/OrchardCore.Search.Elasticsearch/Exten…

3f632a7

…sions/ElasticsearchOptionsExtensions.cs Co-authored-by: Hisham Bin Ateya <[email protected]>

Update src/OrchardCore.Modules/OrchardCore.Search.Elasticsearch/Exten…

f8d82d3

…sions/ElasticsearchOptionsExtensions.cs Co-authored-by: Hisham Bin Ateya <[email protected]>

Update src/OrchardCore.Modules/OrchardCore.Search.Elasticsearch/Exten…

4622246

…sions/ElasticsearchOptionsExtensions.cs Co-authored-by: Hisham Bin Ateya <[email protected]>

Update src/OrchardCore.Modules/OrchardCore.Search.Elasticsearch/Start…

0edcd4c

…up.cs Co-authored-by: Hisham Bin Ateya <[email protected]>

Fixed codestyle ElasticsearchOptionsExtensions.cs

465d88b

Fixed namespace ElasticsearchOptionsExtensions.cs

808f8db

Merge branch 'main' into issue16384

94a3d14

github-actions bot added the merge conflict label Oct 17, 2024

Resolve conflicts

9fbfe86

github-actions bot removed the merge conflict label Oct 17, 2024

sebastienros approved these changes Oct 17, 2024

View reviewed changes

Merge branch 'main' into issue16384

4c30b21

hishamco approved these changes Oct 17, 2024

View reviewed changes

hishamco enabled auto-merge (squash) October 17, 2024 21:52

hishamco merged commit 99c8b77 into OrchardCMS:main Oct 17, 2024
11 checks passed

allcontributors bot mentioned this pull request Oct 17, 2024

add denispetrische as a contributor for code #16902

Merged

This was referenced Nov 1, 2024

Monthly community metrics report for 2024-10-01..2024-10-31 #16941

Open

Monthly community metrics report for 2024-10-01..2024-10-31 NinaRocket/OrchardCore#10

Open

MikeAlhayek mentioned this pull request Nov 14, 2024

Custom analyzers in Elasticsearch no longer work #17012

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support of Elasticsearch Token Filters #16843

Add support of Elasticsearch Token Filters #16843

denispetrische commented Oct 7, 2024 •

edited by Skrypt

Loading

github-actions bot commented Oct 7, 2024

denispetrische commented Oct 7, 2024

Piedone commented Oct 7, 2024

Piedone commented Oct 14, 2024

Skrypt commented Oct 15, 2024

Piedone commented Oct 15, 2024

Skrypt Oct 15, 2024

Skrypt Oct 15, 2024 •

edited

Loading

Skrypt Oct 15, 2024

denispetrische Oct 16, 2024

denispetrische Oct 16, 2024 •

edited by Skrypt

Loading

denispetrische Oct 16, 2024

Skrypt Oct 16, 2024

Skrypt Oct 15, 2024

denispetrische Oct 16, 2024 •

edited

Loading

Skrypt Oct 16, 2024

Skrypt Oct 15, 2024

Skrypt commented Oct 15, 2024 •

edited

Loading

Skrypt left a comment

github-actions bot commented Oct 17, 2024

hishamco commented Oct 17, 2024

denispetrische commented Oct 17, 2024

github-actions bot commented Oct 17, 2024

allcontributors bot commented Oct 17, 2024

hishamco commented Oct 17, 2024

Add support of Elasticsearch Token Filters #16843

Add support of Elasticsearch Token Filters #16843

Conversation

denispetrische commented Oct 7, 2024 • edited by Skrypt Loading

github-actions bot commented Oct 7, 2024

denispetrische commented Oct 7, 2024

Piedone commented Oct 7, 2024

Piedone commented Oct 14, 2024

Skrypt commented Oct 15, 2024

Piedone commented Oct 15, 2024

Choose a reason for hiding this comment

Skrypt Oct 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

denispetrische Oct 16, 2024 • edited by Skrypt Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

denispetrische Oct 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Skrypt commented Oct 15, 2024 • edited Loading

Skrypt left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 17, 2024

hishamco commented Oct 17, 2024

denispetrische commented Oct 17, 2024

github-actions bot commented Oct 17, 2024

allcontributors bot commented Oct 17, 2024

hishamco commented Oct 17, 2024

denispetrische commented Oct 7, 2024 •

edited by Skrypt

Loading

Skrypt Oct 15, 2024 •

edited

Loading

denispetrische Oct 16, 2024 •

edited by Skrypt

Loading

denispetrische Oct 16, 2024 •

edited

Loading

Skrypt commented Oct 15, 2024 •

edited

Loading