Crawler doesn't generate anchor in record urls

[ArthurFlag]

Description

Hi,
I'm indexing this website based on Docusaurus 2 and Redoc for the API docs. Since I moved to the new infra, my search hits for API docs are leading to the correct page, without taking me to the correct header.

Steps to reproduce

1. Go to this website
2. Search for Get user
3. The first search result's link is https://docs.talon.one/management-api/ which is not as accurate as it should be, it should be https://docs.talon.one/management-api/#operation/getUser.

The record doesn't contain the full url:

Record #20 

url: "https://docs.talon.one/management-api/"
url_without_variables: "https://docs.talon.one/management-api/"
url_without_anchor: "https://docs.talon.one/management-api/"  <----
anchor: ""
content: null
content_camel: null
lang: "en"
language: "en"
type: "lvl2"
...

My action looks like this:

    {
      indexName: "talon",
      pathsToMatch: ["https://docs.talon.one/management-api/"],
      recordExtractor: ({ $, helpers }) => {
        return helpers.docsearch({
          recordProps: {
            lvl0: {
              selectors: "",
              defaultValue: "Management API reference docs",
            },
            lvl1: ".api-content h1",
            lvl2: ".api-content h2",
            lvl3: ".api-content h3",
            lvl4: ".api-content h4",
            content: ".api-content h2 + div p",
          },
          indexHeadings: {from:1,to:3}
        });
      },
    },

The HTML to crawl look like this, so H2 has all the info the Crawler would need, I suppose.

What am I doing wrong 🙁 ?

[ArthurFlag]

Sorry to ping you directly @shortcuts, but any clue?

[shortcuts]

Hey,

(I'm off this week so not able to look deeply/no computer)

We changed our way to detect anchors, could you show a snippet of an anchor in your DOM?

It's possible that your text is not a children of your anchor (sibling or other level), so we can't find it

Edit: just saw the snippet, it's definitely this. We need to look into it.

[philipvollet]

Also here for the same problem!

[philipvollet]

Any update here?

[shortcuts]

Hey, all opened issue will be reviewed next week

[axilleas]

👋 just chiming it to say that it would be nice to have some docs about:

We changed our way to detect anchors

🙂

[shortcuts]

Update: we found the issue and will do a bit more of testing before pushing it to prod

@axilleas no docs needed for that case, it's a mistake 😬

[shortcuts]

Hey there,

A fix has been deployed, feel free to test (start a new crawl) and give feedbacks :D

[philipvollet]

For us, it's still not working? Do we need to recreate the index or something else?

[shortcuts]

Hey, what is your appId?

[philipvollet]

Y1LB128RON

[shortcuts]

No crawl was started for Y1LB128RON after #1282 (comment), you need to start a new one in order to see the fix applied.

[ArthurFlag]

Works for us, thanks @shortcuts ❤️

[shortcuts]

Cool! Seems to be fixed for everyone then, please feel free to let us know if you encounter any issue with the new indexing!