Url preview endpoint

[adnull]

This PR introduces the missing URL Preview functionality ( Issue #1303 )

According to the specification that endpoint requests require authentication and are rate limited. There are both endpoints (v1 and v3) as some clients still use deprecated endpoints for url preview requests.

Key features:

• takes info from html pages or image links. Max image size downloadable is configurable. Timeout is configurable
• stores the url's image as a thumbnail in media storage if the thumbnail size for url previews is configured. Otherwise stores the original image (to make it sy-test compatible).
• uses the maxThumbnailGenerators variable to prevent overload. Waits 30s for free generator.
• keeps an in-memory cache of url preview responses. The cache entry expiration time is configurable
• stores the the url preview json responses in the media storage to prevent any extra load
• url preview blacklist is configurable as a list of regexps in the config

Missing:

• tests
• synapse's ability to query/extract oEmbed resources

I'd like somebody to take a look at the code.

• I have added Go unit tests or Complement integration tests for this PR or I have justified why this PR doesn't need tests
• Pull request includes a sign off below using a legally identifiable name or I have already signed off privately

Signed-off-by: Aleksandr Dubovikov <[email protected]>

[codecov]

Codecov Report

Attention: Patch coverage is 68.23529% with 135 lines in your changes missing coverage. Please review.

Project coverage is 67.74%. Comparing base (6cd1285) to head (f58da42).

Files with missing lines	Patch %	Lines
mediaapi/routing/url_preview.go	68.46%	74 Missing and 49 partials ⚠️
mediaapi/thumbnailer/thumbnailer_nfnt.go	54.16%	7 Missing and 4 partials ⚠️
mediaapi/routing/download.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3437      +/-   ##
==========================================
- Coverage   67.74%   67.74%   -0.01%     
==========================================
  Files         518      519       +1     
  Lines       47457    47881     +424     
==========================================
+ Hits        32150    32437     +287     
- Misses      11311    11393      +82     
- Partials     3996     4051      +55     

Flag	Coverage Δ
unittests	53.06% <68.23%> (+0.21%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[S7evinK]

Thanks for contributing to Dendrite!

The repo has been cloned into element-hq/dendrite and made the switch to AGPL as per https://matrix.org/blog/2023/11/06/future-of-synapse-dendrite/.
For more information, feel free to read this and this

If you still wish to contribute under the new license, feel free to create a new PR over at element-hq/dendrite.