Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Azure and AWS-specific Image Caches of ImageSharp (Lombiq Technologies: OCORE-136) #15028

Merged
merged 61 commits into from
Apr 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
1d1d03a
Adding basic prototype
Piedone Jan 9, 2024
831a55e
Adding the first draft of the Azure Media ImageSharp Image Cache feature
Piedone Jan 9, 2024
f63afd5
Docs
Piedone Jan 9, 2024
b97171c
Refactoring
Piedone Jan 9, 2024
d92ce82
Fixing the docs link in CONTRIBUTING.md
Piedone Jan 10, 2024
db5b9b0
Merge remote-tracking branch 'official/main' into issue/OCORE-136
Piedone Jan 17, 2024
930d41d
MD syntax fix
Piedone Jan 17, 2024
20beeaa
TemplateContext factory method instead of initialization method
Piedone Jan 17, 2024
572412a
Code styling
Piedone Jan 17, 2024
254848b
Eager option access
Piedone Jan 17, 2024
eabc330
Lazy initialization for FluidParserHelper
Piedone Jan 17, 2024
60f16e3
Simplified option binding
Piedone Jan 17, 2024
d7fb2ec
Using ToLowerInvariant() to generate the container names instead of t…
Piedone Jan 17, 2024
73aa34d
Factoring out option validation check to method
Piedone Jan 17, 2024
6b05f4e
File-scoped namespaces
Piedone Jan 17, 2024
57b7c84
Using method lookups for configuration values instead of the dictiona…
Piedone Jan 17, 2024
9cf574c
Merge remote-tracking branch 'origin/main' into issue/OCORE-136
Piedone Mar 8, 2024
bea7554
Adding BasePath support for Azure Blob ImageSharp cache
Piedone Mar 8, 2024
f22fc72
Merge remote-tracking branch 'official/main' into issue/OCORE-136
Piedone Mar 13, 2024
2f1378c
"Blob" typos
Piedone Mar 19, 2024
da23340
Merge remote-tracking branch 'origin/main' into issue/OCORE-136
Piedone Mar 19, 2024
9f77673
Removing empty docs
Piedone Mar 19, 2024
d33b463
Code styling
Piedone Mar 19, 2024
3042c8d
Shorter config binding
Piedone Mar 19, 2024
171b686
Formatting
Piedone Mar 19, 2024
8609f53
Removing the "Microsoft" prefix from some Azure wording
Piedone Mar 19, 2024
cf3abea
Docs wording
Piedone Mar 23, 2024
f73dba3
Merge remote-tracking branch 'origin/main' into issue/OCORE-136
Piedone Mar 23, 2024
bb7f327
Moving S3 services to a namespace in anticipation of new services
Piedone Mar 24, 2024
548a8df
Code styling
Piedone Mar 24, 2024
e47985c
Docs link
Piedone Mar 24, 2024
502cd27
Better name for Azure Blob services
Piedone Apr 6, 2024
96abf72
Even better names
Piedone Apr 6, 2024
564fec1
Basics of AWS ImageSharp cache
Piedone Apr 7, 2024
23af590
Making it possible to use local S3 emulators
Piedone Apr 7, 2024
aa1330f
Fixing S3 file uploads
Piedone Apr 7, 2024
d0f41ab
Documenting using local S3 emulators
Piedone Apr 7, 2024
f2c8598
Merge remote-tracking branch 'origin/issue/OCORE-153' into issue/OCOR…
Piedone Apr 7, 2024
657e660
Adding dependency on the storage feature due to IAmazonS3
Piedone Apr 7, 2024
94b6fca
Dry config Fluid parsing
Piedone Apr 7, 2024
ce2a097
Rename
Piedone Apr 7, 2024
6d25a32
DRY Trim()
Piedone Apr 7, 2024
d44be33
Fix type parameters
Piedone Apr 7, 2024
0704d81
Fixing copy-paste errors
Piedone Apr 7, 2024
f297e99
TenantEvents DRY
Piedone Apr 7, 2024
13f5b37
Appsettings docs
Piedone Apr 7, 2024
5dd7d33
Docs
Piedone Apr 7, 2024
3a1d010
Fixing docs navigation
Piedone Apr 7, 2024
5fc658b
DRY
Piedone Apr 7, 2024
acb8b06
Merge remote-tracking branch 'official/main' into issue/OCORE-136
Piedone Apr 8, 2024
f94bea9
Merge remote-tracking branch 'official/main' into issue/OCORE-136
Piedone Apr 15, 2024
14750b1
Fixing new analyzer violations
Piedone Apr 15, 2024
3a797ab
Removing unnecessary interface
Piedone Apr 15, 2024
d6efce8
Seal of approval
Piedone Apr 15, 2024
988e7d2
Adding common const for the Media module's Startup
Piedone Apr 15, 2024
5bc8efb
Renaming IsValid() to IsConfigured()
Piedone Apr 15, 2024
00418e7
More specific names for const classes
Piedone Apr 15, 2024
d40f1ba
Better feature description
Piedone Apr 15, 2024
daa1657
Merge remote-tracking branch 'official/main' into issue/OCORE-136
Piedone Apr 16, 2024
fc6d725
Merge remote-tracking branch 'official/main' into issue/OCORE-136
Piedone Apr 23, 2024
a91d5b6
Merge branch 'main' into issue/OCORE-136
Piedone Apr 24, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 11 additions & 8 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -131,13 +131,13 @@ nav:
- Navigate between pages: docs/topics/navigation/README.md
- Query and Search data: docs/topics/search/README.md
- Secure your application: docs/topics/security/README.md
# - Data: docs/topics/data/README.md
# - Configuration: docs/topics/configuration/README.md
- Data: docs/topics/data/README.md
- Configuration: docs/topics/configuration/README.md
- Workflows: docs/topics/workflows/README.md
- Publishing a new release: docs/topics/publishing-releases/README.md
- Using Docker: docs/topics/docker/README.md
- Using local NuGet packages: docs/topics/local-nuget-packages/README.md
- Managing the Orchard Core Red Hat Ecosystem Catalog certification: docs/topics/red-hat-ecosystem-catalog-certification/README.md
- Managing the Orchard Core Red Hat Ecosystem Catalog certification: docs/topics/red-hat-ecosystem-catalog-certification/README.md
- Reference:
- docs/reference/README.md
- CMS Modules:
Expand All @@ -164,11 +164,6 @@ nav:
- Placements: docs/reference/modules/Placements/README.md
- Themes: docs/reference/modules/Themes/README.md
- Liquid: docs/reference/modules/Liquid/README.md
- Indexing: docs/reference/modules/Indexing/README.md
- SQL Indexing: docs/reference/modules/SQLIndexing/README.md
- Lucene: docs/reference/modules/Lucene/README.md
- Elasticsearch: docs/reference/modules/Elasticsearch/README.md
- Queries: docs/reference/modules/Queries/README.md
- Media:
- Media: docs/reference/modules/Media/README.md
- Media Slugify: docs/reference/modules/Media.Slugify/README.md
Expand All @@ -177,6 +172,13 @@ nav:
- ReCaptcha: docs/reference/modules/ReCaptcha/README.md
- Resources: docs/reference/modules/Resources/README.md
- Rules: docs/reference/modules/Rules/README.md
- Search, Indexing, Querying:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Search, Indexing, Querying:
- Search:

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the same as the section on the page: https://docs.orchardcore.net/en/latest/docs/reference/#search-indexing-querying And that looks good so, since while these are tightly related topics, it's not just search (like SQL queries have nothing to do with search, but queries in general do relate to Lucene and Elasticsearch, which are about indexing, and usually, but not necessarily search...)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The term should be Search. `Indexing and Querying are implementation details for search.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indexing maybe, but Queries is about much more than search. SQL Queries (i.e. the module feature) doesn't have anything to do with search nor indexing, and you can use both Lucene and Elasticsearch querying (e.g. listing all content items with the type Blog Post) without utilizing them for full-text search. That's why I'm saying these three are tightly related, but the whole feature set is neither just search, nor just querying, nor just indexing.

- Azure AI Search: docs/reference/modules/AzureAISearch/README.md
- Elasticsearch: docs/reference/modules/Elasticsearch/README.md
- Indexing: docs/reference/modules/Indexing/README.md
- Lucene: docs/reference/modules/Lucene/README.md
- SQL Indexing: docs/reference/modules/SQLIndexing/README.md
- Queries: docs/reference/modules/Queries/README.md
- Shortcodes: docs/reference/modules/Shortcodes/README.md
- Sitemaps: docs/reference/modules/Sitemaps/README.md
- SMS: docs/reference/modules/Sms/README.md
Expand Down Expand Up @@ -214,6 +216,7 @@ nav:
- Placement: docs/reference/core/Placement/README.md
- Data: docs/reference/core/Data/README.md
- Data Migrations: docs/reference/modules/Migrations/README.md
- Diagnostics: docs/reference/modules/Diagnostics/README.md
- Dynamic Cache: docs/reference/modules/DynamicCache/README.md
- Email: docs/reference/modules/Email/README.md
- SMTP Provider: docs/reference/modules/Email.Smtp/README.md
Expand Down
2 changes: 2 additions & 0 deletions src/OrchardCore.Build/Dependencies.props
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@
<PackageManagement Include="Serilog.AspNetCore" Version="8.0.1" />
<PackageManagement Include="Shortcodes" Version="1.3.3" />
<PackageManagement Include="SixLabors.ImageSharp.Web" Version="3.1.2" />
<PackageManagement Include="SixLabors.ImageSharp.Web.Providers.Azure" Version="3.1.1" />
<PackageManagement Include="SixLabors.ImageSharp.Web.Providers.AWS" Version="3.1.1" />
<PackageManagement Include="StackExchange.Redis" Version="2.7.33" />
<PackageManagement Include="StyleCop.Analyzers" Version="1.1.118" />
<PackageManagement Include="System.Linq.Async" Version="6.0.1" />
Expand Down
37 changes: 29 additions & 8 deletions src/OrchardCore.Cms.Web/appsettings.json
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@
// See https://docs.orchardcore.net/en/latest/docs/reference/modules/DataProtection.Azure/#configuration to configure data protection key storage in Azure Blob Storage.
//"OrchardCore_DataProtection_Azure": {
// "ConnectionString": "", // Set to your Azure Storage account connection string.
// "ContainerName": "dataprotection", // Default to dataprotection. Templatable, refer docs.
// "BlobName": "", // Optional, defaults to Sites/tenant_name/DataProtectionKeys.xml. Templatable, refer docs.
// "ContainerName": "dataprotection", // Default to dataprotection. Templatable, refer to docs.
// "BlobName": "", // Optional, defaults to Sites/tenant_name/DataProtectionKeys.xml. Templatable, refer to docs.
// "CreateContainer": true // Creates the container during app startup if it does not already exist.
//},
// See https://docs.orchardcore.net/en/latest/docs/reference/modules/Markdown/#markdown-configuration
Expand Down Expand Up @@ -62,17 +62,38 @@
// "SecretKey": "",
// "AccessKey": ""
// },
// "BasePath": "/media",
// "BasePath": "/media", // Optionally, set to a path to store media in a subdirectory inside your bucket. Templatable, refer to docs.
// "CreateBucket": true,
// "RemoveBucket": true, // Whether the 'Bucket' is deleted if the tenant is removed, false by default.
// "BucketName": ""
// "BucketName": "media" // Set the bucket's name (mandatory). Templatable, refer to docs.
//},
// See https://docs.orchardcore.net/en/latest/docs/reference/modules/Media.AmazonS3/#configuration_1 to configure media storage in Amazon S3 Storage.
//"OrchardCore_Media_AmazonS3_ImageSharp_Cache": {
// "Region": "eu-central-1",
// "Profile": "default",
// "ProfilesLocation": "",
// "Credentials": {
// "SecretKey": "",
// "AccessKey": ""
// },
// "BasePath": "/media", // Optionally, set to a path to store media in a subdirectory inside your bucket. Templatable, refer to docs.
// "CreateBucket": true,
// "RemoveBucket": true, // Whether the 'Bucket' is deleted if the tenant is removed, false by default.
// "BucketName": "imagesharp" // Set the bucket's name (mandatory). Templatable, refer to docs.
//},
// See https://docs.orchardcore.net/en/latest/docs/reference/modules/Media.Azure/#configuration to configure media storage in Azure Blob Storage.
//"OrchardCore_Media_Azure":
//{
//"OrchardCore_Media_Azure": {
// "ConnectionString": "", // Set to your Azure Storage account connection string.
// "ContainerName": "somecontainer", // Set to the Azure Blob container name. Templatable, refer to docs.
// "BasePath": "some/base/path", // Optionally, set to a path to store media in a subdirectory inside your container. Templatable, refer to docs.
// "CreateContainer": true, // Activates an event to create the container if it does not already exist.
// "RemoveContainer": true // Whether the 'Container' is deleted if the tenant is removed, false by default.
//},
// See http://127.0.0.1:8000/docs/reference/modules/Media.Azure/#configuration_1
//"OrchardCore_Media_Azure_ImageSharp_Cache": {
// "ConnectionString": "", // Set to your Azure Storage account connection string.
// "ContainerName": "somecontainer", // Set to the Azure Blob container name. Templatable, refer docs.
// "BasePath": "some/base/path", // Optionally, set to a path to store media in a subdirectory inside your container. Templatable, refer docs.
// "ContainerName": "somecontainer", // Set to the Azure Blob container name. Templatable, refer to docs.
// "BasePath": "some/base/path", // Optionally, set to a path to store media in a subdirectory inside your container. Templatable, refer to docs.
// "CreateContainer": true, // Activates an event to create the container if it does not already exist.
// "RemoveContainer": true // Whether the 'Container' is deleted if the tenant is removed, false by default.
//},
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
namespace OrchardCore.Media.AmazonS3;

internal static class AmazonS3Constants
{
internal static class ValidationMessages
{
public const string BucketNameIsEmpty = "BucketName is required attribute for S3 storage.";
public const string RegionAndServiceUrlAreEmpty = "Region or ServiceURL is a required attribute for S3 storage.";
}

internal static class AwsCredentialParamNames
{
public const string SecretKey = "SecretKey";
public const string AccessKey = "AccessKey";
}

internal static class ConfigSections
{
public const string AmazonS3 = "OrchardCore_Media_AmazonS3";
public const string AmazonS3ImageSharpCache = "OrchardCore_Media_AmazonS3_ImageSharp_Cache";
}
}

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -12,25 +12,25 @@ namespace OrchardCore.Media.AmazonS3;

public static class AwsStorageOptionsExtension
{
public static IEnumerable<ValidationResult> Validate(this AwsStorageOptions options)
public static IEnumerable<ValidationResult> Validate(this AwsStorageOptionsBase options)
{
if (string.IsNullOrWhiteSpace(options.BucketName))
{
yield return new ValidationResult(Constants.ValidationMessages.BucketNameIsEmpty);
yield return new ValidationResult(AmazonS3Constants.ValidationMessages.BucketNameIsEmpty);
}

if (options.AwsOptions is not null)
{
if (options.AwsOptions.Region is null && options.AwsOptions.DefaultClientConfig.ServiceURL is null)
{
yield return new ValidationResult(Constants.ValidationMessages.RegionEndpointIsEmpty);
yield return new ValidationResult(AmazonS3Constants.ValidationMessages.RegionAndServiceUrlAreEmpty);
}
}
}

public static AwsStorageOptions BindConfiguration(this AwsStorageOptions options, IShellConfiguration shellConfiguration, ILogger logger)
public static AwsStorageOptionsBase BindConfiguration(this AwsStorageOptionsBase options, string configSection, IShellConfiguration shellConfiguration, ILogger logger)
{
var section = shellConfiguration.GetSection("OrchardCore_Media_AmazonS3");
var section = shellConfiguration.GetSection(configSection);

if (!section.Exists())
{
Expand All @@ -53,8 +53,8 @@ public static AwsStorageOptions BindConfiguration(this AwsStorageOptions options
var credentials = section.GetSection("Credentials");
if (credentials.Exists())
{
var secretKey = credentials.GetValue(Constants.AwsCredentialParamNames.SecretKey, string.Empty);
var accessKey = credentials.GetValue(Constants.AwsCredentialParamNames.AccessKey, string.Empty);
var secretKey = credentials.GetValue(AmazonS3Constants.AwsCredentialParamNames.SecretKey, string.Empty);
var accessKey = credentials.GetValue(AmazonS3Constants.AwsCredentialParamNames.AccessKey, string.Empty);

if (!string.IsNullOrWhiteSpace(accessKey) ||
!string.IsNullOrWhiteSpace(secretKey))
Expand Down
17 changes: 0 additions & 17 deletions src/OrchardCore.Modules/OrchardCore.Media.AmazonS3/Constants.cs

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
using Fluid;
using OrchardCore.Environment.Shell;

namespace OrchardCore.Media.AmazonS3.Helpers;

// This is almost the same as in OrchardCore.Media.Azure but there isn't really a good common place for it.
MikeAlhayek marked this conversation as resolved.
Show resolved Hide resolved
internal sealed class OptionsFluidParserHelper<TOptions> where TOptions : class
{
// Local instance since it can be discarded once the startup is over.
private readonly FluidParser _fluidParser = new();
private readonly ShellSettings _shellSettings;

private TemplateContext _templateContext;

public OptionsFluidParserHelper(ShellSettings shellSettings)
{
_shellSettings = shellSettings;
}

public string ParseAndFormat(string template)
{
var templateContext = GetTemplateContext();

// Use Fluid directly as this is transient and cannot invoke _liquidTemplateManager.
var parsedTemplate = _fluidParser.Parse(template);
return parsedTemplate.Render(templateContext, NullEncoder.Default)
.Replace("\r", string.Empty)
.Replace("\n", string.Empty)
.Trim();
}

private TemplateContext GetTemplateContext()
{
if (_templateContext == null)
{
var templateOptions = new TemplateOptions();
_templateContext = new TemplateContext(templateOptions);
templateOptions.MemberAccessStrategy.Register<ShellSettings>();
templateOptions.MemberAccessStrategy.Register<TOptions>();
_templateContext.SetValue("ShellSettings", _shellSettings);
}

return _templateContext;
}
}
14 changes: 13 additions & 1 deletion src/OrchardCore.Modules/OrchardCore.Media.AmazonS3/Manifest.cs
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,22 @@
[assembly: Feature(
Id = "OrchardCore.Media.AmazonS3",
Name = "Amazon Media Storage",
Description = "Enables support for storing media files in Amazon S3 Bucket.",
Description = "Enables support for storing media files in Amazon S3.",
Dependencies =
[
"OrchardCore.Media.Cache"
],
Category = "Hosting"
)]

[assembly: Feature(
Id = "OrchardCore.Media.AmazonS3.ImageSharpImageCache",
Name = "Amazon Media ImageSharp Image Cache",
MikeAlhayek marked this conversation as resolved.
Show resolved Hide resolved
Description = "Provides storage of ImageSharp-generated images within the Amazon S3 storage service.",
Dependencies =
[
"OrchardCore.Media",
"OrchardCore.Media.AmazonS3"
],
Category = "Hosting"
)]
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,11 @@
<ProjectReference Include="..\..\OrchardCore\OrchardCore.Media.Core\OrchardCore.Media.Core.csproj" />
<ProjectReference Include="..\..\OrchardCore\OrchardCore.Module.Targets\OrchardCore.Module.Targets.csproj" />
<ProjectReference Include="..\..\OrchardCore\OrchardCore.Navigation.Core\OrchardCore.Navigation.Core.csproj" />
<ProjectReference Include="..\OrchardCore.Media\OrchardCore.Media.csproj" />
</ItemGroup>

<ItemGroup>
<PackageReference Include="SixLabors.ImageSharp.Web.Providers.AWS" />
</ItemGroup>

</Project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
using Microsoft.Extensions.Options;
using OrchardCore.FileStorage.AmazonS3;
using SixLabors.ImageSharp.Web.Caching.AWS;

namespace OrchardCore.Media.AmazonS3.Services;

// Configuration for ImageSharp's own configuration object. We just pass the settings from the Orchard Core
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need for such a comment

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You asked about the Azure version of this previously, why that was necessary in addition to AwsImageSharpImageCacheOptionsConfiguration. This class might be confusing in the same way.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure that the comment is helpful, answer any questions or adds any value here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What comment would make it clear for you here why we need a second, seemingly unnecessary configuration class?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest removing it. but it's up to you.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd keep it really because apparently it's not clear at a first glance. I can't come up with a better comment.

// configuration.
internal sealed class AWSS3StorageCacheOptionsConfiguration : IConfigureOptions<AWSS3StorageCacheOptions>
{
private readonly AwsImageSharpImageCacheOptions _options;

public AWSS3StorageCacheOptionsConfiguration(IOptions<AwsImageSharpImageCacheOptions> options)
{
_options = options.Value;
}

public void Configure(AWSS3StorageCacheOptions options)
{
var credentials = _options.AwsOptions.Credentials.GetCredentials();

// Only Endpoint or Region is necessary.
options.Endpoint = _options.AwsOptions.DefaultClientConfig.ServiceURL;
options.Region = _options.AwsOptions.Region?.SystemName;
options.BucketName = _options.BucketName;
options.AccessKey = credentials.AccessKey;
options.AccessSecret = credentials.SecretKey;
options.CacheFolder = _options.BasePath;
}
}
Loading