Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow describing an index from a given Type. #525

Closed
wants to merge 52 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
9cbde32
Use System.Text.Json as the default serializer (#504)
sebastienros Dec 22, 2023
263f1d4
remake DescribeContext<>.For
hyzx86 Dec 28, 2023
8fa1f8e
fix UnitTest
hyzx86 Dec 28, 2023
cc9cc7e
Update README.md
hyzx86 Dec 29, 2023
c6c65cd
Update README.md
hyzx86 Dec 29, 2023
d4645fa
Avoid InvalidCastException
Dec 29, 2023
2fd4d5b
fix type cache clear
Dec 29, 2023
23f9c6b
Update README.md
hyzx86 Dec 29, 2023
25f4198
clear code style
Dec 29, 2023
e373ca0
Merge branch 'dynamicIndexType' of https://github.com/hyzx86/yessql i…
Dec 29, 2023
482ed90
Update PropertyIndex.cs
hyzx86 Dec 29, 2023
f850380
Simplify and Cleanup
MikeAlhayek Dec 29, 2023
86d4302
Moving the test. Test should fail MySQL for verification.
MikeAlhayek Dec 29, 2023
74936c1
Fix MySQL error
MikeAlhayek Dec 29, 2023
2c69da9
Last tewaks
MikeAlhayek Dec 29, 2023
144d363
If the same name type is already cached, it should be removed from th…
hyzx86 Dec 30, 2023
fbabfc5
add Dynamic Type UnitTest
hyzx86 Dec 31, 2023
1d5e8a6
validate index type instance
hyzx86 Dec 31, 2023
e83d236
remove error check
hyzx86 Dec 31, 2023
5371cd1
Emulating the usage scenario in OrchardCore, the store may be initial…
hyzx86 Dec 31, 2023
5df8f36
update unit test
hyzx86 Dec 31, 2023
df795ab
add IndexTypeCacheProvider
hyzx86 Jan 1, 2024
f74cc3e
Update test/YesSql.Tests/Indexes/DynamicTypeGeneratorSample.cs
hyzx86 Jan 2, 2024
355b69a
Update test/YesSql.Tests/Indexes/DynamicTypeGeneratorSample.cs
hyzx86 Jan 2, 2024
6122ff4
Update test/YesSql.Tests/Indexes/PropertyIndex.cs
hyzx86 Jan 2, 2024
135d110
Update test/YesSql.Tests/Indexes/DynamicTypeGeneratorSample.cs
hyzx86 Jan 2, 2024
5ee8f86
Update test/YesSql.Tests/Indexes/DynamicTypeGeneratorSample.cs
hyzx86 Jan 2, 2024
c500f20
Update test/YesSql.Tests/Indexes/DynamicTypeGeneratorSample.cs
hyzx86 Jan 2, 2024
b074891
Update test/YesSql.Tests/Indexes/DynamicTypeGeneratorSample.cs
hyzx86 Jan 2, 2024
10cdf25
Update test/YesSql.Tests/Indexes/DynamicTypeGeneratorSample.cs
hyzx86 Jan 2, 2024
7c18fb9
Update test/YesSql.Tests/Indexes/DynamicTypeGeneratorSample.cs
hyzx86 Jan 2, 2024
a52b5a4
Update test/YesSql.Tests/CoreTests.cs
hyzx86 Jan 2, 2024
c213254
Update src/YesSql.Abstractions/Indexes/DescribeContext.cs
hyzx86 Jan 2, 2024
01d217f
fix code format
hyzx86 Jan 2, 2024
e458a98
use vitrual methods instead Interface
hyzx86 Jan 2, 2024
554794d
use session.RegisterIndexes to instead store.RegisterIndexes<Property…
hyzx86 Jan 2, 2024
b945de0
Update test/YesSql.Tests/CoreTests.cs
hyzx86 Jan 2, 2024
be8d8f3
Update src/YesSql.Core/Commands/IndexCommand.cs
hyzx86 Jan 2, 2024
ebc0b32
Update src/YesSql.Abstractions/Indexes/IndexTypeCacheProvider.cs
hyzx86 Jan 2, 2024
bc1d2e7
Update src/YesSql.Abstractions/Indexes/IndexTypeCacheProvider.cs
hyzx86 Jan 2, 2024
72002c8
Update src/YesSql.Abstractions/Indexes/DescribeContext.cs
hyzx86 Jan 2, 2024
16acf5e
Update src/YesSql.Abstractions/IStore.cs
hyzx86 Jan 2, 2024
212a333
add UpdateTypeCache Sample
hyzx86 Jan 2, 2024
c40413d
Merge branch 'dynamicIndexType' of https://github.com/hyzx86/yessql i…
hyzx86 Jan 2, 2024
b0bf2ef
update default index cache
Jan 3, 2024
d5429ea
Restoring the test code
Jan 3, 2024
cd0928a
rename
Jan 3, 2024
784b81c
update unit test
Jan 3, 2024
88d1a56
Merge branch 'main' into release/5.0
MikeAlhayek Jan 26, 2024
ac2d719
merg from 5.x
hyzx86 Feb 26, 2024
8e31c26
Merge branch 'dynamicIndexType' of https://github.com/hyzx86/yessql i…
hyzx86 Feb 26, 2024
2b836e8
Merge remote-tracking branch 'origin/main' into dynamicIndexType
hyzx86 Apr 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<PropertyGroup>
<Copyright>Sebastien Ros</Copyright>
<Authors>Sebastien Ros</Authors>
<TargetFrameworks>net6.0</TargetFrameworks>
<TargetFrameworks>net7.0</TargetFrameworks>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<DebugType>portable</DebugType>
<PackageProjectUrl>https://github.com/sebastienros/yessql</PackageProjectUrl>
Expand Down
8 changes: 8 additions & 0 deletions src/YesSql.Abstractions/IConfiguration.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
using System;
using System.Collections.Generic;
using System.Data;
using YesSql.Indexes;

namespace YesSql
{
Expand Down Expand Up @@ -90,6 +91,13 @@ public interface IConfiguration
/// Gets or sets the identity column size. Default is <see cref="IdentityColumnSize.Int32"/>.
/// </summary>
IdentityColumnSize IdentityColumnSize { get; set; }

/// <summary>
/// Index type cache provider
/// This interface needs to be reimplemented when dynamic type and multi-tenant scenarios are used,
/// using tenant separated cache media insteadexTypeCacheProvider
/// </summary>
IndexTypeCacheProvider IndexTypeCacheProvider { get; set; }
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MikeAlhayek We need to add it to the IConfiguration so that it can be overridden by other implementations
So I extracted IndexTypeCacheProvider into the Abstraction project

Copy link
Contributor Author

@hyzx86 hyzx86 Apr 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for taking so long to reply. Because I didn't submit my comments 😢
I always wondered why there was a Pendding mark on my reply. I just figured it out.

}

public static class ConfigurationExtensions
Expand Down
7 changes: 0 additions & 7 deletions src/YesSql.Abstractions/IContentSerializer.cs
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,5 @@ public interface IContentSerializer
/// <param name="type">The type of the object to deserialize.</param>
/// <returns>The deserialized object.</returns>
object Deserialize(string content, Type type);

/// <summary>
/// Deserializes an object to a <c>dynamic</c> instance.
/// </summary>
/// <param name="content">The <see cref="String" /> instance representing the object to deserialize.</param>
/// <returns>The deserialized object.</returns>
dynamic DeserializeDynamic(string content);
}
}
20 changes: 15 additions & 5 deletions src/YesSql.Abstractions/Indexes/DescribeContext.cs
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ namespace YesSql.Indexes
{
public class DescribeContext<T> : IDescriptor
{
private readonly Dictionary<Type, List<IDescribeFor>> _describes = new Dictionary<Type, List<IDescribeFor>>();
private readonly Dictionary<Type, List<IDescribeFor>> _describes = [];

public IEnumerable<IndexDescriptor> Describe(params Type[] types)
{
Expand All @@ -25,21 +25,31 @@ public IEnumerable<IndexDescriptor> Describe(params Type[] types)
});
}

public IMapFor<T, IIndex> For(Type indexType) => For<IIndex, object>(indexType);

public IMapFor<T, TIndex> For<TIndex>() where TIndex : IIndex
{
return For<TIndex, object>();
return For<TIndex, object>(typeof(TIndex));
}
MikeAlhayek marked this conversation as resolved.
Show resolved Hide resolved

public IMapFor<T, TIndex> For<TIndex, TKey>() where TIndex : IIndex
public IMapFor<T, TIndex> For<TIndex, TKey>() where TIndex : IIndex => For<TIndex, object>(typeof(TIndex));

public IMapFor<T, TIndex> For<TIndex, TKey>(Type indexType) where TIndex : IIndex
{
ArgumentNullException.ThrowIfNull(indexType, nameof(indexType));

List<IDescribeFor> descriptors;

if (!_describes.TryGetValue(typeof(T), out descriptors))
{
descriptors = _describes[typeof(T)] = new List<IDescribeFor>();
descriptors = _describes[typeof(T)] = [];
}

var describeFor = new IndexDescriptor<T, TIndex, TKey>();
var describeFor = new IndexDescriptor<T, TIndex, TKey>()
{
IndexType = indexType,
};

descriptors.Add(describeFor);

return describeFor;
Expand Down
3 changes: 2 additions & 1 deletion src/YesSql.Abstractions/Indexes/DescribeFor.cs
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,9 @@ public class IndexDescriptor<T, TIndex, TKey> : IDescribeFor, IMapFor<T, TIndex>
private Func<object, bool> _filter;

public PropertyInfo GroupProperty { get; set; }
public Type IndexType { get { return typeof(TIndex); } }

public Type IndexType { get; internal set; } = typeof(TIndex);

public Func<object, bool> Filter => _filter;

public IGroupFor<TIndex> Map(Func<T, IEnumerable<TIndex>> map)
Expand Down
54 changes: 54 additions & 0 deletions src/YesSql.Abstractions/Indexes/IndexTypeCacheProvider.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
using System;
using System.Collections.Concurrent;
using System.Linq;
using System.Reflection;
using System.Threading.Tasks;
using YesSql.Serialization;

namespace YesSql.Indexes
{
public class IndexTypeCacheProvider
{
private static readonly ConcurrentDictionary<PropertyInfo, PropertyInfoAccessor> PropertyAccessors = new();
private static readonly ConcurrentDictionary<string, PropertyInfo[]> TypeProperties = new();

public virtual PropertyInfoAccessor GetPropertyAccessor(PropertyInfo property) => PropertyAccessors.GetOrAdd(property, p => new PropertyInfoAccessor(p));

public virtual PropertyInfo[] GetTypeProperties(Type type)
{
if (TypeProperties.TryGetValue(type.FullName, out var pis))
{
return pis;
}

var properties = type.GetProperties().Where(IsWriteable).ToArray();
TypeProperties[type.FullName] = properties;

return properties;
hyzx86 marked this conversation as resolved.
Show resolved Hide resolved
}

public virtual void UpdateCachedType(Type type)
{
if (TypeProperties.TryRemove(type.FullName, out var pis))
{
foreach (var prop in pis)
{
PropertyAccessors.TryRemove(prop, out _);
}
}

var properties = type.GetProperties().Where(IsWriteable).ToArray();
TypeProperties[type.FullName] = properties;
}

protected bool IsWriteable(PropertyInfo pi)
{
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MikeAlhayek Because this method is needed for caching in Yessql, I changed the original IIndexTypeCacheProvider to IndexTypeCacheProvider and included some virtual methods

return
pi.Name != nameof(IIndex.Id) &&
// don't read DocumentId when on a MapIndex as it might be used to
// read the DocumentId directly from an Index query
pi.Name != "DocumentId"
;
}
}
}
43 changes: 13 additions & 30 deletions src/YesSql.Core/Commands/IndexCommand.cs
Original file line number Diff line number Diff line change
Expand Up @@ -15,16 +15,15 @@ namespace YesSql.Commands
{
public abstract class IndexCommand : IIndexCommand
{
private const string _separator = ", ";
protected const string ParameterSuffix = "_$$$";

protected readonly IStore _store;
private static IndexTypeCacheProvider IndexTypeCacheProvider;
protected static PropertyInfo[] KeysProperties = [typeof(IIndex).GetProperty("Id")];

private static readonly ConcurrentDictionary<PropertyInfo, PropertyInfoAccessor> PropertyAccessors = new();
private static readonly ConcurrentDictionary<string, PropertyInfo[]> TypeProperties = new();
private static readonly ConcurrentDictionary<CompoundKey, string> InsertsList = new();
private static readonly ConcurrentDictionary<CompoundKey, string> UpdatesList = new();

protected static PropertyInfo[] KeysProperties = new[] { typeof(IIndex).GetProperty("Id") };
protected readonly IStore _store;

public abstract int ExecutionOrder { get; }

Expand All @@ -33,6 +32,7 @@ public IndexCommand(IIndex index, IStore store, string collection)
Index = index;
_store = store;
Collection = collection;
IndexTypeCacheProvider = store.Configuration.IndexTypeCacheProvider;
}

public IIndex Index { get; }
Expand All @@ -47,13 +47,13 @@ public static void ResetQueryCache()
UpdatesList.Clear();
}

protected static void GetProperties(DbCommand command, object item, string suffix, ISqlDialect dialect)
protected void GetProperties(DbCommand command, object item, string suffix, ISqlDialect dialect)
{
var type = item.GetType();

foreach (var property in TypePropertiesCache(type))
{
var accessor = PropertyAccessors.GetOrAdd(property, p => new PropertyInfoAccessor(p));
var accessor = IndexTypeCacheProvider.GetPropertyAccessor(property);

var value = accessor.Get(item);

Expand All @@ -65,16 +65,9 @@ protected static void GetProperties(DbCommand command, object item, string suffi
}
}

protected static PropertyInfo[] TypePropertiesCache(Type type)
protected PropertyInfo[] TypePropertiesCache(Type type)
{
if (TypeProperties.TryGetValue(type.FullName, out var pis))
{
return pis;
}

var properties = type.GetProperties().Where(IsWriteable).ToArray();
TypeProperties[type.FullName] = properties;
return properties;
return IndexTypeCacheProvider.GetTypeProperties(type);
}

protected string Inserts(Type type, ISqlDialect dialect)
Expand Down Expand Up @@ -102,7 +95,7 @@ protected string Inserts(Type type, ISqlDialect dialect)
sbColumnList.Append(dialect.QuoteForColumnName(property.Name));
if (i < allProperties.Length - 1)
{
sbColumnList.Append(", ");
sbColumnList.Append(_separator);
}
}

Expand All @@ -113,14 +106,14 @@ protected string Inserts(Type type, ISqlDialect dialect)
sbParameterList.Append("@").Append(property.Name).Append(ParameterSuffix);
if (i < allProperties.Length - 1)
{
sbParameterList.Append(", ");
sbParameterList.Append(_separator);
}
}

if (typeof(MapIndex).IsAssignableFrom(type))
{
// We can set the document id
sbColumnList.Append(", ").Append(dialect.QuoteForColumnName("DocumentId"));
sbColumnList.Append(_separator).Append(dialect.QuoteForColumnName("DocumentId"));
sbParameterList.Append(", @DocumentId").Append(ParameterSuffix);
}

Expand Down Expand Up @@ -164,7 +157,7 @@ protected string Updates(Type type, ISqlDialect dialect)
values.Append(dialect.QuoteForColumnName(property.Name) + " = @" + property.Name + ParameterSuffix);
if (i < allProperties.Length - 1)
{
values.Append(", ");
values.Append(_separator);
}
}

Expand All @@ -174,16 +167,6 @@ protected string Updates(Type type, ISqlDialect dialect)
return result;
}

private static bool IsWriteable(PropertyInfo pi)
{
return
pi.Name != nameof(IIndex.Id) &&
// don't read DocumentId when on a MapIndex as it might be used to
// read the DocumentId directly from an Index query
pi.Name != "DocumentId"
;
}

public abstract bool AddToBatch(ISqlDialect dialect, List<string> queries, DbCommand batchCommand, List<Action<DbDataReader>> actions, int index);

private record CompoundKey(string Dialect, string Type, string Schema, string Prefix, string Collection);
Expand Down
5 changes: 4 additions & 1 deletion src/YesSql.Core/Configuration.cs
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
using System.Collections.Generic;
using System.Data;
using YesSql.Data;
using YesSql.Indexes;
using YesSql.Serialization;
using YesSql.Services;

Expand All @@ -15,7 +16,7 @@ public Configuration()
{
IdentifierAccessorFactory = new PropertyAccessorFactory("Id");
VersionAccessorFactory = new PropertyAccessorFactory("Version");
ContentSerializer = new JsonContentSerializer();
ContentSerializer = new DefaultContentSerializer();
IdGenerator = new DefaultIdGenerator();
IsolationLevel = IsolationLevel.ReadCommitted;
TablePrefix = "";
Expand All @@ -24,6 +25,7 @@ public Configuration()
Logger = NullLogger.Instance;
ConcurrentTypes = new HashSet<Type>();
TableNameConvention = new DefaultTableNameConvention();
IndexTypeCacheProvider = new IndexTypeCacheProvider();
}

public IAccessorFactory IdentifierAccessorFactory { get; set; }
Expand All @@ -42,5 +44,6 @@ public Configuration()
public ICommandInterpreter CommandInterpreter { get; set; }
public ISqlDialect SqlDialect { get; set; }
public IdentityColumnSize IdentityColumnSize { get; set; } = IdentityColumnSize.Int32;
public IndexTypeCacheProvider IndexTypeCacheProvider { get; set; }
}
}
2 changes: 1 addition & 1 deletion src/YesSql.Core/Provider/ServiceCollectionExtensions.cs
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ public static IServiceCollection AddDbProvider(

var config = new Configuration();
setupAction.Invoke(config);
services.AddSingleton<IStore>(StoreFactory.CreateAndInitializeAsync(config).GetAwaiter().GetResult());
services.AddSingleton(StoreFactory.CreateAndInitializeAsync(config).GetAwaiter().GetResult());

return services;
}
Expand Down
34 changes: 34 additions & 0 deletions src/YesSql.Core/Serialization/DefaultContentSerializer.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
using System;
using System.Text.Json;
using System.Text.Json.Serialization;

namespace YesSql.Serialization
{
public class DefaultContentSerializer : IContentSerializer
{
private readonly JsonSerializerOptions _options;

public DefaultContentSerializer()
{
_options = new();
_options.DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull;
_options.Converters.Add(UtcDateTimeJsonConverter.Instance);
_options.Converters.Add(DynamicJsonConverter.Instance);
}

public DefaultContentSerializer(JsonSerializerOptions options)
{
_options = options;
}

public object Deserialize(string content, Type type)
{
return JsonSerializer.Deserialize(content, type, _options);
}

public string Serialize(object item)
{
return JsonSerializer.Serialize(item, _options);
}
}
}
Loading