Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Convert legacy total score to standardised when importing high scores #135

Merged
merged 5 commits into from
Aug 1, 2023
Merged

Convert legacy total score to standardised when importing high scores #135

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
caa64b0
Convert ScoreV1 to ScoreV2
smoogipoo Jun 6, 2023
86c7c00
Update `osu.Game` packages
peppy Aug 1, 2023
273329e
Use constants for database attributes
peppy Aug 1, 2023
bd5ecd5
Remove rest of `RecalculateScoresCommand` code for now to ensure it i…
peppy Aug 1, 2023
133cc4f
Merge branch 'master' into convert-from-scorev1
peppy Aug 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 60 additions & 29 deletions osu.Server.Queues.ScorePump/Queue/ImportHighScoresCommand.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,11 @@
using MySqlConnector;
using Newtonsoft.Json;
using osu.Game.Beatmaps.Legacy;
using osu.Game.Database;
using osu.Game.Online.API;
using osu.Game.Online.API.Requests.Responses;
using osu.Game.Rulesets;
using osu.Game.Rulesets.Difficulty;
using osu.Game.Rulesets.Judgements;
using osu.Game.Rulesets.Mods;
using osu.Game.Rulesets.Scoring;
Expand Down Expand Up @@ -451,7 +453,9 @@ private async Task<ScoreInfo> createReferenceScore(Ruleset ruleset, HighScore hi
Mods = ruleset.ConvertFromLegacyMods((LegacyMods)highScore.enabled_mods).Append(classicMod).ToArray(),
Statistics = new Dictionary<HitResult, int>(),
MaximumStatistics = new Dictionary<HitResult, int>(),
MaxCombo = highScore.maxcombo
MaxCombo = highScore.maxcombo,
LegacyTotalScore = highScore.score,
IsLegacyScore = true
};

// Populate statistics and accuracy.
Expand Down Expand Up @@ -499,57 +503,84 @@ private async Task<ScoreInfo> createReferenceScore(Ruleset ruleset, HighScore hi
// A special hit result is used to pad out the combo value to match, based on the max combo from the beatmap attributes.
int maxComboFromStatistics = scoreInfo.MaximumStatistics.Where(kvp => kvp.Key.AffectsCombo()).Select(kvp => kvp.Value).DefaultIfEmpty(0).Sum();

// Note that using BeatmapStore will fail if a particular difficulty attribute value doesn't exist in the database.
// To get around this, we'll specifically look up the attribute directly from the database, utilising the most primitive mod lookup value
// for the ruleset in order to prevent an additional failures if some new mod combinations were added as difficulty adjustment mods.
// This is a little bit hacky and wouldn't be required if the databased attributes were always in-line with osu!lazer, but that's not the case.
// Using the BeatmapStore class will fail if a particular difficulty attribute value doesn't exist in the database as a result of difficulty calculation not having been run yet.
// Additionally, to properly fill out attribute objects, the BeatmapStore class would require a beatmap object resulting in another database query.
// To get around both of these issues, we'll directly look up the attribute ourselves.

int difficultyMods = 0;
// The isConvertedBeatmap parameter only affects whether mania key mods are allowed.
// Since we're dealing with high scores, we assume that the database mod values have already been validated for mania-specific beatmaps that don't allow key mods.
int difficultyMods = (int)LegacyModsHelper.MaskRelevantMods((LegacyMods)highScore.enabled_mods, true, ruleset.RulesetInfo.OnlineID);

switch (ruleset.RulesetInfo.OnlineID)
Dictionary<int, BeatmapDifficultyAttribute> dbAttributes = queryAttributes(
new DifficultyAttributesLookup(highScore.beatmap_id, ruleset.RulesetInfo.OnlineID, difficultyMods), connection, transaction);

if (!dbAttributes.TryGetValue(9, out BeatmapDifficultyAttribute? maxComboAttribute))
{
case 0:
case 1:
case 2:
// In these rulesets, every mod combination has the same max combo. So use the attribute with mods=0.
break;

case 3:
// In mania, only keymods affect the number of objects and thus the max combo.
difficultyMods = highScore.enabled_mods & (int)LegacyModsHelper.KEY_MODS;
break;
Comment on lines -511 to -520
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really know why this switch existed. The max combo appears to be identical for non-keymod mania difficulties too.

https://github.com/ppy/osu/blob/1d4380cfd0f7b91df58cc1342998bec0d6654420/osu.Game.Rulesets.Mania/Difficulty/ManiaDifficultyCalculator.cs#L60-L66

I believe this may have been legacy code...

await Console.Error.WriteLineAsync($"{highScore.score_id}: Could not determine max combo from the difficulty attributes of beatmap {highScore.beatmap_id}.");
return scoreInfo;
}

int? maxComboFromAttributes = getMaxCombo(new MaxComboLookup(highScore.beatmap_id, ruleset.RulesetInfo.OnlineID, difficultyMods), connection, transaction);
const int legacy_accuracy_score = 23;
const int legacy_combo_score = 25;
const int legacy_bonus_score_ratio = 27;

if (maxComboFromAttributes == null)
if (!dbAttributes.ContainsKey(legacy_accuracy_score) || !dbAttributes.ContainsKey(legacy_combo_score) || !dbAttributes.ContainsKey(legacy_bonus_score_ratio))
{
await Console.Error.WriteLineAsync($"{highScore.score_id}: Could not find difficulty attributes for beatmap {highScore.beatmap_id} in the database.");
await Console.Error.WriteLineAsync($"{highScore.score_id}: Could not find legacy scoring values in the difficulty attributes of beatmap {highScore.beatmap_id}.");
return scoreInfo;
}

#pragma warning disable CS0618
if (maxComboFromAttributes > maxComboFromStatistics)
scoreInfo.MaximumStatistics[HitResult.LegacyComboIncrease] = maxComboFromAttributes.Value - maxComboFromStatistics;
// Pad the maximum combo.
if ((int)maxComboAttribute.value > maxComboFromStatistics)
scoreInfo.MaximumStatistics[HitResult.LegacyComboIncrease] = (int)maxComboAttribute.value - maxComboFromStatistics;
#pragma warning restore CS0618

ScoreProcessor scoreProcessor = ruleset.CreateScoreProcessor();
int maximumLegacyAccuracyScore = (int)dbAttributes[legacy_accuracy_score].value;
int maximumLegacyComboScore = (int)dbAttributes[legacy_combo_score].value;
double maximumLegacyBonusRatio = dbAttributes[legacy_bonus_score_ratio].value;

// Although the combo-multiplied portion is stored into difficulty attributes, attributes are only present for mod combinations that affect difficulty.
// For example, an incoming highscore may be +HDHR, but only difficulty attributes for +HR exist in the database.
// To handle this case, the combo-multiplied portion is readjusted with the new mod multiplier.
if (difficultyMods != highScore.enabled_mods)
{
double difficultyAdjustmentModMultiplier = ruleset.ConvertFromLegacyMods((LegacyMods)difficultyMods).Select(m => m.ScoreMultiplier).Aggregate(1.0, (c, n) => c * n);
double modMultiplier = scoreInfo.Mods.Select(m => m.ScoreMultiplier).Aggregate(1.0, (c, n) => c * n);
maximumLegacyComboScore = (int)Math.Round(maximumLegacyComboScore * modMultiplier / difficultyAdjustmentModMultiplier);
}

scoreInfo.TotalScore = StandardisedScoreMigrationTools.ConvertFromLegacyTotalScore(scoreInfo, new DifficultyAttributes
{
LegacyAccuracyScore = maximumLegacyAccuracyScore,
LegacyComboScore = maximumLegacyComboScore,
LegacyBonusScoreRatio = maximumLegacyBonusRatio
});

int baseScore = scoreInfo.Statistics.Where(kvp => kvp.Key.AffectsAccuracy()).Sum(kvp => kvp.Value * Judgement.ToNumericResult(kvp.Key));
int maxBaseScore = scoreInfo.MaximumStatistics.Where(kvp => kvp.Key.AffectsAccuracy()).Sum(kvp => kvp.Value * Judgement.ToNumericResult(kvp.Key));

scoreInfo.TotalScore = (int)scoreProcessor.ComputeScore(ScoringMode.Standardised, scoreInfo);
scoreInfo.Accuracy = maxBaseScore == 0 ? 1 : baseScore / (double)maxBaseScore;

return scoreInfo;
}

private static readonly ConcurrentDictionary<MaxComboLookup, int?> max_combo_cache = new ConcurrentDictionary<MaxComboLookup, int?>();
private static readonly ConcurrentDictionary<DifficultyAttributesLookup, Dictionary<int, BeatmapDifficultyAttribute>> attributes_cache =
new ConcurrentDictionary<DifficultyAttributesLookup, Dictionary<int, BeatmapDifficultyAttribute>>();

private static Dictionary<int, BeatmapDifficultyAttribute> queryAttributes(DifficultyAttributesLookup lookup, MySqlConnection connection, MySqlTransaction transaction)
{
if (attributes_cache.TryGetValue(lookup, out Dictionary<int, BeatmapDifficultyAttribute>? existing))
return existing;

IEnumerable<BeatmapDifficultyAttribute> dbAttributes =
connection.Query<BeatmapDifficultyAttribute>(
$"SELECT * FROM {BeatmapDifficultyAttribute.TABLE_NAME} WHERE `beatmap_id` = @BeatmapId AND `mode` = @RulesetId AND `mods` = @Mods", lookup, transaction);

private int? getMaxCombo(MaxComboLookup lookup, MySqlConnection connection, MySqlTransaction transaction) =>
max_combo_cache.GetOrAdd(lookup, l => connection.QuerySingleOrDefault<int?>(
$"SELECT `value` FROM {BeatmapDifficultyAttribute.TABLE_NAME} WHERE `beatmap_id` = @BeatmapId AND `mode` = @RulesetId AND `mods` = @Mods AND `attrib_id` = 9", l, transaction));
return attributes_cache[lookup] = dbAttributes.ToDictionary(a => (int)a.attrib_id, a => a);
}

private record MaxComboLookup(int BeatmapId, int RulesetId, int Mods)
private record DifficultyAttributesLookup(int BeatmapId, int RulesetId, int Mods)
{
public override string ToString()
{
Expand Down
120 changes: 4 additions & 116 deletions osu.Server.Queues.ScorePump/RecalculateScoresCommand.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,9 @@
// See the LICENCE file in the repository root for full licence text.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using Dapper;
using Dapper.Contrib.Extensions;
using McMaster.Extensions.CommandLineUtils;
using osu.Game.Rulesets;
using osu.Game.Rulesets.Judgements;
using osu.Game.Rulesets.Scoring;
using osu.Game.Scoring;
using osu.Server.Queues.ScoreStatisticsProcessor;
using osu.Server.Queues.ScoreStatisticsProcessor.Models;

namespace osu.Server.Queues.ScorePump
{
Expand All @@ -35,113 +25,11 @@ public class RecalculateScoresCommand : BaseCommand
[Option(CommandOptionType.SingleValue)]
public int Delay { get; set; }

public async Task<int> OnExecuteAsync(CancellationToken cancellationToken)
public Task<int> OnExecuteAsync(CancellationToken cancellationToken)
{
while (!cancellationToken.IsCancellationRequested)
{
Console.WriteLine($"Processing next {batch_size} scores starting from {StartId}");

using (var db = Queue.GetDatabaseConnection())
{
int updateCount = 0;

using (var transaction = await db.BeginTransactionAsync(cancellationToken))
{
SoloScore[] scores = (await db.QueryAsync<SoloScore>($"SELECT * FROM {SoloScore.TABLE_NAME} WHERE `id` >= @StartId ORDER BY `id` LIMIT {batch_size}", new
{
StartId = StartId
}, transaction)).ToArray();

if (scores.Length == 0)
break;

foreach (var score in scores)
{
bool requiresUpdate = ensureMaximumStatistics(score);
requiresUpdate |= ensureCorrectTotalScore(score);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this command even make sense anymore?


if (requiresUpdate)
{
await db.UpdateAsync(score, transaction);
updateCount++;
}
}

await transaction.CommitAsync(cancellationToken);

StartId = scores.Max(s => s.id) + 1;
}

Console.WriteLine($"Updated {updateCount} rows");
}

if (Delay > 0)
{
Console.WriteLine($"Waiting {Delay}ms...");
await Task.Delay(Delay, cancellationToken);
}
}

Console.WriteLine("Finished.");
return 0;
}

private bool ensureMaximumStatistics(SoloScore score)
{
if (score.ScoreInfo.MaximumStatistics.Sum(s => s.Value) > 0)
return false;

Ruleset ruleset = LegacyRulesetHelper.GetRulesetFromLegacyId(score.ruleset_id);
HitResult maxBasicResult = ruleset.GetHitResults().Select(h => h.result).Where(h => h.IsBasic()).MaxBy(Judgement.ToNumericResult);

foreach ((HitResult result, int count) in score.ScoreInfo.Statistics)
{
switch (result)
{
case HitResult.LargeTickHit:
case HitResult.LargeTickMiss:
score.ScoreInfo.MaximumStatistics[HitResult.LargeTickHit] = score.ScoreInfo.MaximumStatistics.GetValueOrDefault(HitResult.LargeTickHit) + count;
break;

case HitResult.SmallTickHit:
case HitResult.SmallTickMiss:
score.ScoreInfo.MaximumStatistics[HitResult.SmallTickHit] = score.ScoreInfo.MaximumStatistics.GetValueOrDefault(HitResult.SmallTickHit) + count;
break;

case HitResult.IgnoreHit:
case HitResult.IgnoreMiss:
case HitResult.SmallBonus:
case HitResult.LargeBonus:
break;

default:
score.ScoreInfo.MaximumStatistics[maxBasicResult] = score.ScoreInfo.MaximumStatistics.GetValueOrDefault(maxBasicResult) + count;
break;
}
}

return true;
}

private bool ensureCorrectTotalScore(SoloScore score)
{
Ruleset ruleset = LegacyRulesetHelper.GetRulesetFromLegacyId(score.ruleset_id);
ScoreInfo scoreInfo = score.ScoreInfo.ToScoreInfo(score.ScoreInfo.Mods.Select(m => m.ToMod(ruleset)).ToArray());
scoreInfo.Ruleset = ruleset.RulesetInfo;

ScoreProcessor scoreProcessor = ruleset.CreateScoreProcessor();
scoreProcessor.Mods.Value = scoreInfo.Mods;

long totalScore = scoreProcessor.ComputeScore(ScoringMode.Standardised, scoreInfo);
double accuracy = scoreProcessor.ComputeAccuracy(scoreInfo);

if (totalScore == score.ScoreInfo.TotalScore && Math.Round(accuracy, 2) == Math.Round(score.ScoreInfo.Accuracy, 2))
return false;

score.ScoreInfo.TotalScore = totalScore;
score.ScoreInfo.Accuracy = accuracy;

return true;
// TODO: the logic to actually recalculate scores was removed. should be considered before this command is used.
// see https://github.com/ppy/osu-queue-score-statistics/pull/135.
throw new NotImplementedException();
}
}
}
2 changes: 1 addition & 1 deletion osu.Server.Queues.ScoreStatisticsProcessor/StreamedWorkingBeatmap.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ private StreamedWorkingBeatmap(Beatmap beatmap)
}

protected override IBeatmap GetBeatmap() => beatmap;
protected override Texture GetBackground() => throw new System.NotImplementedException();
public override Texture GetBackground() => throw new System.NotImplementedException();
protected override Track GetBeatmapTrack() => throw new System.NotImplementedException();
protected override ISkin GetSkin() => throw new System.NotImplementedException();
public override Stream GetStream(string storagePath) => throw new System.NotImplementedException();
Expand Down
10 changes: 5 additions & 5 deletions osu.Server.Queues.ScoreStatisticsProcessor/osu.Server.Queues.ScoreStatisticsProcessor.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,11 @@
<ItemGroup>
<PackageReference Include="Dapper" Version="2.0.123" />
<PackageReference Include="Dapper.Contrib" Version="2.0.78" />
<PackageReference Include="ppy.osu.Game" Version="2022.1214.0" />
<PackageReference Include="ppy.osu.Game.Rulesets.Catch" Version="2022.1214.0" />
<PackageReference Include="ppy.osu.Game.Rulesets.Mania" Version="2022.1214.0" />
<PackageReference Include="ppy.osu.Game.Rulesets.Osu" Version="2022.1214.0" />
<PackageReference Include="ppy.osu.Game.Rulesets.Taiko" Version="2022.1214.0" />
<PackageReference Include="ppy.osu.Game" Version="2023.717.0" />
<PackageReference Include="ppy.osu.Game.Rulesets.Catch" Version="2023.717.0" />
<PackageReference Include="ppy.osu.Game.Rulesets.Mania" Version="2023.717.0" />
<PackageReference Include="ppy.osu.Game.Rulesets.Osu" Version="2023.717.0" />
<PackageReference Include="ppy.osu.Game.Rulesets.Taiko" Version="2023.717.0" />
<PackageReference Include="ppy.osu.Server.OsuQueueProcessor" Version="2022.1220.0" />
</ItemGroup>

Expand Down