Simulator RNG Seed produces inconsistent teams

[andregraubner]

Hey everyone, while playing around with the simulator I noticed that something about the way the simulator uses seeds to generate teams appears to be broken. Specifically, using seeds from online replays does not reproduce the correct teams.

To reproduce, I installed pokemon-showdown in a fresh environment like
git clone https://github.com/smogon/pokemon-showdown.git
cd pokemon-showdown
./pokemon-showdown generate-team gen8randombattle 52760,56463,35283,9085

After installing the dependencies, generate-team spits out the following team:

Boltund||FocusSash|StrongJaw|firefang,psychicfangs,thunderfang,bulkup||85,85,85,85,85,85||||85|]
Sudowoodo||Leftovers|RockHead|woodhammer,stealthrock,headsmash,earthquake||85,85,85,85,85,85||||88|]
Dragapult||ChoiceSpecs|Infiltrator|thunderbolt,uturn,dracometeor,shadowball||85,85,85,85,85,85||||77|]
Cobalion||LumBerry|Justified|stoneedge,swordsdance,closecombat,ironhead||85,85,85,85,85,85|N|||78|]
Cursola||Leftovers|WeakArmor|icebeam,shadowball,strengthsap,earthpower||85,,85,85,85,85||,0,,,,||87|]
Milotic||Leftovers|Competitive|toxic,recover,scald,icebeam||85,,85,85,85,85||,0,,,,||82|

This isn't consistent with the team generated from the same seed in https://replay.pokemonshowdown.com/azure-gen8randombattle-363056.
We can tell from the inputlog that in this case p2's seed is [52760,56463,35283,9085] as specified above, but p2 has a completely different team.

The same issue can be replicated by installing pokemon-showdown from npm:
npm install pokemon-showdown
npx pokemon-showdown generate-team gen8randombattle 52760,56463,35283,9085

This gives a completely different, but still wrong team:

Skuntank||LifeOrb|Aftermath|fireblast,suckerpunch,taunt,crunch||85,85,85,85,85,85||||85|]
Buzzwole||ChoiceScarf|BeastBoost|darkestlariat,stoneedge,leechlife,closecombat||85,85,85,85,85,85|N|||77|]
Araquanid||HeavyDutyBoots|WaterBubble|toxic,stickyweb,liquidation,mirrorcoat||85,85,85,85,85,85||||80|]
Absol||LifeOrb|Justified|closecombat,swordsdance,suckerpunch,knockoff||85,85,85,85,85,85||||84|]
Gastrodon||Leftovers|StormDrain|scald,toxic,earthquake,recover||85,85,85,85,85,85||||84|]
Oranguru||Leftovers|InnerFocus|nastyplot,thunderbolt,psychic,focusblast||85,,85,85,85,85||,0,,,,||86|

In both cases the result is deterministic but wrong, so I'm thinking this might be some version-dependent issue with the PRNG.
However, in both cases package.json shows version "version": "0.11.9".
Does anybody have any idea what might be up? A quick search didn't yield any results.
Thanks!

[mia-pi-git]

This isn't consistent with the team generated from the same seed in https://replay.pokemonshowdown.com/azure-gen8randombattle-363056.

This is a side server replay. They could have made all sorts of modifications to the code that we haven't, so comparing it to the main codebase is impossible.

[andregraubner]

Thanks, that explains it :)

[andregraubner]

Sorry for closing and immediately re-opening this, but I might have spoken too soon.
Checking with https://replay.pokemonshowdown.com/gen8randombattle-1010773668
This one looks like it's not a side-server, unless I'm misunderstanding how to identify those. Maybe you can enlighten me :)

Using the seed of player p1, we can run:
./pokemon-showdown generate-team gen8randombattle 36597,3529,46166,37135

This still produces completely wrong teams, and importantly the installation method still appears to influence the outcome of the seeded PRNG (installing directly from GitHub vs installing over npm). Both results are wrong, deterministic and different from each other.

Thanks for the swift response btw!

[Karthik99999]

Sorry for closing and immediately re-opening this, but I might have spoken too soon. Checking with https://replay.pokemonshowdown.com/gen8randombattle-1010773668 This one looks like it's not a side-server, unless I'm misunderstanding how to identify those. Maybe you can enlighten me :)

Using the seed of player p1, we can run: ./pokemon-showdown generate-team gen8randombattle 36597,3529,46166,37135

This still produces completely wrong teams, and importantly the installation method still appears to influence the outcome of the seeded PRNG (installing directly from GitHub vs installing over npm). Both results are wrong, deterministic and different from each other.

Thanks for the swift response btw!

This replay is from 2019. A lot of changes have been made to the sim and randbats code in that time, which could have affected the rng here.

[andregraubner]

Thanks, that makes sense. I checked with a more recent game and it works, but only when installing from GitHub. Seems like the RNG when installing from npm is different. Might be a stretch, but do you have any idea where this difference is coming from?

[mia-pi-git]

Thanks, that makes sense. I checked with a more recent game and it works, but only when installing from GitHub. Seems like the RNG when installing from npm is different. Might be a stretch, but do you have any idea where this difference is coming from?

We haven't published to NPM in a while, nor do we consistently - it's several months old code at the least (if not years).

[andregraubner]

Is there no supported way to reproduce a game state from a replay then? I'm trying to reproduce old games based on their inputlogs, but even for quite recent games (e.g. this one from 2 months ago) the seeds don't reproduce the correct teams. In some cases they appear to do, but I'm a bit stumped about what the deciding factor here is - I'm aware that there's a version in the inputlog, but printing out __version from battle.ts shows 4383e729cb060ae5378271d604 while the previously mentioned game also ends in the same code, so there is no obvious version mismatch here.