GTA : LCS - All character and vehicle models are invisible on M1 Mac

[patrickmetz]

Game or games this happens in

ULUS-10041 GTA:Liberty City Stories

What area of the game / PPSSPP

One minute into the game when one watches the intro, it becomes clear that all characters and vehicles are invisible most of the time, and only partly "flash" into existence for very brief periods.

I also tested OpenEmu which uses 1.10.3. And that version is not affected and shows all characters and vehicles correctly. But in OpenEmu one cannot even access any of PPSSPP's settings, so using that "frontend" is rather pointless.

Also you guys do not provide 1.10.3 readily compiled for M1.
Brew provides PPSSPP 1.10.3, too, but crashes during installation of PPSSPP. See [#72631].(Homebrew/homebrew-core#72631)
And RetroArch for M1 is missing the PPSSPP core entirely.

So the only way to play this game is OpenEmu, which is not configurable and uses 1x-PSP-resolution -.-

What should happen

Characters and vehicles should be clearly visible, when they are on screen.

Logs

No response

Platform

macOS

Mobile phone model or graphics card

M1

PPSSPP version affected

1.12.3

Last working version

1.10.3

Graphics backend (3D API)

OpenGL / GLES and Vulkan

Checklist

• Test in the latest git build in case it's already fixed.
• Search for other reports of the same issue.
• Try resetting settings or older versions and include if the issue is related.
• Try without any cheats and without loading any save states.
• Include logs or screenshots of issue.

[Panderner]

maybe related to #14514?

[patrickmetz]

Ok, meanwhile I managed to get at least version 1.11.3 running on M1 Mac, which also does not have the mentioned issues, by fixing homebrew's failing install of PPSSPP.

Seems homebrew broke because I upgraded to Monterey, so I had to enter xcode-select --install to upgrade my dev tools and brew remove ppsspp to remove rests of older installs, and then brew install ppsspp, which then worked.
(Source: Homebrew/discussions#673 (comment))

Finally I manually started /opt/homebrew/Cellar/ppsspp/1.11.3_1/PPSSPPSDL.app. And pinned the running app to the dock, for quick access, via right clicking on its dock icon.

So I'm currently enjoying the game by means of an "unofficial" homebrew build.

[LunaMoo]

This could simply be broken Vertex Cache which I guess affects OGL on 1.12.3.
You can try artifacts on the bottom of this: https://github.com/hrydgard/ppsspp/actions/runs/1462943557
which is latest and includes macOS build. Or simply disable vertex cache.

[patrickmetz]

@LunaMoo thanks for trying to help.

I tried 1.12.3 with and without vertex cache on Vulkan and OpenGL

And I tried the artifact 1.12.3-232-gb6e7fe1aa only with vertex cache enabled, because the option cannot be disabled there, also on both Vulkan and OpenGL.

The result for both of these mentioned versions is, that characters and vehicles are still invisible, except for tiny time spans.

[unknownbrackets]

If you can compile, it'd be very interesting if you could try a few more builds. See here:
https://github.com/hrydgard/ppsspp/wiki/How-to-bisect-to-find-what-broke-a-game

This does seem similar to #14514, but that happens in v1.11.3. Both seem likely to be graphics driver bugs. If we can know what change introduced this issue, it might help lead to a fix for both.

I don't have the affected devices, but the more you can narrow the haystack down via bisecting, the better. There are about 1700 commits between v1.11.3 and v1.12.3, but with 4 tests you could narrow that down to 100 commits. With a total of 7-8 tests you could narrow it down to 10 or less. That would probably point to where the problem is coming from.

-[Unknown]

[hrydgard]

Confirmed, and it's not working on any version I'm able to build on M1. Very weird!

• CPU core doesn't affect it
• Software skinning / Hardware transform settings don't affect it
• Happens on both OpenGL and Vulkan (MoltenVK)
• Only on ARM-based Macs (and possibly iOS?)
• Doesn't happen on Android ARM64 devices

[unknownbrackets]

If you could post a framedump where it's definitely happening, I'd like to check what's rendering differently for the objects that aren't visible. Since I won't reproduce the problem (don't have M1), a screenshot showing that same framedump (just load it as if it were a game) would be necessary.

The simpler the scene, the better.

It seems like there must be something unique about those objects.

Software renderer ought to be able to validate that it's not CPU related at all. I assume things will show fine there, if slow.

-[Unknown]

[hrydgard]

Need to fix our framedumping to handle GTA's rather unique presentation method first, but will try to get around to it soon.

[patrickmetz]

I just took three framedumps of the bus arriving in the intro scene. On OpenGL, Vulkan and software-rendering.
The Bus is invisible in all three of them.

framedumps.zip

EDIT: on PPSSPP 1.12.3, M1 Mac

[hrydgard]

This does happen with the software renderer, so whatever the issue, it goes deep...

[BlackGor]

I don't know if it can help but i have installed a older version on my iphone (1.9.3) and two of GTA games was fixed , no more invisible characters or graphical issues.

[hrydgard]

Yeah the problem seems to be unique to ARM builds of PPSSPP for Mac. Those older builds would have been x86-64 builds running under Rosetta, which for some reason is not affected. Thanks for reporting though!

[DTibor1986]

I am also experiencing the invisible characters on Intel based Mac, on Big Sur 11.6.1.
Tried 1.12.1-3. all does it.

Maybe a noob question, but how can I rollback to 1.10.3?

[unknownbrackets]

It should be possible to rollback using Xcode. As far as Brew, it seems like you're meant to clone an older version of the Brew repo (which I've done before but is annoying and slow.) You'd basically be trying to run an x86 build through Rosetta, which works.

Since updating the unit test and headless tests, can anyone with M1 try running them and report if any fail? That might give us a much more controlled example of failure that would be easier to fix.

Also, as I remember, it seems more or less like a CPU/math issue. Some ideas to validate:

• If we take a GE frame dump on a Windows PC, does the frame dump render correctly on M1?
• Conversely, if we take a GE frame dump on the M1, does it render wrongly the same way on Windows?
• Assuming yes and yes, is there a scene without movement/animation that replicates this issue (perhaps a paused screen from a save state)?

My goal is to get a frame dump of EXACTLY the same scene from the PC and M1 devices, so we have two frame dumps that ought to be identical, but aren't. Then we can compare and at least know where they're different - matrix values, vertices, something else? But it is important that they would be identical if it weren't for this bug, or it won't help.

Looking at this:
https://developer.apple.com/documentation/xcode/writing-arm64-code-for-apple-platforms

Arguments are slightly different, but it doesn't seem to impact our sin/cos calls. Probably not related anyway, since this occurs even with all jits off.

-[Unknown]

[unknownbrackets]

Another idea / question: if we compile a Debug build (not RelWithDebInfo), or otherwise compile with -O0, does anything change?

-[Unknown]

[hrydgard]

Well the issue is that framedumps are a mess in this game, making it hard to judge. I'm gonna use the mentioned methods to look into it again soon, though, and possibly fix the framedump problem too (and yup, also check unoptimized)

[unknownbrackets]

Well, the framedumps from the "EDIT: on PPSSPP 1.12.3, M1 Mac" comment do work fine (they have two frames in them, but it's hard to tell exactly why - maybe it's actually rendering two frames per vblank right now...) I do think we should fix it to identify "frames" better, but it's hard because of the various presentation methods...

Ideally I just want a PC version of that same scene. I see a lot of cases in that dump of identical verts in prim calls, but I have seen games do that sometimes to null out verts.

-[Unknown]

[timmawsw]

Hey,
since I am also having this issue (and the similiar issue on GTA Vice City Stories) I would like to help finding the cause.
Can I do something for you in order to get you a step forward in this issue? Or has the issue already been identified?

FYI: I do have a Windows PC, M1 and can also test the OpenEmu version against it. Just tell me how long a scene should be and which versions/settings you need the emulator to be on.

[timmawsw]

Hey,
so far I found out:

1. m1 dump works on windows and displays the same graphic bug there
2. windows dump works on m1 and displays the same correct graphics there

See file attached to test it.
recordings.zip
Next up is running the unit tests

[timmawsw]

Hello,
attached the unittest result.

unittest-m1.log

If you need anything else, just ask me. I can for example give you a reproducable dump (With 1 Weapon the upper body of the character always disappears when aiming on a target and always reappears when stopping to aim)

[unknownbrackets]

Windows, green shirt drawn at 1086/1983:
Bone 1 - Bone 5 are NAN on M1, but shouldn't be.

They are explicitly set to NAN in the dump at 09FFA63C, using 2B 7FC000, which is a NAN. On Windows, these are not NAN, and that's the cause of the missing rendering. So it's pretty clear something is resulting in NAN that shouldn't, and likely it's some CPU calculation to generate this display list.

Since the CPU core doesn't affect it, I think an option might be:

1. Get to this scene, or very close, and create a save state on Windows.
2. Make sure the save state generates the issue on M1, but doesn't have the issue on Windows (I assume this will be the case.)
3. Switch both Windows AND M1 to CPU interpreter (not IR or JIT)
4. Log the NANs produced in WriteVector per instruction name using this branch: master...unknownbrackets:m1-nan-log
5. Compare the outputs. This may tell us if there are certain instruction(s) producing NANs only on M1. It's likely there will be multiple (NAN in = NAN out), but at least it gives us a subset to investigate.

-[Unknown]

[hrydgard]

That's a really good idea! Missed that you made a branch first so made my own, master...interpreter-nan-check, might combine them. Tested mine on PC to make sure no spurious nans are logged (had to add some exclusions), will test on Mac and track this down tonight.

[hrydgard]

Alright, had to just have a quick look before doing other stuff today. first hit is Int_VScl which doesn't feel like a likely root cause. Could also be one of the scalar VFPU instructions, they would not be caught by WriteVector... I'll mess around more later.

[unknownbrackets]

Outside transfer/lsu, everything should use WriteVector or WriteMatrix (to apply prefixes.) Maybe worth looking at matrix too, though. Perhaps more likely there (like that compiler bug for LBP back in the day...)

Vscl does seem pretty unlikely. Maybe FPU is more likely than I thought...

-[Unknown]

[hrydgard]

Mac NaN stats on that first scene with the mob boss:

new frame, vfpu nans produced:

• lv.q = 6412
• v(h)tfm3 = 309
• vadd = 1130
• vdot = 131
• vh2f = 10
• vmov = 1792
• vrsq = 64
• vscl = 2079

That's a lot. I do think many are coming from an unused fourth channel, especially vh2f / lv.q.

PC:

new frame, vfpu nans produced:

• lv.q = 34
• vh2f = 10

Yeah, that confirms my theory about vh2f/lv.q, but doesn't quite help pinpointing the source... Six candidates (in addition to matrices and FPU), hm.

In non-surprising news, USE_VFPU_SQRT doesn't fix it.

[unknownbrackets]

I pushed a new version of the branch which does matrix values and tries to avoid logging for nan propagation (didn't really test it, though.)

If VScl is the culprit, maybe it's related to swizzle? In that case, this gets tricky but I'm thinking what I added in the latest commit. That will log the Vscl # per frame that resulted in NAN. If we can catch it right after the save state, we could do something like if (my_isnan(d[i]) || vsclNum == 12345) { for a cleaner compare.

master...unknownbrackets:m1-nan-log

Maybe we'll find it has something to do with denormals?

-[Unknown]

[unknownbrackets]

I pushed a new version of the debugging branch: https://github.com/hrydgard/ppsspp/compare/master...unknownbrackets:ppsspp:m1-nan-log?expand=1

This one logs FPU nan production as well as vrsq 0/inf production.

-[Unknown]

[DTibor1986]

I don't now if it is related to this issue, but both LCS and VCS crashing after intro on 1.14.3. (both with Vulcan and OGL)
I am using an M1 Mac.

[unknownbrackets]

It would help if you could try earlier git builds and find when it started crashing.

-[Unknown]

[hrydgard]

Hm, no problems getting in-game in a fresh build in either Vulkan or OpenGL, on my Macbook Air M1.

[DTibor1986]

I have just tried with 1.14.4 and still crashing both games. Will check earlier builds. (Other games are OK)
Btw I am using an M1 Max with Monterey, but this shouldn't matter if is OK with your M1 Air.

Anyway many thanks for your prompt reply!

[DTibor1986]

1.14-9-g8ed87d48e, Merge pull request #16589 from sum2012/patch #4962 was the last build that I can get in game with invisible bus + characters in the beginning cutscene.

[DTibor1986]

some pictures about the glitch

[unknownbrackets]

1.14-9-g8ed87d48e, Merge pull request #16589 from sum2012/patch #4962 was the last build that I can get in game with invisible bus + characters in the beginning cutscene.

Does that mean v1.14-13-g5406c4f97a doesn't work? That didn't really have any changes except a somewhat minor one to the UI. Are you downloading these builds from GitHub or building locally?

We know the glitch still happens, although we don't know why. More pictures won't help at this time. Some math equation is calculating "not a number" as the result instead of the expected number, and it's only happening on M1. But the equation is complex and involves a lot of equations within the game's code, so we're not quite sure what specific step the M1 macs are calculating an unexpected result for.

-[Unknown]

[DTibor1986]

correct, 1.14-13-g5406c4f97a the first one that doesn't work. I am downloading the artifacts from GitHub.

As for the glitch I know that you are aware of the M1 specific problem, that it isn't easy to track back. (only included the pics to remember myself which was the last "working" build)

[Kethen]

For some reason, on intel hardware, with the same version of moltenvk obtained from https://github.com/hrydgard/ppsspp/releases/tag/v1.14.4, the issue is observed on the native build but not windows build ran under cxwine 22.0.1 (https://github.com/Gcenx/WineskinServer/releases)

[hrydgard]

Wow, that is a super interesting result! Rules out that it's ARM64-specific!

What if you use software rendering on that setup?

[galad87]

Just a shot in the dark: could it be the same issue cemu had, regarding Metal alignment requirements?

cemu-project/Cemu#534
cemu-project/Cemu#445

[Kethen]

Software rendering enabled

The pairs of floating legs on the final screenshots does resemble #14514

With software rendering disabled for a higher resolution rendering

[unknownbrackets]

Does #16816 help that, then? This applies the alignment from a quick look at the cemu pull request.

I'm not sure how that'd help the NANs we saw, but maybe this indicates there's some misaligned read involved. Worth trying that branch with hardware transform off, jit off, etc.

-[Unknown]

[Kethen]

Bisecting on intel hardware, 0ccc63b is where the issue started, while it was issue free up to dbe6658

So it looks like somehow apple provided compilers just breaks the vfpu sin/cos implementation introduced at #14406, at least for intel x86 hardware

[iota97]

Well, from cppreferce: "It is undefined behavior to read from the member of the union that wasn't most recently written. Many compilers implement, as a non-standard language extension, the ability to read inactive members of a union."

I wonder if reinterpret_cast works fine...

Edit: reinterpret_cast is UB too, yeah just memcpy seems fine (or C++20 bit_cast).

[hrydgard]

Oh darn, I thought unions were the "approved" way to bitcast. I guess memcpy it is... Will play with it tomorrow.

[unknownbrackets]

It seems like they aren't on C++ yeah... does #16817 help?

-[Unknown]

[hrydgard]

Some poking around with my convenient partial revert #16927 resulted in this result:

Removing:

	if (k < 0x68) {
		val.i &= 0x80000000;
		return val.f;
	}

from vfpu_sin() in MIPSVFPUUtils.cpp is enough to get things working again on both Intel and ARM64 Macs. Most curious.

[DTibor1986]

Hello, sorry to bother you, but GTA LC & VC still crashing me, no matter whatever backend I am using. I am on MacOS 12.6.3, M1 Max. Tried resetting PPSSPP, different rom (I know..) but still crashing both game right after the intro scene. Any other game in my library just runs perfectly fine.

[DTibor1986]

I am on the build which fixing the 16928 issue.

[hrydgard]

I would guess your ISO is broken, or maybe you're using some cheat that doesn't quite work? I have an ISO file I dumped myself from UMD and it works just fine on both my Macs, as far as I can tell.

[sum2012]

Lucky I just test chinese patched of GTA still work (in windows)