add macOS support #16

JayFoxRox · 2021-02-27T18:28:31Z

Closes #2

This is pretty hacky, as I only needed it for a little project. Very little effort and testing went into this.
I'll add some review comments about things which still need work.

A bunch of code was stolen from https://github.com/obsproject/obs-studio/tree/516ed4458dbc74b27b425533789d4e9dd95b923b/plugins/mac-virtualcam
Ideally we'd probably have a custom DAL plugin - see https://github.com/johnboiles/coremediaio-dal-minimal-example

I also took the UYVY code from stackoverflow or something, but adapted it. I did not check which colorspace is used or expected.

JayFoxRox · 2021-02-27T18:29:01Z

pyvirtualcam/native_macos/Defines.h

+//  obs-mac-virtualcam is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+//  GNU General Public License for more details.


License should be mentioned somewhere more prominently

JayFoxRox · 2021-02-27T18:29:20Z

pyvirtualcam/native_macos/Defines.h

+//  along with obs-mac-virtualcam. If not, see <http://www.gnu.org/licenses/>.
+
+#define PLUGIN_NAME "mac-virtualcam"
+#define PLUGIN_VERSION "1.3.0"


Not sure if these fields are still relevant

pyvirtualcam/native_macos/Defines.h

JayFoxRox · 2021-02-27T18:30:08Z

pyvirtualcam/native_macos/main.mm

+}
+static void UVfromRGB(uint8_t* u, uint8_t* v, uint8_t r, uint8_t g, uint8_t b) {
+    *u = (uint8_t)(-0.148 * r - 0.291 * g + 0.439 * b + 128);
+    *v = (uint8_t)( 0.439 * r - 0.368 * g - 0.071 * b + 128);


Colorspace for these might be bad. Also horrible performance.

Should be more optimized but I didn't care enough.

pyvirtualcam/native_macos/main.mm

JayFoxRox · 2021-02-27T18:31:23Z

pyvirtualcam/native_macos/main.mm

+    cam_output_width = width;
+    cam_output_height = height;
+    cam_fps_num = fps * 1000;
+    cam_fps_den = 1000;


Calculation of FPS values is pretty bad. I'm not even sure if I tested this version of the code. Originally I had fps:1

pyvirtualcam/native_macos/main.mm

JayFoxRox · 2021-02-27T18:32:36Z

pyvirtualcam/native_macos/main.mm

+
+    uint8_t* data = (uint8_t*) malloc(buf.shape[1] * buf.shape[0] * 2);
+
+    // Convert RGBA to UYVY


This should be optimized

setup.py

JayFoxRox · 2021-02-27T18:33:23Z

setup.py

+                "-framework", "Cocoa",
+                "-framework", "CoreMediaIO",
+                "-framework", "IOSurface",
+                "-framework", "IOKit"


These come from the macOS obs plugin CMakeList - most of them shouldn't be required.
I was simply too lazy to filter this.

Fixed in #22.

setup.py

JayFoxRox · 2021-02-27T18:34:38Z

setup.py

    if sys.platform == 'darwin':
-        darwin_opts = ['-stdlib=libc++', '-mmacosx-version-min=10.7']
+        darwin_opts = ['-stdlib=libc++', '-std=gnu++14']


I believe this comes from the macOS obs plugin CMake file; probably not necessary?

Fixed in #22.

JayFoxRox · 2021-02-27T18:35:12Z

setup.py

+            if sys.platform != 'darwin':
+                opts.append(cpp_flag(self.compiler))
+            #if has_flag(self.compiler, '-fvisibility=hidden'):
+            #    opts.append('-fvisibility=hidden')


Commented out during debugging when I had missing-symbol errors, can probably be re-enabled.

Even with this commented out, trying to use this package now again yields this:

$ python3 ./samples/simple.py Traceback (most recent call last): File "pyvirtualcam/./samples/simple.py", line 4, in <module> import pyvirtualcam File "pyvirtualcam/pyvirtualcam/__init__.py", line 3, in <module> from .camera import Camera File "pyvirtualcam/pyvirtualcam/camera.py", line 13, in <module> from pyvirtualcam import _native_macos as _native ImportError: dlopen(pyvirtualcam/pyvirtualcam/_native_macos.cpython-39-darwin.so, 2): Symbol not found: _OBJC_CLASS_$_NSMachBootstrapServer Referenced from: pyvirtualcam/pyvirtualcam/_native_macos.cpython-39-darwin.so Expected in: flat namespace in pyvirtualcam/pyvirtualcam/_native_macos.cpython-39-darwin.so

~~Not sure how I fixed it earlier.~~

(Edit: Looks like setuptools didn't bother recompiling, because I still had a build/-folder and .so-files, so it used old flags - compilation and using worked fine after removing those)

Fixed in #22.

JayFoxRox · 2021-02-27T18:35:47Z

setup.py

-            if has_flag(self.compiler, '-fvisibility=hidden'):
-                opts.append('-fvisibility=hidden')
+            if sys.platform != 'darwin':
+                opts.append(cpp_flag(self.compiler))


This condition was added when I used some .c file. I have removed that file by now. So this change can probably be removed.

Fixed in #22.

JayFoxRox · 2021-02-27T18:36:27Z

setup.py

-            ext.extra_compile_args = opts
-            ext.extra_link_args = link_opts
+            ext.extra_compile_args += opts
+            ext.extra_link_args += link_opts


I had issues with "-framework" lines not reaching the linker. I' believe this was related to that?

Yes, that seems right. I copied this as-is from the pybind11 docs, probably worth reporting it.

Would be nice if you could report it; I never worked with pybind11 (except for the hacks in this PR).

Looks like they factored that into a component now https://github.com/pybind/python_example/blob/master/setup.py. At some point I'll probably switch to that.

JayFoxRox · 2021-02-27T18:39:07Z

pyvirtualcam/native_macos/main.mm

+    NSRunLoop *runLoop;
+  	NSDate *timeout = [NSDate dateWithTimeIntervalSinceNow:0.001];
+    runLoop = [NSRunLoop currentRunLoop];
+    [runLoop runUntilDate:timeout];


I wasted about 1-2 hours to find this solution.
I noticed that someone else had the exact same problem before (https://stackoverflow.com/questions/62253531/cannot-connect-to-nsmachbootstrapserver-from-plugin); at least I found a solution, but it feels very hacky.

There's probably a cleaner way - but I don't usually code on macOS and never really had to do ObjC before.

The timeout value (0.001) seems very janky, too.

I assume the timeout is one of the reasons for failing CI performance tests. Not sure about the actual time-slice resolution, but I could imagine that this blocks for more than 1ms.

I did not check if it still works when providing timeout in the past (so it returns immediately after handling messages).

(Possibly) fixed in #22.

JayFoxRox · 2021-02-27T18:39:33Z

pyvirtualcam/native_macos/main.mm

+    *v = (uint8_t)( 0.439 * r - 0.368 * g - 0.071 * b + 128);
+}
+
+static bool virtual_output_start(int width, int height, double fps, int delay) {


delay argument is being ignored. What's the intention here?

It determines the size of the ring buffer for the Windows variant, roughly small delay = small ring buffer = low latency = potentially more instable. Kind of an advanced option. It's fine to leave it unused here.

I assumed that was the case.

If you wanted support for this, you could probably play with the presentation-timestamp on macOS to achieve something similar (pre-buffer multiple frames).
I currently use the mach_absolute_time function to create a timestamp, but I'm not sure if that has any impact (mach_absolute_time is something I saw mentioned in the OBS plugin, although I did not verify it's the correct timestamp source; and then the DAL plugin might ignore these timestamps anyway).

pyvirtualcam/native_macos/main.mm

JayFoxRox · 2021-02-27T18:51:37Z

pyvirtualcam/native_macos/main.mm

+    m.def("send", &send, R"pbdoc(
+        Send frame to the virtual cam.
+    )pbdoc");
+}


README needs information that macOS is supported

Opened #21.

pyvirtualcam/native_macos/main.mm

JayFoxRox · 2021-02-27T18:55:21Z

pyvirtualcam/native_macos/main.mm

+            uint8_t mixRGB[3];
+            mixRGB[0] = (rgba[0+0] + rgba[4+0]) / 2;
+            mixRGB[1] = (rgba[0+1] + rgba[4+1]) / 2;
+            mixRGB[2] = (rgba[0+2] + rgba[4+2]) / 2;


Doing this in 8-bit will reduce quality.
Also should probably do gamma correction before downsampling.

I've switched to floating-point in #22.

JayFoxRox · 2021-02-27T20:36:24Z

pyvirtualcam/native_macos/main.mm

+    cam_fps_den = 1000;
+
+	  blog(LOG_DEBUG, "output_create");
+	  sMachServer = [[OBSDALMachServer alloc] init];


If the OBS DAL plugin is not installed, then software (Zoom, Quicktime, etc.)won't show a virtual camera and therefore it's unlikely that anything will ever connect to the Mach server.

The OBS plugin would first check if the DAL plugin is installed (and the correct version).
If the DAL plugin is not installed, the OBS plugin would attempt to install it, or report an error.

My code does none of those checks. It will always start the server, regardless if the user has a suitable client.

Ideally we should abort creation and report an error about the missing OBS DAL plugin. - However, I was too lazy to add this.

In practical terms, does that mean for this to work a user would have to use the OBS plugin at least once so that the DAL plugin is installed? I assume so. I agree that checking for /Library/CoreMediaIO/Plug-Ins/DAL/obs-mac-virtualcam.plugin and printing a useful error message makes sense.

The macOS plugin only became part of OBS Studio very recently. Before that, the plugin probably self-installed the DAL plugin to make it easier for the user.

With inclusion of the plugin in OBS Studio I'd assume that the OBS installer installs it automatically these days.

But I'm not sure either.

letmaik · 2021-02-27T21:35:51Z

Thanks for the contribution! I think it's pretty good, hackyness is on par with the Windows variant :) This package is experimental, so that's not a problem. As long as folks can use it for some funny demos or their research prototypes it's good enough.
It sounds like you don't want to invest more time into cleanup, right? I don't own a mac, which complicates it a little but I may be able to rely on GitHub Actions to validate any further changes.

setup.py

letmaik · 2021-02-27T22:56:23Z

I added macOS CI, seems like there are some compile issues.

.github/scripts/build-macos.sh

JayFoxRox · 2021-02-27T23:41:26Z

It sounds like you don't want to invest more time into cleanup, right? I don't own a mac, which complicates it a little but I may be able to rely on GitHub Actions to validate any further changes.

I'd like to see this included in master and am willing to fix the worst bugs I've created.
So a bit of cleanup is fine, but I don't want to spend more than a couple of minutes on it (especially due to #19).

CI sounds like a good idea. Make sure it also catches issues like this: #16 (comment).

letmaik · 2021-02-28T09:52:36Z

CI now passes except for the test that checks whether the target fps are achieved. It seems that it fluctuates a lot judging from the output of the test_sample_simple:

12.0 fps
13.4 fps
11.4 fps
25.3 fps

Compared to Windows:

20.0 fps
20.1 fps
19.9 fps
20.0 fps

Do you have any idea how bad this really was when you tested it? Did it ever cause instability?

JayFoxRox · 2021-02-28T10:09:43Z

Do you have any idea how bad this really was when you tested it?

I only tested this for about a minute or two.

Tests were done with simple.py and https://github.com/JayFoxRox/Fujifilm-FinePix-Driver on an older MacBook Pro.
It's hard to tell framerate from those test cases and I don't have any logs of the FPS numbers at hand, but could get some if really necessary.

The FinePix camera image felt slow, but it's a 320x240 camera from 2002 which sends JPEG frame and is polled at 35ms per frame with naive decoding by my driver.

Did it ever cause instability?

I also had one case where my FinePix camera driver crashed, but I believe it was due to the USB code or a bad USB cable; not pyvirtualcam.

except for the test that checks whether the target fps are achieved

I expect slowness, especially at higher resolutions.

Things which should help:

Improve the runLoop polling (add macOS support #16 (comment)).
Improve UYVY conversion (add macOS support #16 (comment))
Remove memory allocation in UYVY conversion.

However, I don't want to spend time on any of that at the moment.
For now, this is "good enough" for me, and someone can iterate later.

JayFoxRox · 2021-02-28T11:41:51Z

I'm on my MacBook Pro (5-inch, 2017) / 2,8 GHz Quad-Core Intel Core i7 now.

With Zoom preferences opened to see the image:

$ python3 samples/simple.py
21.1 fps
20.1 fps
20.4 fps
20.0 fps
19.9 fps
19.4 fps
20.2 fps
20.4 fps

$ python3 samples/animated_gradient.py
Virtual cam started (1280x720 @ 20fps)
21.8 fps
22.1 fps
20.1 fps
20.5 fps
20.7 fps
20.5 fps
20.5 fps
20.3 fps

$ pytest -v -s ../test
=========================================================================================================== test session starts ============================================================================================================
platform darwin -- Python 3.9.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /usr/local/opt/[email protected]/bin/python3.9
cachedir: .pytest_cache
rootdir: <...>/pyvirtualcam
collected 5 items

../test/test_camera.py::test_consecutive PASSED
../test/test_camera.py::test_parallel virtual camera output already started
PASSED
../test/test_camera.py::test_sleep_until_next_frame PASSED
../test/test_samples.py::test_sample_simple 20.1 fps
19.8 fps
19.8 fps
20.1 fps
PASSED
../test/test_util.py::test_fps_counter PASSED

============================================================================================================ 5 passed in 13.28s ============================================================================================================

So errors likely stem from CI performance fluctuation.

letmaik · 2021-02-28T12:30:05Z

I'm on my MacBook Pro (5-inch, 2017) / 2,8 GHz Quad-Core Intel Core i7 now.
...
../test/test_camera.py::test_sleep_until_next_frame PASSED
...
So errors likely stem from CI performance fluctuation.

Great, I disabled that test in CI for macOS, all green now.