Jframe in macos not working

[vwxyzjn]

Hi, I am having trouble running the GUIs using Jframe in macos. The code I am running is as follows:

import jpype
import jpype.imports

jpype.startJVM()
import java
import javax
from javax.swing import *

def createAndShowGUI():
    frame = JFrame("HelloWorldSwing")
    frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE)
    label = JLabel("Hello World")
    frame.getContentPane().add(label)
    frame.pack()
    frame.setVisible(True)

# Start an event loop thread to handling gui events
@jpype.JImplements(java.lang.Runnable)
class Launch:
    @jpype.JOverride
    def run(self):
        createAndShowGUI()
javax.swing.SwingUtilities.invokeLater(Launch())

And after running the code above, there is a python icon showing up in the task bar, but not really showing anything when I clicked on it as shown in the screenshot.

I can verify this is not an issue on the linux side.

If you guys have issues accessing a mac for debugging, I will be happy to provide a macos on EC2 for the purpose of debugging this issue. Feel free to contact me personally at costa.huang at outlook dot com to get the macos VNC credentials.

[vwxyzjn]

This is also a blocker for Farama-Foundation/MicroRTS-Py#3

[Thrameos]

I believe there are some macros in jpype.gui for this (using AppHelper) though I have never used them. They were never documented and I am not sure if they ever worked.

Unfortunately I don't have access to a mac. It runs fine on windows and linux for me. I should note that your application is falling through to the exit routine so Python may not be in a health state. I would try to add something to make sure that you are not about to exit. Just add a long sleep for now. This would make sure that we don't have problems because the main thread has died and you are running code from a spawned thread. (Which could be a race condition.)

Also try adding some print statement to make sure that it is actually hitting the createAndShowGUI. It is possible that it failed due to a race with main.

[marscher]

I remember that I tried one of these gui examples (Swing) with success a long time ago.
@vwxyzjn Are you sure, you have set up your ssh connection to forward the window to your local X11 (e.g. use ssh -X, no idea about OSX X11, if installed/enabled by default).

[vwxyzjn]

That was done using VNC, and everything seems set up correctly. What Mac OS version did you run it on?

[marscher]

I don't use Apple products. Could it be, that you're using a headless version of the JRE/JDK?

[vwxyzjn]

@marscher Thanks for the reply. On Windows and Linux it seems fine, it’s just on Mac this strange issue appears. I am pretty sure that I used a non headless version.

[marscher]

OK, it then would be very helpful, if you could annotate the script with some print statements to see where the code gets stuck.

[vwxyzjn]

Looks like the script got stuck at import with https://adoptopenjdk.net/releases.html?variant=openjdk8&jvmVariant=hotspot

[vwxyzjn]

Apologies for the screenshots, hard to modify the script over vnc. The following are results for the official java here https://www.oracle.com/java/technologies/javase-jdk15-downloads.html

This is also using Python 3.9, which I don't know if it makes a difference.

[Thrameos]

Sadly I don't know how to begin debugging something like this. It looks like it fails on the first call to create a screen resource.

What happens if you try to create a screen resource in the main thread. Does that work?

[vwxyzjn]

Thanks fro the reply. What does that mean in the main thread? Do you have a code sample I could try it out by any chance?

[marscher]

I think this "invokeLater" method does create a new thread, if you directly invoke your JFrame creating function it should run in the main thread.

[marscher]

I mean you should not wrap it in the Runnable (Thread) interface

[vwxyzjn]

import jpype
import jpype.imports

jpype.startJVM()
print("jvm started")
import java
import javax
from javax.swing import *
print("java swing imported")

def createAndShowGUI():
    print("l1")
    frame = JFrame("HelloWorldSwing")
    print("l2")
    frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE)
    print("l3")
    label = JLabel("Hello World")
    print("l4")
    frame.getContentPane().add(label)
    print("l5")
    frame.pack()
    print("l6")
    frame.setVisible(True)
    print("l7")


createAndShowGUI()

So definitely have some interesting results :) The code is able to finish but still not having the window show up.

[vwxyzjn]

If by chance you are interested in using this mac instance through VNC, feel free to download this private key (two one-time link)

https://file.io/o5eQB6ShSj9x
https://file.io/plInRCcXekvZ

and run

ssh -L 5900:localhost:5900 -i m.pem [email protected]
# now open a vnc to connect `vnc://ec2-user@localhost`

[Thrameos]

I unfortunately have work for now so I can't help with a debugging session. I did try the code you sent and it runs and shows the window as expected. So this is still looking a lot like X session or headless issue. So the next step would be to make sure this isn't a race condition with termination. As your main body just has Python terminated after calling the create... so lets make it

createAndShowGUI()

# Make sure we are not getting into a race
print("post")
java.lang.Thread.sleep(10000)
print("done")

Or we can make sure that X11 is working at all.

import matplotlib.pyplot as plt
plt.plot([0,1])

createAndShowGUI()

# Make sure that X11 is reachable
plt.show()

[vwxyzjn]

second snippet seems very interesting

[vwxyzjn]

So the hello world window only shows when I do the following

import matplotlib.pyplot as plt
plt.plot([0,1])

createAndShowGUI()

# Make sure that X11 is reachable
plt.show()

And the following does not show the hello world window

import matplotlib.pyplot as plt

createAndShowGUI()

# Make sure that X11 is reachable
plt.show()

[vwxyzjn]

And it kind of works with gym-microrts now!! :D

import gym
import gym_microrts
import time
import numpy as np
from gym.wrappers import Monitor
from gym_microrts import microrts_ai
import matplotlib.pyplot as plt

env = gym.make(
    "MicrortsDefeatCoacAIShaped-v3",
    render_theme=2, # optional customization
    frame_skip=0, # optional customization
    ai2=microrts_ai.coacAI, # optional customization
    map_path="maps/16x16/basesWorkers16x16.xml", # optional customization
    reward_weight=np.array([10.0, 1.0, 1.0, 0.2, 1.0, 4.0, 0.0]) # optional customization
    # the above `reward_weight` (in order) means +10 for wining,
    # +1 for each resource gathered or returned, +1 for each worker produced
    # +0.2 for each building produced, +1 for each attack action issued
    # +4 for each combat units produced, +0.0 for getting `closer` to enemy base
)
# env = Monitor(env, f'videos', force=True)
env.action_space.seed(0)
env.reset()
for i in range(100):
    plt.plot([0,1])
    env.render()
    plt.show()
    time.sleep(0.001)
    action = env.action_space.sample()

    # optional: selecting only valid units.
    if len((env.unit_location_mask==1).nonzero()[0]) != 0:
        action[0] = (env.unit_location_mask==1).nonzero()[0][0]

    next_obs, reward, done, info = env.step(action)
    if done:
        env.reset()
env.close()
print("done")

However I have to close the plotted window for the game to continue

[Thrameos]

So that seems to indicate that something has not opened the X11 connection prior to call to JFrame. When we call matplotlib it initializes the X11 connection, and then once initialized Java can use the open connection. But it we don't have an open connection then X11 fails. What happens if you simply clear the figure without asking it to be shown?

So the question we need to resolve is how to get the X11 connection open without depending on matplotlib being called first.

[vwxyzjn]

Thanks so much for helping to debug this. What do you mean by "clearing the figure". Do you mean plt.clf()? So the following does not work

plt.plot([0,1])
createAndShowGUI()

plt.plot([1,2])
createAndShowGUI()
plt.clf()

[Thrameos]

So the show is required. What about?

# Create a figure and proceed
plt.show(block=false)

# Close the figure
plt.close()
``

[vwxyzjn]

import matplotlib.pyplot as plt
plt.plot([1,2])

createAndShowGUI()
# Make sure that X11 is reachable
# Create a figure and proceed
plt.show(block=False)

# Close the figure
plt.close()

does not work

[Thrameos]

Interesting. Well we have some clue of what is going on, but not the root source of the problem. I am sure that it is some osx specific command that is required to start the application loop. When you are using matplotlib on osx it creates the required resource, but when you use Java it is getting missed. Unfortunately beyond pointing you to the jpype/_gui hooks and isolating it to not being a connection problem but rather something missing (because matplotlib succeeds where Java does not), I am at my limit of osx knowledge.

I hope this will point in the direction of useful web searching, though I feel that it is not specific enough yet.

Have you tried the simple experiment of writing the application in pure Java and testing using the same java that is being launched from JPype. Perhaps we can isolate this further?

[vwxyzjn]

Have you tried the simple experiment of writing the application in pure Java and testing using the same java that is being launched from JPype. Perhaps we can isolate this further?

The short answer is yes. The renderer for gym-microrts is written entirely on the java side: https://github.com/vwxyzjn/microrts/blob/3461c3ecf6f20344c89d36b0bf48da18a1843df7/src/tests/JNIClient.java#L105-L120

[ap--]

This issue is extremely interesting to one of our projects: paquo

Essentially we've been facing similar problems when trying to run the QuPath GUI with full control from Python on OSX. I'm not sure if it's helpful, but here are some related links:

See: https://forum.image.sc/t/paquo-read-write-qupath-projects-from-python/41892/14?u=poehlmann
And the referenced thread from above: https://forum.image.sc/t/displaying-imagej-and-napari-ui-simultaneously/32187

If I understand correctly, the main issue for us is that (and I quote @sdvillal):

"macOS constrains the event loop of GUI applications to run in the main thread of the process, which means the python process itself gets blocked".

I have a few experiments with jpype._gui where I can get some functionality to work, but the Python interpreter falls through and is in a non healthy state - kept alive before exit. I'm relatively busy for the next few days, but might be able to post an example the day after tomorrow.

Cheers,
Andreas 😃

[vwxyzjn]

@ap-- Thanks so much for chiming in. By using %gui osx in jupyter, I can indeed get it work with the hello world example

However, still causing issue for gym-microrts

[Thrameos]

@ap-- Can you make the Python main call some kind of handle even loop and wait for event loop to complete before proceeding? I am not sure what call that is in Java side by it would transfer control back to Java which frees the Python interpreter to run code in other threads and would prevent falling into the exit state until after the GUI is complete, but I am sure there should be something.

Perhaps it can be something as simple as adding join statement.

http://www.javased.com/?post=1341699

I know there is a call to check which thread is the event dispatch, but I don't know how to get the thread through swing.

https://docs.oracle.com/javase/8/docs/api/javax/swing/SwingUtilities.html#isEventDispatchThread--

[sdvillal]

@vwxyzjn what happens if instead you use the "%gui qt" magic?
https://forum.image.sc/t/paquo-read-write-qupath-projects-from-python/41892/17

[vwxyzjn]

@sdvillal does not seem to work

[sdvillal]

@vwxyzjn I have the feeling that you do not have a qt implementation installed. You could try to install one, or alternatively give a try to "%gui matplotlib".

[vwxyzjn]

@sdvillal sorry for the delay. Both %gui matplotlib and %gui qt works for the hello world example but not the gym-microrts example.

[vsquared]

The following runs without error on my Mac (Sonoma 14.4.1 Intel). I ran the code in a Thonny editor with the PyObjc plugin added.

import jpype
import jpype.imports

jpype.startJVM()
import java
import javax
from javax.swing import *
from Cocoa import NSApp

def createAndShowGUI():
    frame = JFrame("HelloWorldSwing")
    frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE)
    label = JLabel("Hello World")
    frame.getContentPane().add(label)
    frame.pack()
    frame.setVisible(True)

javax.swing.SwingUtilities.invokeLater(createAndShowGUI)
NSApp.run()

The key to getting the frame to show up on a Mac is the last line which comes from the Cocoa framework.

[Thrameos]

You are correct. The only way for GUIs to run is to have an event servicing loop running. The question remains if the event servicing loop can be a new thread that is spawned or if it must be the main thread. It would be nice if we had a pattern that worked on all machines but given my lack of access to a Mac I will have to depend on users.

[vsquared]

I think this will run across platforms, but am waiting on folks using other operating systems to test it. Please let me know:

import jpype
import jpype.imports

jpype.startJVM()
import java
import javax
import platform
from javax.swing import *
if(platform.system() == "Darwin"):
    from Cocoa import NSApp

def createAndShowGUI():
    frame = JFrame("HelloWorldSwing")
    frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE)
    label = JLabel("Hello World")
    frame.getContentPane().add(label)
    frame.pack()
    frame.setVisible(True)

javax.swing.SwingUtilities.invokeLater(createAndShowGUI)
print(platform.system())
if(platform.system() == "Darwin"):    
    NSApp.run()

run starts the main event loop. Personally, I've never seen a programmatically created mac app that ran from a spawned event loop.

[Thrameos]

Do both playforns execute code after the if statement?

[vsquared]

Do both playforns execute code after the if statement?

I'm not sure I understand the question. If the user has a mac, the main event loop will be started. To the best of my knowledge the other two platforms would not be affected by the last 'if' statement. I have no knowledge of what Linux and Windows use for their event loops. I just know that every mac demo that I ever created programmatically has a 'main' and the last line of code in that main is a call to 'run' (which starts the main event loop).

[Thrameos]

My question is will the script run the same on both platforms. Ie.

import jpype
import jpype.imports

jpype.startJVM(classpath="jars/*")
import java
import javax
import platform
from javax.swing import *
if(platform.system() == "Darwin"):
    from Cocoa import NSApp

def createAndShowGUI():
    frame = JFrame("HelloWorldSwing")
    frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE)
    label = JLabel("Hello World")
    frame.getContentPane().add(label)
    frame.pack()
    frame.setVisible(True)

javax.swing.SwingUtilities.invokeLater(createAndShowGUI)
print(platform.system())
if(platform.system() == "Darwin"):
    NSApp.run()

print("Continues")

If the statement prints "Continues" on both platforms then the code is equivalent. if the mac code stops at NSApp.run() then there is a difference that we need to address. Once we have a working solution that gives the same behavior I can fold it into the documentation and make a call for "jpype.startGUI()" that will call the correct behavior on all systems.

[Thrameos]

If the mac stops at run we may be able to start another thread which calls the run. Though I believe that there is some restriction on which thread can run the event loop which have been the problem for providing a consistent user experience.

[vsquared]

The demo code above (run in Thonny IDE) does not print("Continues") on my mac and I wouldn't expect it to. When I write code in objc everything is above the 'main' and 'run' is the last call in 'main'. I don't recall ever seeing it done any other way; not sure why you would want to do that.

[Thrameos]

It isn't really a matter of why would someone want to do it. It is a mater the people chose to do something with the main thread such as continuing with the interactive shell.

The issue it that the code must run the same regardless of the platform and as all other systems continue execution then the solution for mac must do the same....

Does this work on mac with the same behavior as Linux and Windows.

import jpype
import jpype.imports

jpype.startJVM(classpath="jars/*")
import java
import javax
import platform
from javax.swing import *
if(platform.system() == "Darwin"):
    from Cocoa import NSApp

def createAndShowGUI():
    frame = JFrame("HelloWorldSwing")
    frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE)
    label = JLabel("Hello World")
    frame.getContentPane().add(label)
    frame.pack()
    frame.setVisible(True)

javax.swing.SwingUtilities.invokeLater(createAndShowGUI)
print(platform.system())
if(platform.system() == "Darwin"):
    import threading
    t = threading.Thread(target=NsApp.run)
    t.start()

print("Continues")

If it works then I think we have a solution. If it fails then we remain stuck because interactive sessions and code that expected the thread to continue will fail.

[Thrameos]

BTW I would like to thank you for your assistance.

It is important to remember that when writing multiplatform libraries that features always be universal, not just across platforms but also across time. Thus if I can't get the same behavior for every system or guarantee that same behavior will exist for as long as the library exists (even when the dependencies get upgraded), then I can't include it as a feature of the library. I know that most programmers take a "works for me attitude" but with libraries you have to assume everyone will think about problems in different ways and it is always the unusual pattern that cause edge case fails.

[vsquared]

I'm experimenting with a generic python editor that uses JPype to bridge to Java and will likely use a version of the above code as a template (which potentially could be run on all three platforms). The group of programmers it will be released into normally don't play around with threads so I will be interested to see how it is received. If there are problems I'll remove it from the server.

Thus if I can't get the same behavior for every system or guarantee that same behavior will exist for as long as the library exists (even when the dependencies get upgraded), then I can't include it as a feature of the library. I hear what you are saying and agree. On the other hand, I initially got this code from your website and as written it doesn't run on a mac so by your definition it's not really cross-platform. That's why I think you should put up a disclaimer to that effect until it gets resolved. Just my opinion and I do appreciate all the work that you've put into this project; otherwise I couldn't tap into java at all.

[Thrameos]

Technically there is nothing "wrong" with the example. It is "cross platform" in that is textbook what Java requires. OSX just has added some requirements that violate the Java contract. So OSX being the only one that doesn't service the loop automatically is the odd man out. After all the whole point of javax.swing.SwingUtilities.invokeLater(createAndShowGUI) is to execute the event handler elsewhere so that main thread can continue. Forcing the users to the call something else to service the event loop sort of defeats the purpose. Though that is just my prospective. ¯\_(ツ)_/¯

I believe there was "intended" solution for mac already in the JPype under setupGuiEnvironment. Unfortunately, without a mac, I have can't test it nor verify its function. So sadly I don't know if it works properly as it predates my involvement with the project entirely thus I was never able to add it to the manual given I have no examples of it running. It also has the problem that it breaks the main thread continuing so no way to use it interactively. It is "universal" in that it breaks the flow of all systems equally.

As far as threads, we have many options here. We can launch through python or Java threads and it can be invisible to the user. The question is does it actually service the loop properly with a complex gui. There have been reports that on mac the gui event loop must be main or there will be deadlocks. Unfortunately I don't understand how matplotlib and other plotting modules would work if that were the case.

Once I have a documentable and cross platform solution, I have no problem making it available.

Lets just run through some tests...

1. First does it still run with the service loop in a thread?

2. What happens if the service loop thread is launched prior to the gui set up? (Can we do it proactively?)

import jpype
import jpype.imports

jpype.startJVM(classpath="jars/*")

print(platform.system())
if(platform.system() == "Darwin"):
    import threading
    t = threading.Thread(target=NsApp.run)
    t.start()

import java
import javax
import platform
from javax.swing import *
if(platform.system() == "Darwin"):
    from Cocoa import NSApp

def createAndShowGUI():
    frame = JFrame("HelloWorldSwing")
    frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE)
    label = JLabel("Hello World")
    frame.getContentPane().add(label)
    frame.pack()
    frame.setVisible(True)

javax.swing.SwingUtilities.invokeLater(createAndShowGUI)

print("Continues")

In other words, can I just make it start with jpype.startJVM(gui=True) such that I can just start and forget. On Linux and Windows can just make it a NOP, and on OSX it will start the event loop. This is my preferred solution as it is mostly transparent.

[vsquared]

Won't run on a Mac. First error: NameError: name 'platform' is not defined If I fix that by moving 'import platform' up under 'import jpype.imports', I get this error NameError: name 'NsApp' is not defined which I resolved by moving 'from Cocoa import NSApp' up above 'import threading'. Actually it's NSApp (upper case 's') which harkens back to Steve Jobs and his NextStep endeavors after he initially left Apple. Also the 'run' usually has parentheses after it, eg NSApp.run() in Python. In objc its just run. I tried all the permutations I could think of but it still could not figure out what to do with 'run'. Unresolved error = AttributeError: 'NoneType' object has no attribute 'run'

[Thrameos]

Sorry the problem with code that I can't run is I don't get to see trivial errors (and with dyslexia I don't read code particularly well.)

Lets try one more time....

import jpype
import jpype.imports
import platform

# Start the JVM
jpype.startJVM(classpath="jars/*")

# Check the platform and start event handler for OSX
if(platform.system() == "Darwin"):

    # Define a function to run later
    def deferred():
        # Start servicing in the  new thread
        print(platform.system())
        from Cocoa import NSApp
        NSApp.run()

    # Launch later
    import threading
    thr = threading.Thread(target=deferred)   # No parentheses here because it is supposed run later
    thr.start()

# Begin Java code
import java
import javax
from javax.swing import *

def createAndShowGUI():
    frame = JFrame("HelloWorldSwing")
    frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE)
    label = JLabel("Hello World")
    frame.getContentPane().add(label)
    frame.pack()
    frame.setVisible(True)

javax.swing.SwingUtilities.invokeLater(createAndShowGUI)
print("Continues")

[vsquared]

That works!

Addendum:
Console output looks like what you're trying to achieve, but.... There's no GUI. Swing window is not on screen.

[Thrameos]

Very weird. Does the gui appear when the "if" block is after the invoke later? I am not sure why their event loop would care if it was started before or after the elements were added. After all you have to be able to add GUI elements after starting the event loop.

I believe this is must be the documented problem that the only thread that can service the event loop is the main (which is a horrible restriction because it prohibits interactive python). I still am clueless how matplotlib and other python apps that pop up windows and continue the interactive can work.

[vsquared]

I didn't understand when the "if" block is after the invoke later. Perhaps better if you re-post with the code rearranged the way that you are suggesting.

[Thrameos]

import jpype.imports
import platform

# Start the JVM
jpype.startJVM(classpath="jars/*")


# Begin Java code
import java
import javax
from javax.swing import *

def createAndShowGUI():
    frame = JFrame("HelloWorldSwing")
    frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE)
    label = JLabel("Hello World")
    frame.getContentPane().add(label)
    frame.pack()
    frame.setVisible(True)

javax.swing.SwingUtilities.invokeLater(createAndShowGUI)

# Check the platform and start event handler for OSX
if(platform.system() == "Darwin"):

    # Define a function to run later
    def deferred():
        # Start servicing in the  new thread
        print(platform.system())
        from Cocoa import NSApp
        NSApp.run()

    # Launch later
    import threading
    thr = threading.Thread(target=deferred)   # No parentheses here because it is supposed run later
    thr.start()

print("Continues")

I suspect this also won't work. Cocoa is notorious because they are the only the only system that has this weird restriction.

glfw/glfw#136

[vsquared]

No, there is no GUI with that method either.

[Thrameos]

Thanks for the feedback. It seems that all I can do is put documentation about the work around. There is really no way to support the same pattern that every other system has if they are going to force the event loop onto a specific thread. Python doesn't have a "resume from here" on another thread option so I can't really add an easy to use option.

I see virtually the same complaint going back 22 years in forum. So their bad design choices just live on forever. Much like the "Java doesn't like to be forked" and "Java gives UTF8 encoded UTF16 strings" these poor choices create restrictions that burn users year after year and poor library developers like me are always caught in the middle. Gee if only they had a large amount of money flowing in from a product line so that could address problems.

[vsquared]

I doubt that it will change anything, but below is boilerplate code that I have used on hundreds if not thousands of objc demos:

int main () {
 NSApplication *application = [NSApplication sharedApplication];
 AppDelegate *appDelegate = [[AppDelegate alloc] init];
 [application setDelegate:appDelegate];
 [application run];
return 0;
}

NSApp is just the approved shortcut to an instance of NSApplication.

[Thrameos]

I am going to sleep on it and consider some options. I am trying to make a "friendly" version for users that want to now have the main thread get taken, by simply launching Python in such a way that the main thread a side thread. I may request a few more tests to see if it deals with the issue.

Thanks so much for your help.

[vsquared]

Thanks so much for your help. No problem. Enjoyed working with you. Let me know if you need further testing.

[Thrameos]

I have looked through a good amount of code. OSX is very particular about what they consider to be main. Unless I go to the "jpython" solution in which we make a custom version of Python in which the Python thread is started on a non-main thread.

The key appears to be pthread_main_np() which is checked by Application main. The question is if there is any way to designate some other thread as main later? There seems like some there may be ways to hack it. But given the number of others that have fought the problem I doubt that something hasn't already found a clever solution.

I think the problem remains that doing research on this definitely is something that requires access to a platform. I will once again ping my local workgroup to see if there is anything that can be begged or borrowed.

[vsquared]

I don't know much about threading, but I came across this today and you're probably aware of it, but just in case it could be helpful:
https://docs.python.org/3/library/asyncio-eventloop.html. Macos is a unix system and so is Linux. If you can control Linux one would think you could also control macos. As an aside, I just ran jpype code similar to that above in my prototypic generic editor and it produced a nice window with a button.

Addendum: Link doesn't work for some reason; you'll have to copy/paste it into a browser.