Just a simple "Cancel" and "Exit" buttons would make the experience more like what most users expect.

[CosmicTourist]

Even though I am a more advanced user, my training as a systems analyst always helps me to look at an application from the average user's viewpoint. In that regard, having a Cancel button and an Exit button on the screen would help some users feel that the boxes they've checked/unchecked have, in fact, been processed and they can leave the application gracefully. Just having users check/uncheck boxes and then hit the "X" may make some users wonder whether they made any changes at all.

[xM4ddy]

Please use issues for things like this, also since those buttons are unneeded I don't think that adding more buttons is going to make it clearer but I will consider instructions in the app.

[unfrozencanary]

you could just add a note "changes take effect immediately".