Merge remote-tracking branch 'upstream/main' into add-GUI

.gitignore

@@ -13,3 +13,6 @@ cache
 
 # db
 *.db
+
+# Generated when adding editable dependencies in requirements.txt (-e)
+src

README.md

@@ -7,7 +7,7 @@ with the power of Transformers by:
 
 - Recording screenshots and associated user input
 - Aggregating and visualizing user input and recordings for development
-- Converting screenshots and user input into tokenized format
+- Converting screenshots and user input into tok/enized format
 - Generating synthetic input via transformer model completions
 - Replaying synthetic input to complete tasks
 
@@ -35,6 +35,10 @@ alembic upgrade head
 pytest
 ```
 
+## Permissions
+
+See how to set up system permissions on macOS [here](./permissions_in_macOS.md).
+
 ## Run
 
 ### Record
@@ -178,25 +182,18 @@ If you're interested in getting paid for your work, please mention it in your Pu
 
 ## Troubleshooting
 
-Apple Silicon:
+MacOS: if you encounter system alert messages or find issues when making and replaying recordings, make sure to [set up permissions accordingly](./permissions_in_macOS.md).
 
-```
-$ python openadapt/record.py
-...
-This process is not trusted! Input event monitoring will not be possible until it is added to accessibility clients.
-```
+![MacOS System Alerts](./assets/macOS_permissions_alert.png)
 
-Solution:
-https://stackoverflow.com/a/69673312
+In summary (from https://stackoverflow.com/a/69673312):
 
-```
-Settings -> Security & Privacy
-Click on the Privacy tab
-Scroll and click on the Accessibility Row
-Click the +
-Navigate to /System/Applications/Utilities/ or wherever the Terminal.app is installed
-Click okay.
-```
+1. Settings -> Security & Privacy
+2. Click on the Privacy tab
+3. Scroll and click on the Accessibility Row
+4. Click +
+5. Navigate to /System/Applications/Utilities/ (or wherever Terminal.app is installed)
+6. Click okay.
 
 ## Developing
 

permissions_in_macOS.md

@@ -0,0 +1,23 @@
+# Setting permissions in macOS
+
+If using macOS, you'll need to set up some permissions to ensure that OpenAdapt works as intended. The following instructions apply to macOS Ventura. Earlier version may present different menus.
+
+Note that while you will be prompted on first run for input monitoring and screen recording permissions, you won't be prompted for accessibility permissions needed for replay of actions. If permission is not granted in that case, replay of actions will silently fail.
+
+## Enabling input monitoring
+
+Input monitoring must be enabled in Settings in order to allow capture of mouse and keyboard input. You can do so by enabling the Terminal application under Settings → Privacy and security → Input monitoring → Allow Terminal
+
+![Enabling input monitoring](./assets/macOS_input_monitoring.png)
+
+## Enabling screen recording
+
+Screen recoding must be enabled in Settings in order to allow capture of screenshots. You can do so by enabling the Terminal application under Settings → Privacy and security → Screen recording → Allow Terminal
+
+![Enabling screen recording](./assets/macOS_screen_recording.png)
+
+## Enabling replay of actions
+
+Mouse and keyboard control must be enabled in Settings in order to allow replay of actions by the framework. You can do so by enabling the Terminal application under Settings → Privay and security → Accessibility → Allow Terminal
+
+![Enabling replay of actions](./assets/macOS_accessibility.png)

requirements.txt

@@ -1,6 +1,6 @@
 alembic==1.8.1
 ascii_magic==2.3.0
-git+https://github.com/abrichr/atomacos.git; sys_platform == 'darwin'
+-e git+https://github.com/abrichr/atomacos.git; sys_platform == 'darwin'
 bokeh==2.4.3
 clipboard==0.0.4
 deepdiff[optimize]==6.3.0
@@ -14,9 +14,11 @@ openai==0.27.5
 pandas==2.0.0
 pygetwindow==0.0.9; sys_platform == 'win32'
 pyinstaller
+setuptools-lint
+sphinx
 python-dotenv==1.0.0
 pywin32==306; sys_platform == 'win32'
-git+https://github.com/abrichr/pynput.git
+-e git+https://github.com/abrichr/pynput.git#egg=pynput
 pytest==7.1.3
 rapidocr-onnxruntime==1.2.3
 scikit-learn==1.2.2