Merge branch 'MLDSAI:main' into update

.github/ISSUE_TEMPLATE/bug_form.yml

@@ -0,0 +1,29 @@
+name: Bug Report
+description: File a bug report
+title: "[Bug]: "
+labels: bug
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for taking the time to fill out this bug report!
+  - type: textarea
+    id: describe-bug
+    attributes:
+      label: Describe the bug
+      description: What behaviour was expected and what actually happened? Include screenshots and log output where useful.
+      placeholder: I expected ... but what actually happened was ...
+    validations:
+      required: true
+  - type: textarea
+    id: steps
+    attributes:
+      label: To Reproduce
+      description: What Operating System did you use and steps would we take to reproduce the behaviour?
+      placeholder: "I use [macOS/Microsoft Windows].
+        Steps: 
+        1. Go to '...' 
+        2. Click on '....' 
+        3. Scroll down to '....' "
+    validations:
+      required: true

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1 @@
+blank_issues_enabled: true

.github/ISSUE_TEMPLATE/new_feature.yml

@@ -0,0 +1,20 @@
+name: "Feature Request"
+description: Submit a proposal for a new OpenAdapt feature
+labels: enhancement
+body:
+  - type: textarea
+    id: feature-request
+    validations:
+      required: true
+    attributes:
+      label: Feature request
+      description: A clear and concise description of the feature proposal. Please provide links to any relevant resources.
+
+  - type: textarea
+    id: motivation
+    validations:
+      required: false
+    attributes:
+      label: Motivation
+      description: Please outline the purpose for the proposal (e.g., is it related to a problem?). Add any relevant links (e.g. GitHub issues).
+      placeholder: I'm always frustrated when [...] so this feature would [...].

.github/pull_request_template.md

@@ -0,0 +1,26 @@
+<!-- Thank you for submitting a pull request! To ensure a prompt review of your changes, please provide the following information. -->
+
+**What kind of change does this PR introduce?**
+
+<!-- E.g. a bugfix, feature, refactoring, etc… -->
+
+**Summary**
+
+<!-- Explain the **motivation** for making this change. What existing problem does the pull request solve? Try to link to an open issue. -->
+
+**Checklist**
+* [ ] My code follows the style guidelines of OpenAdapt
+* [ ] I have perfomed a self-review of my code
+* [ ] If applicable, I have added tests to prove my fix is functional/effective
+* [ ] I have linted my code locally prior to submission
+* [ ] I have commented my code, particularly in hard-to-understand areas
+* [ ] I have made corresponding changes to the documentation (e.g. README.md, requirements.txt)
+* [ ] New and existing unit tests pass locally with my changes
+
+**How can your code be run and tested?**
+
+<!-- See the README.md for examples. Include test output. -->
+
+
+**Other information**
+<!-- Delete this subheading if no additional context is needed. -->

.github/workflows/main.yml

@@ -0,0 +1,42 @@
+name: Python CI
+
+on:
+  push:
+    branches:
+      - '**'
+
+jobs:
+  run-ci:
+    runs-on: windows-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v3
+
+      - name: Set up Python
+        uses: actions/setup-python@v3
+        with:
+          python-version: '3.10'
+
+      - name: Install dependencies
+        run: |
+          pip install wheel
+          pip install -r requirements.txt
+          pip install -e .
+
+      - name: Check formatting with Black
+        run: |
+          black --check --exclude "(src|alembic)/" .
+
+      - name: Run headless tests
+        uses: coactions/setup-xvfb@v1
+        with:
+          run: python -m pytest
+          working-directory: ./  # Optional: Specify the working directory if needed
+          options:  # Optional: Add any additional options or arguments for pytest
+
+      - name: flake8 Lint
+        uses: py-actions/flake8@v2
+        with:
+          plugins: "flake8-docstrings"
+          extra-args: "--docstring-convention=google --exclude=alembic/versions"

.gitignore

@@ -13,3 +13,20 @@ cache
 
 # db
 *.db
+*.db-journal
+
+# VSCode
+.VSCode
+.vsCode
+
+#Idea
+.idea
+
+# Generated performance charts
+performance
+
+# Generated when adding editable dependencies in requirements.txt (-e)
+src
+
+# MacOS file
+.DS_Store

CONTRIBUTING.md

@@ -0,0 +1,33 @@
+# How to contribute
+
+We would love to implement your contributions to this project! We simply ask that you observe the following guidelines.  
+
+## Code Style
+
+This project follows the [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html).
+
+
+## Creating Issue
+In order to effectively communicate any bugs or request new features, please select the appropriate form. If none of the options suit your needs, you can click on "Open a blank issue" located at the bottom.
+
+## Testing
+[GitHub Actions](https://github.com/MLDSAI/OpenAdapt/actions/new) are automatically run on each pull request to ensure consistent behaviour and style. The Actions are composed of PyTest, [black](https://github.com/psf/black) and [flake8](https://flake8.pycqa.org/en/latest/user/index.html).
+
+You can run these tests on your own computer by downloading the depencencies in requirements.txt and then running pytest in the root directory. 
+
+## Pull Request Format
+
+To speed up the review process, please use the provided pull request template and create a draft pull request to get initial feedback. 
+
+The pull request template includes areas to explain the changes, and a checklist with boxes for code style, testing, and documenttation.
+
+## Submitting Changes
+
+1. Fork the current repository
+2. Make a branch to work on, or use the main branch
+3. Push desired changes onto the branch in step 2
+4. Submit a pull request with the branch in step 2 as the head ref and the MLDSAI/OpenAdapt main as the base ref
+     - Note: may need to click "compare across forks"
+5. Update pull request using feedback
+     - this step may not be necessary, or may need to be repeated
+6. Celebrate contributing to OpenAdapt!

README.md

@@ -2,6 +2,10 @@
 
 # OpenAdapt: AI-First Process Automation with Transformers
 
+<img width="1512" alt="image" src="https://github.com/MLDSAI/OpenAdapt/assets/774615/60a85160-7b07-41b0-9295-2d1bfa6a0994">
+
+([Slides](https://t.ly/7RGr))
+
 Welcome to OpenAdapt! This Python library implements AI-First Process Automation
 with the power of Transformers by:
 
@@ -15,14 +19,26 @@ The goal is similar to that of
 [Robotic Process Automation](https://en.wikipedia.org/wiki/Robotic_process_automation),
 except that we use transformers instead of conventional RPA tools.
 
-The approach is similar to [adept.ai](https://adept.ai/), with some key differences:
-1. Instead of requiring the user to prompt the model directly, we prompt it behind the
-scenes by observing the user's activities;
-2. We work with all types of desktop applications, not just web apps;
-3. We're open source!
+The direction is adjacent to [Adept.ai](https://adept.ai/), with some key differences:
+1. OpenAdapt is model agnostic;
+1. OpenAdapt generates prompts automatically (auto-prompted, not user-prompted);
+2. OpenAdapt works with all types of desktop GUIs, including virtualized (e.g. Citrix) and web
+3. OpenAdapt is open source! (license TBD, please see https://github.com/MLDSAI/OpenAdapt/issues/246)
 
 ## Install
 
+Recommended: Install with [Poetry](https://python-poetry.org/):
+```
+git clone https://github.com/MLDSAI/OpenAdapt.git
+cd OpenAdapt
+pip install poetry
+poetry install
+poetry shell
+alembic upgrade head
+pytest
+```
+
+Manual:
 ```
 git clone https://github.com/MLDSAI/OpenAdapt.git
 cd OpenAdapt
@@ -31,10 +47,15 @@ source .venv/bin/activate
 pip install wheel
 pip install -r requirements.txt
 pip install -e .
+python -m spacy download en_core_web_trf
 alembic upgrade head
 pytest
 ```
 
+## Permissions
+
+See how to set up system permissions on macOS [here](./permissions_in_macOS.md).
+
 ## Run
 
 ### Record
@@ -75,7 +96,7 @@ python -m openadapt.visualize
 
 This will open your browser. It will look something like this:
 
-![visualize.py](./assets/visualize.png)
+![image](https://github.com/MLDSAI/OpenAdapt/assets/774615/5d7253b7-ae12-477c-94a3-b388e4f37587)
 
 ### Playback
 
@@ -90,10 +111,6 @@ More ReplayStrategies coming soon! (see [Contributing](#Contributing)).
 
 ## Contributing
 
-### Design
-
-![image](https://user-images.githubusercontent.com/774615/236658984-01f9c06b-d132-40ee-a716-205fa76bf3f2.png)
-
 ### Problem Statement
 
 Our goal is to automate the task described and demonstrated in a `Recording`.
@@ -180,25 +197,18 @@ If you're interested in getting paid for your work, please mention it in your Pu
 
 ## Troubleshooting
 
-Apple Silicon:
+MacOS: if you encounter system alert messages or find issues when making and replaying recordings, make sure to [set up permissions accordingly](./permissions_in_macOS.md).
 
-```
-$ python openadapt/record.py
-...
-This process is not trusted! Input event monitoring will not be possible until it is added to accessibility clients.
-```
+![MacOS System Alerts](./assets/macOS_permissions_alert.png)
 
-Solution:
-https://stackoverflow.com/a/69673312
+In summary (from https://stackoverflow.com/a/69673312):
 
-```
-Settings -> Security & Privacy
-Click on the Privacy tab
-Scroll and click on the Accessibility Row
-Click the +
-Navigate to /System/Applications/Utilities/ or wherever the Terminal.app is installed
-Click okay.
-```
+1. Settings -> Security & Privacy
+2. Click on the Privacy tab
+3. Scroll and click on the Accessibility Row
+4. Click +
+5. Navigate to /System/Applications/Utilities/ (or wherever Terminal.app is installed)
+6. Click okay.
 
 ## Developing
 

alembic/env.py

@@ -7,12 +7,12 @@
 
 # this is the Alembic Config object, which provides
 # access to the values within the .ini file in use.
-config = context.config
+alembic_config = context.config
 
 # Interpret the config file for Python logging.
 # This line sets up loggers basically.
-if config.config_file_name is not None:
-    fileConfig(config.config_file_name)
+if alembic_config .config_file_name is not None:
+    fileConfig(alembic_config .config_file_name)
 
 # add your model's MetaData object here
 # for 'autogenerate' support
@@ -21,6 +21,7 @@
 from openadapt.config import DB_URL
 from openadapt.models import *
 from openadapt.db import Base
+
 target_metadata = Base.metadata
 
 # other values from the config, defined by the needs of env.py,
@@ -46,7 +47,7 @@ def run_migrations_offline() -> None:
     script output.
 
     """
-    #url = config.get_main_option("sqlalchemy.url")
+    # url = config.get_main_option("sqlalchemy.url")
     url = get_url()
     context.configure(
         url=url,
@@ -67,7 +68,7 @@ def run_migrations_online() -> None:
     and associate a connection with the context.
 
     """
-    configuration = config.get_section(config.config_ini_section)
+    configuration = alembic_config .get_section(alembic_config .config_ini_section)
     configuration["sqlalchemy.url"] = get_url()
     connectable = engine_from_config(
         configuration=configuration,

alembic/versions/104d4a614d95_add_performancestat.py

@@ -0,0 +1,36 @@
+"""add PerformanceStat
+
+Revision ID: 104d4a614d95
+Revises: b2dc41850120
+Create Date: 2023-05-27 02:59:14.032373
+
+"""
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision = '104d4a614d95'
+down_revision = 'b2dc41850120'
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.create_table('performance_stat',
+    sa.Column('id', sa.Integer(), nullable=False),
+    sa.Column('recording_timestamp', sa.Integer(), nullable=True),
+    sa.Column('event_type', sa.String(), nullable=True),
+    sa.Column('start_time', sa.Integer(), nullable=True),
+    sa.Column('end_time', sa.Integer(), nullable=True),
+    sa.Column('window_id', sa.String(), nullable=True),
+    sa.PrimaryKeyConstraint('id', name=op.f('pk_performance_stat'))
+    )
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_table('performance_stat')
+    # ### end Alembic commands ###