Merge branch 'MLDSAI:main' into expand-on-demo-replay-strategy

.github/ISSUE_TEMPLATE/bug_form.yml

@@ -0,0 +1,29 @@
+name: Bug Report
+description: File a bug report
+title: "[Bug]: "
+labels: bug
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for taking the time to fill out this bug report!
+  - type: textarea
+    id: describe-bug
+    attributes:
+      label: Describe the bug
+      description: What behaviour was expected and what actually happened? Include screenshots and log output where useful.
+      placeholder: I expected ... but what actually happened was ...
+    validations:
+      required: true
+  - type: textarea
+    id: steps
+    attributes:
+      label: To Reproduce
+      description: What Operating System did you use and steps would we take to reproduce the behaviour?
+      placeholder: "I use [macOS/Microsoft Windows].
+        Steps: 
+        1. Go to '...' 
+        2. Click on '....' 
+        3. Scroll down to '....' "
+    validations:
+      required: true

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1 @@
+blank_issues_enabled: true

.github/ISSUE_TEMPLATE/new_feature.yml

@@ -0,0 +1,20 @@
+name: "Feature Request"
+description: Submit a proposal for a new OpenAdapt feature
+labels: enhancement
+body:
+  - type: textarea
+    id: feature-request
+    validations:
+      required: true
+    attributes:
+      label: Feature request
+      description: A clear and concise description of the feature proposal. Please provide links to any relevant resources.
+
+  - type: textarea
+    id: motivation
+    validations:
+      required: false
+    attributes:
+      label: Motivation
+      description: Please outline the purpose for the proposal (e.g., is it related to a problem?). Add any relevant links (e.g. GitHub issues).
+      placeholder: I'm always frustrated when [...] so this feature would [...].

.github/pull_request_template.md

@@ -0,0 +1,26 @@
+<!-- Thank you for submitting a pull request! To ensure a prompt review of your changes, please provide the following information. -->
+
+**What kind of change does this PR introduce?**
+
+<!-- E.g. a bugfix, feature, refactoring, etc… -->
+
+**Summary**
+
+<!-- Explain the **motivation** for making this change. What existing problem does the pull request solve? Try to link to an open issue. -->
+
+**Checklist**
+* [ ] My code follows the style guidelines of OpenAdapt
+* [ ] I have perfomed a self-review of my code
+* [ ] If applicable, I have added tests to prove my fix is functional/effective
+* [ ] I have linted my code locally prior to submission
+* [ ] I have commented my code, particularly in hard-to-understand areas
+* [ ] I have made corresponding changes to the documentation (e.g. README.md, requirements.txt)
+* [ ] New and existing unit tests pass locally with my changes
+
+**How can your code be run and tested?**
+
+<!-- See the README.md for examples. Include test output. -->
+
+
+**Other information**
+<!-- Delete this subheading if no additional context is needed. -->

.github/workflows/main.yml

@@ -0,0 +1,42 @@
+name: Python CI
+
+on:
+  push:
+    branches:
+      - '**'
+
+jobs:
+  run-ci:
+    runs-on: windows-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v3
+
+      - name: Set up Python
+        uses: actions/setup-python@v3
+        with:
+          python-version: '3.10'
+
+      - name: Install dependencies
+        run: |
+          pip install wheel
+          pip install -r requirements.txt
+          pip install -e .
+
+      - name: Check formatting with Black
+        run: |
+          black --check --exclude "(src|alembic)/" .
+
+      - name: Run headless tests
+        uses: coactions/setup-xvfb@v1
+        with:
+          run: python -m pytest
+          working-directory: ./  # Optional: Specify the working directory if needed
+          options:  # Optional: Add any additional options or arguments for pytest
+
+      - name: flake8 Lint
+        uses: py-actions/flake8@v2
+        with:
+          plugins: "flake8-docstrings"
+          extra-args: "--docstring-convention=google --exclude=alembic/versions"

.gitignore

@@ -19,8 +19,14 @@ cache
 .VSCode
 .vsCode
 
+#Idea
+.idea
+
 # Generated performance charts
 performance
 
 # Generated when adding editable dependencies in requirements.txt (-e)
 src
+
+# MacOS file
+.DS_Store

CONTRIBUTING.md

@@ -0,0 +1,33 @@
+# How to contribute
+
+We would love to implement your contributions to this project! We simply ask that you observe the following guidelines.  
+
+## Code Style
+
+This project follows the [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html).
+
+
+## Creating Issue
+In order to effectively communicate any bugs or request new features, please select the appropriate form. If none of the options suit your needs, you can click on "Open a blank issue" located at the bottom.
+
+## Testing
+[GitHub Actions](https://github.com/MLDSAI/OpenAdapt/actions/new) are automatically run on each pull request to ensure consistent behaviour and style. The Actions are composed of PyTest, [black](https://github.com/psf/black) and [flake8](https://flake8.pycqa.org/en/latest/user/index.html).
+
+You can run these tests on your own computer by downloading the depencencies in requirements.txt and then running pytest in the root directory. 
+
+## Pull Request Format
+
+To speed up the review process, please use the provided pull request template and create a draft pull request to get initial feedback. 
+
+The pull request template includes areas to explain the changes, and a checklist with boxes for code style, testing, and documenttation.
+
+## Submitting Changes
+
+1. Fork the current repository
+2. Make a branch to work on, or use the main branch
+3. Push desired changes onto the branch in step 2
+4. Submit a pull request with the branch in step 2 as the head ref and the MLDSAI/OpenAdapt main as the base ref
+     - Note: may need to click "compare across forks"
+5. Update pull request using feedback
+     - this step may not be necessary, or may need to be repeated
+6. Celebrate contributing to OpenAdapt!

README.md

@@ -2,27 +2,50 @@
 
 # OpenAdapt: AI-First Process Automation with Transformers
 
+### Enormous volumes of mental labor are wasted on repetitive GUI workflows.
+
+### Foundation Models (e.g. [GPT-4](https://openai.com/research/gpt-4), [ACT-1](https://www.adept.ai/blog/act-1)) are powerful automation tools.
+
+### OpenAdapt connects Foundation Models to GUIs:
+
+<img width="1473" alt="image" src="https://github.com/MLDSAI/OpenAdapt/assets/774615/5a760e4a-c596-4604-b1a4-a9563dce0fe7">
+
+
+([Slides](https://t.ly/7RGr))
+
 Welcome to OpenAdapt! This Python library implements AI-First Process Automation
 with the power of Transformers by:
 
 - Recording screenshots and associated user input
 - Aggregating and visualizing user input and recordings for development
-- Converting screenshots and user input into tok/enized format
+- Converting screenshots and user input into tokenized format
 - Generating synthetic input via transformer model completions
 - Replaying synthetic input to complete tasks
 
 The goal is similar to that of
 [Robotic Process Automation](https://en.wikipedia.org/wiki/Robotic_process_automation),
 except that we use transformers instead of conventional RPA tools.
 
-The approach is similar to [adept.ai](https://adept.ai/), with some key differences:
-1. Instead of requiring the user to prompt the model directly, we prompt it behind the
-scenes by observing the user's activities;
-2. We work with all types of desktop applications, not just web apps;
-3. We're open source!
+The direction is adjacent to [Adept.ai](https://adept.ai/), with some key differences:
+1. OpenAdapt is model agnostic;
+1. OpenAdapt generates prompts automatically (auto-prompted, not user-prompted);
+2. OpenAdapt works with all types of desktop GUIs, including virtualized (e.g. Citrix) and web
+3. OpenAdapt is open source! (license TBD, please see https://github.com/MLDSAI/OpenAdapt/issues/246)
 
 ## Install
 
+Recommended: Install with [Poetry](https://python-poetry.org/):
+```
+git clone https://github.com/MLDSAI/OpenAdapt.git
+cd OpenAdapt
+pip install poetry
+poetry install
+poetry shell
+alembic upgrade head
+pytest
+```
+
+Manual:
 ```
 git clone https://github.com/MLDSAI/OpenAdapt.git
 cd OpenAdapt
@@ -95,8 +118,6 @@ More ReplayStrategies coming soon! (see [Contributing](#Contributing)).
 
 ## Contributing
 
-### Design
-
 ### Problem Statement
 
 Our goal is to automate the task described and demonstrated in a `Recording`.

install/install_openadapt.ps1

@@ -14,6 +14,7 @@ $VCRedistInstaller = "vc_redist.x64.exe"
 $VCRedistInstallerLoc = "https://aka.ms/vs/17/release/vc_redist.x64.exe"
 $VCRedistRegPath = "HKLM:\SOFTWARE\Microsoft\VisualStudio\14.0\VC\Runtimes\X64"
 
+# TODO: Add Tesseract OCR installation: https://tesseract-ocr.github.io/tessdoc/Installation.html
 
 # Return true if a command/exe is available
 function CheckCMDExists() {

install/install_openadapt.sh

@@ -1,6 +1,8 @@
 #!/bin/bash
 set -e
 
+# TODO: Add Tesseract OCR installation: https://tesseract-ocr.github.io/tessdoc/Installation.html
+
 # Run a command and ensure it did not fail
 RunAndCheck() {
     res=$($1)

openadapt/app/__init__.py

@@ -0,0 +1,3 @@
+from openadapt.app.main import run_app
+
+run_app()

openadapt/app/build.py

@@ -0,0 +1,30 @@
+import os
+import subprocess
+from pathlib import Path
+
+import nicegui
+
+spec = [
+    "pyi-makespec",
+    f"{Path(__file__).parent}/main.py",
+    f"--icon={Path(__file__).parent}/assets/logo.ico",
+    "--name",
+    "OpenAdapt",  # name
+    # "--onefile", # trade startup speed for smaller file size
+    "--onedir",
+    "--windowed",  # prevent console appearing, only use with ui.run(native=True, ...)
+    "--add-data",
+    f"{Path(nicegui.__file__).parent}{os.pathsep}nicegui",
+]
+
+subprocess.call(spec)
+
+# add import sys ; sys.setrecursionlimit(sys.getrecursionlimit() * 5) to line 2 of OpenAdapt.spec
+with open("OpenAdapt.spec", "r+") as f:
+    lines = f.readlines()
+    lines[1] = "import sys ; sys.setrecursionlimit(sys.getrecursionlimit() * 5)\n"
+    f.seek(0)
+    f.truncate()
+    f.writelines(lines)
+
+subprocess.call(["pyinstaller", "OpenAdapt.spec"])

openadapt/app/cards.py

@@ -0,0 +1,88 @@
+import signal
+from nicegui import ui
+from subprocess import Popen
+from openadapt.app.objects.local_file_picker import LocalFilePicker
+from openadapt.app.util import set_dark, sync_switch
+
+PROC = None
+
+
+def settings(dark_mode):
+    with ui.dialog() as settings, ui.card():
+        s = ui.switch("Dark mode", on_change=lambda: set_dark(dark_mode, s.value))
+        sync_switch(s, dark_mode)
+        ui.button("Close", on_click=lambda: settings.close())
+
+    settings.open()
+
+
+def select_import(f):
+    async def pick_file():
+        result = await LocalFilePicker(".")
+        ui.notify(f"Selected {result[0]}" if result else "No file selected.")
+        selected_file.text = result[0] if result else ""
+        import_button.enabled = True if result else False
+
+    with ui.dialog() as import_dialog, ui.card():
+        with ui.column():
+            ui.button("Select File", on_click=pick_file).props("icon=folder")
+            selected_file = ui.label("")
+            selected_file.visible = False
+            import_button = ui.button(
+                "Import", on_click=lambda: f(selected_file.text, delete.value)
+            )
+            import_button.enabled = False
+            delete = ui.checkbox("Delete file after import")
+
+    import_dialog.open()
+
+
+def recording_prompt(options, record_button):
+    if PROC is None:
+        with ui.dialog() as dialog, ui.card():
+            ui.label("Enter a name for the recording: ")
+            ui.input(
+                label="Name",
+                placeholder="test",
+                autocomplete=options,
+                on_change=lambda e: result.set_text(e),
+            )
+            result = ui.label()
+
+            with ui.row():
+                ui.button("Close", on_click=dialog.close)
+                ui.button("Enter", on_click=lambda: on_record())
+
+            dialog.open()
+
+    def terminate():
+        global PROC
+        PROC.send_signal(signal.SIGINT)
+
+        # wait for process to terminate
+        PROC.wait()
+        ui.notify("Stopped recording")
+        record_button._props["name"] = "radio_button_checked"
+        record_button.on("click", lambda: recording_prompt(options, record_button))
+
+        PROC = None
+
+    def begin():
+        name = result.text.__getattribute__("value")
+
+        ui.notify(
+            f"Recording {name}... Press CTRL + C in terminal window to cancel",
+        )
+        PROC = Popen(
+            "python3 -m openadapt.record " + name,
+            shell=True,
+        )
+        record_button._props["name"] = "stop"
+        record_button.on("click", lambda: terminate())
+        record_button.update()
+        return PROC
+
+    def on_record():
+        global PROC
+        dialog.close()
+        PROC = begin()