Merge branch 'MLDSAI:main' into feat/sam_mixin

.github/ISSUE_TEMPLATE/bug_form.yml

@@ -0,0 +1,29 @@
+name: Bug Report
+description: File a bug report
+title: "[Bug]: "
+labels: bug
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for taking the time to fill out this bug report!
+  - type: textarea
+    id: describe-bug
+    attributes:
+      label: Describe the bug
+      description: What behaviour was expected and what actually happened? Include screenshots and log output where useful.
+      value: I expected ... but what actually happened was ...
+    validations:
+      required: true
+  - type: textarea
+    id: steps
+    attributes:
+      label: To Reproduce
+      description: What Operating System did you use and steps would we take to reproduce the behaviour?
+      value: "I use [macOS/Microsoft Windows].
+        Steps: 
+        1. Go to '...' 
+        2. Click on '....' 
+        3. Scroll down to '....' "
+    validations:
+      required: true

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1 @@
+blank_issues_enabled: true

.github/ISSUE_TEMPLATE/new_feature.yml

@@ -0,0 +1,20 @@
+name: "Feature Request"
+description: Submit a proposal for a new OpenAdapt feature
+labels: enhancement
+body:
+  - type: textarea
+    id: feature-request
+    validations:
+      required: true
+    attributes:
+      label: Feature request
+      description: A clear and concise description of the feature proposal. Please provide links to any relevant resources.
+
+  - type: textarea
+    id: motivation
+    validations:
+      required: false
+    attributes:
+      label: Motivation
+      description: Please outline the purpose for the proposal (e.g., is it related to a problem?). Add any relevant links (e.g. GitHub issues).
+      value: I'm always frustrated when [...] so this feature would [...].

.github/pull_request_template.md

@@ -0,0 +1,25 @@
+<!-- Thanks for submitting a pull request! Please provide  information so we can quickly review your pull request. -->
+
+**What kind of change does this PR introduce?**
+
+<!-- E.g. a bugfix, feature, refactoring, etc… -->
+
+**Checklist**
+* [ ] My code follows the style guidelines of this project
+* [ ] I have pefomed a self-review of my code
+* [ ] If applicable, I have added tests to prove my fix is functional/effective
+* [ ] I have linted my code locally prior to submission
+* [ ] I have commented my code, particularly in hard-to-understand areas
+* [ ] I have made corresponding changes to the documentation (e.g. README.md, requirements.txt)
+* [ ] New and existing unit tests pass locally with my changes
+
+**Summary**
+
+<!-- Explain the **motivation** for making this change. What existing problem does the pull request solve? Try to link to an open issue. -->
+
+**How can your code be run and tested?**
+
+<!-- See the README.md for examples. -->
+
+
+**Other information**

.gitignore

@@ -23,4 +23,4 @@ cache
 performance
 
 # Generated when adding editable dependencies in requirements.txt (-e)
-src
+src

CONTRIBUTING.md

@@ -0,0 +1,33 @@
+# How to contribute
+
+We would love to implement your contributions to this project! We simply ask that you observe the following guidelines.  
+
+## Code Style
+
+This project follows the [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html).
+
+
+## Creating Issue
+In order to effectively communicate any bugs or request new features, please select the appropriate form. If none of the options suit your needs, you can click on "Open a blank issue" located at the bottom.
+
+## Testing
+[GitHub Actions](https://github.com/MLDSAI/OpenAdapt/actions/new) are automatically run on each pull request to ensure consistent behaviour and style. The Actions are composed of PyTest, [black](https://github.com/psf/black) and [flake8](https://flake8.pycqa.org/en/latest/user/index.html).
+
+You can run these tests on your own computer by downloading the depencencies in requirements.txt and then running pytest in the root directory. 
+
+## Pull Request Format
+
+To speed up the review process, please use the provided pull request template and create a draft pull request to get initial feedback. 
+
+The pull request template includes areas to explain the changes, and a checklist with boxes for code style, testing, and documenttation.
+
+## Submitting Changes
+
+1. Fork the current repository
+2. Make a branch to work on, or use the main branch
+3. Push desired changes onto the branch in step 2
+4. Submit a pull request with the branch in step 2 as the head ref and the MLDSAI/OpenAdapt main as the base ref
+     - Note: may need to click "compare across forks"
+5. Update pull request using feedback
+     - this step may not be necessary, or may need to be repeated
+6. Celebrate contributing to OpenAdapt!

README.md

@@ -2,24 +2,28 @@
 
 # OpenAdapt: AI-First Process Automation with Transformers
 
+<img width="1512" alt="image" src="https://github.com/MLDSAI/OpenAdapt/assets/774615/60a85160-7b07-41b0-9295-2d1bfa6a0994">
+
+([Slides](https://docs.google.com/presentation/d/e/2PACX-1vStYEzETEMtzWDGCizJt4L2rdQpoAQin8T2cZkLw6WqNBUtmBhaNm-1BrhvGOVNyFV9UPjyVfa0l6IC/pub?start=false&loop=false&delayms=3000))
+
 Welcome to OpenAdapt! This Python library implements AI-First Process Automation
 with the power of Transformers by:
 
 - Recording screenshots and associated user input
 - Aggregating and visualizing user input and recordings for development
-- Converting screenshots and user input into tok/enized format
+- Converting screenshots and user input into tokenized format
 - Generating synthetic input via transformer model completions
 - Replaying synthetic input to complete tasks
 
 The goal is similar to that of
 [Robotic Process Automation](https://en.wikipedia.org/wiki/Robotic_process_automation),
 except that we use transformers instead of conventional RPA tools.
 
-The approach is similar to [adept.ai](https://adept.ai/), with some key differences:
-1. Instead of requiring the user to prompt the model directly, we prompt it behind the
-scenes by observing the user's activities;
-2. We work with all types of desktop applications, not just web apps;
-3. We're open source!
+The direction is adjacent to [Adept.ai](https://adept.ai/), with some key differences:
+1. OpenAdapt is model agnostic;
+1. OpenAdapt generates prompts automatically (auto-prompted, not user-prompted);
+2. OpenAdapt works with all types of desktop GUIs, including virtualized (e.g. Citrix) and web
+3. OpenAdapt is open source! (license TBD, please see https://github.com/MLDSAI/OpenAdapt/issues/246)
 
 ## Install
 
@@ -31,6 +35,7 @@ source .venv/bin/activate
 pip install wheel
 pip install -r requirements.txt
 pip install -e .
+python -m spacy download en_core_web_trf
 alembic upgrade head
 pytest
 ```
@@ -94,8 +99,6 @@ More ReplayStrategies coming soon! (see [Contributing](#Contributing)).
 
 ## Contributing
 
-### Design
-
 ### Problem Statement
 
 Our goal is to automate the task described and demonstrated in a `Recording`.

alembic/env.py

@@ -7,12 +7,12 @@
 
 # this is the Alembic Config object, which provides
 # access to the values within the .ini file in use.
-config = context.config
+alembic_config = context.config
 
 # Interpret the config file for Python logging.
 # This line sets up loggers basically.
-if config.config_file_name is not None:
-    fileConfig(config.config_file_name)
+if alembic_config .config_file_name is not None:
+    fileConfig(alembic_config .config_file_name)
 
 # add your model's MetaData object here
 # for 'autogenerate' support
@@ -21,6 +21,7 @@
 from openadapt.config import DB_URL
 from openadapt.models import *
 from openadapt.db import Base
+
 target_metadata = Base.metadata
 
 # other values from the config, defined by the needs of env.py,
@@ -46,7 +47,7 @@ def run_migrations_offline() -> None:
     script output.
 
     """
-    #url = config.get_main_option("sqlalchemy.url")
+    # url = config.get_main_option("sqlalchemy.url")
     url = get_url()
     context.configure(
         url=url,
@@ -67,7 +68,7 @@ def run_migrations_online() -> None:
     and associate a connection with the context.
 
     """
-    configuration = config.get_section(config.config_ini_section)
+    configuration = alembic_config .get_section(alembic_config .config_ini_section)
     configuration["sqlalchemy.url"] = get_url()
     connectable = engine_from_config(
         configuration=configuration,

install/install_openadapt.ps1

@@ -0,0 +1,178 @@
+# PowerShell script to pull OpenAdapt and install
+
+# Change these if a different version of is required
+$pythonCmd = "python3.10"
+$pythonVerStr = "Python 3.10*"
+$pythonInstaller = "python-3.10.11-amd64.exe"
+$pythonInstallerLoc = "https://www.python.org/ftp/python/3.10.11/python-3.10.11-amd64.exe"
+
+$gitInstaller = "Git-2.40.1-64-bit.exe"
+$gitInstallerLoc = "https://github.com/git-for-windows/git/releases/download/v2.40.1.windows.1/Git-2.40.1-64-bit.exe"
+$gitUninstaller = "C:\Program Files\Git\unins000.exe"
+
+$VCRedistInstaller = "vc_redist.x64.exe"
+$VCRedistInstallerLoc = "https://aka.ms/vs/17/release/vc_redist.x64.exe"
+$VCRedistRegPath = "HKLM:\SOFTWARE\Microsoft\VisualStudio\14.0\VC\Runtimes\X64"
+
+# TODO: Add Tesseract OCR installation: https://tesseract-ocr.github.io/tessdoc/Installation.html
+
+# Return true if a command/exe is available
+function CheckCMDExists() {
+    Param
+    (
+        [Parameter(Mandatory = $true)] [string] $command
+    )
+
+    Get-Command $command -errorvariable getErr -erroraction 'silentlycontinue'
+    If ($getErr -eq $null) {
+        $true;
+    }
+    return $false;
+}
+
+# Get command for python, install python if required version is unavailable
+function GetPythonCMD() {
+    # Use python alias of required version if it exists
+    if (CheckCMDExists($pythonCmd)) {
+        return $pythonCmd
+    }
+
+    # Use python exe if it exists and is the required version
+    $pythonCmd = "python"
+    if (CheckCMDExists($pythonCmd)) {
+        $res = Invoke-Expression "python -V"
+        if ($res -like $pythonVerStr) {
+            return $pythonCmd
+        }
+    }
+
+    # Install required python version
+    Write-Host "Downloading python installer"
+    $ProgressPreference = 'SilentlyContinue'
+    Invoke-WebRequest -Uri $pythonInstallerLoc -OutFile $pythonInstaller
+    $exists = Test-Path -Path $pythonInstaller -PathType Leaf
+    if(!$exists) {
+        Write-Host "Failed to download python installer"
+        Cleanup
+        exit
+    }
+
+    Write-Host "Installing python, click 'Yes' if prompted for permission"
+    Start-Process -FilePath $pythonInstaller -Verb runAs -ArgumentList '/quiet','InstallAllUsers=0','PrependPath=1' -Wait
+
+    #Refresh Path Environment Variable
+    $env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")
+
+    # Make sure python is now available and the right version
+    if (CheckCMDExists($pythonCmd)) {
+        $res = Invoke-Expression "python -V"
+        if ($res -like $pythonVerStr) {
+            Remove-Item $pythonInstaller
+            return $pythonCmd
+        }
+    }
+
+    Write-Host "Error after installing python. Uninstalling, click 'Yes' if prompted for permission"
+    Start-Process -FilePath $pythonInstaller -Verb runAs -ArgumentList '/quiet','/uninstall' -Wait
+    Remove-Item $pythonInstaller
+
+    # Stop OpenAdapt install
+    Cleanup
+    exit
+}
+
+function Cleanup {
+    $exists = Test-Path -Path "..\OpenAdapt"
+    if($exists) {
+        Set-Location ..\
+        Remove-Item -LiteralPath "OpenAdapt" -Force -Recurse
+    }
+}
+
+# Run a command and ensure it did not fail
+function RunAndCheck {
+    Param
+    (
+        [Parameter(Mandatory = $true)] [string] $Command,
+        [Parameter(Mandatory = $true)] [string] $Desc
+    )
+
+    Invoke-Expression $Command
+    if ($LastExitCode) {
+        Write-Host "Failed: $Desc : $LastExitCode"
+        Cleanup
+        exit
+    }
+
+    Write-Host "Success: $Desc"
+}
+
+$gitExists = CheckCMDExists "git"
+if (!$gitExists) {
+    # Install git
+    Write-Host "Downloading git installer"
+    $ProgressPreference = 'SilentlyContinue'
+    Invoke-WebRequest -Uri $gitInstallerLoc -OutFile $gitInstaller
+    $exists = Test-Path -Path $gitInstaller -PathType Leaf
+    if(!$exists) {
+        Write-Host "Failed to download git installer"
+        exit
+    }
+
+    Write-Host "Installing git, click 'Yes' if prompted for permission"
+    Start-Process -FilePath $gitInstaller -Verb runAs -ArgumentList '/VERYSILENT /NORESTART /NOCANCEL /SP- /CLOSEAPPLICATIONS /RESTARTAPPLICATIONS /COMPONENTS="icons,ext\reg\shellhere,assoc,assoc_sh"' -Wait
+    Remove-Item $gitInstaller
+
+    #Refresh Path Environment Variable
+    $env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")
+
+    # Make sure git is now available
+    $gitExists = CheckCMDExists "git"
+    if (!$gitExists) {
+        Write-Host "Error after installing git. Uninstalling, click 'Yes' if prompted for permission"
+        Start-Process -FilePath $gitUninstaller -Verb runAs -ArgumentList '/VERYSILENT','/SUPPRESSMSGBOXES','/NORESTART' -Wait
+        exit
+    }
+}
+
+# Check if Visual C++ Redist is installed
+# Note: Temporarily setting $ErrorActionPreference as -erroraction 'silentlycontinue' doesn't prevent non-terminating error with Get-ItemPropertyValue
+$ErrorActionPreference="SilentlyContinue"
+$vcredistExists = Get-ItemPropertyValue -Path $VCRedistRegPath -erroraction 'silentlycontinue' -Name Installed
+$ErrorActionPreference="Continue"
+
+if (!$vcredistExists) {
+    # Install Visual C++ Redist
+    Write-Host "Downloading Visual C++ Redist"
+    $ProgressPreference = 'SilentlyContinue'
+    Invoke-WebRequest -Uri $VCRedistInstallerLoc -OutFile $VCRedistInstaller
+    $exists = Test-Path -Path $VCRedistInstaller -PathType Leaf
+    if(!$exists) {
+        Write-Host "Failed to download Visual C++ installer"
+        exit
+    }
+
+    Write-Host "Installing Visual C++ Redist, click 'Yes' if prompted for permission"
+    Start-Process -FilePath $VCRedistInstaller -Verb runAs -ArgumentList "/install /q /norestart" -Wait
+    Remove-Item $VCRedistInstaller
+
+    if($LastExitCode) {
+        "Failed to install Visual C++ Redist: $LastExitCode"
+        exit
+    }
+}
+
+RunAndCheck "git clone -q https://github.com/MLDSAI/OpenAdapt.git" "clone git repo"
+
+Set-Location .\OpenAdapt
+
+$python = GetPythonCMD
+
+RunAndCheck "$python -m venv .venv" "create python virtual environment"
+RunAndCheck ".venv\Scripts\Activate.ps1" "enable python virtual environment"
+RunAndCheck "pip install wheel" "pip install wheel"
+RunAndCheck "pip install -r requirements.txt" "pip install -r requirements.txt"
+RunAndCheck "pip install -e ." "pip install -e ."
+RunAndCheck "alembic upgrade head" "alembic upgrade head"
+RunAndCheck "python -m spacy download en_core_web_trf" "python -m spacy download en_core_web_trf"
+RunAndCheck "pytest" "run OpenAdapt tests"