Merge branch 'main' into publishing-with-pypi

.github/ISSUE_TEMPLATE/bug_form.yml

@@ -0,0 +1,29 @@
+name: Bug Report
+description: File a bug report
+title: "[Bug]: "
+labels: bug
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for taking the time to fill out this bug report!
+  - type: textarea
+    id: describe-bug
+    attributes:
+      label: Describe the bug
+      description: What behaviour was expected and what actually happened? Include screenshots and log output where useful.
+      value: I expected ... but what actually happened was ...
+    validations:
+      required: true
+  - type: textarea
+    id: steps
+    attributes:
+      label: To Reproduce
+      description: What Operating System did you use and steps would we take to reproduce the behaviour?
+      value: "I use [macOS/Microsoft Windows].
+        Steps: 
+        1. Go to '...' 
+        2. Click on '....' 
+        3. Scroll down to '....' "
+    validations:
+      required: true

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1 @@
+blank_issues_enabled: true

.github/ISSUE_TEMPLATE/new_feature.yml

@@ -0,0 +1,20 @@
+name: "Feature Request"
+description: Submit a proposal for a new OpenAdapt feature
+labels: enhancement
+body:
+  - type: textarea
+    id: feature-request
+    validations:
+      required: true
+    attributes:
+      label: Feature request
+      description: A clear and concise description of the feature proposal. Please provide links to any relevant resources.
+
+  - type: textarea
+    id: motivation
+    validations:
+      required: false
+    attributes:
+      label: Motivation
+      description: Please outline the purpose for the proposal (e.g., is it related to a problem?). Add any relevant links (e.g. GitHub issues).
+      value: I'm always frustrated when [...] so this feature would [...].

.github/pull_request_template.md

@@ -0,0 +1,25 @@
+<!-- Thanks for submitting a pull request! Please provide  information so we can quickly review your pull request. -->
+
+**What kind of change does this PR introduce?**
+
+<!-- E.g. a bugfix, feature, refactoring, etc… -->
+
+**Checklist**
+* [ ] My code follows the style guidelines of this project
+* [ ] I have pefomed a self-review of my code
+* [ ] If applicable, I have added tests to prove my fix is functional/effective
+* [ ] I have linted my code locally prior to submission
+* [ ] I have commented my code, particularly in hard-to-understand areas
+* [ ] I have made corresponding changes to the documentation (e.g. README.md, requirements.txt)
+* [ ] New and existing unit tests pass locally with my changes
+
+**Summary**
+
+<!-- Explain the **motivation** for making this change. What existing problem does the pull request solve? Try to link to an open issue. -->
+
+**How can your code be run and tested?**
+
+<!-- See the README.md for examples. -->
+
+
+**Other information**

.gitignore

@@ -13,12 +13,14 @@ cache
 
 # db
 *.db
+*.db-journal
 
 # VSCode
 .VSCode
+.vsCode
 
 # Generated performance charts
 performance
 
 # Generated when adding editable dependencies in requirements.txt (-e)
-src
+src

CONTRIBUTING.md

@@ -1,6 +1,25 @@
-# Contributing to Project
+# How to contribute
 
-Welcome to OpenAdapt! We appreciate your interest in contributing. Before getting started, please take a moment to review the guidelines below.
+We would love to implement your contributions to this project! We simply ask that you observe the following guidelines.  
+
+## Code Style
+
+This project follows the [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html).
+
+
+## Creating Issue
+In order to effectively communicate any bugs or request new features, please select the appropriate form. If none of the options suit your needs, you can click on "Open a blank issue" located at the bottom.
+
+## Testing
+[GitHub Actions](https://github.com/MLDSAI/OpenAdapt/actions/new) are automatically run on each pull request to ensure consistent behaviour and style. The Actions are composed of PyTest, [black](https://github.com/psf/black) and [flake8](https://flake8.pycqa.org/en/latest/user/index.html).
+
+You can run these tests on your own computer by downloading the depencencies in requirements.txt and then running pytest in the root directory. 
+
+## Pull Request Format
+
+To speed up the review process, please use the provided pull request template and create a draft pull request to get initial feedback. 
+
+The pull request template includes areas to explain the changes, and a checklist with boxes for code style, testing, and documenttation.
 
 ## Commit Message Format
 
@@ -55,13 +74,13 @@ The subject should be a succinct description of the change. Please follow these
 - Do not capitalize the first letter
 - Do not end with a dot (.)
 
-## Body
+### Body
 
 The body should provide the motivation for the change and contrast it with the previous behavior. Use the imperative, present tense (e.g., "change" not "changed" or "changes").
 
 The body or footer can begin with **BREAKING CHANGE:** followed by a short description to create a major release. For example, a major release would increment the version number from `0.1.0` to `1.1.0`.
 
-## Version Bump
+### Version Bump
 
 We use semantic versioning for our releases. The version number is automatically determined based on the commit types mentioned above. Here's how the version numbers are incremented:
 
@@ -74,4 +93,13 @@ We use semantic versioning for our releases. The version number is automatically
 - **PATCH** version for backward-compatible bug fixes or small changes
   - Increments the patch version number (e.g., 1.0.0 -> 1.0.1)
 
-Thank you for your contributions to the Project!
+## Submitting Changes
+
+1. Fork the current repository
+2. Make a branch to work on, or use the main branch
+3. Push desired changes onto the branch in step 2
+4. Submit a pull request with the branch in step 2 as the head ref and the MLDSAI/OpenAdapt main as the base ref
+     - Note: may need to click "compare across forks"
+5. Update pull request using feedback
+     - this step may not be necessary, or may need to be repeated
+6. Celebrate contributing to OpenAdapt!

README.md

@@ -2,24 +2,28 @@
 
 # OpenAdapt: AI-First Process Automation with Transformers
 
+<img width="1512" alt="image" src="https://github.com/MLDSAI/OpenAdapt/assets/774615/60a85160-7b07-41b0-9295-2d1bfa6a0994">
+
+([Slides](https://docs.google.com/presentation/d/e/2PACX-1vStYEzETEMtzWDGCizJt4L2rdQpoAQin8T2cZkLw6WqNBUtmBhaNm-1BrhvGOVNyFV9UPjyVfa0l6IC/pub?start=false&loop=false&delayms=3000))
+
 Welcome to OpenAdapt! This Python library implements AI-First Process Automation
 with the power of Transformers by:
 
 - Recording screenshots and associated user input
 - Aggregating and visualizing user input and recordings for development
-- Converting screenshots and user input into tok/enized format
+- Converting screenshots and user input into tokenized format
 - Generating synthetic input via transformer model completions
 - Replaying synthetic input to complete tasks
 
 The goal is similar to that of
 [Robotic Process Automation](https://en.wikipedia.org/wiki/Robotic_process_automation),
 except that we use transformers instead of conventional RPA tools.
 
-The approach is similar to [adept.ai](https://adept.ai/), with some key differences:
-1. Instead of requiring the user to prompt the model directly, we prompt it behind the
-scenes by observing the user's activities;
-2. We work with all types of desktop applications, not just web apps;
-3. We're open source!
+The direction is adjacent to [Adept.ai](https://adept.ai/), with some key differences:
+1. OpenAdapt is model agnostic;
+1. OpenAdapt generates prompts automatically (auto-prompted, not user-prompted);
+2. OpenAdapt works with all types of desktop GUIs, including virtualized (e.g. Citrix) and web
+3. OpenAdapt is open source! (license TBD, please see https://github.com/MLDSAI/OpenAdapt/issues/246)
 
 ## Install
 
@@ -31,6 +35,7 @@ source .venv/bin/activate
 pip install wheel
 pip install -r requirements.txt
 pip install -e .
+python -m spacy download en_core_web_trf
 alembic upgrade head
 pytest
 ```
@@ -94,8 +99,6 @@ More ReplayStrategies coming soon! (see [Contributing](#Contributing)).
 
 ## Contributing
 
-### Design
-
 ### Problem Statement
 
 Our goal is to automate the task described and demonstrated in a `Recording`.

alembic/env.py

@@ -7,12 +7,12 @@
 
 # this is the Alembic Config object, which provides
 # access to the values within the .ini file in use.
-config = context.config
+alembic_config = context.config
 
 # Interpret the config file for Python logging.
 # This line sets up loggers basically.
-if config.config_file_name is not None:
-    fileConfig(config.config_file_name)
+if alembic_config .config_file_name is not None:
+    fileConfig(alembic_config .config_file_name)
 
 # add your model's MetaData object here
 # for 'autogenerate' support
@@ -21,6 +21,7 @@
 from openadapt.config import DB_URL
 from openadapt.models import *
 from openadapt.db import Base
+
 target_metadata = Base.metadata
 
 # other values from the config, defined by the needs of env.py,
@@ -46,7 +47,7 @@ def run_migrations_offline() -> None:
     script output.
 
     """
-    #url = config.get_main_option("sqlalchemy.url")
+    # url = config.get_main_option("sqlalchemy.url")
     url = get_url()
     context.configure(
         url=url,
@@ -67,7 +68,7 @@ def run_migrations_online() -> None:
     and associate a connection with the context.
 
     """
-    configuration = config.get_section(config.config_ini_section)
+    configuration = alembic_config .get_section(alembic_config .config_ini_section)
     configuration["sqlalchemy.url"] = get_url()
     connectable = engine_from_config(
         configuration=configuration,

install/install_openadapt.ps1

@@ -14,6 +14,7 @@ $VCRedistInstaller = "vc_redist.x64.exe"
 $VCRedistInstallerLoc = "https://aka.ms/vs/17/release/vc_redist.x64.exe"
 $VCRedistRegPath = "HKLM:\SOFTWARE\Microsoft\VisualStudio\14.0\VC\Runtimes\X64"
 
+# TODO: Add Tesseract OCR installation: https://tesseract-ocr.github.io/tessdoc/Installation.html
 
 # Return true if a command/exe is available
 function CheckCMDExists() {

install/install_openadapt.sh

@@ -1,6 +1,8 @@
 #!/bin/bash
 set -e
 
+# TODO: Add Tesseract OCR installation: https://tesseract-ocr.github.io/tessdoc/Installation.html
+
 # Run a command and ensure it did not fail
 RunAndCheck() {
     res=$($1)

openadapt/config.py

@@ -1,3 +1,14 @@
+"""Script containing configurations for the openadapt application.
+
+Usage:
+
+    from openadapt import config
+    ...
+    config.<setting>
+    ...
+
+"""
+
 import multiprocessing
 import os
 import pathlib
@@ -13,12 +24,64 @@
     "DB_ECHO": False,
     "DB_FNAME": "openadapt.db",
     "OPENAI_API_KEY": "<set your api key in .env>",
-    #"OPENAI_MODEL_NAME": "gpt-4",
+    # "OPENAI_MODEL_NAME": "gpt-4",
     "OPENAI_MODEL_NAME": "gpt-3.5-turbo",
     # may incur significant performance penalty
     "RECORD_READ_ACTIVE_ELEMENT_STATE": False,
     # TODO: remove?
     "REPLAY_STRIP_ELEMENT_STATE": True,
+    # IGNORES WARNINGS (PICKLING, ETC.)
+    "IGNORE_WARNINGS": False,
+    # ACTION EVENT CONFIGURATIONS
+    "ACTION_TEXT_SEP": "-",
+    "ACTION_TEXT_NAME_PREFIX": "<",
+    "ACTION_TEXT_NAME_SUFFIX": ">",
+    # SCRUBBING CONFIGURATIONS
+    "SCRUB_ENABLED": True,
+    "SCRUB_CHAR": "*",
+    "SCRUB_LANGUAGE": "en",
+    # TODO support lists in getenv_fallback
+    "SCRUB_FILL_COLOR": (255, 0, 0),
+    "SCRUB_CONFIG_TRF": {
+        "nlp_engine_name": "spacy",
+        "models": [{"lang_code": "en", "model_name": "en_core_web_trf"}],
+    },
+    "SCRUB_IGNORE_ENTITIES": [
+        # 'US_PASSPORT',
+        # 'US_DRIVER_LICENSE',
+        # 'CRYPTO',
+        # 'UK_NHS',
+        # 'PERSON',
+        # 'CREDIT_CARD',
+        # 'US_BANK_NUMBER',
+        # 'PHONE_NUMBER',
+        # 'US_ITIN',
+        # 'AU_ABN',
+        "DATE_TIME",
+        # 'NRP',
+        # 'SG_NRIC_FIN',
+        # 'AU_ACN',
+        # 'IP_ADDRESS',
+        # 'EMAIL_ADDRESS',
+        "URL",
+        # 'IBAN_CODE',
+        # 'AU_TFN',
+        # 'LOCATION',
+        # 'AU_MEDICARE',
+        # 'US_SSN',
+        # 'MEDICAL_LICENSE'
+    ],
+    "SCRUB_KEYS_HTML": [
+        "text",
+        "canonical_text",
+        "title",
+        "state",
+        "task_description",
+        "key_char",
+        "canonical_key_char",
+        "key_vk",
+        "children",
+    ],
 }
 
 
@@ -38,8 +101,29 @@ def getenv_fallback(var_name):
 ROOT_DIRPATH = pathlib.Path(__file__).parent.parent.resolve()
 DB_FPATH = ROOT_DIRPATH / DB_FNAME
 DB_URL = f"sqlite:///{DB_FPATH}"
+DIRNAME_PERFORMANCE_PLOTS = "performance"
 
 if multiprocessing.current_process().name == "MainProcess":
     for key, val in locals().items():
         if not key.startswith("_") and key.isupper():
             logger.info(f"{key}={val}")
+
+
+def filter_log_messages(data):
+    """
+    This function filters log messages by ignoring any message that contains a specific string.
+
+    Args:
+      data: The input parameter "data" is expected to be data from a loguru logger.
+
+    Returns:
+      a boolean value indicating whether the message in the input data should be ignored or not. If the
+    message contains any of the messages in the `messages_to_ignore` list, the function returns `False`
+    indicating that the message should be ignored. Otherwise, it returns `True` indicating that the
+    message should not be ignored.
+    """
+    # TODO: ultimately, we want to fix the underlying issues, but for now, we can ignore these messages
+    messages_to_ignore = [
+        "Cannot pickle Objective-C objects",
+    ]
+    return not any(msg in data["message"] for msg in messages_to_ignore)