Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RAG] Embed workspace files #102

Open
wants to merge 18 commits into
base: main
Choose a base branch
from

Conversation

anetaj
Copy link
Contributor

@anetaj anetaj commented Oct 15, 2024

  • Workspace files are now embedded and saved to a vector store (if configured).
  • The algorithm respects workspace ignored files (they won't be embedded).
  • Files only get re-embedded if new or modified since the last embedding.
  • For now, embedding only happens in the background when the extension is activated. You have to restart the extension after adding API keys.
  • For now, only a few file types are supported. Unsupported files will be ignored.

Example docker-compose for OpenSearch:

# Reference:
#   https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/#sample-docker-composeyml
version: "3"
services:
  opensearch:
    image: opensearchproject/opensearch:2.6.0
    container_name: opensearch
    environment:
      - cluster.name=opensearch
      - node.name=opensearch
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - "DISABLE_INSTALL_DEMO_CONFIG=true"
      - "DISABLE_SECURITY_PLUGIN=true"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - opensearch_data:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600
    networks:
      - opensearch
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest # Make sure the version of opensearch-dashboards matches the version of opensearch installed on other nodes
    container_name: opensearch-dashboards
    ports:
      - 5601:5601 # Map host port 5601 to container port 5601
    expose:
      - "5601" # Expose port 5601 for web access to OpenSearch Dashboards
    environment:
      OPENSEARCH_HOSTS: '["http://opensearch:9200"]' # Define the OpenSearch nodes that OpenSearch Dashboards will query
      DISABLE_SECURITY_DASHBOARDS_PLUGIN: "true" # disables security dashboards plugin in OpenSearch Dashboards
    networks:
      - opensearch
networks:
  opensearch:
volumes:
  opensearch_data:

@andrewpareles andrewpareles added the new feature New feature or request label Oct 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new feature New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants