GITBOOK-66: Revamp documentation

README.md

@@ -82,10 +82,10 @@ For more on this pipeline, check our [Quickstart Guide](quickstart-guide.md).
 
 ### Guides
 
-* [Updating Aqueduct](guides/updating-aqueduct.md)
+* [Updating Aqueduct](installation-and-configuration/updating-aqueduct.md)
 * [Debugging a Prediction Pipeline](guides/debugging-a-failed-workflow.md)
-* [Running on Airflow](guides/running-airflow-workflow.md)
-* [Changing the Aqueduct Metadata Store](guides/changing-metadata-store.md)
+* [Running on Airflow](broken-reference)
+* [Changing the Aqueduct Metadata Store](broken-reference)
 * [Porting a Workflow to Aqueduct](guides/porting-a-workflow-to-aqueduct.md)
 
 ### API Reference

SUMMARY.md

@@ -3,14 +3,14 @@
 * [Welcome to Aqueduct](README.md)
 * [The Aqueduct Philosophy](the-aqueduct-philosophy.md)
 * [Quickstart Guide](quickstart-guide.md)
-* [Installation and Deployment](installation-and-deployment.md)
-  * [Installing Aqueduct](installation-and-deployment/installing-aqueduct.md)
-  * [Starting Aqueduct](installation-and-deployment/starting-aqueduct.md)
-  * [Running on AWS EC2](installation-and-deployment/running-on-aws-ec2.md)
-  * [Running with Anaconda](installation-and-deployment/running-with-anaconda.md)
-  * [Running with pyenv](installation-and-deployment/running-with-pyenv.md)
-  * [Deploying with Helm Chart](installation-and-deployment/deploying-with-helm.md)
-  * [Configuring Integrations](installation-and-deployment/configuring-integrations.md)
+* [Installation and Configuration](installation-and-configuration/README.md)
+  * [Installing Aqueduct](installation-and-configuration/installing-aqueduct/README.md)
+    * [Running on AWS EC2](installation-and-configuration/installing-aqueduct/running-on-aws-ec2.md)
+    * [Running on Google Compute Engine](installation-and-configuration/installing-aqueduct/running-on-google-compute-engine.md)
+    * [Running on Kubernetes with Helm](installation-and-configuration/installing-aqueduct/deploying-with-helm.md)
+  * [Configuring Aqueduct](installation-and-configuration/configuring-aqueduct.md)
+  * [Configuring your Python environment](installation-and-configuration/configuring-your-python-environment.md)
+  * [Updating Aqueduct](installation-and-configuration/updating-aqueduct.md)
 * [Example Workflows](example-workflows/README.md)
   * [Quickstart Tutorial](example-workflows/quickstart-tutorial.md)
   * [Using Parameters](example-workflows/parameters-tutorial.md)
@@ -23,46 +23,50 @@
 * [Workflows](workflows/README.md)
   * [Creating a Workflow](workflows/creating-a-workflow.md)
   * [Workflow Versions](workflows/workflow-versions.md)
-  * [Managing Workflow Schedules](workflows/managing-workflow-schedules.md)
+  * [Scheduling Workflwos](workflows/managing-workflow-schedules.md)
   * [Editing a Workflow](workflows/editing-a-workflow.md)
   * [Deleting a Workflow](workflows/deleting-a-workflow.md)
-  * [Cascading Workflows](workflows/cascading-workflows.md)
 * [Integrations](integrations/README.md)
   * [On-demand resources](integrations/on-demand-resources/README.md)
     * [On-demand AWS EKS Clusters](integrations/on-demand-resources/on-demand-aws-eks-clusters.md)
-  * [Integration Management Guide](integrations/guide.md)
-  * [Aqueduct Demo Database](integrations/aqueduct-demo-integration.md)
-  * [Adding an Integration](integrations/adding-an-integration/README.md)
-    * [Connecting to Google Cloud Storage](integrations/adding-an-integration/connecting-to-google-cloud-storage.md)
-    * [Connecting to AWS S3](integrations/adding-an-integration/connecting-to-aws-s3.md)
-    * [Connecting to Google BigQuery](integrations/adding-an-integration/connecting-to-google-bigquery.md)
-    * [Connecting to AWS Lambda](integrations/adding-an-integration/connecting-to-aws-lambda.md)
-    * [Connecting to Kubernetes](integrations/adding-an-integration/connecting-to-k8s-cluster.md)
-    * [Connecting to conda](integrations/adding-an-integration/connecting-to-conda.md)
-    * [Connecting to Databricks on AWS](integrations/adding-an-integration/connecting-to-databricks.md)
-  * [Using Integrations](integrations/using-integrations/README.md)
-    * [Compute Integrations](integrations/using-integrations/compute-integrations.md)
-    * [Relational Databases](integrations/using-integrations/relational-databases.md)
-    * [AWS S3](integrations/using-integrations/aws-s3.md)
-    * [Databricks](integrations/using-integrations/databricks.md)
-    * [Spark on AWS EMR](integrations/using-integrations/spark-emr.md)
-  * [Viewing Data in Aqueduct](integrations/viewing-data-in-an-integration.md)
-* [Notifications](notifications/README.md)
-  * [Email](notifications/connecting-to-email.md)
-  * [Slack](notifications/connecting-to-slack.md)
+  * [Data Systems](integrations/data-systems/README.md)
+    * [SQL Systems](integrations/data-systems/sql-systems/README.md)
+      * [Snowflake](integrations/data-systems/sql-systems/snowflake.md)
+      * [Postgres](integrations/data-systems/sql-systems/postgres.md)
+      * [MySQL](integrations/data-systems/sql-systems/mysql.md)
+      * [MariaDB](integrations/data-systems/sql-systems/mariadb.md)
+      * [SQLite](integrations/data-systems/sql-systems/sqlite.md)
+      * [AWS Athena](integrations/data-systems/sql-systems/aws-athena.md)
+      * [AWS Redshift](integrations/data-systems/sql-systems/aws-redshift.md)
+      * [Google BigQuery](integrations/data-systems/sql-systems/google-bigquery.md)
+    * [Non-SQL Systems](integrations/data-systems/non-sql-systems/README.md)
+      * [AWS S3](integrations/data-systems/non-sql-systems/aws-s3.md)
+      * [Google Cloud Storage](integrations/data-systems/non-sql-systems/google-cloud-storage.md)
+      * [MongoDB](integrations/data-systems/non-sql-systems/mongodb.md)
+    * [Aqueduct Demo Database](integrations/data-systems/aqueduct-demo-integration.md)
+  * [Compute Systems](integrations/compute-systems/README.md)
+    * [On-Demand Resources](integrations/compute-systems/on-demand-resources/README.md)
+      * [On-Demand AWS EKS Clusters](integrations/compute-systems/on-demand-resources/on-demand-aws-eks-clusters.md)
+    * [Aqueduct](integrations/compute-systems/aqueduct.md)
+    * [Airflow](integrations/compute-systems/airflow.md)
+    * [Conda](integrations/compute-systems/conda.md)
+    * [Databricks](integrations/compute-systems/databricks.md)
+    * [Kubernetes](integrations/compute-systems/kubernetes.md)
+    * [AWS Lambda](integrations/compute-systems/aws-lambda.md)
+    * [Apache Spark on AWS EMR](integrations/compute-systems/apache-spark-on-aws-emr.md)
+  * [Managing Integrations](integrations/guide.md)
 * [Operators](operators.md)
   * [Creating a Python Operator](operators/creating-a-python-operator.md)
-  * [Specifying a requirements.txt](operators/specifying-a-requirements.txt.md)
-  * [File Dependencies in Python](operators/file-dependencies-in-python.md)
-  * [Improve Dependencies and Python Version Management Using Conda](operators/using-conda.md)
-  * [Lazy vs. Eager Execution](operators/lazy-vs.-eager-execution.md)
+  * [requirements.txt](operators/specifying-a-requirements.txt.md)
+  * [File Dependencies](operators/file-dependencies-in-python.md)
   * [Configuring GPUs, CPUs, and Memory](operators/configuring-resource-constraints.md)
 * [Artifacts](artifacts.md)
-  * [Saving an Artifact](artifacts/saving-an-artifact.md)
 * [Parameters](parameters.md)
+* [Notifications](notifications/README.md)
+  * [Email](notifications/connecting-to-email.md)
+  * [Slack](notifications/connecting-to-slack.md)
 * [Metrics & Checks](metrics-and-checks.md)
-  * [Metrics: Measuring your Predictions](metrics-and-checks/metrics-measuring-your-predictions/README.md)
-    * [System Metrics](metrics-and-checks/metrics-measuring-your-predictions/system-metrics.md)
+  * [Metrics: Measuring your Predictions](metrics-and-checks/metrics-measuring-your-predictions.md)
   * [Checks: Ensuring Correctness](metrics-and-checks/checks-ensuring-correctness.md)
   * [Preset Metrics & Checks](metrics-and-checks/preset-metrics-and-checks.md)
 * [API Reference](api-reference/README.md)
@@ -94,10 +98,6 @@
       * [aqueduct.aqueduct\_client](api-reference/sdk-reference/package-aqueduct/aqueduct.aqueduct\_client.md)
       * [aqueduct.schedule](api-reference/sdk-reference/package-aqueduct/aqueduct.schedule.md)
 * [Guides](guides/README.md)
-  * [Updating Aqueduct](guides/updating-aqueduct.md)
   * [Porting a workflow to Aqueduct](guides/porting-a-workflow-to-aqueduct.md)
-  * [Changing the Aqueduct Metadata Store](guides/changing-metadata-store.md)
   * [Debugging a Failed Workflow](guides/debugging-a-failed-workflow.md)
-  * [Running a Workflow on Airflow](guides/running-airflow-workflow.md)
-* [Usage Stats Collection](usage.md)
 * [FAQs](faqs.md)

api-reference/aqueduct-cli.md

@@ -3,6 +3,7 @@
 This page provide a detailed walkthrough of the Aqueduct CLI. 
 
 * ``[`start`](aqueduct-cli.md#undefined)``
+* ``[`version`](aqueduct-cli.md#version)``
 * ``[`install`](aqueduct-cli.md#install)``
 * ``[`apikey`](aqueduct-cli.md#apikey)``
 * ``[`clear`](aqueduct-cli.md#undefined)``
@@ -19,9 +20,13 @@ This page provide a detailed walkthrough of the Aqueduct CLI. 
 
 `aqueduct start` is a blocking command, meaning that it will start the server in a process that takes over your terminal window.
 
+#### version
+
+`aqueduct version` displays which version of Aqueduct you are running locally. For more details on updating Aqueduct, see [updating-aqueduct.md](../installation-and-configuration/updating-aqueduct.md "mention").
+
 #### install
 
-`aqueduct install <connector>` installs the dependencies required for `<connector>` on your machine. In most cases, these are `pip` packages on a system-by-system basis, but certain connectors (MySQL & Microsoft SQL Server) require special configuration -- see [configuring-integrations.md](../installation-and-deployment/configuring-integrations.md "mention") for more details.
+`aqueduct install <connector>` installs the dependencies required for `<connector>` on your machine. In most cases, these are `pip` packages on a system-by-system basis, but certain connectors (MySQL & Microsoft SQL Server) require special configuration -- see [Broken link](broken-reference "mention") for more details.
 
 #### apikey
 
@@ -30,3 +35,28 @@ This page provide a detailed walkthrough of the Aqueduct CLI.&#x20;
 #### clear
 
 `aqueduct clear` removes the local Aqueduct installation by deleting the server, UI code, and any associated metadata.
+
+#### storage
+
+`aqueduct storage` allows you to configure Aqueduct's artifact storage. Using Aqueduct storage, you can either set your artifact storage to be the local filesystem or an S3 bucket.&#x20;
+
+To set the artifact store to a path on the local file system, use `aqueduct storage --use local`:
+
+```bash
+AQUEDUCT_STORAGE_PATH=$HOME/.aqueduct
+aqueduct storage --use local --path $AQUEDUCT_STORAGE_PATH
+```
+
+To set the artifact store to an S3 bucket, use `aqueduct storage --use s3`:
+
+```bash
+S3_BUCKET=s3://aqueduct_bucket/path/to/storage
+AWS_CREDENTIALS=$HOME/.aws/credentials
+AWS_CREDENTIALS_PROFILE=default
+
+aqueduct storage \
+    --use s3 \
+    --path $S3_BUCKET \
+    --credentials $AWS_CREDENTIALS \
+    --profile $AWS_CREDENTIALS_PROFILE
+```

artifacts.md

@@ -4,7 +4,7 @@ description: Encapsulating data in Aqueduct
 
 # Artifacts
 
-_Artifacts_ are thin wrappers around data objects in Aqueduct. Wrapping data in an Artifact allows Aqueduct to track that artifact across workflow runs help you understand how your data is changing over time. 
+_Artifacts_ are thin wrappers around data objects in Aqueduct. Wrapping data in an Artifact allows Aqueduct to track that artifact across workflow runs help you understand how your data is changing over time.
 
 We support the following data types:
 
@@ -18,15 +18,23 @@ We support the following data types:
 * `bytes`
 * `image`: Corresponds to PIL.Image type.
 * `picklable`: Any object that can be pickled. This is the catch-all type used for all types that are not included above.
-* `untyped`: A special placeholder Artifact used to type the [lazily-executed operators](../operators/lazy-vs-eager-execution.md))
+* `untyped`: A special placeholder Artifact used to type [lazily-executed operators](operators/lazy-vs-eager-execution.md) — **you should never use the type yourself**. 
 
 These types are inferred and tracked by the system, no user input is required! Aqueduct uses this type information to provide useful type enforcement like:
+
 * Enforcing artifact type consistency across workflow runs. An artifact with type `t` will continue to have type `t` in future runs.
 * Catching errors earlier on, such as writing non-relational data to a relational integration (eg. SQL).
 
+An Artifact can be saved to a storage system by calling `.save` on the integration object in the Aqueduct SDK:
+
+```python
+import aqueduct as aq
+client = aq.Client()
+
+db = client.integration('aqueduct_demo')
+wines = db.sql('SELECT * FROM wines;')
 
-There are three operations that can be done on artifacts:
+db.save(wines, 'wines_2', 'replace')
+```
 
-* [Saving an Artifact](artifacts/saving-an-artifact.md)
-* Transforming an Artifact -- this is done by passing an Artifact as an argument to an [Operator](operators.md)
-* Attaching a [Metric or a Check](metrics-and-checks.md) to an artifact -- again, this can be done by passing an Artifact as an argument to the relevant Metric or Check
+The specific requirements for each data system will vary based on what data types they accept and what metadata they need. To learn more, visit the documentation page for the data system you're using.