Docs Release 2024-04-001

CHANGELOG.md

@@ -11,6 +11,14 @@ Changelogs for previous years can be found [here](#changelog-archives).
 * [2022](./changelogs/2022-archive.md)
 * [2023](./changelogs/2023-archive.md)
 
+## 2024-04-02 | 08 APRIL 2024
+
+`BUG FIXES`
+
+- Secrets were being filtered by environment
+- Variables weren't updating in the deployments dialogue
+- External topics weren't being recognised by new deployments
+
 ## 2024-03-02-topic-details-hf | 13 MARCH 2024
 
 `IMPROVEMENTS`
@@ -195,7 +203,7 @@ Changelogs for previous years can be found [here](#changelog-archives).
     - Available the first version of the Quix CLI
     - Try it out from here: https://github.com/quixio/quix-cli
 - Permission system (CLI only)
-    - New permission system allowing granular permission using roles such as Admin, Manager, Editor and Viewer on platform resources like Environments and Projects. More information here: https://quix.io/docs/get-started/cli.html#setting-permissions
+    - New permission system allowing granular permission using roles such as Admin, Manager, Editor and Viewer on platform resources like Environments and Projects. More information here: https://quix.io/docs/kb/cli.html#setting-permissions
 - Added support for Git-Submodules on our Git integration
 - Private Code Samples support for BYOC clusters
 - Project settings refactor

docs/apis/portal-api/overview.md

@@ -1,9 +1,9 @@
 ---
 title: Portal API
-description: The Portal API gives access to the Portal interface, enabling you to programmatically control projects, environments, applications, and deployments.
+description: The Portal API gives access to the Portal interface.
 ---
 
-# Overview
+# Overview - Portal API
 
 The Portal API gives access to the Portal interface, enabling you to programmatically control projects, environments, applications, and deployments.
 

docs/apis/query-api/overview.md

@@ -3,7 +3,7 @@ title: Query API
 description: The Query API enables you to fetch persisted data stored in the Quix platform.
 ---
 
-# Overview
+# Overview - Query API
 
 The Query API enables you to fetch persisted data stored in the Quix platform. You can use it for exploring the platform, prototyping applications, or working with stored data in any language with HTTP capabilities.
 

docs/apis/streaming-reader-api/overview.md

@@ -3,7 +3,7 @@ title: Streaming Reader API
 description: Streaming Reader API supports real-time data streaming over WebSockets.
 ---
 
-# Overview
+# Overview - Streaming Reader API
 
 Quix supports real-time data streaming over WebSockets (or Long Polling depending on client support). 
 

docs/apis/streaming-writer-api/overview.md

@@ -3,7 +3,7 @@ title: Streaming Writer API
 description: Streaming Writer API supports real-time data streaming over HTTP or WebSockets.
 ---
 
-# Overview
+# Overview - Streaming Writer API
 
 The Streaming Writer API enables you to stream data into a Quix topic using HTTP endpoints or Microsoft's [SignalR](https://learn.microsoft.com/en-us/aspnet/signalr/overview/getting-started/introduction-to-signalr){target=_blank} technology. You can use the Streaming Writer API from any language or web client with an HTTP (REST) or WebSockets interface. 
 

docs/create/create-environment.md

@@ -69,7 +69,7 @@ Each of these is described briefly in the following sections.
 
 ### Quix broker
 
-The simplest and most convenient choice is to use Quix-managed Kafka. No installation of Kafka is required, and configuration can be done through the UI if you need to change the sensible default values. Very little knowledge of Kafka is expected, beyond basic familiarity with concepts such as [topics](../get-started/glossary.md#topic).
+The simplest and most convenient choice is to use Quix-managed Kafka. No installation of Kafka is required, and configuration can be done through the UI if you need to change the sensible default values. Very little knowledge of Kafka is expected, beyond basic familiarity with concepts such as [topics](../kb/glossary.md#topic).
 
 There is a small charge for storage for messages persisted in a topic: 
 

docs/deploy/state-management.md

@@ -63,10 +63,7 @@ def count_messages(value: dict, state: State):
 
 load_dotenv()
 
-app = Application.Quix(
-    consumer_group="cpu_load", 
-    auto_create_topics=True,
-)
+app = Application()
 
 topic = app.topic('cpu-load')
 sdf = app.dataframe(topic)

docs/develop/authentication/quix-streams.md

@@ -4,7 +4,6 @@ If you're using Quix Streams for local development, you'll need to set some envi
 
 ```
 Quix__Sdk__Token="sdk-12345"
-Quix__Portal__Api="portal-api.platform.quix.io"
 ```
 
 You can read the documentation on [obtaining your SDK token](./streaming-token.md), also known as the streaming token.

docs/develop/authentication/streaming-token.md

@@ -14,7 +14,7 @@ If you are looking for a bearer token to access the Quix APIs, such as the Porta
 
 The streaming token is primarily used to authenticate the [Quix Streams client library](https://quix.io/docs/quix-streams/introduction.html).
 
-When using it with `QuixStreamingClient`, you no longer need to provide all broker credentials manually, they’ll be acquired when needed and set up automatically.
+When using Quix Streams in Quix Cloud, you no longer need to provide all broker credentials manually, they’ll be acquired when needed and set up automatically.
 
 !!! warning
 
@@ -28,9 +28,9 @@ Having two keys lets you update your services without interruption, as both `Tok
 
 You have two main options regarding how you rotate:
 
-1. The easiest way to rotate comes with some service downtime. This assumes you do not directly set the token for your `QuixStreamingClient`, instead you let Quix take care of it for you by using the default environment variable. In this scenario all you have to do is rotate keys, stop and start all your deployments. Until a service is restarted it’ll try to communicate with Quix using the deactivated token. If you’re using local environments, those need to be updated manually.
+1. The easiest way to rotate comes with some service downtime. This assumes you do not directly set the token for your Quix Streams code, instead you let Quix take care of it for you by using the default environment variable. In this scenario all you have to do is rotate keys, stop and start all your deployments. Until a service is restarted it’ll try to communicate with Quix using the deactivated token. If you’re using local environments, those need to be updated manually.
 
-2. The alternative option is more complicated, but you can achieve no downtime. This requires you to set a new environment variable you control. This should point to the token to be used. Provide the value of this environment variable to `QuixStreamingClient` by passing it as an argument. Once you have that, set the value of this environment variable to `Token 2` and start your services. When you’re sure you replaced the tokens for all services, rotate your keys.
+2. The alternative option is more complicated, but you can achieve no downtime. This requires you to set a new environment variable you control. This should point to the token to be used. Provide the value of this environment variable to your Quix Streams code by passing it as an argument. Once you have that, set the value of this environment variable to `Token 2` and start your services. When you’re sure you replaced the tokens for all services, rotate your keys.
 
 !!! note
 

docs/develop/integrate-data/inbound-webhooks.md

@@ -13,10 +13,7 @@ import json
 import hmac
 import hashlib
 
-app = Application.Quix(
-    consumer_group="sample_webhook_group",
-    auto_create_topics=True,
-)
+app = Application()
 
 serializer = JSONSerializer()
 output_topic = app.topic(os.environ["output"])

docs/develop/integrate-data/polling.md

@@ -7,10 +7,7 @@ from quixstreams import Application
 from quixstreams.models.serializers.quix import JSONSerializer, SerializationContext
 import time, os, requests
 
-app = Application.Quix(
-    consumer_group="sample_consumer_group",
-    auto_create_topics=True,
-)
+app = Application()
 
 serializer = JSONSerializer()
 output_topic = app.topic(os.environ["output"])

docs/develop/integrate-data/quix-streams.md

@@ -21,10 +21,7 @@ def get_cpu_load():
         "timestamp": int(time.time_ns()),
     }
 
-app = Application.Quix(
-    consumer_group="cpu_load", 
-    auto_create_topics=True,
-)
+app = Application()
 
 serializer = JSONSerializer()
 output_topic = app.topic("cpu-load")

docs/develop/integrate-data/read-csv.md

@@ -49,7 +49,7 @@ from quixstreams import Application
 load_dotenv()
 
 # Create an Application to connect to the Quix broker with SDK token
-app = Application.Quix()
+app = Application()
 
 # Define an output topic
 output_topic = app.topic(os.environ["output"])

docs/develop/integrate-data/web-app.md

@@ -18,10 +18,7 @@ from datetime import datetime
 from waitress import serve
 import os, json
 
-app = Application.Quix(
-    consumer_group="sample_consumer_group",
-    auto_create_topics=True,
-)
+app = Application()
 
 serializer = JSONSerializer()
 output_topic = app.topic(os.environ["output"])

docs/develop/process/overview.md

@@ -3,6 +3,6 @@ title: Processing data
 description: This section of the documentation covers processing your data.
 ---
 
-# Overview
+# Overview - Processing data
 
 This section of the documentation covers **processing your data**.

docs/develop/process/timeseries-events.md

@@ -97,5 +97,5 @@ In the above code, the event generating code has been removed for simplicity. No
 
 * [Example code](https://github.com/quixio/tutorial-code/blob/main/generate-events/README.md){target=_blank} - the complete code for the example.
 * [Quix Streams](https://quix.io/docs/quix-streams/introduction.html) - documentation on data formats, publishing, and subscribing to topics.
-* [Quix Tour](../../get-started/quixtour/overview.md) - generates processing based on threshold triggering.
+* [Quix Tour](../../quix-cloud/quixtour/overview.md) - generates processing based on threshold triggering.
 * [Currency alerting](../../tutorials/currency-alerting/currency-alerting.md) - tutorial on generating events based on a threshold.

docs/get-started/build-cli.md

@@ -0,0 +1,92 @@
+# Build a pipeline using Quix CLI
+
+In previous sections of the documentation you explored using Quix Streams. You now continue on your command-line journey by installing the Quix CLI, and then using it to connect with Quix Cloud. You create a simple project on the command line, and sync it with your Quix Cloud pipeline view.
+
+## Step 1: Install Quix CLI
+
+```
+curl -fsSL https://github.com/quixio/quix-cli/raw/main/install.sh | sudo bash
+```
+
+For further details on installation, including instructions for Microsoft Windows, see the [install guide](https://github.com/quixio/quix-cli?tab=readme-ov-file#installation-of-quix-cli){target=_blank}.
+
+## Step 2: Sign up to Quix Cloud for free
+
+Sign up [here](https://portal.platform.quix.io/self-sign-up){target=_blank}.
+
+!!! tip 
+
+    If you are prompted to create a project, you can stop at this point, and proceed to the next step in this tutorial. You will come back to complete the project creation wizard at a later step, because you are going to base your Quix project on a Git project you are yet to create.
+
+## Step 3: Log in using the CLI
+
+```
+quix login
+```
+
+If you're not logged into Cloud, you'll be prompted to log in.
+
+## Step 4: Create a Git repository
+
+Create a Git repo where you can store your files, for example you could use GitHub. Create a repo initialized with a `README.md` file, so it can be cloned more easily. 
+
+## Step 5: Clone your Git repo into your local project directory
+
+For example, if your GitHub repo is named `cli-app`:
+
+```
+git clone <url-to-git>/cli-app
+cd cli-app
+```
+
+## Step 6: Initialize your project as a Quix project
+
+In your Git project directory, enter:
+
+```
+quix local init
+```
+
+This initializes your Quix project with a `quix.yaml` file, which describes your Quix project.
+
+
+## Step 7: Create your application locally
+
+Now create a sample application:
+
+```
+quix local app create starter-transformation
+```
+
+This creates a starter transformation for you. You can explore the files created locally for you. The `main.py` code will look familiar to you if you've tried the [previous sections](./welcome.md) of the documentation.
+
+
+## Step 8: Sync your application
+
+To sync your application, change into the `Starter transformation` directory and enter:
+
+```
+quix local deploy --push --sync
+```
+
+This updates your `quix.yaml` project file, and pushes all changes to your Git repository.
+
+## Step 9: In Quix Cloud create a project
+
+In this step you create a project in Quix Cloud from your Git repository.
+
+1. Return to Quix Cloud.
+2. Select `Quix advanced configuration` to continue creation of your project.
+3. Select your Git provider.
+4. Link the project to your Git repository using the guide provided for your chosen Git provider.
+4. Sync Quix Cloud to your project by clicking the `Sync environment` button.
+
+## Step 10: See your pipeline running
+
+Go to pipeline view, and see your pipeline running, with your Starter transformation.
+
+![Pipeline running](../images/starter-transform.png)
+
+## Next steps
+
+* [Read the Quix CLI documentation](../kb/cli.md).

docs/get-started/consume.md

@@ -0,0 +1,64 @@
+# Consume data
+
+You'll now write some code that simply consumes data from a topic and prints it out.
+
+## Step 1: Write your consumer code
+
+Write your consumer code:
+
+``` python
+from quixstreams import Application
+from datetime import timedelta
+import json
+
+# connect to your local Kafka broker
+app = Application(
+    broker_address="localhost:9092",
+    consumer_group="consume-v1",
+    auto_offset_reset="earliest",
+)
+
+# configure the input topic to subscribe to (you'll read data from this topic)
+input_topic = app.topic("cpu-load")
+
+# consume (read) messages from the input topic
+sdf = app.dataframe(topic=input_topic)
+
+# print every row
+sdf = sdf.update(lambda row: print(json.dumps(row)))
+
+if __name__ == "__main__":
+    # run the application and process all inbound messages using the sdf pipeline
+    app.run(sdf)
+```
+
+Save this in a file names `consumer.py`.
+
+## Step 2: Run your consumer
+
+``` python
+python3 consumer.py
+```
+
+You are now subscribing to data on the `cpu-load` topic.
+
+Each message received is printed out as JSON:
+
+``` json
+{
+    "cpu_load": 5.7,
+    "memory": {
+        "total": 0,
+        "used": 0,
+        "free": 0,
+        "percent": 0,
+        "sin": 90937131008,
+        "sout": 483672064
+    },
+    "timestamp": 1712238254512946000
+}
+```
+
+## Next step
+
+* [Process data](./process.md) - process streaming data in a Kafka topic in real time.