forked from apache/pulsar
-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[feature][doc]PIP-190: Generate 2.10.x/2.9.x/2.8.x docs (apache#17074)
* Generate 2.10.x/2.9.x/2.8.x docs * refresh versions.json
- Loading branch information
1 parent
e67db2a
commit fbd2be6
Showing
519 changed files
with
150,327 additions
and
10 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,56 @@ | ||
--- | ||
slug: / | ||
id: about | ||
title: Welcome to the doc portal! | ||
sidebar_label: "About" | ||
--- | ||
|
||
import BlockLinks from "@site/src/components/BlockLinks"; | ||
import BlockLink from "@site/src/components/BlockLink"; | ||
import { docUrl } from "@site/src/utils/index"; | ||
|
||
|
||
# Welcome to the doc portal! | ||
*** | ||
|
||
This portal holds a variety of support documents to help you work with Pulsar . If you’re a beginner, there are tutorials and explainers to help you understand Pulsar and how it works. | ||
|
||
If you’re an experienced coder, review this page to learn the easiest way to access the specific content you’re looking for. | ||
|
||
## Get Started Now | ||
<BlockLinks> | ||
<BlockLink title="About Pulsar" url="/docs/concepts-overview/" /> | ||
<BlockLink title="Get Started" url="/docs/getting-started-standalone/" /> | ||
<BlockLink title="Install, Deploy, Upgrade" url="/docs/deploy-aws/" /> | ||
<BlockLink title="Pulsar for Developers" url="/docs/develop-tools/" /> | ||
<BlockLink title="How To" url="/docs/functions-develop/" /> | ||
<BlockLink title="References" url="/docs/reference-terminology/" /> | ||
</BlockLinks> | ||
|
||
## Navigation | ||
*** | ||
|
||
There are several ways to get around in the doc portal. The index navigation pane is a table of contents for the entire archive. The archive is divided into sections, like chapters in a book. Click the title of the topic to view it. | ||
|
||
In-context links provide an easy way to immediately reference related topics. Click the underlined term to view the topic. | ||
|
||
Links to related topics can be found at the bottom of each topic page. Click the link to view the topic. | ||
|
||
![Page Linking](/assets/page-linking.png) | ||
|
||
## Continuous Improvement | ||
*** | ||
As you probably know, we are working on a new user experience for our documentation portal that will make learning about and building on top of Apache Pulsar a much better experience. Whether you need overview concepts, how-to procedures, curated guides or quick references, we’re building content to support it. This welcome page is just the first step. We will be providing updates every month. | ||
|
||
## Help Improve These Documents | ||
*** | ||
|
||
You’ll notice an Edit button at the bottom and top of each page. Click it to open a landing page with instructions for requesting changes to posted documents. These are your resources. Participation is not only welcomed – it’s essential! | ||
|
||
## Join the Community! | ||
*** | ||
|
||
The Pulsar community on github is active, passionate, and knowledgeable. Join discussions, voice opinions, suggest features, and dive into the code itself. Find your Pulsar family here at [apache/pulsar](https://github.com/apache/pulsar). | ||
|
||
An equally passionate community can be found in the [Pulsar Slack channel](https://apache-pulsar.slack.com/). You’ll need an invitation to join, but many Github Pulsar community members are Slack members too. Join, hang out, learn, and make some new friends. | ||
|
276 changes: 276 additions & 0 deletions
276
site2/website/versioned_docs/version-2.10.x/adaptors-kafka.md
Large diffs are not rendered by default.
Oops, something went wrong.
91 changes: 91 additions & 0 deletions
91
site2/website/versioned_docs/version-2.10.x/adaptors-spark.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,91 @@ | ||
--- | ||
id: adaptors-spark | ||
title: Pulsar adaptor for Apache Spark | ||
sidebar_label: "Apache Spark" | ||
original_id: adaptors-spark | ||
--- | ||
|
||
## Spark Streaming receiver | ||
The Spark Streaming receiver for Pulsar is a custom receiver that enables Apache [Spark Streaming](https://spark.apache.org/streaming/) to receive raw data from Pulsar. | ||
|
||
An application can receive data in [Resilient Distributed Dataset](https://spark.apache.org/docs/latest/programming-guide.html#resilient-distributed-datasets-rdds) (RDD) format via the Spark Streaming receiver and can process it in a variety of ways. | ||
|
||
### Prerequisites | ||
|
||
To use the receiver, include a dependency for the `pulsar-spark` library in your Java configuration. | ||
|
||
#### Maven | ||
|
||
If you're using Maven, add this to your `pom.xml`: | ||
|
||
```xml | ||
|
||
<!-- in your <properties> block --> | ||
<pulsar.version>@pulsar:version@</pulsar.version> | ||
|
||
<!-- in your <dependencies> block --> | ||
<dependency> | ||
<groupId>org.apache.pulsar</groupId> | ||
<artifactId>pulsar-spark</artifactId> | ||
<version>${pulsar.version}</version> | ||
</dependency> | ||
|
||
``` | ||
|
||
#### Gradle | ||
|
||
If you're using Gradle, add this to your `build.gradle` file: | ||
|
||
```groovy | ||
def pulsarVersion = "@pulsar:version@" | ||
dependencies { | ||
compile group: 'org.apache.pulsar', name: 'pulsar-spark', version: pulsarVersion | ||
} | ||
``` | ||
|
||
### Usage | ||
|
||
Pass an instance of `SparkStreamingPulsarReceiver` to the `receiverStream` method in `JavaStreamingContext`: | ||
|
||
```java | ||
|
||
String serviceUrl = "pulsar://localhost:6650/"; | ||
String topic = "persistent://public/default/test_src"; | ||
String subs = "test_sub"; | ||
|
||
SparkConf sparkConf = new SparkConf().setMaster("local[*]").setAppName("Pulsar Spark Example"); | ||
|
||
JavaStreamingContext jsc = new JavaStreamingContext(sparkConf, Durations.seconds(60)); | ||
|
||
ConsumerConfigurationData<byte[]> pulsarConf = new ConsumerConfigurationData(); | ||
|
||
Set<String> set = new HashSet(); | ||
set.add(topic); | ||
pulsarConf.setTopicNames(set); | ||
pulsarConf.setSubscriptionName(subs); | ||
|
||
SparkStreamingPulsarReceiver pulsarReceiver = new SparkStreamingPulsarReceiver( | ||
serviceUrl, | ||
pulsarConf, | ||
new AuthenticationDisabled()); | ||
|
||
JavaReceiverInputDStream<byte[]> lineDStream = jsc.receiverStream(pulsarReceiver); | ||
|
||
``` | ||
|
||
For a complete example, click [here](https://github.com/apache/pulsar-adapters/blob/master/examples/spark/src/main/java/org/apache/spark/streaming/receiver/example/SparkStreamingPulsarReceiverExample.java). In this example, the number of messages that contain the string "Pulsar" in received messages is counted. | ||
|
||
Note that if needed, other Pulsar authentication classes can be used. For example, in order to use a token during authentication the following parameters for the `SparkStreamingPulsarReceiver` constructor can be set: | ||
|
||
```java | ||
|
||
SparkStreamingPulsarReceiver pulsarReceiver = new SparkStreamingPulsarReceiver( | ||
serviceUrl, | ||
pulsarConf, | ||
new AuthenticationToken("token:<secret-JWT-token>")); | ||
|
||
``` | ||
|
96 changes: 96 additions & 0 deletions
96
site2/website/versioned_docs/version-2.10.x/adaptors-storm.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,96 @@ | ||
--- | ||
id: adaptors-storm | ||
title: Pulsar adaptor for Apache Storm | ||
sidebar_label: "Apache Storm" | ||
original_id: adaptors-storm | ||
--- | ||
|
||
Pulsar Storm is an adaptor for integrating with [Apache Storm](http://storm.apache.org/) topologies. It provides core Storm implementations for sending and receiving data. | ||
|
||
An application can inject data into a Storm topology via a generic Pulsar spout, as well as consume data from a Storm topology via a generic Pulsar bolt. | ||
|
||
## Using the Pulsar Storm Adaptor | ||
|
||
Include dependency for Pulsar Storm Adaptor: | ||
|
||
```xml | ||
|
||
<dependency> | ||
<groupId>org.apache.pulsar</groupId> | ||
<artifactId>pulsar-storm</artifactId> | ||
<version>${pulsar.version}</version> | ||
</dependency> | ||
|
||
``` | ||
|
||
## Pulsar Spout | ||
|
||
The Pulsar Spout allows for the data published on a topic to be consumed by a Storm topology. It emits a Storm tuple based on the message received and the `MessageToValuesMapper` provided by the client. | ||
|
||
The tuples that fail to be processed by the downstream bolts will be re-injected by the spout with an exponential backoff, within a configurable timeout (the default is 60 seconds) or a configurable number of retries, whichever comes first, after which it is acknowledged by the consumer. Here's an example construction of a spout: | ||
|
||
```java | ||
|
||
MessageToValuesMapper messageToValuesMapper = new MessageToValuesMapper() { | ||
|
||
@Override | ||
public Values toValues(Message msg) { | ||
return new Values(new String(msg.getData())); | ||
} | ||
|
||
@Override | ||
public void declareOutputFields(OutputFieldsDeclarer declarer) { | ||
// declare the output fields | ||
declarer.declare(new Fields("string")); | ||
} | ||
}; | ||
|
||
// Configure a Pulsar Spout | ||
PulsarSpoutConfiguration spoutConf = new PulsarSpoutConfiguration(); | ||
spoutConf.setServiceUrl("pulsar://broker.messaging.usw.example.com:6650"); | ||
spoutConf.setTopic("persistent://my-property/usw/my-ns/my-topic1"); | ||
spoutConf.setSubscriptionName("my-subscriber-name1"); | ||
spoutConf.setMessageToValuesMapper(messageToValuesMapper); | ||
|
||
// Create a Pulsar Spout | ||
PulsarSpout spout = new PulsarSpout(spoutConf); | ||
|
||
``` | ||
|
||
For a complete example, click [here](https://github.com/apache/pulsar-adapters/blob/master/pulsar-storm/src/test/java/org/apache/pulsar/storm/PulsarSpoutTest.java). | ||
|
||
## Pulsar Bolt | ||
|
||
The Pulsar bolt allows data in a Storm topology to be published on a topic. It publishes messages based on the Storm tuple received and the `TupleToMessageMapper` provided by the client. | ||
|
||
A partitioned topic can also be used to publish messages on different topics. In the implementation of the `TupleToMessageMapper`, a "key" will need to be provided in the message which will send the messages with the same key to the same topic. Here's an example bolt: | ||
|
||
```java | ||
|
||
TupleToMessageMapper tupleToMessageMapper = new TupleToMessageMapper() { | ||
|
||
@Override | ||
public TypedMessageBuilder<byte[]> toMessage(TypedMessageBuilder<byte[]> msgBuilder, Tuple tuple) { | ||
String receivedMessage = tuple.getString(0); | ||
// message processing | ||
String processedMsg = receivedMessage + "-processed"; | ||
return msgBuilder.value(processedMsg.getBytes()); | ||
} | ||
|
||
@Override | ||
public void declareOutputFields(OutputFieldsDeclarer declarer) { | ||
// declare the output fields | ||
} | ||
}; | ||
|
||
// Configure a Pulsar Bolt | ||
PulsarBoltConfiguration boltConf = new PulsarBoltConfiguration(); | ||
boltConf.setServiceUrl("pulsar://broker.messaging.usw.example.com:6650"); | ||
boltConf.setTopic("persistent://my-property/usw/my-ns/my-topic2"); | ||
boltConf.setTupleToMessageMapper(tupleToMessageMapper); | ||
|
||
// Create a Pulsar Bolt | ||
PulsarBolt bolt = new PulsarBolt(boltConf); | ||
|
||
``` | ||
|
Oops, something went wrong.