apache#17541)

* [ci] Reduce runners load for pulls that affects only doc or cpp changes

* add final jobs to mark required workflows as completed even if they are skipped

* test parametrized job id

* fix latest step

- to reduce load on CI

…che#17551)

* [ci] Skip run flaky tests suite for cpp-only changes

* also ignore doc changes

* use changed_files

…le after 10 minutes (apache#17401)

* [fix][broker] Leader election cache should not invalidate entries

- cache expiration leads to problems and it is better to serve a stale entry than to ever expire the
  entry
- MetadataStore didn't support passing cache configuration for a MetadataCache, it
  was necessary to modify the interface to support that.

* Add test case

* Prevent stale values in leader election cache

Co-authored-by: Enrico Olivelli <[email protected]>

…ache#17553)

* [improve][misc] Pull request template

1. Highlight to retain the documentation section so that committers can correct it if necessary simply with UI.
2. Few people edit the "affect one of the following parts". Change it to checkboxes for easily applying.
3. Comment out some placeholder.

* Update .github/PULL_REQUEST_TEMPLATE.md

* remove `This example uses Pulsar x.x.x`

* Update site2/docs/tiered-storage-aliyun.md

Co-authored-by: momo-jun <[email protected]>

* Update site2/docs/tiered-storage-aws.md

Co-authored-by: momo-jun <[email protected]>

* Update site2/docs/tiered-storage-filesystem.md

Co-authored-by: momo-jun <[email protected]>

* Update site2/docs/tiered-storage-azure.md

Co-authored-by: momo-jun <[email protected]>

* Update site2/docs/tiered-storage-gcs.md

Co-authored-by: momo-jun <[email protected]>

Co-authored-by: momo-jun <[email protected]>

* WIP:move cluster-level failover

* update

* Update site2/docs/client-libraries-java.md

Co-authored-by: momo-jun <[email protected]>

* Update site2/docs/concepts-cluster-level-failover.md

Co-authored-by: momo-jun <[email protected]>

* improve doc

Co-authored-by: momo-jun <[email protected]>

…cos) (apache#17564)

…d topic with active consumer leaves topic metadata inconsistent. (apache#17308)" (apache#17566)

This reverts commit 9529850.

…ger (apache#17569)

Master Issue: apache#16912

Remove `WhiteBox` usage in `pulsar-client` and `pulsar-client-tools`. We then have only usages in `pulsar-broker`, `pulsar-io-elastic-search`, and `testmocks`.

cc @nicoloboschi @lhotari @shoothzj @eolivelli 

### Documentation

<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->

- [ ] `doc-required` 
(Your PR needs to update docs and you will update later)

- [x] `doc-not-needed` 
(Please explain why)

- [ ] `doc` 
(Your PR contains doc changes)

- [ ] `doc-complete`
(Docs have been already added)

…s` greater than `maxUnackedMessages` (apache#17483)

…ces (apache#17421)

…7198)

…sk space issue in saving docker image (apache#17584)

* [fix][CI] Don't run "Pulsar CI checks completed" too early

- "if: always()" caused the job to run also when something failed

* Clean up disk space in "Build Pulsar docker image"

* Show disk space before and after cleaning

* Fix deleting files

)

Master Issue: apache#16912 

### Documentation

<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->

- [ ] `doc-required` 
(Your PR needs to update docs and you will update later)

- [x] `doc-not-needed` 
(Please explain why)

- [ ] `doc` 
(Your PR contains doc changes)

- [ ] `doc-complete`
(Docs have been already added)

…17546)

… too early (apache#17586)

…template (apache#17594)

1. We don't run Pulsar CI for the master branch, so although we can add a badge for it [![ci](https://github.com/apache/pulsar/actions/workflows/pulsar-ci.yaml/badge.svg)](https://github.com/apache/pulsar/actions/workflows/pulsar-ci.yaml), it's inaccurate.
2. Docker build workflow name has been changed. Also, the workflow doesn't reflect a useful status. Replace it with docker pull statistics.

…ache#17591)

- by default Pulsar won't flush log statements to disk after each log line. This is problematic while investigating some issues since the broker might crash without the last log lines.

apache#17562)

…ion number (apache#17349)

…ll (apache#16946)

…pache#17559)

…pache#17597)

* Fix scheduledExecutorProvider not shutdown

* Fix check null

Co-authored-by: Lari Hotari <[email protected]>

…e#17614)

* [fix][cpp] Fix libcurl build failure when building deb package

### Motivation

See apache#17538 (comment)

The root cause is when libcurl is built from source, it uses
[`ld`](https://linux.die.net/man/1/ld) to check if the `libcurl.so`
links to the correct dependencies in runtime. In Linux, a dynamic
library links to the paths of `/etc/ld.so.conf` by default. However,
different from other images like `centos:7` and `alpine`, this file
includes `/usr/lib/x86_64-linux-gnu` in `debian:9`.

```bash
$ cat /etc/ld.so.conf
include /etc/ld.so.conf.d/*.conf

$ cat /etc/ld.so.conf.d/*.conf
/usr/lib/x86_64-linux-gnu/libfakeroot
# libc default configuration
/usr/local/lib
# Multiarch support
/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu
```

When libcurl is compiled, it links to the install path of libopenssl via
the `--with-ssl` option:

https://github.com/apache/pulsar/blob/1f50366768e76f1a5f7084f7972167f989ddd0af/pulsar-client-cpp/pkg/deb/Dockerfile#L85

i.e. `/usr/local/ssl/lib/libopenssl.so`.

However, after the `libcurl.so` is built, it links to
`/usr/lib/x86_64-linux-gnu/libssl.so.1.1`, see the following output:

```bash
$ ldd /usr/local/lib/libcurl.so
/usr/local/lib/libcurl.so: /usr/lib/x86_64-linux-gnu/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by /usr/local/lib/libcurl.so)
```

In `debian:9`, the default libopenssl version is 1.1.0:

```bash
$ strings /usr/lib/x86_64-linux-gnu/libssl.so.1.1 | grep OpenSSL
OpenSSL 1.1.0l  10 Sep 2019
```

The ABI compatibility is not guaranteed between 1.1.0l and 1.1.1n, see
https://abi-laboratory.pro/index.php?view=timeline&l=openssl.

### Modifications

Set the `LD_LIBRARY_PATH` to `/usr/local/ssl/lib` in the Dockerfile to
build deb package.

Actually it's not required for other images like `centos:7`, but it's
also good to add the `LD_LIBRARY_PATH` to them. So this PR set the
environment variable to them as well.

* Fix workflow so that cpp-tests isn't skipped

* Revisit workflow fix to cover doc only workflows too

* Fix multi-line condition

Co-authored-by: Lari Hotari <[email protected]>

…on (apache#17374)

Co-authored-by: fanjianye <[email protected]>

…ons IT (apache#17372)

…che#17602)

…s and ignores memory usage in getMaxResourceUsage() (apache#17598)

…xception (apache#17512)

* [fix][tiered-storage] Don't cleanup data when offload met BadVersion
---

*Motivation*

There have two ways that will cause the offload data cleanup. One is met
offload conflict exception, and another is completeLedgerInfoForOffloaded
reaches max retry time and throws zookeeper exceptions.

We retry the zookeeper operation on connection loss exception. We should
be careful about this exception, because we may loss data if the metadata
update successfully.

When a MetaStore exception happens, we can not make sure the metadata update is
failed or not. Because we have a retry on the connection loss, it is
possible to get a BadVersion or other exception after retrying.

So we don't clean up the data if this happens.

*Modification*

- don't delete data if has meta store exception

* log error when skip deleting

* improve logs

* improve doc

* Update site2/docs/io-quickstart.md

Co-authored-by: momo-jun <[email protected]>

* Update site2/docs/io-quickstart.md

Co-authored-by: Anonymitaet <[email protected]>

Co-authored-by: momo-jun <[email protected]>
Co-authored-by: Anonymitaet <[email protected]>

### Motivation

Expose Chunking configuration C API

### Modifications

* Expose `chunking_enabled`, `max_pending_chunked_message `, `auto_ack_oldest_chunked_message_on_queue_full ` to C API.
* Add unit tests for C API of producer configuration and consumer configuration.

…che#17461)

---

*Motivation*

Update the jsr version in the presto license file

…terface&config&proto -part 1 (apache#17344)

…pache#16758)

Master Issue: apache#15370

### Motivation

see apache#15370

### Modifications

I will complete proposal apache#15370 with these pull requests( *current pull request is a part of step 7-1* ): 

1. Write the batch transaction log handler: `TxnLogBufferedWriter`
2. Configuration changes and protocol changes.
3. Transaction log store enables the batch feature.
4. Pending ack log store enables the batch feature.
5. Supports dynamic configuration.
6. Append admin API for transaction batch log and docs( admin and configuration doc ).
   GET /admin/v3/transactions/coordinatorStats
   GET /admin/v3/transactions/pendingAckStats/:tenant/:namespace:/:topic:/:subName
7. Append metrics support for transaction batch log.
  7-1. Metrics of Txn Buffered Writer.
  7-2. `TransactionLog` and `PendingAckStore` enables the Metrics of Txn Buffered Writer

----

### The desired effect

`TransactionLog` should create `TxnLogBufferedWriter` with params: 

```JSON
{
  "metricsPrefix": "pulsar_txn_tc",
  "labelNames": "coordinatorId",
  "labelValues": "1"
}
```

The  metrics output of `TransactionLog` will like this: 

```
# A metrics for how many batches were triggered due to threshold "batchedWriteMaxRecords".
# TYPE pulsar_txn_tc_batched_log_batched_log_triggering_count_by_records Counter
pulsar_txn_tc_batched_log_batched_log_triggering_count_by_records{coordinatorId="1"} 15
...
...
...
# pulsar_txn_tc_batched_log_records_count_per_entry A metrics for how many records in per batch written by the component[pulsar_txn_tc] per batch.
# TYPE pulsar_txn_tc_batched_log_records_count_per_entry Histogram
pulsar_txn_tc_batched_log_records_count_per_entry_bucket{coordinatorId="1", le="10"} 1
pulsar_txn_tc_batched_log_records_count_per_entry_bucket{coordinatorId="1", le="50"} 3
pulsar_txn_tc_batched_log_records_count_per_entry_bucket{coordinatorId="1", le="100"} 5
pulsar_txn_tc_batched_log_records_count_per_entry_bucket{coordinatorId="1", le="500"} 10
pulsar_txn_tc_batched_log_records_count_per_entry_bucket{coordinatorId="1", le="1000"} 10
pulsar_txn_tc_batched_log_records_count_per_entry_bucket{coordinatorId="1", le="+Inf"} 10
pulsar_txn_tc_batched_log_records_count_per_entry_count{coordinatorId="1", le="+Inf"} 10
pulsar_txn_tc_batched_log_records_count_per_entry_sum{coordinatorId="1", le="+Inf"} 5432
```

`PendingAckStore` is the same. But all the PendingackStores will not differentiate the Subscription labels (because there are too many)

----

### Manage the registered collectors ourselves.

To build Metrics Stat, we need to execute these two steps:
1. Create `Collector` and register to `CollectorRegistry`, perhaps the Collector is `Histogram` or `Counter` 
2. Register labels to `Collector` and get `Collector.child`(holds by Metrics Stat). This step can also be omitted because we can execute `collector.labels(labelValues)` to get `Collector.child`. 

In the Transaction log scenario, multiple Transaction Logs share the same `Collector`, and each has its own `Collector.Child`, so when we build metrics stat for each Transaction Log, we call `collector.labels(labelValues)` to get the `Collector.Child`. However, the CollectorRegistry does not provide an API like this:

```java
public Collector getRegistedCollector(String name);
```

and it will throw IllegalArgumentException when we registering collector with the same name more than once, see:


https://github.com/prometheus/client_java/blob/1966186deaaa83aec496d88ff604a90795bad688/simpleclient/src/main/java/io/prometheus/client/CollectorRegistry.java#L49-L65

So we have to manage the registered collectors ourselves.

----

#### Holds the `Collector.child` by each Metrics stat instance

To save the overhead of `collector.labels(labelValues)`, we make each Metrics Stat hold a reference of `Collector.child`, because this method is not light enough:

https://github.com/prometheus/client_java/blob/1966186deaaa83aec496d88ff604a90795bad688/simpleclient/src/main/java/io/prometheus/client/SimpleCollector.java#L63-L80

----

#### Code will be removed in the next PR (7-2)

In the complete design, we should have two implementations like UML blow, one for enabling the batch feature, and another for disabled:

![uml](https://user-images.githubusercontent.com/25195800/182418488-6469a38f-a96c-44e9-8ee6-01273b58b0cd.jpeg)


To reduce later maintenance costs, I'd like to ditch the 'DisabledMetricsStat' and we'll always use the implementation 'MetricsStatimpl' even if the Txn buffer writer disables the batch feature. This constructor without 'param-metricsStats' and these' null checks' will be removed in the next PR. This is compatible only with split PR, making each PR have less code

### Documentation


- [ ] `doc-required` 
  
- [x] `doc-not-needed` 
  
- [ ] `doc` 

- [ ] `doc-complete`

…tatus (apache#17592)

…pache#17619)

* [fix][cli] Pulsar shell: ensure admin client is recycled or disposed

…ion' case (apache#17432)

…per event thread (apache#17620)

…ed (apache#17543)

…min client (apache#17023)

…17450)

Fixes apache#14848

### Motivation

There were several fixes on `ClientTest.testReferenceCount` but it's
still very flaky.

The root cause is even after a `Reader` went out of the scope and
destructed, there was still a `Reader` object existed in the thread of
the event loop. See
https://github.com/apache/pulsar/blob/845daf5cac23a4dda4a209d91c85804a0bcaf28a/pulsar-client-cpp/lib/ReaderImpl.cc#L88

To verify this point, I added some logs and saw:

```
2022-09-14 03:52:28.427 INFO  [140046042864960] Reader:39 | Reader ctor 0x7fffd2a7c110
# ...
2022-09-14 03:52:28.444 INFO  [140046039774976] Reader:42 | Reader ctor 0x7f5f0273d720 ReaderImpl(0x7f5efc00a9d0, 3)
# ...
2022-09-14 03:52:28.445 INFO  [140046042864960] ClientTest:217 | Reference count of the reader: 4
# ...
2022-09-14 03:52:28.445 INFO  [140046042864960] ClientImpl:490 | Closing Pulsar client with 1 producers and 2 consumers
2022-09-14 03:52:28.445 INFO  [140046039774976] Reader:55 | Reader dtor 0x7f5f0273d720 ReaderImpl(0x7f5efc00a9d0, 3)
```

The first `Reader` object 0x7fffd2a7c110 was constructed in main thread
140046042864960. However, it destructed before another `Reader` object
0x0x7f5f0273d720 that was constructed in event loop thread
140046039774976. When the callback passed to `createReaderAsync`
completed the promise, the `createReader` immediately returns, at the
same time the `Reader` object in the callback was still in the scope and
not destructed.

Since `Reader` holds a `shared_ptr<ReaderImpl>` and `ReaderImpl` holds
a `shared_ptr<ConsumerImpl>`, if we check the reference count too
quickly, the reference count of the underlying consumer is still
positive because the `Reader` was not destructed at the moment.

### Modifications

Since we cannot determine the precise destructed time point because that
`Reader` object is in the event loop thread, we have to wait for a
while. This PR adds a `waitUntil` utility function to wait for at most
some time until the condition is met. Then wait until the reference
count becomes 0 after the `Reader` object goes out of scope.

Replace `ASSERT_EQ` with `EXPECT_EQ` to let the test continue if it
failed.

### Verifying this change

Following the steps here to reproduce:
apache#14848 (comment)

The test never failed even with `--gtest_repeat=100`.

…ecution time (apache#17627)

)

…#17548)

link apache#17548
### Motivation
now delayed features and transaction messages cannot be used together.
When sending a transaction message with a delayed time and commit this transaction, the message will be immediately received by consumers.

Code, eg.
```
    @test
    public void testDelayedTransactionMessages() throws Exception {
        String topic = NAMESPACE1 + "/testDelayedTransactionMessages";

        @cleanup
        Consumer<String> sharedConsumer = pulsarClient.newConsumer(Schema.STRING)
                .topic(topic)
                .subscriptionName("shared-sub")
                .subscriptionType(SubscriptionType.Shared)
                .subscribe();

        @cleanup
        Producer<String> producer = pulsarClient.newProducer(Schema.STRING)
                .topic(topic)
                .enableBatching(false)
                .create();

        Transaction transaction = pulsarClient.newTransaction()
                .withTransactionTimeout(10, TimeUnit.SECONDS).build().get();

        // send delayed messages
        for (int i = 0; i < 10; i++) {
            producer.newMessage(transaction)
                    .value("msg-" + i)
                    .deliverAfter(5, TimeUnit.SECONDS)
                    .sendAsync();
        }

        producer.flush();

        transaction.commit().get();

        Message<String> msg = sharedConsumer.receive(1, TimeUnit.SECONDS);
        // the msg now is not null
        assertNull(msg);
    }
```
This PR will implement clients to send delayed messages with transactions.

### Modifications
make transaction message can be put in `trackDelayedDelivery` to implement client send delayed messages with the transaction.

It is worth noting that the dispatcher sends transaction messages to consumers and should follow the `MaxReadPosition` change—(something about `MaxReadPosition` https://github.com/streamnative/community/blob/master/rfc/rfcs/0003-transaction-buffer-design.md). 

Because of the existence of maxReadPosition, the distribution of transaction messages depends on whether the previous transaction message is completed. This will cause delay time extended, but not shortened

### Verifying this change
add the test

…e failed (apache#17652)

* fix tests

* fix checkstyle

* fix tests

* fix tests

* fix tests

* fix checkstyle

* fix tests

Co-authored-by: gavingaozhangmin <[email protected]>

…ker is closing (apache#17315)

…17603)

…broker (apache#17658)

)

This closes apache#17047.

Upstream patch is apache/maven-shade-plugin#150.

Bump maven-shade-plugin version to 3.4.0 for the upstream fix.

Signed-off-by: tison <[email protected]>

Signed-off-by: tison <[email protected]>

…sts outside of apache/pulsar (apache#17606)

- the if condition was missing for the ssh-access step in pulsar-java-test-image build job

- add additional condition that applies to only pull_request events.
  - this is a better default

…sts (apache#17410)

### Motivation

Currently the operation timeout only works for requests other than
lookup, like SEND and FLOW. However, the lookup requests, which are sent
by `LookupService`, should also apply the operation timeout, see

https://github.com/apache/pulsar/blob/7075a5ce0d4a70f52625ac8c3d0c48894442b72a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PulsarClientImpl.java#L1019-L1025

In addition, no attempts would be retried if lookup failed even due to a
retryable reason. For example, if some of all configured brokers were
down, the C++ client would fail immediately.

Therefore, this PR intends to retry lookup for some certain cases:
- The connection cannot be established, except connection timeout, which
  is controlled by the connection timeout.
- A `ServiceNotReady` error is received, except it's caused by
  `PulsarServerException`, e.g. the listener name is wrong.

Then, apply the operation timeout to avoid infinite retries.

### Modifications

- Add a `ResultRetryable` error code, which should only be used
  internally. Complete the futures with this error in the cases said
  previously.
- Add a `RetryableLookupService` implementation to support retries based
  on the backoff policy. Replace the directly usages of `LookupService`
  implementations with this class in `ClientImpl`.

The following tests are added:
- `ClientTest.testMultiBrokerUrl`: verify when multiple brokers are
  configured, even if one of them is not available, the creation of
  producer or consumer could still succeed.
- `LookupService.testRetry`: verify all lookup methods could be retried.
- `LookupService.testTimeout`: verify all lookup methods could be
  completed with `ResultTimeout` if no brokers are available.

### TODO

In future, we should add lookup timeout instead of operation timeout for
lookup requests and separate lookup connection pool, see PIP-91.

* Handle ResultRetryable in handleDisconnection

…rvice (apache#17632)

…he#17648)

…#17275)

### Motivation

There are some wrong steps in the release process and the "Release
workflow" section is just a list and hard to verify directly.

### Modifications

- Remove the "Move master branch to next version" and "Write release
  notes" sections because they should be done in the last few steps and
  they have already been documented.
- Add the steps to build Docker images when running the vote.
- Skip the release of Helm Chart for patch releases.
- Update the step to publish Python clients for macOS because we won't
  need the VirtualBox after apache#15024.

Regenerate the table of contents via `gh-md-toc` to replace the "Release
workflow" section. Then we can verify it by clicking the link to the
corresponding section.

…17269)

…pache#17609)

…Installation" title (apache#17580)

…tioned system topic (apache#17338)

* Create 2022-09-09-Apache-Pulsar-2-7-5.md

* Update site2/website/blog/2022-09-09-Apache-Pulsar-2-7-5.md

Co-authored-by: Anonymitaet <[email protected]>

Co-authored-by: Anonymitaet <[email protected]>

* [improve][doc] add developers-landing page

The Pulsar for Developers content block on the documentation landing page needs to link to this page.

* Update getting-started-standalone.md

* Update getting-started-standalone.md

* move the file to /site2/docs/

* Update about.md

* Update site2/docs/developers-landing.md

Co-authored-by: tison <[email protected]>

* Update site2/docs/getting-started-standalone.md

Co-authored-by: tison <[email protected]>

Co-authored-by: momo-jun <[email protected]>
Co-authored-by: tison <[email protected]>

…while print the producer stats (apache#17671)

…duceConsumeUsageOnRG (apache#17617)

…th a relative path (apache#17675)

* [improve][cli] Pulsar shell: allow to create a new config (--file) with a relative path

* win

* checkstlye

…or last active time not updated in time (apache#17573)