Deploying Apache Pulsar on an OVHcloud Managed Kubernetes cluster

Objective

Apache Pulsar is an open-source, distributed messaging and streaming platform built for the cloud. Companies around the world are using Pulsar to deliver the most robust event-driven architectures for real-time applications. Its decoupled design allows messages to be consumed in high-performance streams or adaptable queues. It natively provides a series of enterprise-grade features, such as multi-tenancy, geo-replication, Pulsar Functions, and a multi-language API.

StreamNative, a company founded by the original creators of Apache Pulsar, partners with software teams worldwide to accelerate time to production and manage Pulsar at scale with StreamNative Cloud, a fully managed Pulsar service. It has developed the StreamNative Operators for Apache Pulsar that provide full lifecycle management of Pulsar deployments on Kubernetes offered under a simple, free Community License. With the operators, you can easily deploy a Pulsar cluster on Kubernetes and manage it through the Kubernetes API and kubectl.

This tutorial demonstrates how to install Apache Pulsar on an OVHcloud Managed Kubernetes cluster using the StreamNative Operators for Apache Pulsar.

Requirements

• An OVHcloud Managed Kubernetes cluster (v1.24+) with a minimum of 3 worker nodes, each having at least 2 vCores and 6 GB of memory. Once you have the cluster ready, you should have a default storage class automatically installed, which is required for Pulsar instances on Kubernetes. You can run kubectl get sc to view your available storage classes.
• kubectl installed and configured (v1.16 or later). For more information, see Configuring kubectl on an OVHcloud Managed Kubernetes cluster.

Instructions

Before installing Pulsar, you need to install the operators first, which can be deployed using the Helm chart or Operator Lifecycle Manager (OLM). Compared with the deployment with Helm, OLM does not require you to manually upgrade custom resource definitions (CRDs) when upgrading the operators.

This tutorial uses OLM to install the StreamNative Operators for Apache Pulsar. For more information about the deployment with Helm, read the documentation.

Installing Operator Lifecycle Manager and the operators

OLM is a popular and powerful tool to manage operators. With OLM, your CRDs can be automatically updated when you upgrade operators.

1. Run the following command to create necessary Kubernetes resources of OLM, including CRDs, ClusterRoles, and Deployments. The script also creates two namespaces - olm and operators.

curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.24.0/install.sh | bash -s v0.24.0

2. Install the CRDs and custom controllers for Pulsar components (brokers, proxies, BookKeeper, and ZooKeeper). The controllers are deployed in the operators namespace by default.

kubectl create -f https://raw.githubusercontent.com/streamnative/charts/master/examples/pulsar-operators/olm-subscription.yaml

3. Verify that the operators are installed successfully. When you deploy the operators, OLM first runs a Job in the olm namespace for each operator. You need to wait for the Jobs to finish before you can see the output.

kubectl get pods -n operators

Expected output:

NAME                                                      READY   STATUS    RESTARTS   AGE
bookkeeper-operator-controller-manager-5b4fd6d856-ckpmw   2/2     Running   0          51s
pulsar-operator-controller-manager-9745c86f9-6dm2q        2/2     Running   0          43s
zookeeper-operator-controller-manager-669d7789-v76vd      2/2     Running   0          74s

Installing Pulsar with the operators

StreamNative provides a quickstart YAML file that contains the manifests of Pulsar brokers, BookKeeper, and ZooKeeper. You can use it to quickly deploy Pulsar’s custom resources.

1. Create a namespace called pulsar where Pulsar workloads will be deployed later. This is the default name in the example YAML file. If you change the namespace in this step, you need to use the same namespace in the next step. This namespace can be different from the one where operators are installed.

kubectl create ns pulsar

2. Choose to deploy Pulsar with or without proxies. The following two YAML files both contain the manifests for ZooKeeperCluster, BookKeeperCluster, and PulsarBroker. Pulsar proxies are optional, serving as a gateway to route traffic to Pulsar brokers.

• Run the following command to install Pulsar with proxies.

kubectl create -f https://raw.githubusercontent.com/streamnative/charts/master/examples/pulsar-operators/proxy.yaml

• Run the following command to install Pulsar without proxies.

kubectl create -f https://raw.githubusercontent.com/streamnative/charts/master/examples/pulsar-operators/quick-start.yaml

3. Verify that the different components of the Pulsar cluster are up and running.

kubectl get pods -n pulsar

Expected output:

NAME                         READY   STATUS    RESTARTS   AGE
bookies-bk-0                 1/1     Running   0          9m8s
bookies-bk-1                 1/1     Running   0          9m8s
bookies-bk-2                 1/1     Running   0          9m8s
bookies-bk-auto-recovery-0   1/1     Running   0          8m20s
brokers-broker-0             1/1     Running   0          9m9s
brokers-broker-1             1/1     Running   0          9m9s
zookeepers-zk-0              1/1     Running   0          12m
zookeepers-zk-1              1/1     Running   0          12m
zookeepers-zk-2              1/1     Running   0          12m

The Pulsar cluster is now ready to serve requests.

Testing the Pulsar cluster

1. Exec into a broker Pod to create a tenant, a namespace, and a topic with 4 partitions.

kubectl exec -n pulsar brokers-broker-0 -- bin/pulsar-admin tenants create apache
kubectl exec -n pulsar brokers-broker-0 -- bin/pulsar-admin namespaces create apache/pulsar
kubectl exec -n pulsar brokers-broker-0 -- bin/pulsar-admin topics create-partitioned-topic apache/pulsar/test-topic -p 4

2. List all the partitioned topics in the namespace apache/pulsar.

kubectl exec -n pulsar brokers-broker-0 -- bin/pulsar-admin topics list-partitioned-topics apache/pulsar

Expected output:

persistent://apache/pulsar/test-topic

3. Create a subscription to consume messages from apache/pulsar/test-topic.

kubectl exec -n pulsar brokers-broker-0 -- bin/pulsar-client consume -s sub apache/pulsar/test-topic -n 0

4. Open a new terminal and run the following command to send 10 messages to the test-topic topic.

kubectl exec -n pulsar brokers-broker-0 -- bin/pulsar-client produce apache/pulsar/test-topic -m "---------hello apache pulsar-------" -n 10

Expected output on the consumer side:

----- got message -----
key:[null], properties:[], content:---------hello apache pulsar-------
----- got message -----
key:[null], properties:[], content:---------hello apache pulsar-------
----- got message -----
key:[null], properties:[], content:---------hello apache pulsar-------
----- got message -----
key:[null], properties:[], content:---------hello apache pulsar-------
----- got message -----
key:[null], properties:[], content:---------hello apache pulsar-------
----- got message -----
key:[null], properties:[], content:---------hello apache pulsar-------
----- got message -----
key:[null], properties:[], content:---------hello apache pulsar-------
----- got message -----
key:[null], properties:[], content:---------hello apache pulsar-------
----- got message -----
key:[null], properties:[], content:---------hello apache pulsar-------
----- got message -----
key:[null], properties:[], content:---------hello apache pulsar-------

Cleaning up

You can uninstall the operators and Pulsar when you no longer need them.

Uninstalling Pulsar

If you already have deployed Pulsar with Pulsar operators, you need to uninstall Pulsar first. You can skip this step if Pulsar is not installed.

• Run the following command to uninstall Pulsar if you installed it with proxies.

kubectl delete -f https://raw.githubusercontent.com/streamnative/charts/master/examples/pulsar-operators/proxy.yaml

• Run the following command to uninstall Pulsar if you installed it without proxies.

kubectl delete -f https://raw.githubusercontent.com/streamnative/charts/master/examples/pulsar-operators/quick-start.yaml

Uninstalling the operators

1. Delete the Subscriptions created by OLM. Subscriptions convey a user’s intent to subscribe to the latest version of an operator. By deleting the Subscriptions associated with Pulsar operators, you let OLM know that you no longer want new versions of operators to be installed.

kubectl delete -f https://raw.githubusercontent.com/streamnative/charts/master/examples/pulsar-operators/olm-subscription.yaml

2. Delete the ClusterServiceVersions (CSVs) of Pulsar operators. You can run kubectl get csv -n operators to check the versions.

kubectl delete csv pulsar-operator.<version> bookkeeper-operator.<version> zookeeper-operator.<version> -n operators

3. Delete the CRDs of Pulsar.

kubectl delete crd pulsarbrokers.pulsar.streamnative.io pulsarproxies.pulsar.streamnative.io bookkeeperclusters.bookkeeper.streamnative.io zookeeperclusters.zookeeper.streamnative.io

Uninstalling OLM

Run the following commands to uninstall OLM:

kubectl delete apiservices.apiregistration.k8s.io v1.packages.operators.coreos.com
kubectl delete -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.24.0/crds.yaml
kubectl delete -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.24.0/olm.yaml

More resources

The StreamNative team looks to provide a better cloud-native experience for Pulsar users with the StreamNative Operators for Apache Pulsar. To learn more about OVHcloud Managed Kubernetes, Pulsar, and Pulsar Operators, check out the following resources:

• OVHcloud Managed Kubernetes documentation
• Apache Pulsar documentation
• StreamNative Operators for Apache Pulsar documentation