If a dedicated ZooKeeper ensemble is still a performance bottleneck, you can address the issue by using SSDs on your ZooKeeper nodes. (You may, for example, have a monitoring console that reads offset information from ZooKeeper.) If you have to use ZooKeeper for offset management, we recommend using a dedicated ZooKeeper ensemble for your Kafka cluster. However, you may choose to use ZooKeeper to store offsets for backwards compatibility. However, users get to choose whether they want offsets managed in Kafka or ZooKeeper, via the consumer config parameter offsets.storage.Ĭloudera highly recommends using Kafka to store offsets. Essentially, it uses a separate Kafka topic to manage recently read offsets, and thus Zookeeper is no longer required for offset management. Starting with version 0.8.2, the high-level consumer can use Kafka itself to manage offsets.
Nevertheless, this issue has been resolved in the current version of Kafka (0.8.2 at the time of this writing). However, this only occurs in extreme cases, when there are many hundreds of consumers using the same ZooKeeper cluster for offset management. If there are many consumers simultaneously reading from Kafka, the read write load on ZooKeeper may exceed its capacity, making ZooKeeper a bottleneck. Older versions of Kafka’s high-level consumer (0.8.1 and older) used ZooKeeper to maintain read offsets (the most recently read offset of each partition of a topic). Is it true that Apache ZooKeeper can become a pain point with a Kafka cluster?
USING KAFKA TOOL CREATE DC1 TO DC2 SOFTWARE
The other option is to use software that provides filesystem-level encryption such as Cloudera Navigator Encrypt, included as part of Cloudera Enterprise, which provides a transparent encryption layer between the application and the filesystem. But that would require the producers to share encryption protocols and keys with the Consumers. Users can always encrypt the payload of the messages written to Kafka-that is, producers encrypt the data before writing to Kafka, and then the consumers can decrypt the received messages. How do I encrypt the data persisted on my Kafka Brokers?Ĭurrently, Kafka does not provide any mechanism to encrypt the data persisted on the brokers (i.e. Sequential reads and writes are heavily optimized by modern operating systems. Thus, disk reads and writes in Kafka are sequential, with very few random seeks.
Using SSDs instead of spinning disks has not been shown to provide a significant performance improvement for Kafka, for two main reasons:
In this post, you’ll find answers to most of those questions. Apache Kafka now is an integrated part of CDH, manageable via Cloudera Manager, and we are witnessing rapid adoption of Kafka across our customer base.Īs more customers adopt Apache Kafka, a common set of questions from them about development and deployment have emerged as a pattern. This post contains answers to common questions about deploying and configuring Apache Kafka as part of a Cloudera-powered enterprise data hub.Ĭloudera added support for Apache Kafka, the open standard for streaming data, in February 2015 after its brief incubation period in Cloudera Labs.
0 kommentar(er)
