Kafka Tutorial: How to Configure & Reset Consumer Offset in Apache Kafka

Kafka guarantees “exactly-once” semantics when used correctly. This ensures that any message published to Kafka will be processed at least once by one consumer, and it will never be redelivered to another consumer or requeued for reprocessing after being acknowledged as processed.

There are use cases where you want to reset the offset for a given topic partition, which means the consumed messages identified by the offset need to be reprocessed from scratch. For this you have to learn Apache Kafka in detail. Learning more about kafta connect might help, too. 

You can reset message offsets manually in zookeeper, or configure your client library’s session to automatically rewind offsets based on retention settings. 

To use this tutorial you should already have set up Apache Kafka with at least one broker instance running. If not, please follow the Kafka Quick Start to set up a single-node cluster. Once your Kafka cluster is running, you’re ready to proceed.

Step 1: Configure Your Consumer Session to Automatically Rewind Offsets When Expired.

Use one of the following Session configuration methods depending on your chosen client library.

Step 2: Manually Reset Topic Partition Offset in Zookeeper after Processing Completely through All Messages.

Make sure you process the entire topic partition completely through all messages before proceeding with these steps. Every Kafka topic has a set of configurable parameters that can be specified and changed via the .properties file on each broker (kafka-broker-1/config/server.properties). These properties can be viewed using the kafka-configs tool.

Step 3: View Message Metadata with kafka-avro-console-consumer.

Viewing message metadata gives you insight into details about messages published to your topics including information about timestamp, partition, offset and key. The Avro console consumer allows you to view this metadata without having to write your own custom application.

Step 4: (Optional) Reset the Topic Partition Offset Manually in Zookeeper for a Given Topic and Partition ID.

This step is not recommended unless you plan on building out a more robust consumer application that needs to rewind offsets based on retention settings as outlined above, or if you want to delete messages from the partition manually.

Step 5: View Offset Ranges with `kafka-offset-checker`

Kafka offset checker is a tool used for comparing expected offsets vs actual offsets to determine if any messages are missing or out of order. Understanding the different types of log records will help you utilize this tool effectively. For more information on Kafka log data, check out this explanation.

Step 6: View Offset Ranges with kafka-console-consumer

The Kafka console consumer is a tool for viewing the topic metadata and offset records in each partition of a given topic. This includes details about message key, timestamp, partition, and offset. The offsets for each record can be viewed visually by clicking on individual records or all together at once using the “Show All” button.

Step 7: View Offset Ranges with `kafka-tools`

Kafka tools will give you a full overview of all your topics including their partitions and associated configuration settings. Using this tool gives you insight into various retention settings which include how long messages will be retained before automatically being deleted from disk, metadata information such as message key, and timestamp.

Step 8: Configure Kafka to Delete Messages from a Topic When the Offset is Greater than Retention Time (Optional)

This step is not recommended unless you plan on building out a more robust consumer application that needs to rewind offsets based on retention settings as outlined above, or if you want to delete messages from the partition manually.

Step 9: (Optional) Reset Topics with Retention Time Using the `kafka-log4j-consumer`

The Kafka log4j consumer is another tool for viewing topic metadata including information about message key, timestamp, partition and offset. It also allows you to set various topics configuration parameters such as retention time which deletes messages after they are acknowledged as processed by consumers. After processing through all messages in each partition of this topic, reset the message offset manually using the kafka console consumer or kafka-offset-checker tool.

Here ends the 9 step guide on Kafka Tutorial: How to Configure & Reset Consumer Offset in Apache Kafka. We hope you have now understood how to configure and reset consumer offset. If you have any further queries, drop it in the comment section and we will review it.