编程知识 cdmana.com

Kafka common mistakes and solutions in Kafka learning notes (topic, production and consumption, startup)

0x01 Initiate error reporting

1.1 First error

2017-02-17 17:25:29,224] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)

kafka.common.KafkaException: Failed to acquire lock on file .lock in /var/log/kafka-logs. A Kafka instance in another process or thread is using this directory.

at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:100)

at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:97)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)

at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)

at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)

at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)

at scala.collection.AbstractTraversable.map(Traversable.scala:104)

at kafka.log.LogManager.lockLogDirs(LogManager.scala:97)

at kafka.log.LogManager.<init>(LogManager.scala:59)

at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:609)

at kafka.server.KafkaServer.startup(KafkaServer.scala:183)

at io.confluent.support.metrics.SupportedServerStartable.startup(SupportedServerStartable.java:100)

at io.confluent.support.metrics.SupportedKafka.main(SupportedKafka.java:49)

resolvent :Failed to acquire lock on file .lock in /var/log/kafka-logs.-- The problem is that there are other processes using kafka,ps -ef|grep kafka, Kill the process that uses the directory ;

1.2 The second mistake : Yes index The file does not have permission

 Change the file permissions to the correct user name and user group ;

 Catalog /var/log/kafka-logs/, among __consumer_offsets-29 It's the offset ;

1.3 The third kind of production and consumption report error :jaas There's a problem with the connection

kafka_client_jaas.conf There is a problem with the file configuration 

16 Environment 

/opt/dataload/filesource_wangjuan/conf Next kafka_client_jaas.conf
 

KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required

useKeyTab=true

storeKey=true

keyTab="/home/client/keytabs/client.keytab"

serviceName="kafka"

principal="client/dcp@DCP.COM";

};

0x02 The producer reported an error

2.1 The first one is : The producer to topic Failed to send message

[2017-03-09 09:16:00,982] [ERROR] [startJob_Worker-10] [DCPKafkaProducer.java line:62] produceR towards topicdf02211 Abnormal sending message 

org.apache.kafka.common.KafkaException: Failed to construct kafka producer

at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:335)

The reason is the configuration file :kafka_client_jaas.conf There's a problem with the configuration ,keyTab The path of is not correct , As a result of ;

2.2 The second kind : Production and consumption report error : Failed to construct kafka producer

Key information of error reporting :Failed to construct kafka producer

resolvent : Configuration file problem :KafkaClient in serviceName Should be kafka, It was configured as zookeeper; After restart , Just fine ;

The configuration file is as follows :

KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required

useKeyTab=true

storeKey=true

useTicketCache=false

serviceName=kafka

keyTab="/etc/security/keytabs/kafka.service.keytab"

principal="kafka/dcp16@DCP.COM";

};

KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required

useKeyTab=true

storeKey=true

serviceName=kafka

keyTab="/etc/security/keytabs/kafka.service.keytab"

principal="kafka/dcp16@DCP.COM";

};

Client {
com.sun.security.auth.module.Krb5LoginModule required

useKeyTab=true

storeKey=true

useTicketCache=false

serviceName=zookeeper

keyTab="/etc/security/keytabs/kafka.service.keytab"

principal="kafka/dcp16@DCP.COM";

};

Problem description :

[kafka@DCP16 bin]$ ./kafka-console-producer --broker-list DCP16:9092 --topic topicin050511 --producer.config ../etc/kafka/producer.properties

org.apache.kafka.common.KafkaException: Failed to construct kafka producer

at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:335)

at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:188)

at kafka.producer.NewShinyProducer.<init>(BaseProducer.scala:40)

at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:45)

at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)

Caused by: org.apache.kafka.common.KafkaException: java.lang.IllegalArgumentException: Conflicting serviceName values found in JAAS and Kafka configs value in JAAS file zookeeper, value in Kafka config kafka

at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86)

at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:70)

at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:83)

at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:277)

... 4 more

Caused by: java.lang.IllegalArgumentException: Conflicting serviceName values found in JAAS and Kafka configs value in JAAS file zookeeper, value in Kafka config kafka

at org.apache.kafka.common.security.kerberos.KerberosLogin.getServiceName(KerberosLogin.java:305)

at org.apache.kafka.common.security.kerberos.KerberosLogin.configure(KerberosLogin.java:103)

at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:45)

at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68)

at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78)

... 7 more

[kafka@DCP16 bin]$ ./kafka-console-producer --broker-list DCP16:9092 --topic topicin050511 --producer.config ../etc/kafka/producer.properties

2.3 Consumption times are wrong : ERROR Unknown error when running consumer: (kafka.tools.ConsoleConsumer$)

[root@DCP16 bin]# ./kafka-console-consumer --zookeeper dcp18:2181,dcp16:2181,dcp19:2181/kafkakerberos --from-beginning --topic topicout050511 --new-consumer --consumer.config ../etc/kafka/consumer.properties --bootstrap-server DCP16:9092

[2017-05-07 22:24:37,479] ERROR Unknown error when running consumer: (kafka.tools.ConsoleConsumer$)

org.apache.kafka.common.KafkaException: Failed to construct kafka consumer

at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:702)

at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:587)

at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:569)

at kafka.consumer.NewShinyConsumer.<init>(BaseConsumer.scala:53)

at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:64)

at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:51)

at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)

Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user

at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86)

at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:70)

at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:83)

at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:623)

... 6 more

Caused by: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user

at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:899)

at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)

at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)

at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)

at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)

at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)

at javax.security.auth.login.LoginContext.login(LoginContext.java:595)

at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:69)

at org.apache.kafka.common.security.kerberos.KerberosLogin.login(KerberosLogin.java:110)

at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:46)

at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68)

at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78)

Derivative problems :

kafka Production news will be wrong :

[2017-05-07 23:17:16,240] ERROR Error when sending message to topic topicin050511 with key: null, value: 0 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)

org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

hold KafkaClient Change to the following configuration , Can 了 :

KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required

useTicketCache=true;

};

0x03 The consumer reported an error

3.1 First error :replication factor: 1 larger than available brokers: 0

Consumption times are wrong :Error while executing topic command : replication factor: 1 larger than available brokers: 0

terms of settlement :/confluent-3.0.0/bin Next restart daemon

./kafka-server-stop -daemon ../etc/kafka/server.properties

./kafka-server-start -daemon ../etc/kafka/server.properties

then zk restart ;sh zkCli.sh -server ai186;

/usr/hdp/2.4.2.0-258/zookeeper/bin/zkCli.sh -- Script Directory

If there is a mistake , You can view the following configuration in the configuration file :

zookeeper.connect=dcp18:2181/kafkakerberos; -- yes group name

3.2 The second mistake :TOPIC_AUTHORIZATION_FAILED

./bin/kafka-console-consumer --zookeeper DCP185:2181,DCP186:2181,DCP187:2181/kafka --from-beginning --topic wangjuan_topic1 --new-consumer --consumer.config ./etc/kafka/consumer.properties --bootstrap-server DCP187:9092

[2017-03-02 13:44:38,398] WARN The configuration zookeeper.connection.timeout.ms = 6000 was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)

[2017-03-02 13:44:38,575] WARN Error while fetching metadata with correlation id 1 : {wangjuan_topic1=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)

[2017-03-02 13:44:38,677] WARN Error while fetching metadata with correlation id 2 : {wangjuan_topic1=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)

[2017-03-02 13:44:38,780] WARN Error while fetching metadata with correlation id 3 : {wangjuan_topic1=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)

resolvent : In the following parameters in the configuration file User Of U It must be in capital ;

super.users=User:kafka

Or it could be server.properties Medium adver.listen Of IP It is not right , It could be dead in the code IP;

3.3 The third possible solution to the mistake :

Can't consume , View kafka Error information in the startup log of : The log file belongs to the wrong group , Should be hadoop;

perhaps , see kafka Corresponding zookeeper Configuration suffix of , Whether it has been changed , If changed , be topic It needs to be regenerated ;

3.4 The fourth mistake : The consumption of tomcat Report errors

[2017-04-01 06:37:21,823] [INFO] [Thread-5] [AbstractCoordinator.java line:542] Marking the coordinator DCP187:9092 (id: 2147483647 rack: null) dead for group test-consumer-group

[2017-04-01 06:37:21,825] [WARN] [Thread-5] [ConsumerCoordinator.java line:476] Auto offset commit failed for group test-consumer-group: Commit offsets failed with retriable exception. You should retry committing offsets.

Change the code ,tomcat The heartbeat timeout of is as follows :

There is no change before :;

./webapps/web/WEB-INF/classes/com/ai/bdx/dcp/hadoop/service/impl/DCPKafkaConsumer.class;

After restart , Show in log :

[2017-04-01 10:14:56,167] [INFO] [Thread-5] [AbstractCoordinator.java line:542] Marking the coordinator DCP187:9092 (id: 2147483647 rack: null) dead for group test-consumer-group

[2017-04-01 10:14:56,286] [INFO] [Thread-5] [AbstractCoordinator.java line:505] Discovered coordinator DCP187:9092 (id: 2147483647 rack: null) for group test-consumer-group.

0x04 establish topic Time error

establish topic Times wrong :

[2017-04-10 10:32:23,776] WARN SASL configuration failed: javax.security.auth.login.LoginException: Checksum failed Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)

Exception in thread "main" org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentication failure

at org.I0Itec.zkclient.ZkClient.waitForKeeperState(ZkClient.java:946)

at org.I0Itec.zkclient.ZkClient.waitUntilConnected(ZkClient.java:923)

at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1230)

at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)

at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)

at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:75)

at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)

at kafka.admin.TopicCommand$.main(TopicCommand.scala:54)

at kafka.admin.TopicCommand.main(TopicCommand.scala)

Problem location : yes jaas There is something wrong with the document. :

resolvent :server.properties In the document super.user Want to be with jaas In the document keytab Of principle Agreement ;

server.properties:super.users=User:client

kafka_server_jaas.conf Change the document to :

 

KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required

useKeyTab=true

storeKey=true

useTicketCache=false

serviceName=kafka

keyTab="/data/data1/confluent-3.0.0/kafka.keytab"

principal="kafka@DCP.COM";

};

 

KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required

useKeyTab=true

storeKey=true

keyTab="/home/client/client.keytab"

principal="client/DCP187@DCP.COM";

};

 

Client {
com.sun.security.auth.module.Krb5LoginModule required

useKeyTab=true

storeKey=true

useTicketCache=false

serviceName=zookeeper

keyTab="/home/client/client.keytab"

principal="client/DCP187@DCP.COM";

};

Participation of this paper Tencent cloud media sharing plan , You are welcome to join us , share .

版权声明
本文为[Jetpropelledsnake21]所创,转载请带上原文链接,感谢
https://cdmana.com/2020/12/20201225111921904c.html

Scroll to Top