编程知识 cdmana.com

HBase deletes oldWAL online to free up disk space

HBase的WAL日志会从WAL转移到oldWAL目录,oldWALThe retention time can be set by parameters,The screenshot below will beoldWAL设置为7天,通过配置hbase.master.logcleaner.ttl指定,This may result in a larger disk space usage.If the online environment encounters a large amount of disk space,Consider removing sections based on date criteriaoldWAL日志.(如果是直接删除oldWAL目录下面的所有文件,Possibly because there is currentlyWALcaused by movementRegioniServer宕机)
请添加图片描述
Use the following steps to delete by dateoldWAL文件:

  1. 切换到hdfs用户
su - hdfs
  1. Check if it is a list of files with old dates that you want to delete
hdfs dfs -ls /hbase/oldWALs | grep 2022-07-2[2-3]  <-- Lists the full list of file properties with date fields
hdfs dfs -ls /hbase/oldWALs | grep 2022-07-2[2-3] | awk '{print $8}' <-- Only output within the corresponding dateoldWAL文件名列表
  1. 执行删除操作
hdfs dfs -ls /hbase/oldWALs | grep 2022-07-2[2-3] | awk '{print $8}' | xargs hdfs dfs -rm -skipTrash
  1. 运行检查hdfs健康情况
hdfs fsck /
..............................Status: HEALTHY
Total size:    1202147250905 B (Total open files size: 415 B)
Total dirs:    88969
Total files:   147730
Total symlinks:                0 (Files currently being written: 6)
Total blocks (validated):      139316 (avg. block size 8628924 B) (Total open file blocks (not validated): 5)
Minimally replicated blocks:   139316 (100.0 %)
Over-replicated blocks:        0 (0.0 %)
Under-replicated blocks:       0 (0.0 %)
Mis-replicated blocks:         0 (0.0 %)
Default replication factor:    3
Average block replication:     3.0
Corrupt blocks:                0
Missing replicas:              0 (0.0 %)
Number of data-nodes:          4
Number of racks:               1
FSCK ended at Sun Jul 24 11:43:03 CST 2022 in 1421 milliseconds
The filesystem under path '/' is HEALTHY
  1. 运行hbase hbck检查
hbase hbck
Number of Tables: 3176
22/07/24 12:06:53 INFO util.HBaseFsck: Loading region directories from HDFS
....................................
22/07/24 12:06:57 INFO util.HBaseFsck: Loading region information from HDFS
..........................................
22/07/24 12:07:07 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x46591e98 connecting to ZooKeeper ensemble=esggk01.esgyncn.local:2181,esggk04.esgyncn.local:2181,esggk05.esgyncn.local:2181
22/07/24 12:07:07 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=esggk01.esgyncn.local:2181,esggk04.esgyncn.local:2181,esggk05.esgyncn.local:2181 sessionTimeout=90000 watcher=hconnection-0x46591e980x0, quorum=esggk01.esgyncn.local:2181,esggk04.esgyncn.local:2181,esggk05.esgyncn.local:2181, baseZNode=/hbase
22/07/24 12:07:07 INFO zookeeper.ClientCnxn: Opening socket connection to server esggk05.esgyncn.local/10.11.40.45:2181. Will not attempt to authenticate using SASL (unknown error)
22/07/24 12:07:07 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.11.40.42:53664, server: esggk05.esgyncn.local/10.11.40.45:2181
22/07/24 12:07:07 INFO zookeeper.ClientCnxn: Session establishment complete on server esggk05.esgyncn.local/10.11.40.45:2181, sessionid = 0xff821f58ad583f3e, negotiated timeout = 90000
22/07/24 12:07:07 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0xff821f58ad583f3e
22/07/24 12:07:07 INFO zookeeper.ZooKeeper: Session: 0xff821f58ad583f3e closed
22/07/24 12:07:07 INFO zookeeper.ClientCnxn: EventThread shut down
22/07/24 12:07:07 INFO util.HBaseFsck: Checking and fixing region consistency
22/07/24 12:07:11 INFO util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
22/07/24 12:07:11 INFO util.HBaseFsck: Computing mapping of all store files
..................................................
22/07/24 12:07:15 INFO util.HBaseFsck: Validating mapping using HDFS state
22/07/24 12:07:16 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
22/07/24 12:07:16 INFO replication.ReplicationPeersZKImpl: Added new peer cluster=esggk10.esgyncn.local,esggk11.esgyncn.local,esggk08.esgyncn.local:2181:/hbase
Summary:
Table TRAFODION.LTTSV7.KCEP_PZJBEI_IDX2 is okay.
    Number of regions: 1
..................Other table output is omitted here.....................
Table TRAF_RSRVD_1:TRAFODION._PRIVMGR_MD_.SCHEMA_PRIVILEGES is okay.
    Number of regions: 1
    Deployed on:  esggk02.esgyncn.local,60020,1658383339653
0 inconsistencies detected.
Status: OK
22/07/24 12:07:16 INFO zookeeper.ZooKeeper: Session: 0xff821f58acfc3eb2 closed
22/07/24 12:07:16 INFO zookeeper.ClientCnxn: EventThread shut down
22/07/24 12:07:16 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
22/07/24 12:07:16 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0xff821f58ad583f3b
22/07/24 12:07:16 INFO zookeeper.ZooKeeper: Session: 0xff821f58ad583f3b closed
22/07/24 12:07:16 INFO zookeeper.ClientCnxn: EventThread shut down

版权声明
本文为[post_yuan]所创,转载请带上原文链接,感谢
https://cdmana.com/2022/218/202208060748092437.html

Scroll to Top