编程知识 cdmana.com

[Hadoop 3. X] layered case of massive bank transfer data

 Insert picture description here

The bank has a lot of transfers every day 、 Transaction needs to be saved 、 Handle . Every time a user makes a transaction or transfer , Banks need to save all relevant information about users' transfer .
Four big banks :
 Insert picture description here

Banks have a lot of users , The big four banks have a number of 10 Million users . The amount of data to be saved can be imagined . If there are data , All treat equally , In order to ensure the performance of using data , Using high-performance storage , This will be a great waste of resources . actually , Data over a certain period of time , The frequency of data access is much lower . for example : User query 5 Transfer records years ago 、 It's better than querying 1 The frequency of transfer records in the year category is much lower .

therefore , In order to make better use of resources , Data needs to be layered . That is, data in different time ranges , On different layers ( Cold and hot temperature ) in .
5.1 Storage tiering strategy
Follow the configuration below , stay HDFS Of source Create the following folders in the directory :

Folder path Storage strategy explain

/source/bank/transfer/log_lte1y DISK Store data collected within one year /source/bank/transfer/log_gt1y ARCHIVE Storage 1 More than years of data

5.2 test

Create folder

hdfs dfs -mkdir -p /source/bank/transfer/log_lte1y
hdfs dfs -mkdir -p /source/bank/transfer/log_gt1y

Specify storage policy

hdfs storagepolicies -setStoragePolicy -path /source/bank/transfer/log_lte1y -policy HOT
hdfs storagepolicies -setStoragePolicy -path /source/bank/transfer/log_gt1y -policy COLD

View storage policy

hdfs storagepolicies -getStoragePolicy -path /source/bank/transfer/log_lte1y
hdfs storagepolicies -getStoragePolicy -path /source/bank/transfer/log_gt1y

Upload file test

Upload files to linux

rz

A year

hdfs dfs -put /root/bank_record.csv 
/source/bank/transfer/log_lte1y/bank_record_2020_9.csv

Five years

hdfs dfs -put /root/bank_record.csv 
/source/bank/transfer/log_gt1y/bank_record_2015_9.csv

Suppose it's now 2021 year 10 year , We can move the previous data to log_gt1y

hdfs dfs -mv /source/bank/transfer/log_lte1y/bank_record_2020_9.csv 
/source/bank/transfer/log_gt1y/bank_record_2020_9.csv

版权声明
本文为[Manor's big data struggle]所创,转载请带上原文链接,感谢
https://cdmana.com/2021/10/20211002003321646p.html

Scroll to Top