编程知识 cdmana.com

[Hadoop 3. X series] use of HDFS rest HTTP API (II) httpfs

 Insert picture description here

Preface

Current blog Hadoop Most articles stay in Hadoop2.x Stage , This series will be based on the big data of dark horse programmers Hadoop3.x A full set of tutorial , Yes 2.x There are no new features to supplement and update , One click three times plus attention , Don't get lost next time !

Article history

[hadoop3.x series ]HDFS REST HTTP API Use ( One )WebHDFS

HTTPFS summary

l HttpHDFS In essence, WebHDFS It's the same , They are all provided HTTP REST API function , But the difference is HttpHDFS yes HttpFS Is an independent from HadoopNameNode Service for , It is itself Java JettyWeb Applications .

l Because it can be deployed independently , So you can be right HttpHDFS Set up firewall , And avoid NameNode Exposed to the wall , For some systems with high security requirements ,HttpHDFS It would be better .

l HttpFS It's a kind of server , It provides REST HTTP gateway , Support all HDFS File system operations ( Read and write ). And it can work with WebhdfsREST HTTPAPI

l HttpFS Can be used to run different versions Hadoop( Overcome RPC Version control problem ) Transfer data between clusters , For example, using HadoopDiscreCP.

l HttpFS It can be used to access on the cluster behind the firewall HDFS Data in (HttpFS The server acts as a gateway , Is the only system that allows access to the cluster across the firewall ).

l HttpFS have access to HTTP Utilities ( for example curl and wget) And from the Java In other languages HTTP library Perl To visit HDFS Data in .

l This ****Webhdfs**** The client file system implementation can use Hadoop File system command access HttpFS(hdfs dfs) Line tools and use Hadoop file system JavaAPI Of Java Applications .

l HttpFS Built in support Hadoop Pseudo authentication and HTTP、SPNEGO Kerberos And other pluggable authentication mechanisms . It also provides Hadoop Agent user support .

1.1.1 HTTPFS How it works

l HttpFS Is an independent from HadoopNameNode Service for .

l HttpFS Itself is Java JettyWeb Applications .

l HttpFS HTTP Web service API Call yes HTTPREST call , Mapping to HDFS File system operations . for example , Use curl/Unix command :

$curl ‘http://httpfs-host:14000/webhdfs/v1/user/foo/README.txt?op=OPEN&user.name=foo’
return HDFS The content of /user/foo/README.txt archives .

$curl ‘http://httpfs-host:14000/webhdfs/v1/user/foo?op=LISTSTATUS&user.name=foo’
return HDFS The content of /user/foo In the directory JSON Format .

$curl ‘http://httpfs-host:14000/webhdfs/v1/user/foo?op=GETTRASHROOT&user.name=foo’
Return path /user/foo/.trash, If / It's an encrypted area , Then return to the path ./.Trash/Foo. See more details about the garbage path in the encrypted area .

$curl -X POST‘http://httpfs-host:14000/webhdfs/v1/user/foo/bar?op=MKDIRS&user.name=foo’
establish HDFS/user/foo/bar Catalog .

l HttpFS The default port number is 14000

1.1.2 To configure Hadoop

edit Hadoop Of core-site.xml, And will run HttpFS Server's Unix User defined as proxyuser. for example :

 <property>  
     <name>hadoop.proxyuser.#HTTPFSUSER#.hosts</name>  
     <value>httpfs-host.foo.com</value> 
     </property> <property>
     <name>hadoop.proxyuser.#HTTPFSUSER#.groups</name>  
     <value>*</value> 
 </property>

important : Replace #HTTPFSUSER# Using will start HttpFS Server's Unix user .

for example :

<property>  
<name>hadoop.proxyuser.root.hosts</name>  
<value>*</value></property><property>  
<name>hadoop.proxyuser.root.groups</name>  
<value>*</value>
</property>

1.1.3 restart Hadoop

restart Hadoop, And activate the proxy user configuration

1.1.4 start-up HttpFS

hdfs --daemon start httpfs

1.1.5 test HttpFS Work

http://node1.itcast.cn:14000/webhdfs/v1?user.name=root&op=LISTSTATUS

img

1.1.6 HTTP Default service

Name\ Description
/conf Display configuration properties
/jmx Java JMX management interface
/logLevel Get or set log level per class
/logs Display log files
/stacks Display JVM stacks
/static/index.html The static home page

http://node1.itcast.cn:14000/conf?user.name=root

Postscript

Blog home page :https://manor.blog.csdn.net
Welcome to thumb up Collection Leaving a message. Please correct any mistakes !
This paper is written by manor original , First appeared in CSDN Blog

版权声明
本文为[Manor's big data struggle]所创,转载请带上原文链接,感谢
https://cdmana.com/2021/10/20211002003321651x.html

Scroll to Top