hadoop安装和libhdfs使用
QQQ 科研 51
  1. 根据Apache Hadoop官方文档操作,第一步配置好JAVA,ubuntu可以直接下载:openjdk8 devel,也可以查看本机已安装的java环境:sudo update-alternatives --config java
  2. 配置ssh免密登录:
sudo apt install ssh
sudo apt install pdsh
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys
ssh localhost
  1. 下载Hadoop运行文件,有src和binary两种选择,src适合要对hadoop做修改的人群,binaray可以直接运行,一般下载binaray即可。3.3.4解压并进入hadoop文件夹。
  2. 修改etc/hadoop-env.sh,根据自己情况修改
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export JRE_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre
export HDFS_NAMENODE_USER="root"
export HDFS_DATANODE_USER="root"
export HDFS_SECONDARYNAMENODE_USER="root"
export YARN_RESOURCEMANAGER_USER="root"
export YARN_NODEMANAGER_USER="root"
  1. 配置系统环境变量/etc/profile(全局修改),/bash/.bashrc(当前用户更改),我选择全局更改,有如下:
export PDSH_RCMD_TYPE=ssh

export HADOOP_HOME=/home/***/下载/hadoop-3.3.4

export CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath --glob):$CLASSPATH

export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native:$LD_LIBRARY_PATH:$JAVA_HOME/jre/lib/amd64/server

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

export JRE_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre
  1. 启动并初始化单节点伪集群
    图片
#初始格式化
bin/hdfs namenode -format
#启动集群
sbin/start-dfs.sh
#关闭集群
sbin/stop-dfs.sh
  1. 尝试编译hdfs c++例子:g++ -o hdfs hdfs.cpp -L $HADOOP_HOME/lib/native -lhdfs -I $HADOOP_HOME/include -L $JAVA_HOME/jre/lib/amd64/server/ -ljvm
#include "hdfs.h"
#include "stdio.h"
//#hdfs.cpp
int main(int argc, char **argv)
{
    hdfsFS fs = hdfsConnect("localhost", 9000);
    if (!fs) {
        fprintf(stderr, "connect fail\n");
        return -1; 
    }   
    hdfsFile writeFile = hdfsOpenFile(fs, "/first.txt", O_WRONLY, 4096, 0, 0); 
    if (!writeFile) {
        fprintf(stderr, "openfile fali\n");
        return -1; 
    }   
    hdfsWrite(fs, writeFile, "hello hdfs", 10);
    hdfsCloseFile(fs, writeFile);
    hdfsDisconnect(fs);
    return 0;
}

结果入图:
图片-1660809488548


hadoop安装和libhdfs使用
https://blog.427221.xyz/archives/hadoop-an-zhuang-he-libhdfs-shi-yong
作者
qqq
发布于
更新于
许可