Apache Hadoop Installation

This guide is a collection or a summary on how to install and use a footprint of Apache Hadoop. I tried to follow an old version 2.7.1 guide that I created few years ago and adjusted this to use the latest version. Apache Hadoop 3.5.0 is used below; check the Apache releases page before future
installations.

These instructions target Linux (Ubuntu/Debian) for development or testing.
Production clusters need Kerberos, network controls, encryption, monitoring,
backups, and an upgrade plan. Do not expose HDFS or YARN ports to the internet.

Native single-node installation

Prerequisites

sudo apt-get update
sudo apt-get install -y openjdk-17-jdk openssh-client openssh-server pdsh curl tar
java -version

Enter fullscreen mode Exit fullscreen mode

Hadoop requires Java and SSH; pdsh is recommended by the current Apache
single-node documentation. Find JAVA_HOME if needed:

readlink -f "$(command -v java)" | sed 's:/bin/java::'

Enter fullscreen mode Exit fullscreen mode

Download and install

Pin the version for repeatable installs and verify Apache’s SHA-512 checksum:

export HADOOP_VERSION=3.5.0
cd /tmp
curl -fLO "https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz"
curl -fLO "https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz.sha512"
sha512sum -c "hadoop-${HADOOP_VERSION}.tar.gz.sha512"
sudo tar -xzf "hadoop-${HADOOP_VERSION}.tar.gz" -C /opt
sudo ln -sfn "/opt/hadoop-${HADOOP_VERSION}" /opt/hadoop
sudo chown -R "$USER":"$USER" "/opt/hadoop-${HADOOP_VERSION}"

Enter fullscreen mode Exit fullscreen mode

Add this to ~/.bashrc, adjusting JAVA_HOME if necessary:

export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
export HADOOP_HOME=/opt/hadoop
export HADOOP_CONF_DIR="$HADOOP_HOME/etc/hadoop"
export HADOOP_HDFS_HOME="$HADOOP_HOME"
export HADOOP_YARN_HOME="$HADOOP_HOME"
export HADOOP_MAPRED_HOME="$HADOOP_HOME"
export PATH="$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin"

Enter fullscreen mode Exit fullscreen mode

Then load and verify it:

source ~/.bashrc
sed -i "s|^# export JAVA_HOME=.*|export JAVA_HOME=${JAVA_HOME}|" "$HADOOP_HOME/etc/hadoop/hadoop-env.sh"
hadoop version

Enter fullscreen mode Exit fullscreen mode

Configure pseudo-distributed mode

mkdir -p /opt/hadoop-data/{name,data}

cat > "$HADOOP_HOME/etc/hadoop/core-site.xml" <<'EOF'
<configuration>
  <property><name>fs.defaultFS</name><value>hdfs://localhost:9000</value></property>
</configuration>
EOF

cat > "$HADOOP_HOME/etc/hadoop/hdfs-site.xml" <<'EOF'
<configuration>
  <property><name>dfs.replication</name><value>1</value></property>
  <property><name>dfs.namenode.name.dir</name><value>file:///opt/hadoop-data/name</value></property>
  <property><name>dfs.datanode.data.dir</name><value>file:///opt/hadoop-data/data</value></property>
</configuration>
EOF

cat > "$HADOOP_HOME/etc/hadoop/mapred-site.xml" <<'EOF'
<configuration>
  <property><name>mapreduce.framework.name</name><value>yarn</value></property>
  <property><name>mapreduce.application.classpath</name><value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value></property>
</configuration>
EOF

cat > "$HADOOP_HOME/etc/hadoop/yarn-site.xml" <<'EOF'
<configuration>
  <property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property>
  <property><name>yarn.nodemanager.env-whitelist</name><value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME,HADOOP_HOME,HADOOP_MAPRED_HOME,PATH,LANG,TZ</value></property>
</configuration>
EOF

Enter fullscreen mode Exit fullscreen mode

The data paths may be moved to a separate disk. Do not use /tmp for production
data.

Enable local SSH and start Hadoop

sudo systemctl enable --now ssh
test -f ~/.ssh/id_ed25519 || ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_ed25519
cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
ssh localhost exit

Enter fullscreen mode Exit fullscreen mode

Formatting destroys NameNode metadata. Run this only for a new test cluster:

hdfs namenode -format
start-dfs.sh
start-yarn.sh

Enter fullscreen mode Exit fullscreen mode

Web interfaces:

• NameNode: http://localhost:9870/
• ResourceManager: http://localhost:8088/

Run a test MapReduce job:

hdfs dfs -mkdir -p "/user/$USER" input
hdfs dfs -put "$HADOOP_HOME/etc/hadoop"/*.xml input
hadoop jar "$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-${HADOOP_VERSION}.jar" grep input output 'dfs[a-z.]+'
hdfs dfs -cat output/*

Enter fullscreen mode Exit fullscreen mode

Check and stop services:

jps
hdfs dfsadmin -report
stop-yarn.sh
stop-dfs.sh

Enter fullscreen mode Exit fullscreen mode

Logs are normally in $HADOOP_HOME/logs.

Docker installation

Docker avoids installing Java and Hadoop on the host. Apache publishes
apache/hadoop:3.5.0:

docker pull apache/hadoop:3.5.0
docker run --rm -it --name hadoop --hostname hadoop apache/hadoop:3.5.0 bash

Enter fullscreen mode Exit fullscreen mode

For HDFS/YARN as multiple containers, use Apache’s Compose setup. It is
intended for local testing and can be scaled with DataNodes:

git clone --depth 1 --branch branch-3.5 https://github.com/apache/hadoop.git
cd hadoop
mvn clean install -Dmaven.javadoc.skip=true -DskipTests -DskipShade -Pdist,src
cd hadoop-dist/target/hadoop-3.5.0/compose/hadoop
docker compose up -d --scale datanode=3
docker compose ps
docker compose exec datanode hdfs dfs -mkdir -p /test
docker compose exec datanode hdfs dfs -ls /
docker compose down

Enter fullscreen mode Exit fullscreen mode

The Compose setup is preferable for testing a real NameNode/DataNode topology.
Use explicit persistent volumes for NameNode metadata and DataNode storage if
data must survive container recreation. A single disposable container is not a
production cluster.

Resource sizing

The PDF’s example for 4 cores, 4 GB RAM, and two disks reserves 1 GB for the
system and calculates two 512 MB containers using:

containers = min(2 * cores, 1.8 * disks, available RAM / minimum container size)

Enter fullscreen mode Exit fullscreen mode

This is only a planning example. Do not configure more YARN memory than the
machine or Docker VM has; leave room for Hadoop JVMs and the operating system.

Common abbreviations in the Hadoop guide:

Term Meaning HDFS Hadoop Distributed File System YARN Yet Another Resource Negotiator SSH Secure Shell JVM Java Virtual Machine XML Extensible Markup Language RPC Remote Procedure Call UI User Interface HTTP Hypertext Transfer Protocol CPU Central Processing Unit RAM Random Access Memory OS Operating System VM Virtual Machine PPA Personal Package Archive SHA-512 Secure Hash Algorithm with a 512-bit digest

Some Hadoop components are names rather than abbreviations:

• MapReduce — Hadoop’s distributed data-processing model.
• NameNode — manages HDFS metadata.
• DataNode — stores HDFS data blocks.
• ResourceManager — manages cluster resources.
• NodeManager — manages resources on an individual node.