Configuring and Starting Hadoop, Stratosphere and Spark Servers for bioKepler or DDP

Hadoop

Hadoop module 1.1

Hadoop module 1.1 is developed based on Hadoop 2.2.0. The BioKepler and DDP suites include the binaries, libraries, and configuration files necessary to run a Hadoop server. These files are located in $HOME/KeplerData/workflows/module/hadoop-1.1.0/tools.

The following steps describe how to configure and start the Hadoop server included in the bioKepler/DDP. If Hadoop does not start, look at the log files in logs/. 

Linux and Mac

1. Set JAVA_HOME in etc/hadoop/hadoop-env.sh to where java is installed on your computer.
2. Make sure all files in bin/ and sbin/ and etc/ are executable.

 chmod a+x bin/* sbin/* etc/*

3. Before starting Hadoop for the first time, format the namenode by running:

 bin/format-namenode.sh

4.  Start Hadoop.

bin/start-hadoop.sh

1.  Under Mac OS X, if start-hadoop.sh fails with "localhost: ssh: connect to host localhost port 22: Connection refused", then go to the System Preferences, select Sharing and then enable Remote Login.
5. Stop Hadoop:

bin/stop-hadoop.sh

Windows

1. Download and install Cygwin, including OpenSSH and Open ssl packages in "C:\cygwin" and update your PATH environment variable:

   c:\cygwin\bin;c:\cygwin\usr\bin

2. Hadoop requires password-less SSH access to manage its nodes. Set up authorization keys to be used by hadoop when ssh’ing to localhost:

ssh-keygen -t rsa -P ""
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

3. Set JAVA_HOME parameter in conf/hadoop-env.sh (line  8) to where java is installed on your computer.

4. Set cygwin path translation in bin/hadoop-config.sh  (line 184):

# cygwin path translation
if $cygwin; then
    JAVA_HOME=`cygpath -w "$JAVA_HOME"`
    CLASSPATH=`cygpath -wp "$CLASSPATH"`
    HADOOP_HOME=`cygpath -w "$HADOOP_HOME"`
    HADOOP_LOG_DIR=`cygpath -w "$HADOOP_LOG_DIR"`
    JAVA_LIBRARY_PATH=`cygpath -w "$JAVA_LIBRARY_PATH"`
    TOOL_PATH=`cygpath -wp "$TOOL_PATH"`
fi

5. Before starting Hadoop for the first time, format the namenode by running:

./bin/format-namenode.sh

6.  Start Hadoop:

bin/start-hadoop.sh

7.  Stop Hadoop:

bin/stop-hadoop.sh

 

Hadoop module 1.0

The BioKepler and DDP suites include the binaries, libraries, and configuration files necessary to run a Hadoop server. These files are located in $HOME/KeplerData/workflows/module/hadoop-1.0.0/tools.

The following steps describe how to configure and start the Hadoop server included in the bioKepler/DDP. If Hadoop does not start, look at the log files in logs/. 

Linux and Mac

1. Set JAVA_HOME in conf/hadoop-env.sh to where java is installed on your computer.
2. Make sure all files in bin/ and conf/hadoop-env.sh are executable.
3. Before starting Hadoop for the first time, format the namenode by running:

 echo "Y" | bin/hadoop namenode -format

4.  Start Hadoop:

bin/start-hadoop.sh

Windows

1. Download and install Cygwin, including OpenSSH and Open ssl packages in "C:\cygwin" and update your PATH environment variable:

   c:\cygwin\bin;c:\cygwin\usr\bin

2. Hadoop requires password-less SSH access to manage its nodes. Set up authorization keys to be used by hadoop when ssh’ing to localhost:

ssh-keygen -t rsa -P ""
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

3. Set JAVA_HOME parameter in conf/hadoop-env.sh (line  8) to where java is installed on your computer.

4. Set cygwin path translation in bin/hadoop-config.sh  (line 184):

# cygwin path translation
if $cygwin; then
    JAVA_HOME=`cygpath -w "$JAVA_HOME"`
    CLASSPATH=`cygpath -wp "$CLASSPATH"`
    HADOOP_HOME=`cygpath -w "$HADOOP_HOME"`
    HADOOP_LOG_DIR=`cygpath -w "$HADOOP_LOG_DIR"`
    JAVA_LIBRARY_PATH=`cygpath -w "$JAVA_LIBRARY_PATH"`
    TOOL_PATH=`cygpath -wp "$TOOL_PATH"`
fi

5. Before starting Hadoop for the first time, format the namenode by running:

./bin/hadoop namenode –format

6.  Start Hadoop:

bin/start-hadoop.sh

 

Stratosphere

Linux and Mac