Hortonworks Data Platform Installation errata – Missing manual
- Pre-requisites
- Creating service users and databases in MySQL
- JDBC connector error during ambari-server setup with MySQL
- MySql connection failing during ambari automated installation
Some useful bash scripts
- Create database and user on mysql for services like ambari, oozie, hue, etc.
1. Pre-requisites
- Setup password-less ssh for root user
- Install some basic packages. Some could be environment specific, for e.g. mysql
yum install -y ntp telnet curl wget unzip mysql-server mysql-connector-java
- On CentOS 6, disable THP
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
- Configure services
chkconfig iptables off chkconfig mysqld on chkconfig ntpd on service mysqld start service ntpd start service iptables stop
- Install Oracle JDKhttp://www.oracle.com/technetwork/java/javase/downloads/index.html
- Ambari.repo format
#VERSION_NUMBER=2.2.2.0-460 [Updates-ambari-2.2.2.0] name=ambari-2.2.2.0 - Updates #baseurl=http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.2.0 baseurl=http://192.168.56.1:8000/AMBARI-2.2.2.0/centos6/2.2.2.0-460 gpgcheck=0 gpgkey=http://public-repo-1.hortonworks.com/ambari/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1
- If using python http server, fix repo URLs
http://192.168.56.1:8000/AMBARI-2.2.2.0/centos6/2.2.2.0-460 http://192.168.56.1:8000/centos6-hdp-2.3.4.7/HDP/centos6/2.x/updates/2.3.4.7 http://192.168.56.1:8000/centos6-hdp-2.3.4.7/HDP-UTILS-1.1.0.20/repos/centos6
2. Creating service users and databases in MySQL
Automation script in this article below.
CREATE USER '<SERVICEUSER>'@'%' IDENTIFIED BY '<SERVICEPASSWORD>'; GRANT ALL PRIVILEGES ON *.* TO '<SERVICEUSER>'@'%'; CREATE USER '<SERVICEUSER>'@'localhost' IDENTIFIED BY '<SERVICEPASSWORD>'; GRANT ALL PRIVILEGES ON *.* TO '<SERVICEUSER>'@'localhost'; CREATE USER '<SERVICEUSER>'@'<SERVERFQDN>' IDENTIFIED BY '<SERVICEPASSWORD>'; GRANT ALL PRIVILEGES ON *.* TO '<SERVICEUSER>'@'<SERVERFQDN>'; FLUSH PRIVILEGES;
Login with the same user using a password to verify, then create the database
$ mysql -u <SERVICEUSER> -p CREATE DATABASE <SERVICEDATABASE>; USE <SERVICEDATABASE>;
3. JDBC connector error during ambari-server setup with MySQL
ambari-server setup ... ... Enter advanced database configuration [y / n] (n)? Y ================================================== ============================ Choose one of the following options: [1] - PostgreSQL (Embedded) [2] - Oracle [3] - MySQL [4] - PostgreSQL ================================================== ============================ Enter choice (1): 3 Hostname (localhost): Port (3306): Database Name (ambari): Username (ambari): Enter Database Password (bigdata): WARNING: Before starting Ambari Server, you must copy the MySQL JDBC driver JAR file to /usr/share/java. Press to continue. ERROR: Before starting Ambari Server, you must copy the MySQL JDBC driver JAR file to /usr/share/java. ERROR: Exiting with exit code -1. REASON: Before starting Ambari Server, you must copy the MySQL JDBC driver JAR file to /usr/share/java.
To fix the error install mysql-connector-java using yum
yum install -y mysql-connector-java
4. MySql connection failing during ambari automated installation
Check if the jdbc driver has been registered with ambari-server
sudo ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
Some useful bash scripts
Create database and user on mysql for services like ambari, oozie, hue, etc.
usage: sudo ./create_db-2.sh -u userambari -p ambari -d ambari -s hdp2-3-4-7-c7.sandbox
if [[ $EUID -ne 0 ]]; then echo "This script must be run as root" 1>&2 exit 1 fi while [[ $# > 1 ]] do key="$1" case $key in -u|--user) _SERVICEUSER="$2" shift # past argument ;; -p|--password) _SERVICEPASSWORD="$2" shift # past argument ;; -d|--database) _SERVICEDATABASE="$2" shift # past argument ;; -s|--server) _SERVERFQDN="$2" shift # past argument ;; *) # unknown option printf "Usage: %s: [-u <user> -p <password> -d <database name> -s <server fqdn> \n" $0 exit 2 ;; esac shift # past argument or value done if [ -z $_SERVICEUSER ] || [ -z $_SERVICEPASSWORD ] || [ -z $_SERVICEDATABASE ] || [ -z $_SERVERFQDN ] ; then printf "Error: Missing arguments.\nUsage: %s: [-u <user> -p <password> -d <database name> -s <server fqdn>]\n" $0 exit 2 fi echo -e "### Following commands will be executed in mysql. ###\n" echo "CREATE DATABASE $_SERVICEDATABASE;" echo "CREATE USER '$_SERVICEUSER'@'%' IDENTIFIED BY '$_SERVICEPASSWORD';" echo "GRANT ALL PRIVILEGES ON $_SERVICEDATABASE.* TO '$_SERVICEUSER'@'%';" echo "CREATE USER '$_SERVICEUSER'@'localhost' IDENTIFIED BY '$_SERVICEPASSWORD';" echo "GRANT ALL PRIVILEGES ON $_SERVICEDATABASE.* TO '$_SERVICEUSER'@'localhost';" echo "CREATE USER '$_SERVICEUSER'@'$_SERVERFQDN' IDENTIFIED BY '$_SERVICEPASSWORD';" echo "GRANT ALL PRIVILEGES ON $_SERVICEDATABASE.* TO '$_SERVICEUSER'@'$_SERVERFQDN';" echo -e "FLUSH PRIVILEGES;\n" read -p "### Continue (y/n)? " answer case ${answer:0:1} in y|Y ) eval $(echo mysql -u root -e \""CREATE DATABASE $_SERVICEDATABASE;\"") eval $(echo mysql -u root -e \""CREATE USER '$_SERVICEUSER'@'%' IDENTIFIED BY '$_SERVICEPASSWORD';\"") eval $(echo mysql -u root -e \""GRANT ALL PRIVILEGES ON $_SERVICEDATABASE.* TO '$_SERVICEUSER'@'%';\"") eval $(echo mysql -u root -e \""CREATE USER '$_SERVICEUSER'@'localhost' IDENTIFIED BY '$_SERVICEPASSWORD';\"") eval $(echo mysql -u root -e \""GRANT ALL PRIVILEGES ON $_SERVICEDATABASE.* TO '$_SERVICEUSER'@'localhost';\"") eval $(echo mysql -u root -e \""CREATE USER '$_SERVICEUSER'@'$_SERVERFQDN' IDENTIFIED BY '$_SERVICEPASSWORD';\"") eval $(echo mysql -u root -e \""GRANT ALL PRIVILEGES ON $_SERVICEDATABASE.* TO '$_SERVICEUSER'@'$_SERVERFQDN';\"") eval $(echo mysql -u root -e \""FLUSH PRIVILEGES;\"") ;; * ) echo Exiting. ;; esac