Knowledge is Power
Stuck with creating Hive tables on compressed files? Well the documentation on apache.org suggests that Hive natively supports compressed file – https://cwiki.apache.org/confluence/display/Hive/CompressedStorage Lets try that out. Store a snappy compressed file on HDFS. … thinking, I do not have such file… Wait!...
There are times when we want to parse a string that is actually a JSON. Usually that could be done with built in functions of Hive such as get_json_object(). Though get_json_object cannot parse JSON Array from my experience. These array...
Using JsonSerDe in Hive Download JSON Serde – https://github.com/rcongiu/Hive-JSON-Serde Compile command for hive 1.2.1 – “mvn -Pcdh5 -Dcdh5.hive.version=1.2.1 clean package” . change hive version per the environment Copy json-serde/target/json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar (or similar) to hive/lib Restart hive Sample JSON with test HiveQLs...
Source: https://community.hortonworks.com/articles/16846/how-to-identify-what-is-consuming-space-in-hdfs.html Script #!/usr/bin/env bash max_depth=5 largest_root_dirs=$(hdfs dfs -du -s ‘/*’ | sort -nr | perl -ane ‘print “$F[1] “‘) printf “%15s %s\n” “bytes” “directory” for ld in $largest_root_dirs; do printf “%15.0f %s\n” $(hdfs dfs -du -s $ld| cut -d’ ‘...
Source: https://access.redhat.com/documentation/en-US/Fuse_MQ_Enterprise/7.1/html/Security_Guide/files/LDAP-AddUserEntries.html Overview The basic prerequisite for using LDAP authentication in the broker is to have an X.500 directory server running and configured with a collection of user entries and group entries. For users who are unfamiliar with X.500 directory...
Source: http://krams915.blogspot.com/2011/01/ldap-apache-directory-studio-basic.html In this tutorial we will setup a basic LDAP structure containing users and roles. We will be using the excellent Apache Directory Studio IDE. This tutorial will be the basis for our other Spring LDAP integration tutorials. What is...
Following expect script uses beeline interface to fetch statistics of tables within a database. Use username and queuename with your environment values. #!/usr/bin/expect -f # hive_statistics, v0.1, 2016-05, [email protected] # Usage: ./hive_statistics [database_name] set _database [lindex $argv 0] if {...
Following shell script will get the schema information from Hive using WebHCat server. #!/bin/sh # fetch_webhcat.sh, v0.1, 2016-04-00, [email protected] # Pre-requisites: jq, curl, python (json.tool) _WEBHCAT_SERVER=”server:50111″ _USER_NAME=”JohnDoe” while [[ $# > 1 ]] do key=”$1″ case $key in -u|–user)...
Following is an awk script that I use in TextWrangler as a Text Filter. This script generates the required awk and grep commands to lookup /etc/passwd file. #!/bin/sh # gawk ‘{match($0,”([a-zA-Z]+).acl_submit_applications=(.*)”,a); if(a[1] != “”) print a[1] “\t” a[2] }’ #...
Source: https://pulsatingtaurus.wordpress.com/2015/02/15/clean-uninstall-hortonworks-hdp-2-2/ I love Hadoop and Hortonworks is one of my favored Hadoop distributuion. However while experimenting with the hadoop installation, I had many instances when I needed to start afresh on the set of physical as well as virtual Hadoop cluster. Hortonworks provide...
More
