Monthly Archive: September 2016

0

brew packages and cask packages

brew packages Basic Apps $ brew install bash $ brew install bash-completion $ brew install maven $ brew install openssl $ brew install ssh-copy-id $ brew install wget $ brew install gawk Big data Apps $ brew install hadoop $...

0

Setting up password-less ssh across all nodes in a cluster

Pre-requisites User account for which passwordless ssh will be setup, should be present on all nodes Password of the account should be same across all nodes pdsh and ssh-copy-id commands should be available Prepare 2 files file_of_hosts.txt – containing all...

0

Hive on Tez Performance Tuning – Determining Reducer Counts

Source: https://community.hortonworks.com/articles/22419/hive-on-tez-performance-tuning-determining-reducer.html   Short Description: Some practical steps in Hive Tez tuning Article How Does Tez determine the number of reducers? How can I control this for performance? In this article, I will attempt to answer this while executing and tuning...

0

Hive query tips

Date operations Data operations Headers in Beeline Unlock hive tables Check partitions used in hive query   Debugging Hive Long (query length) queries submitted to Hive Occurrence of thread printing in hiveserver2 log file Capture classes used in hiveserver2 log...

0

Adding compression codec to Hortonworks data platform

Lately I tried installing xz/lzma codec on my local VM setup. The compression ratios are pretty awesome. Won’t do a benchmark here, try it out yourself 😉   Steps Download codec JAR – https://github.com/yongtang/hadoop-xz or https://mvnrepository.com/artifact/io.sensesecure/hadoop-xz Copy downloaded JAR to HDPs’ libs...

0

Good looking .hiverc file

Following is the .hiverc from one of the hadoop environments I work on, — additional .jar includes like the one below — add jar hdfs://ualprod/tmp/json-serde-1.3.7-jar-with-dependencies.jar; set hive.exec.dynamic.partition.mode=nonstrict; set hive.auto.convert.join.noconditionaltask=true; set hive.optimize.sort.dynamic.partition=true; set hive.exec.max.dynamic.partitions=100000; set hive.exec.max.dynamic.partitions.pernode=10000; — large mem?? set hive.tez.container.size=10240;...

0

Fix SNX SSL Network Extender on Mac

Apr 2017 As of Apr, 2017 SNX + Firefox ESR, fails to connect with Java version 1.8.0_131. Resolution Downgrade Java version to 1.8.0.121 Steps Delete JDK, $ sudo rm -rf /Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk Delete JavaAppletPlugin.plugin, $ sudo rm -rf /Library/Internet Plug-Ins/JavaAppletPlugin.plugin Download older version...

0

Kafka on OSX / macOS

Source: https://dtflaneur.wordpress.com/2015/10/05/installing-kafka-on-mac-osx/   Apache Kafka is a highly-scalable publish-subscribe messaging system that can serve as the data backbone in distributed applications. With Kafka’s Producer-Consumer model it becomes easy to implement multiple data consumers that do live monitoring as well persistent...

0

top command on linux

Usage Basic $ top View by process name # top -p $(pgrep -d’,’ <process name>) # sample below, top -p $(pgrep -d’,’ http) View by user top -u <user name>   Sort by mem # Inside top Shift+m View command...

0

Pretty print JSON on mac

Using python with json.tool, $python -m json.tool <filename.json> Or use something like in line 2, below echo ‘{“employee_id”:1,”full_name”:”Sheri Nowmer”,”first_name”:”Sheri”,”last_name”:”Nowmer”,”position_id”:1,”position_title”:”President”,”store_id”:0,”department_id”:1,”birth_date”:”1961-08-26″,”hire_date”:”1994-12-01 00:00:00.0″,”end_date”:null,”salary”:80000.0000,”supervisor_id”:0,”education_level”:”Graduate Degree”,”marital_status”:”S”,”gender”:”F”,”management_role”:”Senior Management”}’ > /tmp/emp.json cat /tmp/emp.json | python -m json.tool     Related posts: AWK quick reference How-to: Install Hue...