Knowledge is Power
————————————————————————- HANDY ONE-LINERS FOR AWK (Unix stream editor) Apr. 17, 2001 compiled by Eric Pement <[email protected]> version 0.1 Latest version of this file is usually at: http://www.student.northpark.edu/pemente/awk/awk1line.txt USAGE: Unix: awk ‘/pattern/ {print “$1”}’ # standard Unix shells DOS/Win: awk ‘/pattern/...
Source: http://beekeeperdata.com/posts/hadoop/2015/07/26/Hive-UDTF-Tutorial.html Author: Matthew Rathbone Co-author: Elena Akhmatova Article Hadoop Hive UDTF Tutorial – Extending Apache Hive with Table Functions While working with both Primitive types and Embedded Data Structures was discussed in part one, the UDF interfaces are limited to...
Yarn jobs hangs on local mac environment Missing node in Resource manager UI There are no nodes in the node list https://www.robin.eu.org:8088/cluster/nodes File: yarn-site.xml Location: /usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/yarn-site.xml <property> <name>yarn.resourcemanager.hostname</name> <value>localhost</value> </property> Resource Manager UI displaying ipv6 for local address Hadoop does...
Source: https://community.hortonworks.com/articles/42695/how-to-create-a-hive-udf-in-scala.html This article will focus on creating a custom HIVE UDF in the Scala programming language. Intellij IDEA 2016 was used to create the project and artifacts. Creation and testing of the UDF was performed on the Hortonworks...
Looking to add custom SerDe and custom or third party codecs to Hortonworks HDP? Only auxlib folder trick worked for me after having tried lot of alternatives. The places where we need to add that auxlib folder containing JARs is,...
Problems Following are some problems we might come across while working in a large setup of hadoop clusters, Namenode restarts taking long time (http://nn-host:50070/dfshealth.html#tab-startup-progress) Namenode startup goes to safemode for a long time after restart Best practices for Namenode &...
At times we need to setup local development domains for our test environment. I use it to maintain a local copy of my hosted website. It is very convenient to make changes to local and then export to hosted website...
brew packages Basic Apps $ brew install bash $ brew install bash-completion $ brew install maven $ brew install openssl $ brew install ssh-copy-id $ brew install wget $ brew install gawk Big data Apps $ brew install hadoop $...
Pre-requisites User account for which passwordless ssh will be setup, should be present on all nodes Password of the account should be same across all nodes pdsh and ssh-copy-id commands should be available Prepare 2 files file_of_hosts.txt – containing all...
Source: https://community.hortonworks.com/articles/22419/hive-on-tez-performance-tuning-determining-reducer.html Short Description: Some practical steps in Hive Tez tuning Article How Does Tez determine the number of reducers? How can I control this for performance? In this article, I will attempt to answer this while executing and tuning...
More
