Knowledge is Power
Source: https://docs.hortonworks.com/HDPDocuments/SS1/SmartSense-1.4.0/bk_installation/content/SSL_troubleshooting.html SmartSense SSL Troubleshooting SmartSense components use SSL for protecting communications between the HST server and agents, and between the HST server and SmartSense Gateway. If installation issues arise, you can reset these SSL certificates. HST Server To reset the...
Compressing files in hadoop Okay, well.. It may or may not be the fastest. Email me if you find a better alternate 😉 Short background, The technique uses simple Pig script Make Pig use tez engine (set the queue name...
Ambari configuration over REST Ambari configuration over REST API Need to login to ambari Access below URL, http://ambari-host:8080/api/v1/services/AMBARI/components/AMBARI_SERVER Related posts: Adding compression codec to Hortonworks data platform Permanently add jars to hadoop HDFS disk consumption – Find what...
Source: https://discuss.pivotal.io/hc/en-us/articles/203272527-Namenode-failed-while-loading-fsimage-with-GC-overhead-limit-exceeded Namenode failed while loading fsimage with GC overhead limit exceeded Problem During startup namenode failed to load fsimage into memory 2014-05-14 17:36:56,806 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file /data/hadoop/nn/dfs/name/current/fsimage_0000000000252211550 using no compression 2014-05-14 17:36:56,806 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files =...
Below python code will help you extract statistics from a set of Sqoop log files for transfer analysis, #!/usr/bin/env python import fnmatch import os import datetime def find_files(directory, pattern): for root, dirs, files in os.walk(directory): for basename in files: if...
The current version of Ambari (2.4.2) does not provide an option to move the Storm Nimbus service from one node to another. However, the steps mentioned the following workaround can be used to perform this. Workaround To move Nimbus service...
NOTE It is not safe to move components like Journal node or Zookeeper. This method is to be used only for components like Storm, Kafka , Falcon or Flume etc. Following steps would help in moving components from one Node...
In case you would like to move JournalNode service to another host, here are the steps to do so: Put HDFS in safemode su hdfs c ‘hdfs dfsadmin fs hdfs://<active node>:8020 safemode enter’ Execute a save namespace of the...
Hive Lookup what killed Hive server $ grep –color=always -nr -B 1 ‘Exception|Service:HiveServer2 is started|java.lang.OutOfMemoryError’ /var/log/hive/hiveserver2.log | less -N Above command looks up the log file for exceptions and startup of hive and print one line above the search term....
1, 2 & 3 .. Lets go! 1. SHELL echo “1345653,110909316904:1341894546|221065796761:1341887508” > /tmp/20170317_array_inputfile.txt hdfs dfs -mkdir -p /tmp/20170317/array_test/input hdfs dfs -put /tmp/20170317_array_inputfile.txt /tmp/20170317/array_test/input rm /tmp/20170317_array_inputfile.txt 2. HIVE drop table SAMPLE; CREATE external TABLE SAMPLE( id BIGINT, record array<struct<col1:string,col2:string>> )row format...
More
