adhocshare

Knowledge is Power

Author: robin

0

Windows

Chocolatey package manager for Windows

You will find a few similarities with Homebrew (https://brew.sh/) and linuxbrew (http://linuxbrew.sh/). Chocolatey (https://chocolatey.org/) is a similar package manager for windows.   Installation Installation command is a little weird than other OS. Inside an administrative command prompt, type below >”%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe” -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command “iex ((New-Object System.Net.WebClient).DownloadString(‘https://chocolatey.org/install.ps1’))” && SET “PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin”...

0

Uncategorized

Etrade Pro Strategy Scanner

WN=Under+$5,+High+Vol+High+Volatility&Sh_CMU=on&Sh_BBU=on&Sh_SV=on&QSV=10&Sh_NR7_1=on&MaxPrice=5&MinVol3M=500000&MinRV=1.5&MinVWVP=0.3&MinRange120P=2&MinRange5DP=3&MinFCD=0.01&MinBB=1.5&X_NYSE=on&X_ARCA=on&X_AMEX=on&XN=on   The objective Use the above etrade scanner for finding, Stocks NOT on the OTCBB or Pinks, US big board stocks only, SP between $.01 and $5.00. Bullish stocks having high volume and volatility This is good for taking...

0

Bigdata / Hadoop

Hive msck repair not working

Okay, so msck repair is not working and you saw something as below, 0: jdbc:hive2://hive_server:10000> msck repair table mytable; Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask (state=08S01,code=1) It seems to appear because of higher...

0

Bigdata / Hadoop

​DistCp Between HA Clusters

Source: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_administration/content/distcp_between_ha_clusters.html ​DistCp Between HA Clusters To copy data between HA clusters, use the dfs.internal.nameservices property in the hdfs-site.xml file to explicitly specify the name services belonging to the local cluster, while continuing to use the dfs.nameservices property to specify all of the name services in...

0

Bigdata / Hadoop

Hive export to CSV

Bash function to export Hive table data to local CSV file Usage: hive_export_csv <db.table> <output.csv> [queue] Recommendation: Add to .bash_profile   hive_export_csv () { if [ -z “$2” ]; then echo “Bad arguments. Usage: ${FUNCNAME[0]} <db.table> <output.csv> [queue]” else uuid=$(uuidgen)...

0

Bigdata / Hadoop

HiveAccessControlException Permission denied user [user] does not have [WRITE] privilege on …

Source: https://community.hortonworks.com/questions/112754/insert-overwrite-directory-beeline.html Error Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [user] does not have [WRITE] privilege on [/tmp/*] (state=42000,code=40000) The above error appears, even though you’ve setup ranger policies, hdfs policies are set up. You’ve checked everything and...

0

Programming

python web scraping (stockfetcher.com pull)

Python code below, It fetches all contents from 5 urls, general discussion, filter exchange, public filters, stock picks and indicators. Destination dir: /tmp File type: csv   #!/usr/bin/env python from lxml import html import requests import csv import sys import...

0

Bigdata / Hadoop

How a newline can ruin your Hive

Source: http://marcel.is/how-newline-can-ruin-your-hive/ If you do not fully understand how Hive/Impala stores your data, it might cost you badly. Symptom #1: Weird values in ingested Hive table You double-checked with select distinct(gender) from customers that the gender column in your source RDBMS really contains only values male, female and NULL....

0

Bigdata

Elasticsearch tips and tricks

Find record having max value for a field Get latest record from Elasticsearch Latest record with ES _timestamp value in results Get record count from last x mins Max value GET http://elasticsearch-server:9200/my_index_name_*/_search?size=0 { “aggs” : { “max_timestamp” : { “max”...

0

Hadoop

Hadoop hdfs tips and tricks

Finding active namenode in a cluster   Active namenode in a cluster # lookup active nn nn_list=`hdfs getconf -namenodes` echo Namenodes found: $nn_list active_node=‘’ #for nn in $( hdfs getconf -namenodes ); do for nn in $nn_list ; do echo...