Author: robin

0

python web scraping (finviz pull)

Python code below, #!/usr/bin/env python from lxml import html import requests import csv import sys import os import humanize # Helper URL # http://python-docs.readthedocs.io/en/latest/scenarios/scrape.html curr_arg = 1 for arg in sys.argv: print “Argument ” + str(curr_arg) + “: ” +...

0

IntelliJ IDEA tips & tricks

Setting JAVA version Coming soon…   1. Setting JAVA version Method 1 press:CTRL + SHIFT + A (For Mac command + shift + A) type: java compiler press: ENTER In the setting window set the Target bytecode to 1.8 Method...

0

Bash scripting tricks

Bash tricks Check bash version in script if ((BASH_VERSINFO[0] < 4)) then echo “Sorry, you need at least bash-4.0 to run this script.” exit 1 fi   Reading columns from a file in bash array table_collection=( $(cut -d $’\t’ -f1...

0

Old bash version on MacOS ?

What? An older bash version? Are you seeing and still not believing on something like below, MacBook-Pro:~$ bash –version GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin16) Copyright (C) 2007 Free Software Foundation, Inc. Well that is my MBP as of Mar 29,...

0

Replace invalid code signature in Mac apps

What? Replacing the invalid code signature in mac apps with a new one How? Example: sudo codesign –force –sign – /Applications/DBeaver.app Symptoms MacOS gives messages like – “Dbeaver” is damaged and can’t be opened. You should move it to the Trash.  ...

0

Nested collections in Hive

1, 2 & 3 .. Lets go! 1. SHELL echo “1345653,110909316904:1341894546|221065796761:1341887508” > /tmp/20170317_array_inputfile.txt hdfs dfs -mkdir -p /tmp/20170317/array_test/input hdfs dfs -put /tmp/20170317_array_inputfile.txt /tmp/20170317/array_test/input rm /tmp/20170317_array_inputfile.txt 2. HIVE drop table SAMPLE; CREATE external TABLE SAMPLE( id BIGINT, record array<struct<col1:string,col2:string>> )row format...

0

Connecting to Apache Phoenix

  Syntax used for connecting to Hbase using sqlline through phoenix is as listed below. ./sqlline.py 10.10.20.60:2181:/hbase-unsecure 10.10.20.60 – My Zookeeper 2181 – Zookeeper client port hbase-unsecure – value configured as zookeeper.znode.parent in hbase-site.xml   Sample query /usr/hdp/current/phoenix-client/bin/psql.py zookeeperserver:2181:/hbase-unsecure /usr/hdp/current/phoenix-client/doc/examples/WEB_STAT.sql...

0

Hive Vertex failure

Vertex failure while running Hive queries? Let’s see what can be done… Not sure..Change, hive.fetch.task.conversion=more; to hive.fetch.task.conversion=none;   Was the data on hdfs in ORC files? and error being similar to below? Vertex failed, vertexName= at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java: ) Try changing...

0

Fixing large mysql ibdata1 resulting from ranger audits

Table Partitioning in MySQL: (Version 5.1.6 or above) Note: Before starting backup/restore please stop all running application which usage XA_ACCESS_AUDIT table. this will be help for keeping snapshot of XA_ACCESS_AUDIT for particular timestamp. Table Partitioning in MySQL:- Partitioned tables created...