Hive msck repair not working
Okay, so msck repair is not working and you saw something as below,
0: jdbc:hive2://hive_server:10000> msck repair table mytable; Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask (state=08S01,code=1)
It seems to appear because of higher number of partitions – https://issues.apache.org/jira/browse/HIVE-12859
This is what worked for me,
set hive.msck.path.validation=ignore;
MSCK REPAIR TABLE table_name;
robin@hive_server:~$ hive --hiveconf hive.msck.path.validation=ignore hive> use mydatabase; OK Time taken: 1.084 seconds hive> msck repair table mytable; OK Partitions not in metastore: mytable:location=00S mytable:location=03S Repair: Added partition to metastore mytable:location=00S Repair: Added partition to metastore mytable:location=03S Time taken: 22.039 seconds, Fetched: 1277 row(s) hive>
References
- https://stackoverflow.com/questions/40043986/hdinsight-hive-msck-repair-table-table-name-throwing-error
- https://stackoverflow.com/questions/38886511/handle-subfolders-after-partitions-in-hive