Configure log files on HDP platform

 

1. Kafka
2. Storm
3. Ranger
4. HDFS
5. Zookeeper
6. Oozie
7. Knox
8. Hive & Hive metastore

 

1. Kafka

Kafka currently uses org.apache.log4j.DailyRollingFileAppender, which doesn’t allow us to specify the max backup index, max file size. And by default, rolls every hour creating 24 files a day.

Appender: org.apache.log4j. RollingFileAppender
Roll criteria: Every 100MB.
Retention: 9 backup logs.

Steps:

1. Login to Ambari
2. Goto the config tab of the Kafka broker.
3. Expand Advanced kafka-log4j
4. Change the appender from org.apache.log4j.DailyRollingFileAppender to org.apache.log4j.RollingFileAppender.
5. Comment out the appender’s date pattern:

   #log4j.appender.kafkaAppender.DatePattern='.'yyyy-MM-dd-HH
   #log4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender

6. Add the following lines for each appender (change the appender name):

   log4j.appender.kafkaAppender=org.apache.log4j.RollingFileAppender
   log4j.appender.kafkaAppender.MaxFileSize=100MB
   log4j.appender.kafkaAppender.MaxBackupIndex=9

7. Repeat the steps for all the appenders in the section.
8. Rolling restart Kafka brokers for the changes to take effect.

Issues/Further Investigation:

The kafka.out doesn’t roll and grows huge. There is a open ticket with HortonWorks on this issue.

 

2. Storm

We need to set the log viewer settings to update the clean up frequency

logviewer.cleanup.age.mins – Value in mins for the worker logs to age for cleaning.

logviewer.cleanup.interval.secs – Value in secs for frequency for the logviewer cleanup process to trigger.

Steps:

1. Login to Ambari
2. Goto the config tab of the Kafka broker.
3. Expand Custom storm-site
4. Add the following two properties:

   logviewer.cleanup.age.mins : XXX (Value in mins for the worker logs to age for cleaning)
   logviewer.cleanup.interval.secs : XXX (Value in secs for frequency for the logviewer cleanup process to trigger)

5. Restart Storm components. (esp.. Supervisors – rolling restart)

 

3. Ranger

Ranger logs are divided into 2 parts: admin and usersync

admin – Ranger admin logs – access logs, xa log and xa sql log

usersync – User sync logs

SSH to the server where Ranger Admin is installed and make the following changes:

1. Admin logs : /usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/log4j.xml
   Change that to RollingFileAppender, see example below:

   <appender name="xa_log_appender">
   <param name="file" value="${catalina.base}/logs/xa_portal.log" />
   <param name="datePattern" value="'.'yyyy-MM-dd" />
   <param name="append" value="true" />
   <layout>
   <param name="ConversionPattern" value="%d [%t] %-5p %C{6} (%F:%L) - %m%n" />
   </layout>
   </appender>

   Change the above to:

   <appender name="xa_log_appender">
   <param name="file" value="${catalina.base}/logs/xa_portal.log" />
   <param name="maxFileSize" value="100MB" />
   <param name="maxBackupIndex" value="9" />
   <layout>
   <param name="ConversionPattern" value="%d [%t] %-5p %C{6} (%F:%L) - %m%n" />
   </layout>
   </appender>

   Repeat the process for sql_appender as well.

2. UserSync logs (usr/hdp/current/ranger-usersync/conf/log4j.xml):
   Currently it uses DailyRollingFileAppender which doesn’t clean up the old files.

   xm<appender name="logFile">
   <param name="file" value="${logdir}/usersync.log" />
   <param name="DatePattern" value="'.'yyyy-MM-dd" />
   <layout>
   <param name="ConversionPattern" value="%d{dd MMM yyyy HH:mm:ss} %5p %c{1} [%t] - %m%n"/>
   </layout>
   </appender>

   Change the above to:

   <appender name="logFile">
   <param name="file" value="${logdir}/usersync.log" />
   <param name="maxFileSize" value="100MB" />
   <param name="maxBackupIndex" value="9" />
   <layout>
   <param name="ConversionPattern" value="%d{dd MMM yyyy HH:mm:ss} %5p %c{1} [%t] - %m%n"/>
   </layout>
   </appender>

3. Restart Ranger.

 

4. HDFS

Audit Logs:

This log is currently configured with DailyRollingFileAppender, have to change this to RollingFileAppender to enforce size based rolling and cleanup.

Steps:

1. Login to Ambari
2. Goto the config tab of HDFS component.
3. Filter for Advanced hdfs-log4j
4. Update the following log appenders:

   #log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFA=org.apache.log4j.RollingFileAppender
   ...
   # Rollver at midnight
   #log4j.appender.DRFA.DatePattern=.yyyy-MM-dd
   ...
   #log4j.appender.DRFA.MaxBackupIndex=30
   log4j.appender.DRFA.MaxFileSize=256MB
   ...
   #log4j.appender.DRFAS=org.apache.log4j.DailyRollingFileAppender
   log4j.appender.DRFAS=org.apache.log4j.RollingFileAppender log4j.appender.DRFAS.File=${hadoop.log.dir}/${hadoop.security.log.file} log4j.appender.DRFAS.layout=org.apache.log4j.PatternLayout
   log4j.appender.DRFAS.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
   #log4j.appender.DRFAS.DatePattern=.yyyy-MM-dd
   log4j.appender.DRFAS.MaxFileSize=100MB
   log4j.appender.DRFAS.MaxBackupIndex=9
   hdfs.audit.logger=INFO,console
   log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=${hdfs.audit.logger}
   log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false
   #log4j.appender.DRFAAUDIT=org.apache.log4j.DailyRollingFileAppender
   log4j.appender.DRFAAUDIT=org.apache.log4j.RollingFileAppender
   log4j.appender.DRFAAUDIT.File=${hadoop.log.dir}/hdfs-audit.log
   log4j.appender.DRFAAUDIT.layout=org.apache.log4j.PatternLayout
   log4j.appender.DRFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
   #log4j.appender.DRFAAUDIT.DatePattern=.yyyy-MM-dd
   log4j.appender.DRFAAUDIT.MaxFileSize=100MB
   log4j.appender.DRFAAUDIT.MaxBackupIndex=9
   mapred.audit.logger=INFO,console
   log4j.logger.org.apache.hadoop.mapred.AuditLogger=${mapred.audit.logger}
   log4j.additivity.org.apache.hadoop.mapred.AuditLogger=false
   #log4j.appender.MRAUDIT=org.apache.log4j.DailyRollingFileAppender
   log4j.appender.MRAUDIT=org.apache.log4j.RollingFileAppender
   log4j.appender.MRAUDIT.File=${hadoop.log.dir}/mapred-audit.log
   log4j.appender.MRAUDIT.layout=org.apache.log4j.PatternLayout
   log4j.appender.MRAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
   #log4j.appender.MRAUDIT.DatePattern=.yyyy-MM-dd
   log4j.appender.MRAUDIT.MaxFileSize=100MB
   log4j.appender.MRAUDIT.MaxBackupIndex=9

 

5. ZooKeeper

# Max log file size of 10MB
log4j.appender.ROLLINGFILE.MaxFileSize=256MB
# uncomment the next line to limit number of backup files
#log4j.appender.ROLLINGFILE.MaxBackupIndex=10

 

6. Oozie

Steps:

1. Login to Ambari
2. Goto the config tab of Oozie component.
3. Filter for oozie.instance.id}] %m%n

   #Commented this to add RollingFileAppender
   #log4j.appender.oozieops=org.apache.log4j.DailyRollingFileAppender
   #log4j.appender.oozieops.DatePattern='.'yyyy-MM-dd
   log4j.appender.oozieops=org.apache.log4j.RollingFileAppender
   log4j.appender.oozieops.MaxFileSize=256MB
   log4j.appender.oozieops.MaxBackupIndex=20
   log4j.appender.oozieops.File=${oozie.log.dir}/oozie-ops.log
   log4j.appender.oozieops.Append=true
   log4j.appender.oozieops.layout=org.apache.log4j.PatternLayout log4j.appender.oozieops.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n
   
   #Commented this to add RollingFileAppender
   #log4j.appender.oozieinstrumentation=org.apache.log4j.DailyRollingFileAppender #log4j.appender.oozieinstrumentation.DatePattern='.'yyyy-MM-dd log4j.appender.oozieinstrumentation=org.apache.log4j.RollingFileAppender log4j.appender.oozieinstrumentation.MaxFileSize=256MB
   log4j.appender.oozieinstrumentation.MaxBackupIndex=20
   log4j.appender.oozieinstrumentation.File=${oozie.log.dir}/oozie-instrumentation.log log4j.appender.oozieinstrumentation.Append=true
   log4j.appender.oozieinstrumentation.layout=org.apache.log4j.PatternLayout log4j.appender.oozieinstrumentation.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n
   
   #Commented this to add RollingFileAppender
   #log4j.appender.oozieaudit=org.apache.log4j.DailyRollingFileAppender
   #log4j.appender.oozieaudit.DatePattern='.'yyyy-MM-dd
   log4j.appender.oozieaudit =org.apache.log4j.RollingFileAppender
   log4j.appender.oozieaudit.MaxFileSize=256MB
   log4j.appender.oozieaudit.MaxBackupIndex=20
   log4j.appender.oozieaudit.File=${oozie.log.dir}/oozie-audit.log
   log4j.appender.oozieaudit.Append=true
   log4j.appender.oozieaudit.layout=org.apache.log4j.PatternLayout log4j.appender.oozieaudit.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n
   
   #Commented this to add RollingFileAppender
   #log4j.appender.openjpa=org.apache.log4j.DailyRollingFileAppender
   #log4j.appender.openjpa.DatePattern='.'yyyy-MM-dd
   log4j.appender.openjpa=org.apache.log4j.RollingFileAppender
   log4j.appender.openjpa.MaxFileSize=256MB
   log4j.appender.openjpa.MaxBackupIndex=20
   log4j.appender.openjpa.File=${oozie.log.dir}/oozie-jpa.log
   log4j.appender.openjpa.Append=true
   log4j.appender.openjpa.layout=org.apache.log4j.PatternLayout
   log4j.appender.openjpa.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n

 

7. Knox

1. Login to Ambari
2. Goto the config tab of Oozie component.
3. Filter for gateway-log4j
4. Update the following log appenders:

   #Commented this to add RollingFileAppender
   #log4j.rootLogger=ERROR, drfa
   log4j.rootLogger=ERROR, rfa
   log4j.logger.org.apache.hadoop.gateway=INFO#log4j.logger.org.apache.hadoop.gateway=DEBUG
   #log4j.logger.org.eclipse.jetty=DEBUG
   #log4j.logger.org.apache.shiro=DEBUG
   #log4j.logger.org.apache.http=DEBUG
   #log4j.logger.org.apache.http.client=DEBUG
   #log4j.logger.org.apache.http.headers=DEBUG
   #log4j.logger.org.apache.http.wire=DEBUG
   log4j.appender.stdout=org.apache.log4j.ConsoleAppender
   log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
   log4j.appender.stdout.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
   
   #Commented this to add RollingFileAppender
   #log4j.appender.drfa=org.apache.log4j.DailyRollingFileAppender
   #log4j.appender.drfa.File=${app.log.dir}/${app.log.file}
   #log4j.appender.drfa.DatePattern=.yyyy-MM-dd
   #log4j.appender.drfa.layout=org.apache.log4j.PatternLayout
   #log4j.appender.drfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
   log4j.appender.rfa=org.apache.log4j.RollingFileAppender
   log4j.appender.rfa.File=${app.log.dir}/${app.log.file}
   log4j.appender.rfa.MaxFileSize=256MB
   log4j.appender.rfa.MaxBackupIndex=10
   log4j.appender.rfa.layout=org.apache.log4j.PatternLayout
   log4j.appender.rfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
   
   log4j.logger.audit=INFO, auditfile
   
   #Commented this to add RollingFileAppender
   #log4j.appender.auditfile=org.apache.log4j.DailyRollingFileAppender log4j.appender.auditfile=org.apache.log4j.RollingFileAppender
   log4j.appender.auditfile.MaxFileSize=256MB
   log4j.appender.auditfile.MaxBackupIndex=10
   log4j.appender.auditfile.File=${app.log.dir}/${app.audit.file}
   log4j.appender.auditfile.Append = true
   #log4j.appender.auditfile.DatePattern = '.'yyyy-MM-dd
   log4j.appender.auditfile.layout = org.apache.hadoop.gateway.audit.log4j.layout.AuditLayout

5. Filter for ldap-log4jUpdate the following log appenders:

   #Commented this to add RollingFileAppender
   #log4j.rootLogger=ERROR, drfa
   log4j.rootLogger=ERROR, rfa
   log4j.logger.org.apache.directory.server.ldap.LdapServer=INFO log4j.logger.org.apache.directory=WARN
   log4j.appender.stdout=org.apache.log4j.ConsoleAppender
   log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
   log4j.appender.stdout.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
   
   #Commented this to add RollingFileAppender
   #log4j.appender.drfa=org.apache.log4j.DailyRollingFileAppender
   #log4j.appender.drfa.File=${app.log.dir}/${app.log.file}
   #log4j.appender.drfa.DatePattern=.yyyy-MM-dd
   #log4j.appender.drfa.layout=org.apache.log4j.PatternLayout
   #log4j.appender.drfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
   
   log4j.appender.rfa=org.apache.log4j.RollingFileAppender
   log4j.appender.rfa.File=${app.log.dir}/${app.log.file}
   log4j.appender.rfa.MaxFileSize=256MB
   log4j.appender.rfa.MaxBackupIndex=10
   log4j.appender.rfa.layout=org.apache.log4j.PatternLayout log4j.appender.rfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n

    

8. Hive & Hive metastore

1. Login to Ambari
2. Goto the config tab of Hive component.
3. Filter for Advanced hive-log4j
4. Change,
   • DRFA -> RFA in all places
   • log4j.appender.RFA=org.apache.log4j.DailyRollingFileAppender -> log4j.appender.RFA=org.apache.log4j.RollingFileAppender
5. Add
   1. log4j.appender.RFA.MaxFileSize=100MB
   2. log4j.appender.RFA.MaxBackupIndex=10
6. Sample below,

   # Licensed to the Apache Software Foundation (ASF) under one
   # or more contributor license agreements.  See the NOTICE file
   # distributed with this work for additional information
   # regarding copyright ownership.  The ASF licenses this file
   # to you under the Apache License, Version 2.0 (the
   # "License"); you may not use this file except in compliance
   # with the License.  You may obtain a copy of the License at
   #
   #     http://www.apache.org/licenses/LICENSE-2.0
   #
   # Unless required by applicable law or agreed to in writing, software
   # distributed under the License is distributed on an "AS IS" BASIS,
   # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   # See the License for the specific language governing permissions and
   # limitations under the License.
   
   # Define some default values that can be overridden by system properties
   hive.log.threshold=ALL
   hive.root.logger=INFO,RFA
   hive.log.dir=${java.io.tmpdir}/${user.name}
   hive.log.file=hive.log
   
   # Define the root logger to the system property "hadoop.root.logger".
   log4j.rootLogger=${hive.root.logger}, EventCounter
   
   # Logging Threshold
   log4j.threshold=${hive.log.threshold}
   
   #
   # Daily Rolling File Appender changed to Rolling File Appender (DRFA -> RFA)
   #
   # Use the PidDailyerRollingFileAppend class instead if you want to use separate log files
   # for different CLI session.
   #
   # log4j.appender.RFA=org.apache.hadoop.hive.ql.log.PidDailyRollingFileAppender
   
   log4j.appender.RFA=org.apache.log4j.RollingFileAppender
   
   log4j.appender.RFA.File=${hive.log.dir}/${hive.log.file}
   
   # Rollver at midnight
   #log4j.appender.RFA.DatePattern=.yyyy-MM-dd
   log4j.appender.RFA.MaxFileSize=100MB
   
   # 30-day backup
   log4j.appender.RFA.MaxBackupIndex=10
   log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
   
   # Pattern format: Date LogLevel LoggerName LogMessage
   #log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
   # Debugging Pattern format
   log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t]: %c{2} (%F:%M(%L)) - %m%n
   
   
   #
   # console
   # Add "console" to rootlogger above if you want to use this
   #
   
   log4j.appender.console=org.apache.log4j.ConsoleAppender
   log4j.appender.console.target=System.err
   log4j.appender.console.layout=org.apache.log4j.PatternLayout
   log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} [%t]: %p %c{2}: %m%n
   log4j.appender.console.encoding=UTF-8
   
   #custom logging levels
   #log4j.logger.xxx=DEBUG
   
   #
   # Event Counter Appender
   # Sends counts of logging messages at different severity levels to Hadoop Metrics.
   #
   log4j.appender.EventCounter=org.apache.hadoop.hive.shims.HiveEventCounter
   
   
   log4j.category.DataNucleus=ERROR,RFA
   log4j.category.Datastore=ERROR,RFA
   log4j.category.Datastore.Schema=ERROR,RFA
   log4j.category.JPOX.Datastore=ERROR,RFA
   log4j.category.JPOX.Plugin=ERROR,RFA
   log4j.category.JPOX.MetaData=ERROR,RFA
   log4j.category.JPOX.Query=ERROR,RFA
   log4j.category.JPOX.General=ERROR,RFA
   log4j.category.JPOX.Enhancer=ERROR,RFA
   
   
   # Silence useless ZK logs
   log4j.logger.org.apache.zookeeper.server.NIOServerCnxn=WARN,RFA
   log4j.logger.org.apache.zookeeper.ClientCnxnSocketNIO=WARN,RFA

    

 

References

1. https://community.hortonworks.com/articles/8882/how-to-control-size-of-log-files-for-various-hdp-c.html
2. https://discuss.pivotal.io/hc/en-us/articles/202296718-How-to-Change-Hadoop-Daemon-log4j-properties