logging - How can I see the LOG.debug output from CopyMapper of DistCp job in Hadoop? -
i'm using recent version of hadoop yarn, , after running distcp job successfully, i'm trying see output of log.debug lines copymapper.java , retriablefilecopycommand.java, though i've enabled debug logging in log4j.properties (and of course copied file nodes in cluster), cannot see output of these lines.
the log.debug statements i'm interested are:
log.debug("distcpmapper::map(): received " + sourcepath + ", " + relpath); (from: copymapper.java )
log.debug("copying " + sourcefilestatus.getpath() + " " + target); log.debug("target file path: " + targetpath); (from retriablefilecopycommand.java)
the distcp job copies 20 files 1 cluster , reports success. check yarn web ui , see job listed under finished jobs. when click on that, application_1454924704123_0001 in case, see 1 entry in list, such as
appattempt_1454924704123_0001_000001 mon feb 8 10:51:27 +0100 2016 http://hadoop10:8042 logs and when click on "logs" see there's "syslog : total file length 165516 bytes." , when examine contents don't see debug lines (i see lot of info lines), don't see strings such distcpmapper or target file path should have been produced copymapper , retriablefilecopycommand.
i sshed hadoop10 node, , did grep still couldn't find such debug output, e.g.:
grep -r "target file" /var/log/hadoop/ returns no result.
in log4j.properties, have lines such as:
hadoop.root.logger=debug,console,rfa log4j.logger.org.apache.hadoop.tools.mapred=debug and in hadoop-env.sh have following line:
export hadoop_daemon_root_logger=debug,rfa is not enough see output of of expected log.debug statements of classes in org.apache.hadoop.tools.mapred package such copymapper , retriablefilecopycommand? or looking @ wrong directory?
you can see contents of log4j.properties , hadoop-env.sh files @ end of message, made sure same on of nodes in cluster.
log4j.properties:
# define default values can overridden system properties #hadoop.root.logger=info,console #hadoop.root.logger=info,console,rfa hadoop.root.logger=debug,console,rfa hadoop.log.dir=. hadoop.log.file=hadoop.log # define root logger system property "hadoop.root.logger". log4j.rootlogger=${hadoop.root.logger}, eventcounter # logging threshold log4j.threshold=all # null appender log4j.appender.nullappender=org.apache.log4j.varia.nullappender # # rolling file appender - cap space usage @ 5gb. # hadoop.log.maxfilesize=256mb hadoop.log.maxbackupindex=20 log4j.appender.rfa=org.apache.log4j.rollingfileappender log4j.appender.rfa.file=${hadoop.log.dir}/${hadoop.log.file} log4j.appender.rfa.maxfilesize=${hadoop.log.maxfilesize} log4j.appender.rfa.maxbackupindex=${hadoop.log.maxbackupindex} log4j.appender.rfa.layout=org.apache.log4j.patternlayout # pattern format: date loglevel loggername logmessage log4j.appender.rfa.layout.conversionpattern=%d{iso8601} %p %c: %m%n # debugging pattern format #log4j.appender.rfa.layout.conversionpattern=%d{iso8601} %-5p %c{2} (%f:%m(%l)) - %m%n # # daily rolling file appender # log4j.appender.drfa=org.apache.log4j.dailyrollingfileappender log4j.appender.drfa.file=${hadoop.log.dir}/${hadoop.log.file} # rollover @ midnight log4j.appender.drfa.datepattern=.yyyy-mm-dd log4j.appender.drfa.layout=org.apache.log4j.patternlayout # pattern format: date loglevel loggername logmessage log4j.appender.drfa.layout.conversionpattern=%d{iso8601} %p %c: %m%n # debugging pattern format #log4j.appender.drfa.layout.conversionpattern=%d{iso8601} %-5p %c{2} (%f:%m(%l)) - %m%n # # console # add "console" rootlogger above if want use # log4j.appender.console=org.apache.log4j.consoleappender log4j.appender.console.target=system.err log4j.appender.console.layout=org.apache.log4j.patternlayout log4j.appender.console.layout.conversionpattern=%d{yy/mm/dd hh:mm:ss} %p %c{2}: %m%n # # tasklog appender # #default values hadoop.tasklog.taskid=null hadoop.tasklog.iscleanup=false hadoop.tasklog.nokeepsplits=4 hadoop.tasklog.totallogfilesize=100 hadoop.tasklog.purgelogsplits=true hadoop.tasklog.logsretainhours=12 log4j.appender.tla=org.apache.hadoop.mapred.tasklogappender log4j.appender.tla.taskid=${hadoop.tasklog.taskid} log4j.appender.tla.iscleanup=${hadoop.tasklog.iscleanup} log4j.appender.tla.totallogfilesize=${hadoop.tasklog.totallogfilesize} log4j.appender.tla.layout=org.apache.log4j.patternlayout log4j.appender.tla.layout.conversionpattern=%d{iso8601} %p %c: %m%n # # hdfs block state change log block manager # # uncomment following suppress normal block state change # messages blockmanager in namenode. #log4j.logger.blockstatechange=warn # #security appender # hadoop.security.logger=info,nullappender hadoop.security.log.maxfilesize=256mb hadoop.security.log.maxbackupindex=20 log4j.category.securitylogger=${hadoop.security.logger} hadoop.security.log.file=securityauth-${user.name}.audit log4j.appender.rfas=org.apache.log4j.rollingfileappender log4j.appender.rfas.file=${hadoop.log.dir}/${hadoop.security.log.file} log4j.appender.rfas.layout=org.apache.log4j.patternlayout log4j.appender.rfas.layout.conversionpattern=%d{iso8601} %p %c: %m%n log4j.appender.rfas.maxfilesize=${hadoop.security.log.maxfilesize} log4j.appender.rfas.maxbackupindex=${hadoop.security.log.maxbackupindex} # # daily rolling security appender # log4j.appender.drfas=org.apache.log4j.dailyrollingfileappender log4j.appender.drfas.file=${hadoop.log.dir}/${hadoop.security.log.file} log4j.appender.drfas.layout=org.apache.log4j.patternlayout log4j.appender.drfas.layout.conversionpattern=%d{iso8601} %p %c: %m%n log4j.appender.drfas.datepattern=.yyyy-mm-dd # # hadoop configuration logging # # uncomment following line turn off configuration deprecation warnings. # log4j.logger.org.apache.hadoop.conf.configuration.deprecation=warn # # hdfs audit logging # hdfs.audit.logger=info,nullappender hdfs.audit.log.maxfilesize=256mb hdfs.audit.log.maxbackupindex=20 log4j.logger.org.apache.hadoop.hdfs.server.namenode.fsnamesystem.audit=${hdfs.audit.logger} log4j.additivity.org.apache.hadoop.hdfs.server.namenode.fsnamesystem.audit=false log4j.appender.rfaaudit=org.apache.log4j.rollingfileappender log4j.appender.rfaaudit.file=${hadoop.log.dir}/hdfs-audit.log log4j.appender.rfaaudit.layout=org.apache.log4j.patternlayout log4j.appender.rfaaudit.layout.conversionpattern=%d{iso8601} %p %c{2}: %m%n log4j.appender.rfaaudit.maxfilesize=${hdfs.audit.log.maxfilesize} log4j.appender.rfaaudit.maxbackupindex=${hdfs.audit.log.maxbackupindex} # # namenode metrics logging. # default retain 2 namenode-metrics.log files 64mb each. # namenode.metrics.logger=info,nullappender log4j.logger.namenodemetricslog=${namenode.metrics.logger} log4j.additivity.namenodemetricslog=false log4j.appender.nnmetricsrfa=org.apache.log4j.rollingfileappender log4j.appender.nnmetricsrfa.file=${hadoop.log.dir}/namenode-metrics.log log4j.appender.nnmetricsrfa.layout=org.apache.log4j.patternlayout log4j.appender.nnmetricsrfa.layout.conversionpattern=%d{iso8601} %m%n log4j.appender.nnmetricsrfa.maxbackupindex=1 log4j.appender.nnmetricsrfa.maxfilesize=64mb # # datanode metrics logging. # default retain 2 datanode-metrics.log files 64mb each. # datanode.metrics.logger=info,nullappender log4j.logger.datanodemetricslog=${datanode.metrics.logger} log4j.additivity.datanodemetricslog=false log4j.appender.dnmetricsrfa=org.apache.log4j.rollingfileappender log4j.appender.dnmetricsrfa.file=${hadoop.log.dir}/datanode-metrics.log log4j.appender.dnmetricsrfa.layout=org.apache.log4j.patternlayout log4j.appender.dnmetricsrfa.layout.conversionpattern=%d{iso8601} %m%n log4j.appender.dnmetricsrfa.maxbackupindex=1 log4j.appender.dnmetricsrfa.maxfilesize=64mb # # mapred audit logging # mapred.audit.logger=info,nullappender mapred.audit.log.maxfilesize=256mb mapred.audit.log.maxbackupindex=20 log4j.logger.org.apache.hadoop.mapred.auditlogger=${mapred.audit.logger} log4j.additivity.org.apache.hadoop.mapred.auditlogger=false log4j.appender.mraudit=org.apache.log4j.rollingfileappender log4j.appender.mraudit.file=${hadoop.log.dir}/mapred-audit.log log4j.appender.mraudit.layout=org.apache.log4j.patternlayout log4j.appender.mraudit.layout.conversionpattern=%d{iso8601} %p %c{2}: %m%n log4j.appender.mraudit.maxfilesize=${mapred.audit.log.maxfilesize} log4j.appender.mraudit.maxbackupindex=${mapred.audit.log.maxbackupindex} # custom logging levels #log4j.logger.org.apache.hadoop.mapred.jobtracker=debug #log4j.logger.org.apache.hadoop.mapred.tasktracker=debug #log4j.logger.org.apache.hadoop.hdfs.server.namenode.fsnamesystem.audit=debug # jets3t library log4j.logger.org.jets3t.service.impl.rest.httpclient.rests3service=error # aws sdk & s3a filesystem log4j.logger.com.amazonaws=error log4j.logger.com.amazonaws.http.amazonhttpclient=error #log4j.logger.org.apache.hadoop.fs.s3a.s3afilesystem=warn log4j.logger.org.apache.hadoop.fs.s3a.s3afilesystem=debug log4j.logger.org.apache.hadoop.tools.mapred=debug #log4j.logger.org.apache.hadoop=debug # # event counter appender # sends counts of logging messages @ different severity levels hadoop metrics. # log4j.appender.eventcounter=org.apache.hadoop.log.metrics.eventcounter # # job summary appender # # use following logger send summary separate file defined # hadoop.mapreduce.jobsummary.log.file : # hadoop.mapreduce.jobsummary.logger=info,jsa # hadoop.mapreduce.jobsummary.logger=${hadoop.root.logger} hadoop.mapreduce.jobsummary.log.file=hadoop-mapreduce.jobsummary.log hadoop.mapreduce.jobsummary.log.maxfilesize=256mb hadoop.mapreduce.jobsummary.log.maxbackupindex=20 log4j.appender.jsa=org.apache.log4j.rollingfileappender log4j.appender.jsa.file=${hadoop.log.dir}/${hadoop.mapreduce.jobsummary.log.file} log4j.appender.jsa.maxfilesize=${hadoop.mapreduce.jobsummary.log.maxfilesize} log4j.appender.jsa.maxbackupindex=${hadoop.mapreduce.jobsummary.log.maxbackupindex} log4j.appender.jsa.layout=org.apache.log4j.patternlayout log4j.appender.jsa.layout.conversionpattern=%d{yy/mm/dd hh:mm:ss} %p %c{2}: %m%n log4j.logger.org.apache.hadoop.mapred.jobinprogress$jobsummary=${hadoop.mapreduce.jobsummary.logger} log4j.additivity.org.apache.hadoop.mapred.jobinprogress$jobsummary=false # # shuffle connection log shufflehandler # uncomment following line enable logging of shuffle connections # log4j.logger.org.apache.hadoop.mapred.shufflehandler.audit=debug # # yarn resourcemanager application summary log # # set resourcemanager summary log filename yarn.server.resourcemanager.appsummary.log.file=rm-appsummary.log # set resourcemanager summary log level , appender yarn.server.resourcemanager.appsummary.logger=${hadoop.root.logger} #yarn.server.resourcemanager.appsummary.logger=info,rmsummary # enable appsummarylogging rm, # set yarn.server.resourcemanager.appsummary.logger # <level>,rmsummary in hadoop-env.sh # appender resourcemanager application summary log # requires following properties set # - hadoop.log.dir (hadoop log directory) # - yarn.server.resourcemanager.appsummary.log.file (resource manager app summary log filename) # - yarn.server.resourcemanager.appsummary.logger (resource manager app summary log level , appender) log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.rmappmanager$applicationsummary=${yarn.server.resourcemanager.appsummary.logger} log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.rmappmanager$applicationsummary=false log4j.appender.rmsummary=org.apache.log4j.rollingfileappender log4j.appender.rmsummary.file=${hadoop.log.dir}/${yarn.server.resourcemanager.appsummary.log.file} log4j.appender.rmsummary.maxfilesize=256mb log4j.appender.rmsummary.maxbackupindex=20 log4j.appender.rmsummary.layout=org.apache.log4j.patternlayout log4j.appender.rmsummary.layout.conversionpattern=%d{iso8601} %p %c{2}: %m%n # hs audit log configs #mapreduce.hs.audit.logger=info,hsaudit #log4j.logger.org.apache.hadoop.mapreduce.v2.hs.hsauditlogger=${mapreduce.hs.audit.logger} #log4j.additivity.org.apache.hadoop.mapreduce.v2.hs.hsauditlogger=false #log4j.appender.hsaudit=org.apache.log4j.dailyrollingfileappender #log4j.appender.hsaudit.file=${hadoop.log.dir}/hs-audit.log #log4j.appender.hsaudit.layout=org.apache.log4j.patternlayout #log4j.appender.hsaudit.layout.conversionpattern=%d{iso8601} %p %c{2}: %m%n #log4j.appender.hsaudit.datepattern=.yyyy-mm-dd # http server request logs #log4j.logger.http.requests.namenode=info,namenoderequestlog #log4j.appender.namenoderequestlog=org.apache.hadoop.http.httprequestlogappender #log4j.appender.namenoderequestlog.filename=${hadoop.log.dir}/jetty-namenode-yyyy_mm_dd.log #log4j.appender.namenoderequestlog.retaindays=3 #log4j.logger.http.requests.datanode=info,datanoderequestlog #log4j.appender.datanoderequestlog=org.apache.hadoop.http.httprequestlogappender #log4j.appender.datanoderequestlog.filename=${hadoop.log.dir}/jetty-datanode-yyyy_mm_dd.log #log4j.appender.datanoderequestlog.retaindays=3 #log4j.logger.http.requests.resourcemanager=info,resourcemanagerrequestlog #log4j.appender.resourcemanagerrequestlog=org.apache.hadoop.http.httprequestlogappender #log4j.appender.resourcemanagerrequestlog.filename=${hadoop.log.dir}/jetty-resourcemanager-yyyy_mm_dd.log #log4j.appender.resourcemanagerrequestlog.retaindays=3 #log4j.logger.http.requests.jobhistory=info,jobhistoryrequestlog #log4j.appender.jobhistoryrequestlog=org.apache.hadoop.http.httprequestlogappender #log4j.appender.jobhistoryrequestlog.filename=${hadoop.log.dir}/jetty-jobhistory-yyyy_mm_dd.log #log4j.appender.jobhistoryrequestlog.retaindays=3 #log4j.logger.http.requests.nodemanager=info,nodemanagerrequestlog #log4j.appender.nodemanagerrequestlog=org.apache.hadoop.http.httprequestlogappender #log4j.appender.nodemanagerrequestlog.filename=${hadoop.log.dir}/jetty-nodemanager-yyyy_mm_dd.log #log4j.appender.nodemanagerrequestlog.retaindays=3 # appender viewing information errors , warnings yarn.ewma.cleanupinterval=300 yarn.ewma.messageagelimitseconds=86400 yarn.ewma.maxuniquemessages=250 log4j.appender.ewma=org.apache.hadoop.yarn.util.log4jwarningerrormetricsappender log4j.appender.ewma.cleanupinterval=${yarn.ewma.cleanupinterval} log4j.appender.ewma.messageagelimitseconds=${yarn.ewma.messageagelimitseconds} log4j.appender.ewma.maxuniquemessages=${yarn.ewma.maxuniquemessages} hadoop-env.sh:
export java_home=/usr/lib/jvm/java-8-oracle export hadoop_prefix=/opt/hadoop-3.0.0-snapshot export hadoop_conf_dir=$hadoop_prefix/etc/hadoop export hadoop_os_type=${hadoop_os_type:-$(uname -s)} case ${hadoop_os_type} in darwin*) export hadoop_opts="${hadoop_opts} -djava.security.krb5.realm= " export hadoop_opts="${hadoop_opts} -djava.security.krb5.kdc= " export hadoop_opts="${hadoop_opts} -djava.security.krb5.conf= " ;; esac export hadoop_classpath=$hadoop_classpath:$hadoop_prefix/share/hadoop/tools/lib/* export hadoop_log_dir=/var/log/hadoop export hadoop_daemon_root_logger=debug,rfa
Comments
Post a Comment