Skip to main content

Configuring such that avoid too many log files

Configuring Hadoop logging to avoid too many log files



I'm having a problem with Hadoop producing too many log files in $HADOOP_LOG_DIR/userlogs (the Ext3 filesystem allows only 32000 subdirectories) which looks like the same problem in this question: Error in Hadoop MapReduce
My question is: does anyone know how to configure Hadoop to roll the log dir or otherwise prevent this? I'm trying to avoid just setting the "mapred.userlog.retain.hours" and/or "mapred.userlog.limit.kb" properties because I want to actually keep the log files.

I was also hoping to configure this in log4j.properties, but looking at the Hadoop 0.20.2 source, it writes directly to logfiles instead of actually using log4j. Perhaps I don't understand how it's using log4j fully.
Solution:
Unfortunately, there isn't a configurable way to prevent that. Every task for a job gets one directory in history/userlogs, which will hold the stdout, stderr, and syslog task log output files. The retain hours will help keep too many of those from accumulating, but you'd have to write a good log rotation tool to auto-tar them.
We had this problem too when we were writing to an NFS mount, because all nodes would share the same history/userlogs directory. This means one job with 30,000 tasks would be enough to break the FS. Logging locally is really the way to go when your cluster actually starts processing a lot of data.

If you are already logging locally and still manage to process 30,000+ tasks on one machine in less than a week, then you are probably creating too many small files, causing too many mappers to spawn for each job.
Solution2:
Our solution is to modify our data collection process to concatenate files before running any jobs. 
Solution3:
 Set the environment variable "HADOOP_ROOT_LOGGER=WARN,console" before starting Hadoop.
export HADOOP_ROOT_LOGGER="WARN,console"
hadoop jar start.jar

Comments

Post a Comment

Popular posts from this blog

Error handling in hadoop map reduce

Based on the documentation, there are a few ways, how the error handling is performed in map reduce. Below are the few: a. Custom counters using enum - increment for every failed record. b. Log error and analyze later. Counters give the number of failed records. However to get the identifier of the failed record(may be its unique key), and details of the exception occurred, node on which the error occurred - we need to perform centralized log analysis and there are many nodes running. Logstash is on which is available. Apart from these, are there any other ways to handle the error scenarios, without manual intervention. Any tools, references, best practices are welcome. I think the same technique applies to any distributed applications, with minor changes. Few questions to ask, when working with error handling: Should the job be stopped if an error occurred in data validation. Most of the Big data use cases might be ok to leave few bad records. But if your usecase wa...
Post a Comment
Read more

LeaseExpired Exception

So finally i Could determine What is going on /2018/week1/abc /2018/week2/abc /2018/week3/abc /2018/week1/xyz /2018/week2/xyz /2018/week3/xyz Notice the same file names across different weeks. And then each of these files were being written to HDFS using the DFSClient. So essentially multiple mappers were trying to write the "same file"(observe filename abc, xyz) even though the files were actually different. As the client has to acquire a lease before writing the file and as the client writing the first of the "abc" file was not releasing the lease while during the writing process, the other client trying to write the other "abc" (lets say of the week2)  was  throwing the LeaseExpiredException with the Lease Mismatch Message. But this still does not explain why the client which first acquired the lease for the write did not succeed. I mean i would expect in such a case that the first writers of every such files to succeed.No idea This type ...
Post a Comment
Read more

Handling csv with enclosed doubled quotes and separated by comma

Incase if you need to load the csv file in which fields are terminated by ',' or any thing and it also includes the quotes char in it.  Below is the format we need to follow with SERDE Create table tbl_netflix(show_id int,title string,director string,casting string,country string,date_added date,release_year int,rating string,duration string,listed_in string,description string,type string) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde' WITH SERDEPROPERTIES (    "separatorChar" = ",",    "quoteChar"     = "\"" )Stored as TextFile;
Post a Comment
Read more