Skip to main content

Which is better Tez or MR

MR

Read the data from the file (disk access =1)
Run mappers
Write map output (disk access =2)
Run shuffle and sort (read intermediate o/p of mapper, disk access =3)
write shuffle and sort  (disk access =4)
run reducers which reads sorted data (disk access =5)
write reducers output (disk access =6)

TEZ

Irrespective of the tasks it first creates DAG(Directed Acyclic Graph)
It is similar to Spark but developed well before than spark.

Executes the plan but no need to read data from disk.
Once ready to do some calculations, get the data from the disk and perform all the steps and produce the output.
One read and one write

Pros: One read and one write
Efficient as it wont access the disk multiple times and stores intermediate results in memory.
Vectorization is enabled on top of it.

Last but not the least
If the table is partitioned and there are delta files (from updates, for eg.), I think mr works but not tez. You may have to run compaction to convert the delta files into base files and then tez will work.

Comments

Post a Comment

Popular posts from this blog

Error handling in hadoop map reduce

Based on the documentation, there are a few ways, how the error handling is performed in map reduce. Below are the few: a. Custom counters using enum - increment for every failed record. b. Log error and analyze later. Counters give the number of failed records. However to get the identifier of the failed record(may be its unique key), and details of the exception occurred, node on which the error occurred - we need to perform centralized log analysis and there are many nodes running. Logstash is on which is available. Apart from these, are there any other ways to handle the error scenarios, without manual intervention. Any tools, references, best practices are welcome. I think the same technique applies to any distributed applications, with minor changes. Few questions to ask, when working with error handling: Should the job be stopped if an error occurred in data validation. Most of the Big data use cases might be ok to leave few bad records. But if your usecase wa...
Post a Comment
Read more

Custom SerDe

Steps invoved We start by implementing the  SerDe   interface and setting up the internal state variables needed by other methods. public class ColumnarMapSerDe implements SerDe { private List < String > columnNames; private ObjectInspector objectInspector; private Map < String, String > rowMap; The initialize method is called when a table is created. It is responsible for verifying that the table definition is compatible with the underlying serialization and deserialization mechanism. In this case, we only support strings. @Override public void initialize( Configuration conf, Properties tableProperties) throws SerDeException { final List < TypeInfo > columnTypes = TypeInfoUtils.getTypeInfosFromTypeString( tableProperties.getProperty(LIST_COLUMN_TYPES)); // verify types for (TypeInfo type : columnTypes) { if (!type.getCategory().equals(PRIMITIVE) || The  serialize  method is c...
Post a Comment
Read more

LeaseExpired Exception

So finally i Could determine What is going on /2018/week1/abc /2018/week2/abc /2018/week3/abc /2018/week1/xyz /2018/week2/xyz /2018/week3/xyz Notice the same file names across different weeks. And then each of these files were being written to HDFS using the DFSClient. So essentially multiple mappers were trying to write the "same file"(observe filename abc, xyz) even though the files were actually different. As the client has to acquire a lease before writing the file and as the client writing the first of the "abc" file was not releasing the lease while during the writing process, the other client trying to write the other "abc" (lets say of the week2)  was  throwing the LeaseExpiredException with the Lease Mismatch Message. But this still does not explain why the client which first acquired the lease for the write did not succeed. I mean i would expect in such a case that the first writers of every such files to succeed.No idea This type ...
Post a Comment
Read more