Spark-Transformation
RDD Transformations are lazy evaluation and is used to transform/update from one RDD into another. When executed on RDD, it results in a single or multiple new RDD.
Since RDD are immutable in nature, transformations always create a new RDD without updating an existing one hence, a chain of RDD transformations creates an RDD lineage
RDD Lineage is also known as the RDD operator graph or RDD dependency graph
In this tutorial, you will learn lazy transformations, types of transformations, a complete list of transformation functions using wordcount example.
- What is a lazy transformation
- Transformation types
- Narrow transformation
- Wider transformation
- Transformation functions
- Transformation functions with word count examples
RDD Transformations are Lazy
RDD Transformations are lazy operations meaning none of the transformations get executed until you call an action on PySpark RDD. Since RDD’s are immutable, any transformations on it result in a new RDD leaving the current one unchanged.
RDD Transformation Types
There are two types of transformations.
-
Narrow Transformation
Narrow transformations are the result of map() and filter() functions and these compute data that live on a single partition meaning there will not be any data movement between partitions to execute narrow transformations.
Functions such as map(), mapPartition(), flatMap(), filter(), union() are some examples of narrow transformation
-
Wider Transformation
Wider transformations are the result of groupByKey() and reduceByKey() functions and these compute data that live on many partitions meaning there will be data movements between partitions to execute wider transformations. Since these shuffles the data, they also called shuffle transformations.
Functions such as groupByKey(), aggregateByKey(), aggregate(), join(), repartition() are some examples of a wider transformations.
Note: When compared to Narrow transformations, wider transformations are expensive operations due to shuffling
|
TRANSFORMATION METHODS |
METHOD USAGE AND DESCRIPTION |
|
cache() |
Caches the RDD |
|
filter() |
Returns a new RDD after applying filter function on source dataset. |
|
flatMap() |
Returns flattern map meaning if you have a dataset with array, it converts each elements in a array as a row. In other words it return 0 or more items in output for each element in dataset. |
|
map() |
Applies transformation function on dataset and returns same number of elements in distributed dataset. |
|
mapPartitions() |
Similar to map, but executs transformation function on each partition, This gives better performance than map function |
|
mapPartitionsWithIndex() |
Similar to map Partitions, but also provides func with an integer value representing the index of the partition. |
|
randomSplit() |
Splits the RDD by the weights specified in the argument. For example rdd.randomSplit(0.7,0.3) |
|
union() |
Comines elements from source dataset and the argument and returns combined dataset. This is similar to union function in Math set operations. |
|
sample() |
Returns the sample dataset. |
|
intersection() |
Returns the dataset which contains elements in both source dataset and an argument |
|
distinct() |
Returns the dataset by eliminating all duplicated elements. |
|
repartition() |
Return a dataset with number of partition specified in the argument. This operation reshuffles the RDD randamly, It could either return lesser or more partioned RDD based on the input supplied. |
|
coalesce() |
Similar to repartition by operates better when we want to the decrease the partitions. Betterment acheives by reshuffling the data from fewer nodes compared with all nodes by repartition. |