![3
1-Hierarchical
indexing
[By Amina Delali]
Data Wrangling and hierarchical indexingData Wrangling and hierarchical indexing
●
Data wrangling is the process of cleaning and unifying messy
and complex data sets for easy access and analysis. (from:
https://www.datawatch.com/what-is-data-wrangling/)
●
Hierarchical indexing: is the use of multiple indexes at
different levels
hind
i1, i2 , i3 will be
In the level A
ser1.index
ser1
level 1 level 2
Indices of level 1 Indices of level 2](https://image.slidesharecdn.com/aaa-ped-7-190413160728/85/Aaa-ped-8-Data-manipulation-Data-wrangling-aggregation-and-group-operations-3-320.jpg)
![4
1-Hierarchical
indexing
[By Amina Delali]
Reordering and sortingReordering and sorting
●
Reordering enables interchanging the index levels using the
swaplevel method
●
Sorting enables sorting the data by sorting one level
values, using the sort_index method.
The order of the data (so
the indexes too)
remains the same
The hierarchy of
the indexes changed
The hierarchy of the
Indexes didn’t change
The order of the data
(so the indexes too)
changed: the_level1
index was sorted](https://image.slidesharecdn.com/aaa-ped-7-190413160728/85/Aaa-ped-8-Data-manipulation-Data-wrangling-aggregation-and-group-operations-4-320.jpg)
![5
1-Hierarchical
indexing
[By Amina Delali]
Operations by levelOperations by level
●
df1
df1
If sum was applied on number,
It would perform and addition instead
Of this concatenation
A new column
addedsorted](https://image.slidesharecdn.com/aaa-ped-7-190413160728/85/Aaa-ped-8-Data-manipulation-Data-wrangling-aggregation-and-group-operations-5-320.jpg)
![6
1-Hierarchical
indexing
[By Amina Delali]
indexingindexing
●
df2 The previous df1 columns
are now indexes
The indexes are
converted into
columns](https://image.slidesharecdn.com/aaa-ped-7-190413160728/85/Aaa-ped-8-Data-manipulation-Data-wrangling-aggregation-and-group-operations-6-320.jpg)
![7
2-Combiningand
mergingDataSets
[By Amina Delali]
mergemerge
●
df1
df3
Each row from df1 with “green”
value in “colors” column, will be
combined with each row from
df3 with “green” value in
“mycolors” column.
Both df1 and df2 have “codes”
column, so “_x” and “_y”
suffixes were added.
“red” rows weren’t included](https://image.slidesharecdn.com/aaa-ped-7-190413160728/85/Aaa-ped-8-Data-manipulation-Data-wrangling-aggregation-and-group-operations-7-320.jpg)
![8
2-Combiningand
mergingDataSets
[By Amina Delali]
mergemerge
●
df1
df3
Specifying the argument “how” as
“left”,all rows from df1 were
included (even if no matching
value exists in df3)
The suffixes argument used to customize the
suffixes added to columns with the same name](https://image.slidesharecdn.com/aaa-ped-7-190413160728/85/Aaa-ped-8-Data-manipulation-Data-wrangling-aggregation-and-group-operations-8-320.jpg)
![9
2-Combiningand
mergingDataSets
[By Amina Delali]
mergemerge
●
df3
df11
Combining df11
And df3 by matching
Values from “number
Column of df11,
with values from
Index values of df3](https://image.slidesharecdn.com/aaa-ped-7-190413160728/85/Aaa-ped-8-Data-manipulation-Data-wrangling-aggregation-and-group-operations-9-320.jpg)
![10
2-Combiningand
mergingDataSets
[By Amina Delali]
joinjoin
●
● df3df11
You have to
specify the
suffixes if the
dataframes
have coulmns
with same
names
By default all the
values of df11 were
added
The dataframes
are combined by
matching indexes
values](https://image.slidesharecdn.com/aaa-ped-7-190413160728/85/Aaa-ped-8-Data-manipulation-Data-wrangling-aggregation-and-group-operations-10-320.jpg)
![11
3-Reshapingand
pivoting
[By Amina Delali]
concatconcat
●
combine_firstcombine_first
ser11
ser2
ser3
ser4ser3
ser4
ser3 values
were chosen
over ser4
values
In ser3, “a” corresponding
value == nan and in ser4
“a” corresponding value
is not null, so it was chosen](https://image.slidesharecdn.com/aaa-ped-7-190413160728/85/Aaa-ped-8-Data-manipulation-Data-wrangling-aggregation-and-group-operations-11-320.jpg)
![12
3-Reshapingand
pivoting
[By Amina Delali]
stack & unstackstack & unstack
●
stack: pivot columns label to rows indexes
●
unstack: pivot rows indexes to columns labels](https://image.slidesharecdn.com/aaa-ped-7-190413160728/85/Aaa-ped-8-Data-manipulation-Data-wrangling-aggregation-and-group-operations-12-320.jpg)
![13
3-Reshapingand
pivoting
[By Amina Delali]
●
meltmelt
pivotpivot
Only 2 unique
values for
Code_type
3 unique values
3 columns
The values are obtained
from “value”by matching
“Code_type”and
“number” values
df3](https://image.slidesharecdn.com/aaa-ped-7-190413160728/85/Aaa-ped-8-Data-manipulation-Data-wrangling-aggregation-and-group-operations-13-320.jpg)
![14
4-GroupbyMechanics
[By Amina Delali]
groupbygroupby
●
df11
5 + 7 +8 = 20
i
i
j
j](https://image.slidesharecdn.com/aaa-ped-7-190413160728/85/Aaa-ped-8-Data-manipulation-Data-wrangling-aggregation-and-group-operations-14-320.jpg)
![15
4-GroupbyMechanics
[By Amina Delali]
groupbygroupby
●
myDict
Gr1 values are summed
together.
And Gr2 values are also
summed together](https://image.slidesharecdn.com/aaa-ped-7-190413160728/85/Aaa-ped-8-Data-manipulation-Data-wrangling-aggregation-and-group-operations-15-320.jpg)
![16
5-Dataaggregation
[By Amina Delali]
aggagg
●
Became an index
Remains a column
Columns kept columnsColumns kept columns
We could just write:
groupby(“Code_type”)](https://image.slidesharecdn.com/aaa-ped-7-190413160728/85/Aaa-ped-8-Data-manipulation-Data-wrangling-aggregation-and-group-operations-16-320.jpg)
![17
6-Otheraggregation
operations
[By Amina Delali]
●
We see that the number
values in each interval is
different from the others
We see that the number
values in each interval are
all the same == 2
The data values
in “value3” column
are grouped by
intervals created
by cut(same length)
and qcut (same
size)
cut & qcutcut & qcut](https://image.slidesharecdn.com/aaa-ped-7-190413160728/85/Aaa-ped-8-Data-manipulation-Data-wrangling-aggregation-and-group-operations-17-320.jpg)
![18
6-Otheraggregation
operations
[By Amina Delali]
●
crosstabcrosstab
df11
For “number” value ==1,
corresponds: 2 values == green
and 0 value in blue and red in
“colors”](https://image.slidesharecdn.com/aaa-ped-7-190413160728/85/Aaa-ped-8-Data-manipulation-Data-wrangling-aggregation-and-group-operations-18-320.jpg)
Aaa ped-8- Data manipulation: Data wrangling, aggregation, and group operations
- 1. Data manipulation: Data wrangling, aggregation, and group operations. AAA-Python Edition
- 2. Plan ● 1- Hierarchical indexing ● 2- Combining and merging Data Sets ● 3- Reshaping and pivoting ● 4- Group by Mechanics ● 5- Data aggregation ● 6- Other aggregation operations
- 3. 3 1-Hierarchical indexing [By Amina Delali] Data Wrangling and hierarchical indexingData Wrangling and hierarchical indexing ● Data wrangling is the process of cleaning and unifying messy and complex data sets for easy access and analysis. (from: https://www.datawatch.com/what-is-data-wrangling/) ● Hierarchical indexing: is the use of multiple indexes at different levels hind i1, i2 , i3 will be In the level A ser1.index ser1 level 1 level 2 Indices of level 1 Indices of level 2
- 4. 4 1-Hierarchical indexing [By Amina Delali] Reordering and sortingReordering and sorting ● Reordering enables interchanging the index levels using the swaplevel method ● Sorting enables sorting the data by sorting one level values, using the sort_index method. The order of the data (so the indexes too) remains the same The hierarchy of the indexes changed The hierarchy of the Indexes didn’t change The order of the data (so the indexes too) changed: the_level1 index was sorted
- 5. 5 1-Hierarchical indexing [By Amina Delali] Operations by levelOperations by level ● df1 df1 If sum was applied on number, It would perform and addition instead Of this concatenation A new column addedsorted
- 6. 6 1-Hierarchical indexing [By Amina Delali] indexingindexing ● df2 The previous df1 columns are now indexes The indexes are converted into columns
- 7. 7 2-Combiningand mergingDataSets [By Amina Delali] mergemerge ● df1 df3 Each row from df1 with “green” value in “colors” column, will be combined with each row from df3 with “green” value in “mycolors” column. Both df1 and df2 have “codes” column, so “_x” and “_y” suffixes were added. “red” rows weren’t included
- 8. 8 2-Combiningand mergingDataSets [By Amina Delali] mergemerge ● df1 df3 Specifying the argument “how” as “left”,all rows from df1 were included (even if no matching value exists in df3) The suffixes argument used to customize the suffixes added to columns with the same name
- 9. 9 2-Combiningand mergingDataSets [By Amina Delali] mergemerge ● df3 df11 Combining df11 And df3 by matching Values from “number Column of df11, with values from Index values of df3
- 10. 10 2-Combiningand mergingDataSets [By Amina Delali] joinjoin ● ● df3df11 You have to specify the suffixes if the dataframes have coulmns with same names By default all the values of df11 were added The dataframes are combined by matching indexes values
- 11. 11 3-Reshapingand pivoting [By Amina Delali] concatconcat ● combine_firstcombine_first ser11 ser2 ser3 ser4ser3 ser4 ser3 values were chosen over ser4 values In ser3, “a” corresponding value == nan and in ser4 “a” corresponding value is not null, so it was chosen
- 12. 12 3-Reshapingand pivoting [By Amina Delali] stack & unstackstack & unstack ● stack: pivot columns label to rows indexes ● unstack: pivot rows indexes to columns labels
- 13. 13 3-Reshapingand pivoting [By Amina Delali] ● meltmelt pivotpivot Only 2 unique values for Code_type 3 unique values 3 columns The values are obtained from “value”by matching “Code_type”and “number” values df3
- 14. 14 4-GroupbyMechanics [By Amina Delali] groupbygroupby ● df11 5 + 7 +8 = 20 i i j j
- 15. 15 4-GroupbyMechanics [By Amina Delali] groupbygroupby ● myDict Gr1 values are summed together. And Gr2 values are also summed together
- 16. 16 5-Dataaggregation [By Amina Delali] aggagg ● Became an index Remains a column Columns kept columnsColumns kept columns We could just write: groupby(“Code_type”)
- 17. 17 6-Otheraggregation operations [By Amina Delali] ● We see that the number values in each interval is different from the others We see that the number values in each interval are all the same == 2 The data values in “value3” column are grouped by intervals created by cut(same length) and qcut (same size) cut & qcutcut & qcut
- 18. 18 6-Otheraggregation operations [By Amina Delali] ● crosstabcrosstab df11 For “number” value ==1, corresponds: 2 values == green and 0 value in blue and red in “colors”
- 19. References ● Datawatch. What is data wrangling? On-line at https://www.datawatch.com/what-is-data-wrangling/. Accessed on 31-10-2018. ● Wes McKinney. Python for data analysis: Data wrangling with Pandas, NumPy, and IPython. O’Reilly Media, Inc, 2018. ● pydata.org. Pandas documentation. On-line at https://pandas.pydata. org/. Accessed on 19-10-2018.
- 20. Thank you! FOR ALL YOUR TIME