![PythonForDataScience Cheat Sheet
Pandas
Learn Python for Data Science Interactively at www.DataCamp.com
Reshaping Data
DataCamp
Learn Python for Data Science Interactively
Advanced Indexing
Reindexing
>>> s2 = s.reindex(['a','c','d','e','b'])
>>> s3 = s.reindex(range(5),
method='bfill')
0 3
1 3
2 3
3 3
4 3
Forward Filling Backward Filling
>>> df.reindex(range(4),
method='ffill')
Country Capital Population
0 Belgium Brussels 11190846
1 India New Delhi 1303171035
2 Brazil Brasília 207847528
3 Brazil Brasília 207847528
Pivot
Stack / Unstack
Melt
Combining Data
>>> pd.melt(df2, Gather columns into rows
id_vars=["Date"],
value_vars=["Type", "Value"],
value_name="Observations")
>>> stacked = df5.stack() Pivot a level of column labels
>>> stacked.unstack() Pivot a level of index labels
>>> df3= df2.pivot(index='Date', Spread rows into columns
columns='Type',
values='Value')
>>> arrays = [np.array([1,2,3]),
np.array([5,4,3])]
>>> df5 = pd.DataFrame(np.random.rand(3, 2), index=arrays)
>>> tuples = list(zip(*arrays))
>>> index = pd.MultiIndex.from_tuples(tuples,
names=['first', 'second'])
>>> df6 = pd.DataFrame(np.random.rand(3, 2), index=index)
>>> df2.set_index(["Date", "Type"])
Missing Data
>>> df.dropna() Drop NaN values
>>> df3.fillna(df3.mean()) Fill NaN values with a predetermined value
>>> df2.replace("a", "f") Replace values with others
2016-03-01 a
2016-03-02 b
2016-03-01 c
11.432
13.031
20.784
2016-03-03 a
2016-03-02 a
2016-03-03 c
99.906
1.303
20.784
Date Type Value
0
1
2
3
4
5
Type
Date
2016-03-01
2016-03-02
2016-03-03
a
11.432
1.303
99.906
b
NaN
13.031
NaN
c
20.784
NaN
20.784
Selecting
>>> df3.loc[:,(df3>1).any()] Select cols with any vals >1
>>> df3.loc[:,(df3>1).all()] Select cols with vals > 1
>>> df3.loc[:,df3.isnull().any()] Select cols with NaN
>>> df3.loc[:,df3.notnull().all()] Select cols without NaN
Indexing With isin
>>> df[(df.Country.isin(df2.Type))] Find same elements
>>> df3.filter(items=”a”,”b”]) Filter on values
>>> df.select(lambda x: not x%5) Select specific elements
Where
>>> s.where(s > 0) Subset the data
Query
>>> df6.query('second > first') Query DataFrame
Pivot Table
>>> df4 = pd.pivot_table(df2, Spread rows into columns
values='Value',
index='Date',
columns='Type'])
Merge
Join
Concatenate
>>> pd.merge(data1,
data2,
how='left',
on='X1')
>>> data1.join(data2, how='right')
Vertical
>>> s.append(s2)
Horizontal/Vertical
>>> pd.concat([s,s2],axis=1, keys=['One','Two'])
>>> pd.concat([data1, data2], axis=1, join='inner')
1
2
3
5
4
3
0
1
0
1
0
1
0.233482
0.390959
0.184713
0.237102
0.433522
0.429401
1
2
3
5
4
3
0
0.233482
0.184713
0.433522
1
0.390959
0.237102
0.429401
Stacked
Unstacked
2016-03-01 a
2016-03-02 b
2016-03-01 c
11.432
13.031
20.784
2016-03-03 a
2016-03-02 a
2016-03-03 c
99.906
1.303
20.784
Date Type Value
0
1
2
3
4
5
2016-03-01 Type
2016-03-02 Type
2016-03-01 Type
a
b
c
2016-03-03 Type
2016-03-02 Type
2016-03-03 Type
a
a
c
Date Variable Observations
0
1
2
3
4
5
2016-03-01 Value
2016-03-02 Value
2016-03-01 Value
11.432
13.031
20.784
2016-03-03 Value
2016-03-02 Value
2016-03-03 Value
99.906
1.303
20.784
6
7
8
9
10
11
Iteration
>>> df.iteritems() (Column-index, Series) pairs
>>> df.iterrows() (Row-index, Series) pairs
data1
a
b
c
11.432
1.303
99.906
X1 X2
a
b
d
20.784
NaN
20.784
data2
X1 X3
a
b
c
11.432
1.303
99.906
20.784
NaN
NaN
X1 X2 X3
>>> pd.merge(data1,
data2,
how='outer',
on='X1')
>>> pd.merge(data1,
data2,
how='right',
on='X1')
a
b
d
11.432
1.303
NaN
20.784
NaN
20.784
X1 X2 X3
>>> pd.merge(data1,
data2,
how='inner',
on='X1')
a
b
11.432
1.303
20.784
NaN
X1 X2 X3
a
b
c
11.432
1.303
99.906
20.784
NaN
NaN
X1 X2 X3
d NaN 20.784
Setting/Resetting Index
>>> df.set_index('Country') Set the index
>>> df4 = df.reset_index() Reset the index
>>> df = df.rename(index=str, Rename DataFrame
columns={"Country":"cntry",
"Capital":"cptl",
"Population":"ppltn"})
Duplicate Data
>>> s3.unique() Return unique values
>>> df2.duplicated('Type') Check duplicates
>>> df2.drop_duplicates('Type', keep='last') Drop duplicates
>>> df.index.duplicated() Check index duplicates
Grouping Data
Aggregation
>>> df2.groupby(by=['Date','Type']).mean()
>>> df4.groupby(level=0).sum()
>>> df4.groupby(level=0).agg({'a':lambda x:sum(x)/len(x),
'b': np.sum})
Transformation
>>> customSum = lambda x: (x+x%2)
>>> df4.groupby(level=0).transform(customSum)
MultiIndexing
Dates
Visualization
Also see NumPy Arrays
>>> s.plot()
>>> plt.show()
Also see Matplotlib
>>> import matplotlib.pyplot as plt
>>> df2.plot()
>>> plt.show()
>>> df2['Date']= pd.to_datetime(df2['Date'])
>>> df2['Date']= pd.date_range('2000-1-1',
periods=6,
freq='M')
>>> dates = [datetime(2012,5,1), datetime(2012,5,2)]
>>> index = pd.DatetimeIndex(dates)
>>> index = pd.date_range(datetime(2012,2,1), end, freq='BM')](https://image.slidesharecdn.com/3pandasadvanced-210204025522/85/3-pandasadvanced-1-320.jpg)
3 pandasadvanced
- 1. PythonForDataScience Cheat Sheet Pandas Learn Python for Data Science Interactively at www.DataCamp.com Reshaping Data DataCamp Learn Python for Data Science Interactively Advanced Indexing Reindexing >>> s2 = s.reindex(['a','c','d','e','b']) >>> s3 = s.reindex(range(5), method='bfill') 0 3 1 3 2 3 3 3 4 3 Forward Filling Backward Filling >>> df.reindex(range(4), method='ffill') Country Capital Population 0 Belgium Brussels 11190846 1 India New Delhi 1303171035 2 Brazil Brasília 207847528 3 Brazil Brasília 207847528 Pivot Stack / Unstack Melt Combining Data >>> pd.melt(df2, Gather columns into rows id_vars=["Date"], value_vars=["Type", "Value"], value_name="Observations") >>> stacked = df5.stack() Pivot a level of column labels >>> stacked.unstack() Pivot a level of index labels >>> df3= df2.pivot(index='Date', Spread rows into columns columns='Type', values='Value') >>> arrays = [np.array([1,2,3]), np.array([5,4,3])] >>> df5 = pd.DataFrame(np.random.rand(3, 2), index=arrays) >>> tuples = list(zip(*arrays)) >>> index = pd.MultiIndex.from_tuples(tuples, names=['first', 'second']) >>> df6 = pd.DataFrame(np.random.rand(3, 2), index=index) >>> df2.set_index(["Date", "Type"]) Missing Data >>> df.dropna() Drop NaN values >>> df3.fillna(df3.mean()) Fill NaN values with a predetermined value >>> df2.replace("a", "f") Replace values with others 2016-03-01 a 2016-03-02 b 2016-03-01 c 11.432 13.031 20.784 2016-03-03 a 2016-03-02 a 2016-03-03 c 99.906 1.303 20.784 Date Type Value 0 1 2 3 4 5 Type Date 2016-03-01 2016-03-02 2016-03-03 a 11.432 1.303 99.906 b NaN 13.031 NaN c 20.784 NaN 20.784 Selecting >>> df3.loc[:,(df3>1).any()] Select cols with any vals >1 >>> df3.loc[:,(df3>1).all()] Select cols with vals > 1 >>> df3.loc[:,df3.isnull().any()] Select cols with NaN >>> df3.loc[:,df3.notnull().all()] Select cols without NaN Indexing With isin >>> df[(df.Country.isin(df2.Type))] Find same elements >>> df3.filter(items=”a”,”b”]) Filter on values >>> df.select(lambda x: not x%5) Select specific elements Where >>> s.where(s > 0) Subset the data Query >>> df6.query('second > first') Query DataFrame Pivot Table >>> df4 = pd.pivot_table(df2, Spread rows into columns values='Value', index='Date', columns='Type']) Merge Join Concatenate >>> pd.merge(data1, data2, how='left', on='X1') >>> data1.join(data2, how='right') Vertical >>> s.append(s2) Horizontal/Vertical >>> pd.concat([s,s2],axis=1, keys=['One','Two']) >>> pd.concat([data1, data2], axis=1, join='inner') 1 2 3 5 4 3 0 1 0 1 0 1 0.233482 0.390959 0.184713 0.237102 0.433522 0.429401 1 2 3 5 4 3 0 0.233482 0.184713 0.433522 1 0.390959 0.237102 0.429401 Stacked Unstacked 2016-03-01 a 2016-03-02 b 2016-03-01 c 11.432 13.031 20.784 2016-03-03 a 2016-03-02 a 2016-03-03 c 99.906 1.303 20.784 Date Type Value 0 1 2 3 4 5 2016-03-01 Type 2016-03-02 Type 2016-03-01 Type a b c 2016-03-03 Type 2016-03-02 Type 2016-03-03 Type a a c Date Variable Observations 0 1 2 3 4 5 2016-03-01 Value 2016-03-02 Value 2016-03-01 Value 11.432 13.031 20.784 2016-03-03 Value 2016-03-02 Value 2016-03-03 Value 99.906 1.303 20.784 6 7 8 9 10 11 Iteration >>> df.iteritems() (Column-index, Series) pairs >>> df.iterrows() (Row-index, Series) pairs data1 a b c 11.432 1.303 99.906 X1 X2 a b d 20.784 NaN 20.784 data2 X1 X3 a b c 11.432 1.303 99.906 20.784 NaN NaN X1 X2 X3 >>> pd.merge(data1, data2, how='outer', on='X1') >>> pd.merge(data1, data2, how='right', on='X1') a b d 11.432 1.303 NaN 20.784 NaN 20.784 X1 X2 X3 >>> pd.merge(data1, data2, how='inner', on='X1') a b 11.432 1.303 20.784 NaN X1 X2 X3 a b c 11.432 1.303 99.906 20.784 NaN NaN X1 X2 X3 d NaN 20.784 Setting/Resetting Index >>> df.set_index('Country') Set the index >>> df4 = df.reset_index() Reset the index >>> df = df.rename(index=str, Rename DataFrame columns={"Country":"cntry", "Capital":"cptl", "Population":"ppltn"}) Duplicate Data >>> s3.unique() Return unique values >>> df2.duplicated('Type') Check duplicates >>> df2.drop_duplicates('Type', keep='last') Drop duplicates >>> df.index.duplicated() Check index duplicates Grouping Data Aggregation >>> df2.groupby(by=['Date','Type']).mean() >>> df4.groupby(level=0).sum() >>> df4.groupby(level=0).agg({'a':lambda x:sum(x)/len(x), 'b': np.sum}) Transformation >>> customSum = lambda x: (x+x%2) >>> df4.groupby(level=0).transform(customSum) MultiIndexing Dates Visualization Also see NumPy Arrays >>> s.plot() >>> plt.show() Also see Matplotlib >>> import matplotlib.pyplot as plt >>> df2.plot() >>> plt.show() >>> df2['Date']= pd.to_datetime(df2['Date']) >>> df2['Date']= pd.date_range('2000-1-1', periods=6, freq='M') >>> dates = [datetime(2012,5,1), datetime(2012,5,2)] >>> index = pd.DatetimeIndex(dates) >>> index = pd.date_range(datetime(2012,2,1), end, freq='BM')