pandas directories on the python language.pptx

• Pandas is a Python library.
• Pandas is used to analyze data.

What is Pandas?
• Pandas is a Python library used for working with data
sets.
• It has functions for analyzing, cleaning, exploring, and
manipulating data.
• The name "Pandas" has a reference to both "Panel
Data", and "Python Data Analysis" and was created by
Wes McKinney in 2008.

Why Use Pandas?
• Pandas allows us to analyze big data and make
conclusions based on statistical theories.
• Pandas can clean messy data sets, and make them
readable and relevant.
• Relevant data is very important in data science.
• Data Science: is a branch of computer science where
we study how to store, use and analyze data for
deriving information from it.

What Can Pandas Do?
• Pandas gives you answers about the data. Like:
• Is there a correlation between two or more columns?
• What is average value?
• Max value?
• Min value?
• Pandas are also able to delete rows that are not
relevant, or contains wrong values, like empty or NULL
values. This is called cleaning the data.

Installation of Pandas
• If you have Python and PIP already installed on a system, then
installation of Pandas is very easy.
• Install it using this command:
C:UsersYour Name>pip install pandas
If this command fails, then use a python distribution that already
has Pandas installed like, Anaconda, Spyder etc.
Import Pandas
Once Pandas is installed, import it in your applications by adding
the import keyword:
import pandas

• ExampleGet your own Python Server
import pandas
mydataset = {
'cars': ["BMW", "Volvo", "Ford"],
'passings': [3, 7, 2]
}
myvar = pandas.DataFrame(mydataset)
print(myvar)

Pandas as pd
Pandas is usually imported under the pd alias.
Create an alias with the as keyword while importing:
import pandas as pd

Checking Pandas Version
• The version string is stored under __version__ attribute.
• Example
import pandas as pd
print(pd.__version__)

Pandas Series
• What is a Series?
• A Pandas Series is like a column in a table.
• It is a one-dimensional array holding data of any type.
• Create a simple Pandas Series from a list:
import pandas as pd
a = [1, 7, 2]
myvar = pd.Series(a)
print(myvar)
0 1
1 7
2 2
dtype: int64

• Labels
• If nothing else is specified, the values are labeled with their
index number. First value has index 0, second value has
index 1 etc.
• This label can be used to access a specified value.
import pandas as pd
a = [1, 7, 2]
myvar = pd.Series(a)
print(myvar[0])
Output: 1
Returns first value of the series.

Create Labels
With the index argument, you can name your own labels.
• Create your own labels:
import pandas as pd
a = [1, 7, 2]
myvar = pd.Series(a, index = ["x", "y", "z"])
print(myvar)

• When you have created labels, you can access an item
by referring to the label.
Example
• Return the value of "y":
import pandas as pd
a = [1, 7, 2]
myvar = pd.Series(a, index = ["x", "y", "z"])
print(myvar["y"])
Output:7

Key/Value Objects as Series
• You can also use a key/value object, like a dictionary,
when creating a Series.
• Create a simple Pandas Series from a dictionary:
import pandas as pd
calories = {"day1": 420, "day2": 380, "day3": 390}
myvar = pd.Series(calories)
print(myvar)

• Note: The keys of the dictionary become the labels.
• To select only some of the items in the dictionary, use the index
argument and specify only the items you want to include in the
Series.
• Example
• Create a Series using only data from "day1" and "day2":
import pandas as pd
calories = {"day1": 420, "day2": 380, "day3": 390}
myvar = pd.Series(calories, index = ["day1", "day2"])
print(myvar)

DataFrames
• Data sets in Pandas are usually multi-dimensional tables, called DataFrames.
• Series is like a column, a DataFrame is the whole table.
• Example
• Create a DataFrame from two Series:
import pandas as pd
data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}
myvar = pd.DataFrame(data)
print(myvar)

Pandas DataFrames
• What is a DataFrame?
• A Pandas DataFrame is a 2 dimensional data structure, like a 2 dimensional array, or a
table with rows and columns.
• Example
• Create a simple Pandas DataFrame:
import pandas as pd
data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}
#load data into a DataFrame object:
df = pd.DataFrame(data)
print(df)

• As you can see from the result above, the DataFrame is like a table with rows and columns.
• Pandas use the loc attribute to return one or more specified row(s)
• Example
• Return row 0:
import pandas as pd
data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}
#load data into a DataFrame object:
print(df.loc[0])
Result
calories 420
duration 50
Name: 0, dtype: int64

• Named Indexes
• With the index argument, you can name your own indexes.
• Example
• Add a list of names to give each row a name:
import pandas as pd
data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}
df = pd.DataFrame(data, index = ["day1", "day2", "day3"])
print(df)

Locate Named Indexes
Use the named index in the loc attribute to return the specified row(s).
Example
Return "day2":
import pandas as pd
data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}
df = pd.DataFrame(data, index = ["day1", "day2", "day3"])
print(df.loc["day2"])

Load Files Into a DataFrame
• If your data sets are stored in a file, Pandas can load
them into a DataFrame.
• Example
• Load a comma separated file (CSV file) into a
DataFrame:
import pandas as pd
df = pd.read_csv('data.csv')
print(df)

Pandas Read CSV
• Read CSV Files
• A simple way to store big data sets is to use CSV files
(comma separated files).
• CSV files contains plain text and is a well know format
that can be read by everyone including Pandas.
• In our examples we will be using a CSV file called
'data.csv'.

• Example
• Load the CSV into a DataFrame:
import pandas as pd
print(df.to_string())
If you have a large DataFrame with many rows,
Pandas will only return the first 5 rows, and the last 5
rows.
Tip: use to_string() to print the entire DataFrame.

max_rows
• The number of rows returned is defined in Pandas option settings.
• You can check your system's maximum rows with the
pd.options.display.max_rows statement.

• In my system the number is 60, which means that if the DataFrame
contains more than 60 rows, the print(df) statement will return only
the headers and the first and last 5 rows.
• You can change the maximum rows number with the same statement.
• Example
• Increase the maximum number of rows to display the entire
DataFrame:
import pandas as pd
pd.options.display.max_rows = 9999
print(df)

Pandas Read JSON
• Read JSON
• Big data sets are often stored, or extracted as JSON.
• JSON is plain text, but has the format of an object, and
is well known in the world of programming, including
Pandas.
• In our examples we will be using a JSON file called
'data.json'.

Load the JSON file into a DataFrame:
import pandas as pd
df = pd.read_json('data.json')

Dictionary as JSON
• JSON = Python Dictionary
• JSON objects have the same format as Python
dictionaries.
• If your JSON code is not in a file, but in a Python
Dictionary, you can load it into a DataFrame directly:
• Example
• Load a Python Dictionary into a DataFrame:

import pandas as pd
data = {
"Duration":{
"0":60,
"1":60,
"2":60,
"3":45,
"4":45,
"5":60
},
"Pulse":{
"0":110,
"1":117,
"2":103,
"3":109,
"4":117,
"5":102
},
"Maxpulse":{
"0":130,
"1":145,
"2":135,
"3":175,
"4":148,
"5":127
},
"Calories":{
"0":409,
"1":479,
"2":340,
"3":282,
"4":406,
"5":300
}
}
print(df)

Pandas - Analyzing DataFrames
• Viewing the Data
• One of the most used method for getting a quick overview of the
DataFrame, is the head() method.
• The head() method returns the headers and a specified number of rows,
starting from the top.
• Get a quick overview by printing the first 10 rows of the
DataFrame:
import pandas as pd
print(df.head(10))
Note: if the number of rows is not specified,
the head() method will return the top 5 rows.

• Print the first 5 rows of the DataFrame:
import pandas as pd
print(df.head())
• There is also a tail() method for viewing the last rows of the DataFrame.
• The tail() method returns the headers and a specified number of rows,
starting from the bottom.
• Example
Print the last 5 rows of the DataFrame:
print(df.tail())

import pandas as pd
print(df.tail())

Info About the Data
• The DataFrames object has a method called info(), that gives you
more information about the data set.
• Example
Print information about the data:
print(df.info())

Result Explained
• The result tells us there are 169 rows and 4 columns:
RangeIndex: 169 entries, 0 to 168
Data columns (total 4 columns):
• And the name of each column, with the data type:
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Duration 169 non-null int64
1 Pulse 169 non-null int64
2 Maxpulse 169 non-null int64
3 Calories 164 non-null float64

Null Values
• The info() method also tells us how many Non-Null values there are present
in each column, and in our data set it seems like there are 164 of 169 Non-
Null values in the "Calories" column.
• Which means that there are 5 rows with no value at all, in the "Calories"
column, for whatever reason.
• Empty values, or Null values, can be bad when analyzing data, and you
should consider removing rows with empty values. This is a step towards
what is called cleaning data, and you will learn more about that in the next
chapters.

Pandas - Cleaning Data
Data Cleaning
• Data cleaning means fixing bad data in your data set.
• Bad data could be:
• Empty cells
• Data in wrong format
• Wrong data
• Duplicates
• In this tutorial you will learn how to deal with all of them.

Pandas - Cleaning Empty Cells
• Empty Cells
• Empty cells can potentially give you a wrong result
when you analyze data.
• Remove Rows
• One way to deal with empty cells is to remove rows
that contain empty cells.
• This is usually OK, since data sets can be very big, and
removing a few rows will not have a big impact on the
result.

• Return a new Data Frame with no empty cells:
import pandas as pd
new_df = df.dropna()
print(new_df.to_string())
Note: By default, the dropna() method returns a new DataFrame, and will not change the original.

• If you want to change the original DataFrame, use the inplace = True
argument:
• Remove all rows with NULL values:
import pandas as pd
df.dropna(inplace = True)

Replace Empty Values
• Another way of dealing with empty cells is to insert a new value
instead.
• This way you do not have to delete entire rows just because of some
empty cells.
• The fillna() method allows us to replace empty cells with a value:
• Replace NULL values with the number 130:
import pandas as pd
df.fillna(130, inplace = True)

Replace Only For Specified Columns
• The example above replaces all empty cells in the whole
Data Frame.
• To only replace empty values for one column, specify
the column name for the DataFrame:
• Replace NULL values in the "Calories" columns with the
number 130:
import pandas as pd
df["Calories"].fillna(130, inplace = True)

Replace Using Mean, Median, or Mode
• A common way to replace empty cells, is to calculate the mean,
median or mode value of the column.
• Pandas uses the mean() median() and mode() methods to calculate
the respective values for a specified column:
• Calculate the MEAN, and replace any empty values with
it:
import pandas as pd
x = df["Calories"].mean()
df["Calories"].fillna(x, inplace = True)
Mean = the average value (the sum of all values divided by number of values).

• Calculate the MEDIAN, and replace any empty values
with it:
import pandas as pd
x = df["Calories"].median()
Median = the value in the middle, after you have sorted all values ascending.

• Calculate the MODE, and replace any empty values with
it:
import pandas as pd
x = df["Calories"].mode()[0]
Mode = the value that appears most frequently.

Pandas - Cleaning Data of Wrong Format
• Data of Wrong Format
• Cells with data of wrong format can make it difficult, or
even impossible, to analyze data.
• To fix it, you have two options: remove the rows, or convert
all cells in the columns into the same format.
• Convert Into a Correct Format
• In our Data Frame, we have two cells with the wrong
format. Check out row 24 and 28, the 'Date' column should
be a string that represents a date:
• Let's try to convert all cells in the 'Date' column into dates.
• Pandas has a to_datetime() method for this:

Convert to date:
import pandas as pd
df['Date'] = pd.to_datetime(df['Date'])
• As you can see from the result, the date in row 26 was
fixed, but the empty date in row 22 got a NaT (Not a Time)
value, in other words an empty value. One way to deal
with empty values is simply removing the entire row.

Pandas - Fixing Wrong Data
• Wrong Data
• "Wrong data" does not have to be "empty cells" or "wrong format", it
can just be wrong, like if someone registered "199" instead of "1.99".
• Sometimes you can spot wrong data by looking at the data set,
because you have an expectation of what it should be.
• If you take a look at our data set, you can see that in row 7, the
duration is 450, but for all the other rows the duration is between 30
and 60.
• It doesn't have to be wrong, but taking in consideration that this is the
data set of someone's workout sessions, we conclude with the fact that
this person did not work out in 450 minutes.

Replacing Values
• One way to fix wrong values is to replace them with
something else.
• In our example, it is most likely a typo, and the value
should be "45" instead of "450", and we could just
insert "45" in row 7:
import pandas as pd
df.loc[7,'Duration'] = 45

• For small data sets you might be able to replace the wrong data one by
one, but not for big data sets.
• To replace wrong data for larger data sets you can create some rules,
e.g. set some boundaries for legal values, and replace any values that
are outside of the boundaries.
• Loop through all values in the "Duration" column.
• If the value is higher than 120, set it to 120:
import pandas as pd
for x in df.index:
if df.loc[x, "Duration"] > 120:
df.loc[x, "Duration"] = 120

Pandas - Removing Duplicates
• Discovering Duplicates
• Duplicate rows are rows that have been registered
more than one time.

• By taking a look at our test data set, we can assume that row 11 and
12 are duplicates.
• To discover duplicates, we can use the duplicated() method.
• The duplicated() method returns a Boolean values for each row:
• ExampleGet your own Python Server
• Returns True for every row that is a duplicate, othwerwise False:
import pandas as pd
print(df.duplicated())

Removing Duplicates
• To remove duplicates, use the drop_duplicates() method.
import pandas as pd
df.drop_duplicates(inplace = True)
Remember: The (inplace = True) will make sure that the method does NOT return a new DataFrame, but it will remove all duplicates from the original DataFrame.

Plotting
• Pandas uses the plot() method to create diagrams.
• We can use Pyplot, a submodule of the Matplotlib library to visualize
the diagram on the screen.
• Import pyplot from Matplotlib and visualize our
DataFrame:
import pandas as pd
import matplotlib.pyplot as plt
df.plot()
plt.show()

pandas directories on the python language.pptx

More Related Content

Similar to pandas directories on the python language.pptx (20)

Recently uploaded (20)

pandas directories on the python language.pptx