Numerical Computing with Python and Numpy
This tutorial series is a beginner-friendly introduction to programming and data analysis using the Python programming language. These tutorials take a practical and coding-focused approach. The best way to learn the material is to execute the code and experiment with it yourself.
This tutorial covers the following topics:
- Working with numerical data in Python
- Going from Python lists to Numpy arrays
- Multi-dimensional Numpy arrays and their benefits
- Array operations, broadcasting, indexing, and slicing
- Working with CSV data files using Numpy
This tutorial is an executable Jupyter notebook. Click the Open in Colab button at the top of this page to execute the code.
Jupyter Notebooks: This notebook is made of cells. Each cell can contain code written in Python or explanations in plain English. You can execute code cells and view the results instantly within the notebook. Jupyter is a powerful platform for experimentation and analysis. Don't be afraid to mess around with the code & break things - you'll learn a lot by encountering and fixing errors. You can use the "Kernel > Restart & Clear Output" menu option to clear all outputs and start again from the top.
Working with numerical data
The "data" in Data Analysis typically refers to numerical data, e.g., stock prices, sales figures, sensor measurements, sports scores, database tables, etc. The Numpy library provides specialized data structures, functions, and other tools for numerical computing in Python. Let's work through an example to see why & how to use Numpy for working with numerical data.
Suppose we want to use climate data like the temperature, rainfall, and humidity to determine if a region is well suited for growing apples. A simple approach for doing this would be to formulate the relationship between the annual yield of apples (tons per hectare) and the climatic conditions like the average temperature (in degrees Fahrenheit), rainfall (in millimeters) & average relative humidity (in percentage) as a linear equation.
yield_of_apples = w1 * temperature + w2 * rainfall + w3 * humidity
We're expressing the yield of apples as a weighted sum of the temperature, rainfall, and humidity. This equation is an approximation since the actual relationship may not necessarily be linear, and there may be other factors involved. But a simple linear model like this often works well in practice.
Based on some statical analysis of historical data, we might come up with reasonable values for the weights w1, w2, and w3. Here's an example set of values:
w1, w2, w3 = 0.3, 0.2, 0.5Given some climate data for a region, we can now predict the yield of apples. Here's some sample data:
To begin, we can define some variables to record climate data for a region.
kanto_temp = 73
kanto_rainfall = 67
kanto_humidity = 43We can now substitute these variables into the linear equation to predict the yield of apples.
kanto_yield_apples = kanto_temp * w1 + kanto_rainfall * w2 + kanto_humidity * w3
kanto_yield_apples56.8print("The expected yield of apples in Kanto region is {} tons per hectare.".format(kanto_yield_apples))The expected yield of apples in Kanto region is 56.8 tons per hectare.
To make it slightly easier to perform the above computation for multiple regions, we can represent the climate data for each region as a vector, i.e., a list of numbers.
kanto = [73, 67, 43]
johto = [91, 88, 64]
hoenn = [87, 134, 58]
sinnoh = [102, 43, 37]
unova = [69, 96, 70]The three numbers in each vector represent the temperature, rainfall, and humidity data, respectively.
We can also represent the set of weights used in the formula as a vector.
weights = [w1, w2, w3]We can now write a function crop_yield to calcuate the yield of apples (or any other crop) given the climate data and the respective weights.
def crop_yield(region, weights):
result = 0
for x, w in zip(region, weights):
result += x * w
return resultcrop_yield(kanto, weights)56.8crop_yield(johto, weights)76.9crop_yield(unova, weights)74.9Going from Python lists to Numpy arrays
The calculation performed by the crop_yield (element-wise multiplication of two vectors and taking a sum of the results) is also called the dot product. Learn more about dot product here: https://www.khanacademy.org/math/linear-algebra/vectors-and-spaces/dot-cross-products/v/vector-dot-product-and-vector-length .
The Numpy library provides a built-in function to compute the dot product of two vectors. However, we must first convert the lists into Numpy arrays.
Let's install the Numpy library using the pip package manager.
!pip install numpy --upgrade --quietNext, let's import the numpy module. It's common practice to import numpy with the alias np.
import numpy as npWe can now use the np.array function to create Numpy arrays.
kanto = np.array([73, 67, 43])kantoarray([73, 67, 43])weights = np.array([w1, w2, w3])weightsarray([0.3, 0.2, 0.5])Numpy arrays have the type ndarray.
type(kanto)numpy.ndarraytype(weights)numpy.ndarrayJust like lists, Numpy arrays support the indexing notation [].
weights[0]0.3kanto[2]43Operating on Numpy arrays
We can now compute the dot product of the two vectors using the np.dot function.
np.dot(kanto, weights)56.8We can achieve the same result with low-level operations supported by Numpy arrays: performing an element-wise multiplication and calculating the resulting numbers' sum.
(kanto * weights).sum()56.8The * operator performs an element-wise multiplication of two arrays if they have the same size. The sum method calculates the sum of numbers in an array.
arr1 = np.array([1, 2, 3])
arr2 = np.array([4, 5, 6])arr1 * arr2array([ 4, 10, 18])arr2.sum()15Benefits of using Numpy arrays
Numpy arrays offer the following benefits over Python lists for operating on numerical data:
- Ease of use: You can write small, concise, and intuitive mathematical expressions like
(kanto * weights).sum()rather than using loops & custom functions likecrop_yield. - Performance: Numpy operations and functions are implemented internally in C++, which makes them much faster than using Python statements & loops that are interpreted at runtime
Here's a comparison of dot products performed using Python loops vs. Numpy arrays on two vectors with a million elements each.
# Python lists
arr1 = list(range(1000000))
arr2 = list(range(1000000, 2000000))
# Numpy arrays
arr1_np = np.array(arr1)
arr2_np = np.array(arr2)%%time
result = 0
for x1, x2 in zip(arr1, arr2):
result += x1*x2
resultCPU times: user 151 ms, sys: 1.35 ms, total: 153 ms
Wall time: 152 ms
833332333333500000%%time
np.dot(arr1_np, arr2_np)CPU times: user 1.96 ms, sys: 751 µs, total: 2.71 ms
Wall time: 1.6 ms
833332333333500000As you can see, using np.dot is 100 times faster than using a for loop. This makes Numpy especially useful while working with really large datasets with tens of thousands or millions of data points.
Multi-dimensional Numpy arrays
We can now go one step further and represent the climate data for all the regions using a single 2-dimensional Numpy array.
climate_data = np.array([[73, 67, 43],
[91, 88, 64],
[87, 134, 58],
[102, 43, 37],
[69, 96, 70]])climate_dataarray([[ 73, 67, 43],
[ 91, 88, 64],
[ 87, 134, 58],
[102, 43, 37],
[ 69, 96, 70]])If you've taken a linear algebra class in high school, you may recognize the above 2-d array as a matrix with five rows and three columns. Each row represents one region, and the columns represent temperature, rainfall, and humidity, respectively.
Numpy arrays can have any number of dimensions and different lengths along each dimension. We can inspect the length along each dimension using the .shape property of an array.
# 2D array (matrix)
climate_data.shape(5, 3)weightsarray([0.3, 0.2, 0.5])# 1D array (vector)
weights.shape(3,)# 3D array
arr3 = np.array([
[[11, 12, 13],
[13, 14, 15]],
[[15, 16, 17],
[17, 18, 19.5]]])arr3.shape(2, 2, 3)All the elements in a numpy array have the same data type. You can check the data type of an array using the .dtype property.
weights.dtypedtype('float64')climate_data.dtypedtype('int64')If an array contains even a single floating point number, all the other elements are also converted to floats.
arr3.dtypedtype('float64')We can now compute the predicted yields of apples in all the regions, using a single matrix multiplication between climate_data (a 5x3 matrix) and weights (a vector of length 3). Here's what it looks like visually:
You can learn about matrices and matrix multiplication by watching the first 3-4 videos of this playlist: https://www.youtube.com/watch?v=xyAuNHPsq-g&list=PLFD0EB975BA0CC1E0&index=1 .
We can use the np.matmul function or the @ operator to perform matrix multiplication.
np.matmul(climate_data, weights)array([56.8, 76.9, 81.9, 57.7, 74.9])climate_data @ weightsarray([56.8, 76.9, 81.9, 57.7, 74.9])Working with CSV data files
Numpy also provides helper functions reading from & writing to files. Let's download a file climate.txt, which contains 10,000 climate measurements (temperature, rainfall & humidity) in the following format:
temperature,rainfall,humidity
25.00,76.00,99.00
39.00,65.00,70.00
59.00,45.00,77.00
84.00,63.00,38.00
66.00,50.00,52.00
41.00,94.00,77.00
91.00,57.00,96.00
49.00,96.00,99.00
67.00,20.00,28.00
...
This format of storing data is known as comma-separated values or CSV.
CSVs: A comma-separated values (CSV) file is a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas. A CSV file typically stores tabular data (numbers and text) in plain text, in which case each line will have the same number of fields. (Wikipedia)
To read this file into a numpy array, we can use the genfromtxt function.
import urllib.request
urllib.request.urlretrieve(
'https://gist.github.com/BirajCoder/a4ffcb76fd6fb221d76ac2ee2b8584e9/raw/4054f90adfd361b7aa4255e99c2e874664094cea/climate.csv',
'climate.txt')('climate.txt', <http.client.HTTPMessage at 0x7ffa9c1e2668>)climate_data = np.genfromtxt('climate.txt', delimiter=',', skip_header=1)climate_dataarray([[25., 76., 99.],
[39., 65., 70.],
[59., 45., 77.],
...,
[99., 62., 58.],
[70., 71., 91.],
[92., 39., 76.]])climate_data.shape(10000, 3)We can now perform a matrix multiplication using the @ operator to predict the yield of apples for the entire dataset using a given set of weights.
weights = np.array([0.3, 0.2, 0.5])yields = climate_data @ weightsyieldsarray([72.2, 59.7, 65.2, ..., 71.1, 80.7, 73.4])yields.shape(10000,)Let's add the yields to climate_data as a fourth column using the np.concatenate function.
climate_results = np.concatenate((climate_data, yields.reshape(10000, 1)), axis=1)climate_resultsarray([[25. , 76. , 99. , 72.2],
[39. , 65. , 70. , 59.7],
[59. , 45. , 77. , 65.2],
...,
[99. , 62. , 58. , 71.1],
[70. , 71. , 91. , 80.7],
[92. , 39. , 76. , 73.4]])There are a couple of subtleties here:
-
Since we wish to add new columns, we pass the argument
axis=1tonp.concatenate. Theaxisargument specifies the dimension for concatenation. -
The arrays should have the same number of dimensions, and the same length along each except the dimension used for concatenation. We use the
np.reshapefunction to change the shape ofyieldsfrom(10000,)to(10000,1).
Here's a visual explanation of np.concatenate along axis=1 (can you guess what axis=0 results in?):
The best way to understand what a Numpy function does is to experiment with it and read the documentation to learn about its arguments & return values. Use the cells below to experiment with np.concatenate and np.reshape.
Let's write the final results from our computation above back to a file using the np.savetxt function.
climate_resultsarray([[25. , 76. , 99. , 72.2],
[39. , 65. , 70. , 59.7],
[59. , 45. , 77. , 65.2],
...,
[99. , 62. , 58. , 71.1],
[70. , 71. , 91. , 80.7],
[92. , 39. , 76. , 73.4]])np.savetxt('climate_results.txt',
climate_results,
fmt='%.2f',
delimiter=',',
header='temperature,rainfall,humidity,yeild_apples',
comments='')The results are written back in the CSV format to the file climate_results.txt.
temperature,rainfall,humidity,yeild_apples
25.00,76.00,99.00,72.20
39.00,65.00,70.00,59.70
59.00,45.00,77.00,65.20
84.00,63.00,38.00,56.80
...
Numpy provides hundreds of functions for performing operations on arrays. Here are some commonly used functions:
- Mathematics:
np.sum,np.exp,np.round, arithemtic operators - Array manipulation:
np.reshape,np.stack,np.concatenate,np.split - Linear Algebra:
np.matmul,np.dot,np.transpose,np.eigvals - Statistics:
np.mean,np.median,np.std,np.max
How to find the function you need? The easiest way to find the right function for a specific operation or use-case is to do a web search. For instance, searching for "How to join numpy arrays" leads to this tutorial on array concatenation.
You can find a full list of array functions here: https://numpy.org/doc/stable/reference/routines.html
Arithmetic operations, broadcasting and comparison
Numpy arrays support arithmetic operators like +, -, *, etc. You can perform an arithmetic operation with a single number (also called scalar) or with another array of the same shape. Operators make it easy to write mathematical expressions with multi-dimensional arrays.
arr2 = np.array([[1, 2, 3, 4],
[5, 6, 7, 8],
[9, 1, 2, 3]])arr3 = np.array([[11, 12, 13, 14],
[15, 16, 17, 18],
[19, 11, 12, 13]])# Adding a scalar
arr2 + 3array([[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[12, 4, 5, 6]])# Element-wise subtraction
arr3 - arr2array([[10, 10, 10, 10],
[10, 10, 10, 10],
[10, 10, 10, 10]])# Division by scalar
arr2 / 2array([[0.5, 1. , 1.5, 2. ],
[2.5, 3. , 3.5, 4. ],
[4.5, 0.5, 1. , 1.5]])# Element-wise multiplication
arr2 * arr3array([[ 11, 24, 39, 56],
[ 75, 96, 119, 144],
[171, 11, 24, 39]])# Modulus with scalar
arr2 % 4array([[1, 2, 3, 0],
[1, 2, 3, 0],
[1, 1, 2, 3]])Array Broadcasting
Numpy arrays also support broadcasting, allowing arithmetic operations between two arrays with different numbers of dimensions but compatible shapes. Let's look at an example to see how it works.
arr2 = np.array([[1, 2, 3, 4],
[5, 6, 7, 8],
[9, 1, 2, 3]])arr2.shape(3, 4)arr4 = np.array([4, 5, 6, 7])arr4.shape(4,)arr2 + arr4array([[ 5, 7, 9, 11],
[ 9, 11, 13, 15],
[13, 6, 8, 10]])When the expression arr2 + arr4 is evaluated, arr4 (which has the shape (4,)) is replicated three times to match the shape (3, 4) of arr2. Numpy performs the replication without actually creating three copies of the smaller dimension array, thus improving performance and using lower memory.
Broadcasting only works if one of the arrays can be replicated to match the other array's shape.
arr5 = np.array([7, 8])arr5.shape(2,)arr2 + arr5---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-72-c22e92053c39> in <module>
----> 1 arr2 + arr5
ValueError: operands could not be broadcast together with shapes (3,4) (2,) In the above example, even if arr5 is replicated three times, it will not match the shape of arr2. Hence arr2 + arr5 cannot be evaluated successfully. Learn more about broadcasting here: https://numpy.org/doc/stable/user/basics.broadcasting.html .
Array Comparison
Numpy arrays also support comparison operations like ==, !=, > etc. The result is an array of booleans.
arr1 = np.array([[1, 2, 3], [3, 4, 5]])
arr2 = np.array([[2, 2, 3], [1, 2, 5]])arr1 == arr2array([[False, True, True],
[False, False, True]])arr1 != arr2array([[ True, False, False],
[ True, True, False]])arr1 >= arr2array([[False, True, True],
[ True, True, True]])arr1 < arr2array([[ True, False, False],
[False, False, False]])Array comparison is frequently used to count the number of equal elements in two arrays using the sum method. Remember that True evaluates to 1 and False evaluates to 0 when booleans are used in arithmetic operations.
(arr1 == arr2).sum()3Array indexing and slicing
Numpy extends Python's list indexing notation using [] to multiple dimensions in an intuitive fashion. You can provide a comma-separated list of indices or ranges to select a specific element or a subarray (also called a slice) from a Numpy array.
arr3 = np.array([
[[11, 12, 13, 14],
[13, 14, 15, 19]],
[[15, 16, 17, 21],
[63, 92, 36, 18]],
[[98, 32, 81, 23],
[17, 18, 19.5, 43]]])arr3.shape(3, 2, 4)# Single element
arr3[1, 1, 2]36.0# Subarray using ranges
arr3[1:, 0:1, :2]array([[[15., 16.]],
[[98., 32.]]])# Mixing indices and ranges
arr3[1:, 1, 3]array([18., 43.])# Mixing indices and ranges
arr3[1:, 1, :3]array([[63. , 92. , 36. ],
[17. , 18. , 19.5]])# Using fewer indices
arr3[1]array([[15., 16., 17., 21.],
[63., 92., 36., 18.]])# Using fewer indices
arr3[:2, 1]array([[13., 14., 15., 19.],
[63., 92., 36., 18.]])# Using too many indices
arr3[1,3,2,1]---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
<ipython-input-87-fbde713646b3> in <module>
1 # Using too many indices
----> 2 arr3[1,3,2,1]
IndexError: too many indices for array: array is 3-dimensional, but 4 were indexedThe notation and its results can seem confusing at first, so take your time to experiment and become comfortable with it. Use the cells below to try out some examples of array indexing and slicing, with different combinations of indices and ranges. Here are some more examples demonstrated visually:
Other ways of creating Numpy arrays
Numpy also provides some handy functions to create arrays of desired shapes with fixed or random values. Check out the official documentation or use the help function to learn more.
# All zeros
np.zeros((3, 2))array([[0., 0.],
[0., 0.],
[0., 0.]])# All ones
np.ones([2, 2, 3])array([[[1., 1., 1.],
[1., 1., 1.]],
[[1., 1., 1.],
[1., 1., 1.]]])# Identity matrix
np.eye(3)array([[1., 0., 0.],
[0., 1., 0.],
[0., 0., 1.]])# Random vector
np.random.rand(5)array([0.8208279 , 0.93801216, 0.44954753, 0.17816933, 0.32049174])# Random matrix
np.random.randn(2, 3) # rand vs. randn - what's the difference?array([[ 0.83775 , 1.13851471, -0.79147694],
[ 0.56320765, 1.00386056, -0.42502339]])# Fixed value
np.full([2, 3], 42)array([[42, 42, 42],
[42, 42, 42]])# Range with start, end and step
np.arange(10, 90, 3)array([10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, 43, 46, 49, 52, 55, 58,
61, 64, 67, 70, 73, 76, 79, 82, 85, 88])# Equally spaced numbers in a range
np.linspace(3, 27, 9)array([ 3., 6., 9., 12., 15., 18., 21., 24., 27.])Exercises
Try the following exercises to become familiar with Numpy arrays and practice your skills:
- Assignment on Numpy array functions: https://jovian.com/aakashns/numpy-array-operations
- (Optional) 100 numpy exercises: https://jovian.com/aakashns/100-numpy-exercises
Summary and Further Reading
With this, we complete our discussion of numerical computing with Numpy. We've covered the following topics in this tutorial:
- Going from Python lists to Numpy arrays
- Operating on Numpy arrays
- Benefits of using Numpy arrays over lists
- Multi-dimensional Numpy arrays
- Working with CSV data files
- Arithmetic operations and broadcasting
- Array indexing and slicing
- Other ways of creating Numpy arrays
Check out the following resources for learning more about Numpy:
- Official tutorial: https://numpy.org/devdocs/user/quickstart.html
- Numpy tutorial on W3Schools: https://www.w3schools.com/python/numpy_intro.asp
- Advanced Numpy (exploring the internals): http://scipy-lectures.org/advanced/advanced_numpy/index.html
Questions for Revision
Try answering the following questions to test your understanding of the topics covered in this notebook:
- What is a vector?
- How do you represent vectors using a Python list? Give an example.
- What is a dot product of two vectors?
- Write a function to compute the dot product of two vectors.
- What is Numpy?
- How do you install Numpy?
- How do you import the
numpymodule? - What does it mean to import a module with an alias? Give an example.
- What is the commonly used alias for
numpy? - What is a Numpy array?
- How do you create a Numpy array? Give an example.
- What is the type of Numpy arrays?
- How do you access the elements of a Numpy array?
- How do you compute the dot product of two vectors using Numpy?
- What happens if you try to compute the dot product of two vectors which have different sizes?
- How do you compute the element-wise product of two Numpy arrays?
- How do you compute the sum of all the elements in a Numpy array?
- What are the benefits of using Numpy arrays over Python lists for operating on numerical data?
- Why do Numpy array operations have better performance compared to Python functions and loops?
- Illustrate the performance difference between Numpy array operations and Python loops using an example.
- What are multi-dimensional Numpy arrays?
- Illustrate the creation of Numpy arrays with 2, 3, and 4 dimensions.
- How do you inspect the number of dimensions and the length along each dimension in a Numpy array?
- Can the elements of a Numpy array have different data types?
- How do you check the data type of the elements of a Numpy array?
- What is the data type of a Numpy array?
- What is the difference between a matrix and a 2D Numpy array?
- How do you perform matrix multiplication using Numpy?
- What is the
@operator used for in Numpy? - What is the CSV file format?
- How do you read data from a CSV file using Numpy?
- How do you concatenate two Numpy arrays?
- What is the purpose of the
axisargument ofnp.concatenate? - When are two Numpy arrays compatible for concatenation?
- Give an example of two Numpy arrays that can be concatenated.
- Give an example of two Numpy arrays that cannot be concatenated.
- What is the purpose of the
np.reshapefunction? - What does it mean to “reshape” a Numpy array?
- How do you write a numpy array into a CSV file?
- Give some examples of Numpy functions for performing mathematical operations.
- Give some examples of Numpy functions for performing array manipulation.
- Give some examples of Numpy functions for performing linear algebra.
- Give some examples of Numpy functions for performing statistical operations.
- How do you find the right Numpy function for a specific operation or use case?
- Where can you see a list of all the Numpy array functions and operations?
- What are the arithmetic operators supported by Numpy arrays? Illustrate with examples.
- What is array broadcasting? How is it useful? Illustrate with an example.
- Give some examples of arrays that are compatible for broadcasting?
- Give some examples of arrays that are not compatible for broadcasting?
- What are the comparison operators supported by Numpy arrays? Illustrate with examples.
- How do you access a specific subarray or slice from a Numpy array?
- Illustrate array indexing and slicing in multi-dimensional Numpy arrays with some examples.
- How do you create a Numpy array with a given shape containing all zeros?
- How do you create a Numpy array with a given shape containing all ones?
- How do you create an identity matrix of a given shape?
- How do you create a random vector of a given length?
- How do you create a Numpy array with a given shape with a fixed value for each element?
- How do you create a Numpy array with a given shape containing randomly initialized elements?
- What is the difference between
np.random.randandnp.random.randn? Illustrate with examples. - What is the difference between
np.arangeandnp.linspace? Illustrate with examples.
# Ucomment the next line if you need install numpy
# !pip install numpy --upgrade 17. What is the result of the following expression? (★☆☆)
0 * np.nan
np.nan == np.nan
np.inf > np.nan
np.nan - np.nan
np.nan in set([np.nan])
0.3 == 3 * 0.1
26. What is the output of the following script? (★☆☆)
# Author: Jake VanderPlas
print(sum(range(5),-1))
from numpy import *
print(sum(range(5),-1))
27. Consider an integer vector Z, which of these expressions are legal? (★☆☆)
Z**Z
2 << Z >> 2
Z <- Z
1j*Z
Z/1/1
Z<Z>Z
28. What are the result of the following expressions?
np.array(0) / np.array(0)
np.array(0) // np.array(0)
np.array([np.nan]).astype(int).astype(float)
32. Is the following expressions true? (★☆☆)
np.sqrt(-1) == np.emath.sqrt(-1)
54. How to read the following file? (★★☆)
1, 2, 3, 4, 5
6, , , 7, 8
, , 9,10,11
Congratulations on completing the 100 exercises, you're now an expert on Numpy!!
What to do next?
- Share your completed notebook on Facebook, LinkedIn or Twitter and challenge your friends.
- Check out our course on "Data Analysis with Python: Zero to Pandas" - https://jovian.com/learn/data-analysis-with-python-zero-to-pandas
This tutorial covers the following topics:
- Interacting with the filesystem using the
osmodule - Downloading files from the internet using the
urllibmodule - Reading and processing data from text files
- Parsing data from CSV files into dictionaries & lists
- Writing formatted data back to text files
Interacting with the OS and filesystem
The os module in Python provides many functions for interacting with the OS and the filesystem. Let's import it and try out some examples.
import osWe can check the present working directory using the os.getcwd function.
os.getcwd()'/home/jovyan'To get the list of files in a directory, use os.listdir. You pass an absolute or relative path of a directory as the argument to the function.
help(os.listdir)Help on built-in function listdir in module posix:
listdir(path=None)
Return a list containing the names of the files in the directory.
path can be specified as either str, bytes, or a path-like object. If path is bytes,
the filenames returned will also be bytes; in all other circumstances
the filenames returned will be str.
If path is None, uses the path='.'.
On some platforms, path may also be specified as an open file descriptor;\
the file descriptor must refer to a directory.
If this functionality is unavailable, using it raises NotImplementedError.
The list is in arbitrary order. It does not include the special
entries '.' and '..' even if they are present in the directory.
os.listdir('.') # relative path['.profile',
'.bashrc',
'.bash_logout',
'python-os-and-filesystem.ipynb',
'.local',
'.cache',
'data',
'.jupyter',
'.jovian',
'.ipython',
'.ipynb_checkpoints',
'.empty',
'work',
'.config',
'.conda',
'.git',
'.yarn',
'.jovianrc']os.listdir('/usr') # absolute path['lib32',
'games',
'libx32',
'bin',
'src',
'sbin',
'lib64',
'include',
'local',
'share',
'lib']You can create a new directory using os.makedirs. Let's create a new directory called data, where we'll later download some files.
os.makedirs('./data', exist_ok=True)Can you figure out what the argument exist_ok does? Try using the help function or read the documentation.
Let's verify that the directory was created and is currently empty.
'data' in os.listdir('.')Trueos.listdir('./data')['loans2.txt',
'emis2.txt',
'loans3.txt',
'emis1.txt',
'emis3.txt',
'movies.csv',
'loans1.txt']Let us download some files into the data directory using the urllib module.
url1 = 'https://gist.githubusercontent.com/aakashns/257f6e6c8719c17d0e498ea287d1a386/raw/7def9ef4234ddf0bc82f855ad67dac8b971852ef/loans1.txt'
url2 = 'https://gist.githubusercontent.com/aakashns/257f6e6c8719c17d0e498ea287d1a386/raw/7def9ef4234ddf0bc82f855ad67dac8b971852ef/loans2.txt'
url3 = 'https://gist.githubusercontent.com/aakashns/257f6e6c8719c17d0e498ea287d1a386/raw/7def9ef4234ddf0bc82f855ad67dac8b971852ef/loans3.txt'from urllib.request import urlretrieveurlretrieve(url1, './data/loans1.txt')('./data/loans1.txt', <http.client.HTTPMessage at 0x7f2d5d298220>)urlretrieve(url2, './data/loans2.txt')('./data/loans2.txt', <http.client.HTTPMessage at 0x7f2d5d298d30>)urlretrieve(url3, './data/loans3.txt')('./data/loans3.txt', <http.client.HTTPMessage at 0x7f2d47bd32e0>)Let's verify that the files were downloaded.
os.listdir('./data')['loans2.txt',
'emis2.txt',
'loans3.txt',
'emis1.txt',
'emis3.txt',
'movies.csv',
'loans1.txt']You can also use the requests library to dowload URLs, although you'll need to write some additional code to save the contents of the page to a file.
Reading from a file
To read the contents of a file, we first need to open the file using the built-in open function. The open function returns a file object and provides several methods for interacting with the file's contents.
file1 = open('./data/loans1.txt', mode='r')The open function also accepts a mode argument to specifies how we can interact with the file. The following options are supported:
========= ===============================================================
Character Meaning
--------- ---------------------------------------------------------------
'r' open for reading (default)
'w' open for writing, truncating the file first
'x' create a new file and open it for writing
'a' open for writing, appending to the end of the file if it exists
'b' binary mode
't' text mode (default)
'+' open a disk file for updating (reading and writing)
'U' universal newline mode (deprecated)
========= ===============================================================
To view the contents of the file, we can use the read method of the file object.
file1_contents = file1.read()print(file1_contents)amount,duration,rate,down_payment
100000,36,0.08,20000
200000,12,0.1,
628400,120,0.12,100000
4637400,240,0.06,
42900,90,0.07,8900
916000,16,0.13,
45230,48,0.08,4300
991360,99,0.08,
423000,27,0.09,47200
The file contains information about loans. It is a set of comma-separated values (CSV).
CSVs: A comma-separated values (CSV) file is a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas. A CSV file typically stores tabular data (numbers and text) in plain text, in which case each line will have the same number of fields. (Wikipedia)
The first line of the file is the header, indicating what each of the numbers on the remaining lines represents. Each of the remaining lines provides information about a loan. Thus, the second line 10000,36,0.08,20000 represents a loan with:
- an amount of
$10000, - duration of
36months, - rate of interest of
8%per annum, and - a down payment of
$20000
The CSV is a standard file format used for sharing data for analysis and visualization. Over the course of this tutorial, we will read the data from these CSV files, process it, and write the results back to files. Before we continue, let's close the file using the close method (otherwise, Python will continue to hold the entire file in the RAM)
file1.close()Once a file is closed, you can no longer read from it.
file1.read()---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-19-2fcf3b5e742f> in <module>
----> 1 file1.read()
ValueError: I/O operation on closed file.Closing files automatically using with
To close a file automatically after you've processed it, you can open it using the with statement.
with open('./data/loans2.txt') as file2:
file2_contents = file2.read()
print(file2_contents)amount,duration,rate,down_payment
828400,120,0.11,100000
4633400,240,0.06,
42900,90,0.08,8900
983000,16,0.14,
15230,48,0.07,4300
Once the statements within the with block are executed, the .close method on file2 is automatically invoked. Let's verify this by trying to read from the file object again.
file2.read()---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-21-4a9205324152> in <module>
----> 1 file2.read()
ValueError: I/O operation on closed file.Reading a file line by line
File objects provide a readlines method to read a file line-by-line.
with open('./data/loans3.txt', 'r') as file3:
file3_lines = file3.readlines()file3_lines['amount,duration,rate,down_payment\n',
'45230,48,0.07,4300\n',
'883000,16,0.14,\n',
'100000,12,0.1,\n',
'728400,120,0.12,100000\n',
'3637400,240,0.06,\n',
'82900,90,0.07,8900\n',
'316000,16,0.13,\n',
'15230,48,0.08,4300\n',
'991360,99,0.08,\n',
'323000,27,0.09,4720010000,36,0.08,20000\n',
'528400,120,0.11,100000\n',
'8633400,240,0.06,\n',
'12900,90,0.08,8900']Processing data from files
Before performing any operations on the data stored in a file, we need to convert the file's contents from one large string into Python data types. For the file loans1.txt containing information about loans in a CSV format, we can do the following:
- Read the file line by line
- Parse the first line to get a list of the column names or headers
- Split each remaining line and convert each value into a float
- Create a dictionary for each loan using the headers as keys
- Create a list of dictionaries to keep track of all the loans
Since we will perform the same operations for multiple files, it would be useful to define a function read_csv. We'll also define some helper functions to build up the functionality step by step.
Let's start by defining a function parse_header that takes a line as input and returns a list of column headers.
def parse_headers(header_line):
return header_line.strip().split(',')The strip method removes any extra spaces and the newline character \n. The split method breaks a string into a list using the given separator (, in this case).
file3_lines[0]'amount,duration,rate,down_payment\n'headers = parse_headers(file3_lines[0])headers['amount', 'duration', 'rate', 'down_payment']Next, let's define a function parse_values that takes a line containing some data and returns a list of floating-point numbers.
def parse_values(data_line):
values = []
for item in data_line.strip().split(','):
values.append(float(item))
return valuesfile3_lines[1]'45230,48,0.07,4300\n'parse_values(file3_lines[1])[45230.0, 48.0, 0.07, 4300.0]The values were parsed and converted to floating point numbers, as expected. Let's try it for another line from the file, which does not contain a value for the down payment.
file3_lines[2]'883000,16,0.14,\n'parse_values(file3_lines[2])---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-32-b170e8564609> in <module>
----> 1 parse_values(file3_lines[2])
<ipython-input-28-8be4024619c8> in parse_values(data_line)
2 values = []
3 for item in data_line.strip().split(','):
----> 4 values.append(float(item))
5 return values
ValueError: could not convert string to float: ''The code above leads to a ValueError because the empty string '' cannot be converted to a float. We can enhance the parse_values function to handle this edge case. We will also handle the case where the value is not a float.
def parse_values(data_line):
values = []
for item in data_line.strip().split(','):
if item == '':
values.append(0.0)
else:
try:
values.append(float(item))
except ValueError:
values.append(item)
return valuesfile3_lines[2]'883000,16,0.14,\n'parse_values(file3_lines[2])[883000.0, 16.0, 0.14, 0.0]Next, let's define a function create_item_dict that takes a list of values and a list of headers as inputs and returns a dictionary with the values associated with their respective headers as keys.
def create_item_dict(values, headers):
result = {}
for value, header in zip(values, headers):
result[header] = value
return resultCan you figure out what the Python built-in function zip does? Try out an example, or read the documentation.
for item in zip([1,2,3], ['a', 'b', 'c']):
print(item)(1, 'a')
(2, 'b')
(3, 'c')
Let's try out create_item_dict with a couple of examples.
file3_lines[1]'45230,48,0.07,4300\n'values1 = parse_values(file3_lines[1])
create_item_dict(values1, headers){'amount': 45230.0, 'duration': 48.0, 'rate': 0.07, 'down_payment': 4300.0}file3_lines[2]'883000,16,0.14,\n'values2 = parse_values(file3_lines[2])
create_item_dict(values2, headers){'amount': 883000.0, 'duration': 16.0, 'rate': 0.14, 'down_payment': 0.0}As expected, the values & header are combined to create a dictionary with the appropriate key-value pairs.
We are now ready to put it all together and define the read_csv function.
def read_csv(path):
result = []
# Open the file in read mode
with open(path, 'r') as f:
# Get a list of lines
lines = f.readlines()
# Parse the header
headers = parse_headers(lines[0])
# Loop over the remaining lines
for data_line in lines[1:]:
# Parse the values
values = parse_values(data_line)
# Create a dictionary using values & headers
item_dict = create_item_dict(values, headers)
# Add the dictionary to the result
result.append(item_dict)
return resultLet's try it out!
with open('./data/loans2.txt') as file2:
print(file2.read())amount,duration,rate,down_payment
828400,120,0.11,100000
4633400,240,0.06,
42900,90,0.08,8900
983000,16,0.14,
15230,48,0.07,4300
read_csv('./data/loans2.txt')[{'amount': 828400.0,
'duration': 120.0,
'rate': 0.11,
'down_payment': 100000.0},
{'amount': 4633400.0, 'duration': 240.0, 'rate': 0.06, 'down_payment': 0.0},
{'amount': 42900.0, 'duration': 90.0, 'rate': 0.08, 'down_payment': 8900.0},
{'amount': 983000.0, 'duration': 16.0, 'rate': 0.14, 'down_payment': 0.0},
{'amount': 15230.0, 'duration': 48.0, 'rate': 0.07, 'down_payment': 4300.0}]The file is read and converted to a list of dictionaries, as expected. The read_csv file is generic enough that it can parse any file in the CSV format, with any number of rows or columns. Here's the full code for read_csv along with the helper functions:
def parse_headers(header_line):
return header_line.strip().split(',')
def parse_values(data_line):
values = []
for item in data_line.strip().split(','):
if item == '':
values.append(0.0)
else:
try:
values.append(float(item))
except ValueError:
values.append(item)
return values
def create_item_dict(values, headers):
result = {}
for value, header in zip(values, headers):
result[header] = value
return result
def read_csv(path):
result = []
# Open the file in read mode
with open(path, 'r') as f:
# Get a list of lines
lines = f.readlines()
# Parse the header
headers = parse_headers(lines[0])
# Loop over the remaining lines
for data_line in lines[1:]:
# Parse the values
values = parse_values(data_line)
# Create a dictionary using values & headers
item_dict = create_item_dict(values, headers)
# Add the dictionary to the result
result.append(item_dict)
return resultTry to create small, generic, and reusable functions whenever possible. They will likely be useful beyond just the problem at hand and save you significant effort in the future.
In the previous tutorial, we defined a function to calculate the equal monthly installments for a loan. Here's what it looked like:
import math
def loan_emi(amount, duration, rate, down_payment=0):
"""Calculates the equal montly installment (EMI) for a loan.
Arguments:
amount - Total amount to be spent (loan + down payment)
duration - Duration of the loan (in months)
rate - Rate of interest (monthly)
down_payment (optional) - Optional intial payment (deducted from amount)
"""
loan_amount = amount - down_payment
try:
emi = loan_amount * rate * ((1+rate)**duration) / (((1+rate)**duration)-1)
except ZeroDivisionError:
emi = loan_amount / duration
emi = math.ceil(emi)
return emiWe can use this function to calculate EMIs for all the loans in a file.
loans2 = read_csv('./data/loans2.txt')loans2[{'amount': 828400.0,
'duration': 120.0,
'rate': 0.11,
'down_payment': 100000.0},
{'amount': 4633400.0, 'duration': 240.0, 'rate': 0.06, 'down_payment': 0.0},
{'amount': 42900.0, 'duration': 90.0, 'rate': 0.08, 'down_payment': 8900.0},
{'amount': 983000.0, 'duration': 16.0, 'rate': 0.14, 'down_payment': 0.0},
{'amount': 15230.0, 'duration': 48.0, 'rate': 0.07, 'down_payment': 4300.0}]for loan in loans2:
loan['emi'] = loan_emi(loan['amount'],
loan['duration'],
loan['rate']/12, # the CSV contains yearly rates
loan['down_payment'])loans2[{'amount': 828400.0,
'duration': 120.0,
'rate': 0.11,
'down_payment': 100000.0,
'emi': 10034},
{'amount': 4633400.0,
'duration': 240.0,
'rate': 0.06,
'down_payment': 0.0,
'emi': 33196},
{'amount': 42900.0,
'duration': 90.0,
'rate': 0.08,
'down_payment': 8900.0,
'emi': 504},
{'amount': 983000.0,
'duration': 16.0,
'rate': 0.14,
'down_payment': 0.0,
'emi': 67707},
{'amount': 15230.0,
'duration': 48.0,
'rate': 0.07,
'down_payment': 4300.0,
'emi': 262}]You can see that each loan now has a new key emi, which provides the EMI for the loan. We can extract this logic into a function so that we can use it for other files too.
def compute_emis(loans):
for loan in loans:
loan['emi'] = loan_emi(
loan['amount'],
loan['duration'],
loan['rate']/12, # the CSV contains yearly rates
loan['down_payment'])Writing to files
Now that we have performed some processing on the data, it would be good to write the results back to a CSV file. We can create/open a file in w mode using open and write to it using the .write method. The string format method will come in handy here.
loans2 = read_csv('./data/loans2.txt')compute_emis(loans2)loans2[{'amount': 828400.0,
'duration': 120.0,
'rate': 0.11,
'down_payment': 100000.0,
'emi': 10034},
{'amount': 4633400.0,
'duration': 240.0,
'rate': 0.06,
'down_payment': 0.0,
'emi': 33196},
{'amount': 42900.0,
'duration': 90.0,
'rate': 0.08,
'down_payment': 8900.0,
'emi': 504},
{'amount': 983000.0,
'duration': 16.0,
'rate': 0.14,
'down_payment': 0.0,
'emi': 67707},
{'amount': 15230.0,
'duration': 48.0,
'rate': 0.07,
'down_payment': 4300.0,
'emi': 262}]with open('./data/emis2.txt', 'w') as f:
for loan in loans2:
f.write('{},{},{},{},{}\n'.format(
loan['amount'],
loan['duration'],
loan['rate'],
loan['down_payment'],
loan['emi']))Let's verify that the file was created and written to as expected.
os.listdir('data')['loans2.txt',
'emis2.txt',
'loans3.txt',
'emis1.txt',
'emis3.txt',
'movies.csv',
'loans1.txt']with open('./data/emis2.txt', 'r') as f:
print(f.read())828400.0,120.0,0.11,100000.0,10034
4633400.0,240.0,0.06,0.0,33196
42900.0,90.0,0.08,8900.0,504
983000.0,16.0,0.14,0.0,67707
15230.0,48.0,0.07,4300.0,262
Great, looks like the loan details (along with the computed EMIs) were written into the file.
Let's define a generic function write_csv which takes a list of dictionaries and writes it to a file in CSV format. We will also include the column headers in the first line.
def write_csv(items, path):
# Open the file in write mode
with open(path, 'w') as f:
# Return if there's nothing to write
if len(items) == 0:
return
# Write the headers in the first line
headers = list(items[0].keys())
f.write(','.join(headers) + '\n')
# Write one item per line
for item in items:
values = []
for header in headers:
values.append(str(item.get(header, "")))
f.write(','.join(values) + "\n")Do you understand how the function works? If now, try executing each statement by line by line or a different cell to figure out how it works.
Let's try it out!
loans3 = read_csv('./data/loans3.txt')compute_emis(loans3)write_csv(loans3, './data/emis3.txt')with open('./data/emis3.txt', 'r') as f:
print(f.read())amount,duration,rate,down_payment,emi
45230.0,48.0,0.07,4300.0,981
883000.0,16.0,0.14,0.0,60819
100000.0,12.0,0.1,0.0,8792
728400.0,120.0,0.12,100000.0,9016
3637400.0,240.0,0.06,0.0,26060
82900.0,90.0,0.07,8900.0,1060
316000.0,16.0,0.13,0.0,21618
15230.0,48.0,0.08,4300.0,267
991360.0,99.0,0.08,0.0,13712
323000.0,27.0,0.09,4720010000.0,-193751447
528400.0,120.0,0.11,100000.0,5902
8633400.0,240.0,0.06,0.0,61853
12900.0,90.0,0.08,8900.0,60
With just four lines of code, we can now read each downloaded file, calculate the EMIs, and write the results back to new files:
for i in range(1,4):
loans = read_csv('./data/loans{}.txt'.format(i))
compute_emis(loans)
write_csv(loans, './data/emis{}.txt'.format(i))os.listdir('./data')['loans2.txt',
'emis2.txt',
'loans3.txt',
'emis1.txt',
'emis3.txt',
'movies.csv',
'loans1.txt']Isn't that wonderful? Once all the functions are defined, we can calculate EMIs for thousands or even millions of loans across many files in seconds with just a few lines of code. Now we're starting to see the real power of using a programming language like Python for processing data!
Using Pandas to Read and Write CSVs
There are some limitations to the read_csv and write_csv functions we've defined above:
- The
read_csvfunction fails to create a proper dictionary if any of the values in the CSV files contains commas - The
write_csvfunction fails to create a proper CSV if any of the values to be written contains commas
When a value in a CSV file contains a comma (,), the value is generally placed within double quotes. Double quotes (") in values are converted into two double quotes (""). Here's an example:
title,description
Fast & Furious,"A movie, a race, a franchise"
The Dark Knight,"Gotham, the ""Batman"", and the Joker"
Memento,A guy forgets everything every 15 minutes
Let's try it out.
movies_url = "https://gist.githubusercontent.com/aakashns/afee0a407d44bbc02321993548021af9/raw/6d7473f0ac4c54aca65fc4b06ed831b8a4840190/movies.csv"urlretrieve(movies_url, 'data/movies.csv')('data/movies.csv', <http.client.HTTPMessage at 0x7f2d3bc8c550>)movies = read_csv('data/movies.csv')movies[{'title': 'Fast & Furious', 'description': '"A movie'},
{'title': 'The Dark Knight', 'description': '"Gotham'},
{'title': 'Memento',
'description': 'A guy forgets everything every 15 minutes'}]As you can seen above, the movie descriptions weren't parsed properly.
To read this CSV properly, we can use the pandas library.
!pip install pandas --upgrade --quietimport pandas as pdThe pd.read_csv function can be used to read the CSV file into a pandas data frame: a spreadsheet-like object for analyzing and processing data. We'll learn more about data frames in a future lesson.
movies_dataframe = pd.read_csv('data/movies.csv')movies_dataframeA dataframe can be converted into a list of dictionaries using the to_dict method.
movies = movies_dataframe.to_dict('records')movies[{'title': 'Fast & Furious', 'description': 'A movie, a race, a franchise'},
{'title': 'The Dark Knight',
'description': 'Gotham, the "Batman", and the Joker'},
{'title': 'Memento',
'description': 'A guy forgets everything every 15 minutes'}]If you don't pass the arguments records, you get a dictionary of lists instead.
movies_dict = movies_dataframe.to_dict()movies_dict{'title': {0: 'Fast & Furious', 1: 'The Dark Knight', 2: 'Memento'},
'description': {0: 'A movie, a race, a franchise',
1: 'Gotham, the "Batman", and the Joker',
2: 'A guy forgets everything every 15 minutes'}}Let's try using the write_csv function to write the data in movies back to a CSV file.
write_csv(movies, 'movies2.csv')!head movies2.csvtitle,description
Fast & Furious,A movie, a race, a franchise
The Dark Knight,Gotham, the "Batman", and the Joker
Memento,A guy forgets everything every 15 minutes
As you can see above, the CSV file is not formatted properly. This can be verified by attempting to read the file using pd.read_csv.
pd.read_csv('movies2.csv')To convert a list of dictionaries into a dataframe, you can use the pd.DataFrame constructor.
df2 = pd.DataFrame(movies)df2It can now be written to a CSV file using the .to_csv method of a dataframe.
df2.to_csv('movies3.csv', index=None)Can you guess what the argument index=None does? Try removing it and observing the difference in output.
!head movies3.csvtitle,description
Fast & Furious,"A movie, a race, a franchise"
The Dark Knight,"Gotham, the ""Batman"", and the Joker"
Memento,A guy forgets everything every 15 minutes
The CSV file is formatted properly. We can verify this by trying to read it back.
pd.read_csv('movies3.csv')We're able to write and read the file properly with pandas.
In general, it's always a better idea to use libraries like Pandas for reading and writing CSV files.
Exercise - Processing CSV files using a dictionary of lists
We defined the functions read_csv and write_csv above to convert a CSV file into a list of dictionaries and vice versa. In this exercise, you'll transform the CSV data into a dictionary of lists instead, with one list for each column in the file.
For example, consider the following CSV file:
amount,duration,rate,down_payment
828400,120,0.11,100000
4633400,240,0.06,
42900,90,0.08,8900
983000,16,0.14,
15230,48,0.07,4300
We'll convert it into the following dictionary of lists:
{
amount: [828400, 4633400, 42900, 983000, 15230],
duration: []120, 240, 90, 16, 48],
rate: [0.11, 0.06, 0.08, 0.14, 0.07],
down_payment: [100000, 0, 8900, 0, 4300]
}
Complete the following tasks using the empty cells below:
- Download three CSV files to the folder
data2using the URLs listed in the code cell below, and verify the downloaded files. - Define a function
read_csv_columnarthat reads a CSV file and returns a dictionary of lists in the format shown above. - Define a function
compute_emisthat adds another keyemiinto the dictionary with a list of EMIs computed for each row of data. - Define a function
write_csv_columnarthat writes the data from the dictionary of lists into a correctly formatted CSV file. - Process all three downloaded files and write the results by creating new files in the directory
data2.
Define helper functions wherever required.
url1 = 'https://gist.githubusercontent.com/aakashns/257f6e6c8719c17d0e498ea287d1a386/raw/7def9ef4234ddf0bc82f855ad67dac8b971852ef/loans1.txt'
url2 = 'https://gist.githubusercontent.com/aakashns/257f6e6c8719c17d0e498ea287d1a386/raw/7def9ef4234ddf0bc82f855ad67dac8b971852ef/loans2.txt'
url3 = 'https://gist.githubusercontent.com/aakashns/257f6e6c8719c17d0e498ea287d1a386/raw/7def9ef4234ddf0bc82f855ad67dac8b971852ef/loans3.txt' Summary and Further Reading
With this, we complete our discussion of reading from and writing to files in Python. We've covered the following topics in this tutorial:
- Interacting with the file system using the
osmodule - Downloading files from URLs using the
urllibmodule - Opening files using the
openbuilt-in function - Reading the contents of a file using
.read - Closing a file automatically using
with - Reading a file line by line using
readlines - Processing data from a CSV file by defining functions
- Using helper functions to build more complex functions
- Writing data to a file using
.write
This tutorial on working with files in Python is by no means exhaustive. Following are some more resources you should check out:
- Python Tutorial at W3Schools: https://www.w3schools.com/python/
- Practical Python Programming: https://dabeaz-course.github.io/practical-python/Notes/Contents.html
- Python official documentation: https://docs.python.org/3/tutorial/index.html
Questions for Revision
Try answering the following questions to test your understanding of the topics covered in this notebook:
- What is the purpose of the
osmodule in Python? - How do you identify the current working directory in a Jupyter notebook?
- How do you retrieve the list of files within a directory using Python?
- How do you create a directory using Python?
- How do you check whether a file or directory exists on the filesystem? Hint:
os.path.exists. - Where can you find the full list of functions contained in the
osmodule? - Give examples of 5 useful functions from the
osandos.pathmodules. - How do you download a file from a URL using Python?
- How do you open a file using Python? Give an example?
- What are the different modes for opening a file in Python?
- Can you open a file in multiple modes? Illustrate with an example.
- What is the file object? How is it useful?
- How do you read the contents of a file into a string?
- What is a CSV file? Give an example.
- How do you close an open file?
- Why is it essential to close a file after processing it?
- How do you ensure that files are closed automatically after processing? Give an example.
- How is the
withstatement useful for working with files? - What happens if you try to read from a closed file?
- How do you read the contents of a file line by line?
- Write a function to convert the contents of a CSV file into a list of dictionaries (one dictionary for each row of the file).
- Write a function to convert the contents of a CSV file into a dictionary of lists (one dictionary for each column of the file).
- How do you write to a file using Python?
- How is the string
.formatmethod for writing data to a file in CSV format? - Write a function to write data from a list of dictionaries into a CSV file.
- Write a function to write data from a dictionary of lists into a CSV file.
- Where can you learn about the methods supported by the file object in Python?
- How can you read from and write to CSV files using Pandas?