Log In

Exploratory Data Analysis using Python - A Case Study

Open In Colab

Analyzing responses from the Stack Overflow Annual Developer Survey 2020

This tutorial is an executable Jupyter notebook. Click the Open in Colab button at the top of this page to execute the code.

Jupyter Notebooks: This notebook is made of cells. Each cell can contain code written in Python or explanations in plain English. You can execute code cells and view the results instantly within the notebook. Jupyter is a powerful platform for experimentation and analysis. Don't be afraid to mess around with the code & break things - you'll learn a lot by encountering and fixing errors. You can use the "Kernel > Restart & Clear Output" menu option to clear all outputs and start again from the top.

Introduction

We'll use the StackOverflow developer survey dataset for our analysis. This is an annual survey conducted by SO, and you find the raw data & results here: https://insights.stackoverflow.com/survey.

Let's retrive the files using urlretrieve.

url1 = "https://raw.githubusercontent.com/JovianHQ/notebooks/refs/heads/main/data-analysis-with-python-zero-to-pandas/lesson-6-exploratory-data-analysis-a-case-study/survey_results_public.csv"
url2 = "https://raw.githubusercontent.com/JovianHQ/notebooks/refs/heads/main/data-analysis-with-python-zero-to-pandas/lesson-6-exploratory-data-analysis-a-case-study/survey_results_schema.csv"
url3 = "https://raw.githubusercontent.com/JovianHQ/notebooks/refs/heads/main/data-analysis-with-python-zero-to-pandas/lesson-6-exploratory-data-analysis-a-case-study/README.txt"
from urllib.request import urlretrieve
import os
# Let's create the directory to save the files in.
os.makedirs("stackoverflow-developer-survey-2020", exist_ok = True)

# Let's retrive the files
urlretrieve(url1, './stackoverflow-developer-survey-2020/survey_results_public.csv')
urlretrieve(url2, './stackoverflow-developer-survey-2020/survey_results_schema.csv')
urlretrieve(url3, './stackoverflow-developer-survey-2020/README.txt')
('./stackoverflow-developer-survey-2020/README.txt',
 <http.client.HTTPMessage at 0x7f6f92c8f290>)

Let's verify that the dataset was downloaded into the directory stackoverflow-developer-survey-2020, and retrieve the list of files in the dataset.

os.listdir('stackoverflow-developer-survey-2020')
['survey_results_schema.csv', 'survey_results_public.csv', 'README.txt']

You can through the downloaded files using File > Open menu option in Jupyter. It seems like the dataset contains 3 files:

  • README.txt - containing information about the dataset
  • survey_results_schema.csv - containing the list of questions and the short codes for each question
  • survey_results_public.csv - the full list of responses to the questions

Let's load the CSV files using the Pandas library.

import pandas as pd
survey_raw_df = pd.read_csv('stackoverflow-developer-survey-2020/survey_results_public.csv')
survey_raw_df

The dataset contains over 64,000 responses to 60 questions (although many questions are optional). The responses have been anonymized and there's no personally identifiable information about respondents - just a Randomized respondent ID.

survey_raw_df.columns
Index(['Respondent', 'MainBranch', 'Hobbyist', 'Age', 'Age1stCode', 'CompFreq',
       'CompTotal', 'ConvertedComp', 'Country', 'CurrencyDesc',
       'CurrencySymbol', 'DatabaseDesireNextYear', 'DatabaseWorkedWith',
       'DevType', 'EdLevel', 'Employment', 'Ethnicity', 'Gender', 'JobFactors',
       'JobSat', 'JobSeek', 'LanguageDesireNextYear', 'LanguageWorkedWith',
       'MiscTechDesireNextYear', 'MiscTechWorkedWith',
       'NEWCollabToolsDesireNextYear', 'NEWCollabToolsWorkedWith', 'NEWDevOps',
       'NEWDevOpsImpt', 'NEWEdImpt', 'NEWJobHunt', 'NEWJobHuntResearch',
       'NEWLearn', 'NEWOffTopic', 'NEWOnboardGood', 'NEWOtherComms',
       'NEWOvertime', 'NEWPurchaseResearch', 'NEWPurpleLink', 'NEWSOSites',
       'NEWStuck', 'OpSys', 'OrgSize', 'PlatformDesireNextYear',
       'PlatformWorkedWith', 'PurchaseWhat', 'Sexuality', 'SOAccount',
       'SOComm', 'SOPartFreq', 'SOVisitFreq', 'SurveyEase', 'SurveyLength',
       'Trans', 'UndergradMajor', 'WebframeDesireNextYear',
       'WebframeWorkedWith', 'WelcomeChange', 'WorkWeekHrs', 'YearsCode',
       'YearsCodePro'],
      dtype='object')

Short codes are used as column names. We can refer to the schema file to see the full text of each question.

schema_fname = 'stackoverflow-developer-survey-2020/survey_results_schema.csv'
schema_raw = pd.read_csv(schema_fname, index_col='Column').QuestionText
schema_raw
Column
Respondent            Randomized respondent ID number (not in order ...
MainBranch            Which of the following options best describes ...
Hobbyist                                        Do you code as a hobby?
Age                   What is your age (in years)? If you prefer not...
Age1stCode            At what age did you write your first line of c...
                                            ...                        
WebframeWorkedWith    Which web frameworks have you done extensive d...
WelcomeChange         Compared to last year, how welcome do you feel...
WorkWeekHrs           On average, how many hours per week do you wor...
YearsCode             Including any education, how many years have y...
YearsCodePro          NOT including education, how many years have y...
Name: QuestionText, Length: 61, dtype: object
schema_raw['YearsCodePro']
'NOT including education, how many years have you coded professionally (as a part of your work)?'

We've now loaded the dataset, and are ready to move on to the next step of preprocessing & cleaning the data for our analysis.

Data Preparation & Cleaning

While the survey responses contain a wealth of information, we'll limit our analysis to the following areas:

  • Demographics of the survey respondents & the global programming community
  • Distribution of programming skills, experience and preferences
  • Employment-related information & preferences

Let's select a subset of columns with the relevant data.

selected_columns = [
    # Demographics
    'Country',
    'Age',
    'Gender',
    'EdLevel',
    'UndergradMajor',
    # Programming experience
    'Hobbyist',
    'Age1stCode',
    'YearsCode',
    'YearsCodePro',
    'LanguageWorkedWith',
    'LanguageDesireNextYear',
    'NEWLearn',
    'NEWStuck',
    # Employment
    'Employment',
    'DevType',
    'WorkWeekHrs',
    'JobSat',
    'JobFactors',
    'NEWOvertime',
    'NEWEdImpt'
]
len(selected_columns)
20

Let's extract a copy of the data from these columns into a new data frame survey_df, which we can continue to modify further without affecting the original data frame.

survey_df = survey_raw_df[selected_columns].copy()
schema = schema_raw[selected_columns]

Let's view some basic information about the data frame.

survey_df.shape
(64461, 20)
survey_df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 64461 entries, 0 to 64460 Data columns (total 20 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 Country 64072 non-null object 1 Age 45446 non-null float64 2 Gender 50557 non-null object 3 EdLevel 57431 non-null object 4 UndergradMajor 50995 non-null object 5 Hobbyist 64416 non-null object 6 Age1stCode 57900 non-null object 7 YearsCode 57684 non-null object 8 YearsCodePro 46349 non-null object 9 LanguageWorkedWith 57378 non-null object 10 LanguageDesireNextYear 54113 non-null object 11 NEWLearn 56156 non-null object 12 NEWStuck 54983 non-null object 13 Employment 63854 non-null object 14 DevType 49370 non-null object 15 WorkWeekHrs 41151 non-null float64 16 JobSat 45194 non-null object 17 JobFactors 49349 non-null object 18 NEWOvertime 43231 non-null object 19 NEWEdImpt 48465 non-null object dtypes: float64(2), object(18) memory usage: 9.8+ MB

Most columns have the data type object, either because they contain values of different types, or they contain empty values, which are represented np.NaN. It appears that every column contains some empty values, since the Non-Null count for every column is lower than the total number of rows (64461). We'll need to deal with empty values and manually adjust the data type for each column on a case-by-case basis.

Only two of the columns were detected as numeric columns (Age and WorkWeekHrs), even though there are a few other columns which have mostly numeric values. To make our analysis easier, let's convert some other columns into numeric data types, while ignoring any non-numeric value (they will get converted to NaNs)

survey_df['Age1stCode'] = pd.to_numeric(survey_df.Age1stCode, errors='coerce')
survey_df['YearsCode'] = pd.to_numeric(survey_df.YearsCode, errors='coerce')
survey_df['YearsCodePro'] = pd.to_numeric(survey_df.YearsCodePro, errors='coerce')

Let's now view some basic statistics about the the numeric columns.

survey_df.describe()

There seems to be a problem with the age column, as the minimum value is 1 and max value is 279. This is a common issues with surveys: responses may contain invalid values due to accidental or intentional errors while responding. A simple fix would be ignore the rows where the value in the age column is higher than 100 years or lower than 10 years as invalid survey responses.

survey_df.drop(survey_df[survey_df.Age < 10].index, inplace=True)
survey_df.drop(survey_df[survey_df.Age > 100].index, inplace=True)

The same hold true for WorkWeekHrs. Let's ignore entries where the value for the column is higher than 140 hours.

survey_df.drop(survey_df[survey_df.WorkWeekHrs > 140].index, inplace=True)

The gender column also allows picking multiple options, but to simplify our analysis, we'll remove values containing options.

survey_df['Gender'].value_counts()
Gender
Man                                                            45895
Woman                                                           3835
Non-binary, genderqueer, or gender non-conforming                385
Man;Non-binary, genderqueer, or gender non-conforming            121
Woman;Non-binary, genderqueer, or gender non-conforming           92
Woman;Man                                                         73
Woman;Man;Non-binary, genderqueer, or gender non-conforming       25
Name: count, dtype: int64
import numpy as np
survey_df.where(~(survey_df.Gender.str.contains(';', na=False)), np.nan, inplace=True)

We've now cleaned up and prepared the dataset for analysis.

survey_df

Exploratory Data Analysis

Country

Let's look at country stats

import seaborn as sns
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline

sns.set_style('darkgrid')
matplotlib.rcParams['font.size'] = 14
matplotlib.rcParams['figure.figsize'] = (9, 5)
matplotlib.rcParams['figure.facecolor'] = '#00000000'
survey_df.Country.nunique()
183

Let's also check the fraction of empty values in this column.

survey_df.Country.isna().mean()
np.float64(0.010885453923428608)

Here are the countries with the highest number of respondents.

top_countries = survey_df.Country.value_counts().head(15)
top_countries
Country
United States         12371
India                  8364
United Kingdom         3881
Germany                3864
Canada                 2175
France                 1884
Brazil                 1804
Netherlands            1332
Poland                 1259
Australia              1199
Spain                  1157
Italy                  1115
Russian Federation     1085
Sweden                  879
Pakistan                802
Name: count, dtype: int64

We can visualize this using a bar chart.

plt.figure(figsize=(12,6))
plt.xticks(rotation=75)
sns.barplot(x=top_countries.index, y=top_countries.values);
Notebook output

It appears the that a disproportionately high number of respondents are from USA & India - which one might expect since the Survey is in English.

Age

plt.figure(figsize=(12, 6))
plt.title('Distribution of Age')
plt.xlabel('Age')
plt.ylabel('Number of respondents')

plt.hist(survey_df.Age, bins=np.arange(10,80,5), color='purple');
Notebook output

Gender

gender_counts = survey_df.Gender.value_counts()
gender_counts
Gender
Man                                                  45895
Woman                                                 3835
Non-binary, genderqueer, or gender non-conforming      385
Name: count, dtype: int64

A pie chart would be a good way to represent this split.

plt.figure(figsize=(12,6))
plt.pie(gender_counts, labels=gender_counts.index, autopct='%1.1f%%', startangle=180);
Notebook output

Education Level

Let's compare the education levels using a horizontal bar plot

sns.countplot(data=survey_df, y='EdLevel')
plt.xticks(rotation=75);
plt.title(schema['EdLevel'])
plt.ylabel(None);
Notebook output

Let's also plot undergraduate majors, but this time use percentages, and sort by it.

undergrad_pct = survey_df.UndergradMajor.value_counts() * 100 / survey_df.UndergradMajor.count()

sns.barplot(x=undergrad_pct.values, y=undergrad_pct.index)

plt.title(schema.UndergradMajor)
plt.ylabel(None);
plt.xlabel('Percentage')

Text(0.5, 0, 'Percentage')
Notebook output

Employment

We can also plot directly using Pandas

(survey_df.Employment.value_counts(normalize=True, ascending=True)*100).plot(kind='barh', color='g')
plt.title(schema.Employment)
plt.xlabel('Percentage');
Notebook output

The DevType field is also relevant, but since the question allows multiple answers, the column contains lists of values separated by ;, which makes it a bit harder to analyze directly.

schema.DevType
'Which of the following describe you? Please select all that apply.'
survey_df.DevType.value_counts()
DevType
Developer, full-stack                                                                                                                                                                                                                                                                                                                                                                                                                                               4396
Developer, back-end                                                                                                                                                                                                                                                                                                                                                                                                                                                 3056
Developer, back-end;Developer, front-end;Developer, full-stack                                                                                                                                                                                                                                                                                                                                                                                                      2214
Developer, back-end;Developer, full-stack                                                                                                                                                                                                                                                                                                                                                                                                                           1465
Developer, front-end                                                                                                                                                                                                                                                                                                                                                                                                                                                1390
                                                                                                                                                                                                                                                                                                                                                                                                                                                                    ... 
Data or business analyst;Data scientist or machine learning specialist;Database administrator;Designer;Developer, back-end;Developer, desktop or enterprise applications;Developer, front-end;Developer, full-stack;Developer, game or graphics;Developer, mobile;Developer, QA or test;DevOps specialist;Engineer, data;Engineer, site reliability;Engineering manager;Marketing or sales professional;Product manager;Senior executive/VP;System administrator       1
Data or business analyst;Developer, back-end;DevOps specialist;Engineering manager                                                                                                                                                                                                                                                                                                                                                                                     1
Academic researcher;Data or business analyst;DevOps specialist;Engineer, data                                                                                                                                                                                                                                                                                                                                                                                          1
Developer, embedded applications or devices;Developer, QA or test;Engineering manager                                                                                                                                                                                                                                                                                                                                                                                  1
Academic researcher;Designer;Developer, back-end;Developer, desktop or enterprise applications;Developer, front-end;Developer, full-stack;Developer, game or graphics;Developer, mobile;Developer, QA or test;Educator                                                                                                                                                                                                                                                 1
Name: count, Length: 8213, dtype: int64

Let's define a helper function which turns a column containing lists of values into a data frame with one column for each option.

def split_multicolumn(col_series):
    result_df = col_series.to_frame().copy()
    options = []
    for idx, value in col_series[col_series.notna()].items():
        for option in value.split(';'):
            option = option.strip()
            if option not in result_df.columns:
                options.append(option)
                result_df[option] = False
            result_df.at[idx, option] = True
    return result_df[options]

dev_type_df = split_multicolumn(survey_df.DevType)
dev_type_df
dev_type_totals = dev_type_df.sum().sort_values(ascending=False)
dev_type_totals
Developer, back-end                              26996
Developer, full-stack                            26915
Developer, front-end                             18128
Developer, desktop or enterprise applications    11687
Developer, mobile                                 9406
DevOps specialist                                 5915
Database administrator                            5658
Designer                                          5262
System administrator                              5185
Developer, embedded applications or devices       4701
Data or business analyst                          3970
Data scientist or machine learning specialist     3939
Developer, QA or test                             3893
Engineer, data                                    3700
Academic researcher                               3502
Educator                                          2895
Developer, game or graphics                       2751
Engineering manager                               2699
Product manager                                   2471
Scientist                                         2060
Engineer, site reliability                        1921
Senior executive/VP                               1292
Marketing or sales professional                    625
dtype: int64

And we can now plot it

plt.figure(figsize=(12, 12))
sns.barplot(x=dev_type_totals.values, y=dev_type_totals.index)
plt.title(schema.DevType);
plt.xlabel('count')

Text(0.5, 0, 'count')
Notebook output

Let's also create a scatter plot of YearsCode and YearsCodePro.

sns.scatterplot(data=survey_df, x='YearsCode', y='YearsCodePro', hue='Hobbyist')
plt.xlabel("Years of coding experience")
plt.ylabel("Years of professional coding experience");
Notebook output

There seems to be no correlation as such.

Asking and Answering Questions

To answer, this we can use the LanguageWorkedWith column. Similar to DevType it allowed choosing multiple

Inferences and Conclusion

References and Future Work

 
 
 
 
selected_columns = [
    # Demographics
    'Country',
    'Age',
    'Gender',
    'EdLevel',
    'UndergradMajor',
    # Programming experience
    'Hobbyist',
    'Age1stCode',
    'DevType',
    'YearsCode',
    'YearsCodePro',
    'LanguageWorkedWith',
    'LanguageDesireNextYear',
    'NEWLearn',
    'NEWStuck',
    # Employment
    'Employment',
    'WorkWeekHrs',
    'JobSat',
    'JobFactors',
    'NEWOvertime',
    'NEWEdImpt'
]