Spark Up Your Data with PySpark

Understand the fundamentals of PySpark. Learn how to manipulate large-scale data using PySpark. Develop skills in data cleaning, transformation, and analysis with PySpark. Gain proficiency in building and deploying scalable data processing pipelines. Apply PySpark to real-world data analytics and machine learning tasks.

Instructors:

SJ Sushmitha J

Start Learning

2 Enrolled

53 Lessons

This course introduces you to PySpark, a powerful Python library for big data processing. Through hands-on exercises and projects, learners will explore PySpark's capabilities in handling large-scale data sets, mastering techniques for data cleaning, transformation, and analysis. They will also delve into building scalable data processing pipelines and applying PySpark to real-world data analytics and machine learning tasks. By the end of the course, you will be equipped with the skills to efficiently manage and analyze big data using PySpark.

Course Content

Introduction

Intro

Note

Fundamentals of PySpark

What is Apache Spark

Apache Spark Architecture

How to become a Spark Developer

Introduction to Apache Spark with Python

PySpark Installation

PySpark RDD

PySpark Dataframes

PySpark Programming - SQL & Streaming

PySpark MLlib

Working with PySpark - Part 1

Install PySpark on Windows 10

Create RDD

DataFrame from RDD

Create Sparkcontext in PySpark

Create SparkSession in PySpark

Create PySpark DataFrame

PySpark Read CSV

PySpark Read JSON

createOrReplaceTempView PySpark

createGlobalTempView PySpark

Working with PySpark - Part 2

PySpark Read Text File

PySpark Read Parquet File

PySpark Write CSV File

PySpark Download CSV

PySpark DataFrame Functions

PySpark SQL

PySpark Window Function

PySpark Correlation Analysis

PySpark Chi Square Test

PySpark Covariance

Working with PySpark - Part 3

withColumn, Rename Column

Alias, Distinct, OrderBy

Missing Values in PySpark

Grouped Data Aggregation

Contains, Describe, Cast, Head

Count Distinct, Concat, Length, Collect List

like, rlike, isin, substr

PySpark dense_rank vs rank

PySpark Date Function

PySpark dayofmonth dayofyear dayofweek

Working with PySpark - Part 4

PySpark lag and lead function

PySpark Linear Regression

PySpark Logistic Regression

PySpark Decision Tree

PySpark Random Forest

PySpark K Means Clustering

PySpark StandardScaler

PySpark VectorAssembler

PySpark StringIndexer

PySpark OneHotEncode

PySpark Expr

Assessment on PySpark

Reviews

Review the course

Help us improve our course material.

Write a review