Cooding Dessign
Posted in Machine Learning

Autoencoder Feature Extraction for Regression

Tweet Share Share Autoencoder is a type of neural network that can be used to learn a compressed representation of raw data. An autoencoder is…

Continue Reading... Autoencoder Feature Extraction for Regression
Cooding Dessign
Posted in Machine Learning

PySpark Read JSON file into DataFrame

PySpark SQL provides read.json(“path”) to read a single line or multiline (multiple lines) JSON file into PySpark DataFrame and write.json(“path”) to save or write to JSON file, In this…

Continue Reading... PySpark Read JSON file into DataFrame
Cooding Dessign
Posted in Machine Learning

PySpark fillna() & fill() – Replace NULL Values

In PySpark, DataFrame.fillna() or DataFrameNaFunctions.fill() is used to replace NULL values on the DataFrame columns with either with zero(0), empty string, space, or any constant…

Continue Reading... PySpark fillna() & fill() – Replace NULL Values
Cooding Dessign
Posted in Machine Learning

PySpark How to Filter Rows with NULL Values

While working on PySpark SQL DataFrame we often need to filter rows with NULL/None values on columns, you can do this by checking IS NULL…

Continue Reading... PySpark How to Filter Rows with NULL Values
Cooding Dessign
Posted in Machine Learning

Spark Installation on Linux Ubuntu

Let’s learn how to do Apache Spark Installation on Linux based Ubuntu server, same steps can be used to setup Centos, Debian e.t.c. In real-time…

Continue Reading... Spark Installation on Linux Ubuntu
Cooding Dessign
Posted in Machine Learning

PySpark Random Sample with Example

PySpark provides a pyspark.sql.DataFrame.sample(), pyspark.sql.DataFrame.sampleBy(), RDD.sample(), and RDD.takeSample() methods to get the random sampling subset from the large dataset, In this article I will explain…

Continue Reading... PySpark Random Sample with Example
Cooding Dessign
Posted in Machine Learning

Spark SQL Sampling with Examples

Spark sampling is a mechanism to get random sample records from the dataset, this is helpful when you have a larger dataset and wanted to…

Continue Reading... Spark SQL Sampling with Examples
Cooding Dessign
Posted in Machine Learning

Apache Spark Installation on Windows

In this article, I will explain step-by-step how to do Apache Spark Installation on windows os 7, 10, and the latest version and also explains…

Continue Reading... Apache Spark Installation on Windows
Cooding Dessign
Posted in Machine Learning

PySpark Drop Rows with NULL or None Values

In PySpark, pyspark.sql.DataFrameNaFunctions class provides several functions to deal with NULL/None values, among these drop() function is used to remove/drop rows with NULL values in DataFrame…

Continue Reading... PySpark Drop Rows with NULL or None Values
Cooding Dessign
Posted in Machine Learning

How to Run Spark Examples from IntelliJ

Here, I will explain how to run Apache Spark Application examples explained in this blog on windows using Scala & Maven from IntelliJ IDEA. Since…

Continue Reading... How to Run Spark Examples from IntelliJ