What Is Apache Spark?

A computing platform for clusters designed to perform tasks quickly and for general purpose use To put it in plain terms for those unfamiliar with technical jar

GGAMI2021-12-10

Spark Core

Spark's Storage Layer

Contents

Integrated Components
Spark Core
Spark SQL
Spark Streaming
MLlib
GraphX
Cluster Manager
Spark's Storage Layer

A computing platform for clusters designed to perform tasks quickly and for general-purpose use
To put it in plain terms for those unfamiliar with technical jargon, it is an application that lets you integrate different language environments when analyzing or processing data, so you can flexibly handle tasks such as data processing and analysis (supports multiple languages: Java, Python, SQL)
It effectively supports machine learning and various computational models.
It can also be installed and run within a Hadoop cluster (HDFS).

Integrated Components#

Spark's engine lets high-level components such as SQL and machine learning run.
One advantage of integrating components is that different data-processing models can be neatly combined to create a single application.
For example, if you build a machine learning model that classifies data in real time, analysts can run SQL queries on the results in real time.
Alternatively, you can access it through a Python shell to perform immediate analysis.

Spark components

Spark Core#

Consists of basic functions such as job scheduling, memory management, fault recovery, and integration with storage systems

Spark SQL#

A Spark package for processing structured data
Connects SQL with complex analytical tasks

Spark Streaming#

Enables the processing of real-time data streams (flows of data arriving sequentially).
Supports APIs for manipulating data streams.
Provides the same level of fault management, throughput, and scalability as Spark Core.

MLlib#

Distributed with libraries that provide machine learning capabilities.
Supports core low-level machine learning techniques, including classification, regression, collaborative filtering, and optimization algorithms.

GraphX#

GraphX is a library for working with (social network relationships, such as friendships on social media (network models)).
It can create directed graphs with arbitrary attributes added to edges or vertices, and provides a library of related methods for working with graphs.

Cluster Manager#

It can run on various cluster managers, such as Hadoop YARN, Apache Mesos, and the standalone scheduler.

Spark's Storage Layer#

It can create distributed collections from the Hadoop Distributed File System or other storage systems supported by the Hadoop API.
However, Spark does not necessarily require Hadoop.
It also supports various data formats, such as txt, Avro, and Parquet.