Suggest edit — Spark

Title

Name

Note

---
title: "Spark"
type: concept
related: [Spark, Common Spark Transformations, Optimizing Spark, Vector, Common Spark Actions]
source: https://www.jemoka.com/posts/kbhspark/
confidence: high
status: active
---

Spark is not a database. Importantly, its a &ldquo;framework&rdquo; of data:
Programming platform Distributed file system Prallel execution environment Software ecosystem It gives you the &ldquo;parallel&rdquo; search/sort needed to navigate a large database. It is based on the Hadoop ecosystem. Spark operates on RDDs to do lazy-evaluation.
Quickstart When we start up Spark Shell, it will build you a sc variable which is appropriate for your supercomputer; if you are not, you need to set up the context yourself using the 3 lines noted below to make sc variable:
# build context, IF NOT in a SHELL from myspark import SparkConf, SparkContext conf = SparkConf().setMaster(&#34;local&#34;).setaAppName(&#34;Test_App&#34;) sc = SparkContext(conf=conf) # do shit &#34;transform&#34; my_rdd = sc.textFile(&#34;whatever&#34;) # create an RDD from a datastore filtered_rdd = my_rdd.filter(lambda Line: &#34;hubble&#34; in line) # perform action in rdd # actually perform actions filtered_rdd.count() # 47 filtered_rdd.first() Then, you can submit this script. If you are in a shell its REPL so you don&rsquo;t.
spark-submit Test_App.py Main types of files Spark can handle:
Text (CSV, XML, JSON, etc.) SQL stuff Other NoSQL Stuff Parquet Hive Hadoopy things JDBC, Cassandra, Mongo, etc. etc. etc. Compressed files: tarballs, gzip Main types of filesystems:
yours (local files) HDFS Lustre AWS S3 See also Common Spark Actions, Common Spark Transformations
Spark RDD API The base level API of Spark is one which interacts with vectors or tuples via RDDs. It deals with manipulating unordered sets (i.e. not dataframes).
Create/Load RDD Transform RDD] (performing any operations in theory, lazily, without loading data) Apply some more transforms (filtering, etc.) Perform Actions (actually get data) Transformations move one RDD to another, and Actions load the data from RDD.
Spark is smart to not do anything that&rsquo;s not needed; removing entire stores which isn&rsquo;t needed, caching, dynamically getting them back etc.
Spark can read from whole databases, text files, etc. and move them into an RDD.
Tips and Tricks &ldquo;kv&rdquo; values are kinda like Pandas .groupby without groupyby. You can do most mapping operations by key instead, which means you can always do group.
if you want a certain form, use the .map function to turn it into something else; for instance, if you want to turn something into a kv pair, you can thing.map(lambda x: x-key, x-value) if you need to work with KV data, you should think about swapping key and value if you so desire Spark Anti-Patterns You should never do heavy-duty compute in Python (i.e. if you end up with a for loop somewhere you are probably not using map reduce right) You should never take an action until you absolutely, seriously, need all the data Optimizations See Optimizing Spark
Spark DataFrame API DataFrames are type-safe collections of tables built out of RDDs; they are collections of tuples, where columns have the same type
row_rdd = sc.parallelize([(&#34;thing1&#34;, &#34;thing2&#34;, 3), (&#34;thing1&#34;, &#34;thing2&#34;, 3)]) row_dataframe = spark.createDataFrame(row_rdd, [&#34;col1&#34;, &#34;col2&#34;, &#34;col3&#34;]) row_dataframe.show() You can load entire structured data via:
df = spark.read.json(&#34;jsonfile.json&#34;) And it will create the right schema on your behalf in the DataFrame.
DataFrame Limitations Unlike Pandas, you can&rsquo;t manipulate structured DataFrame well. Hence, you should like. Go back to RDDs if you are manipulating specific data into unstructured form.