# Spark Cluster

A spark cluster similar to the abacus EMR cluster.

## Getting Started

### Run the cluster locally

```sh
# build the local docker image & run.
> docker-compose up -d --build
# - or run without building -
> docker-compose up -d
```

### Stage Data

Data added to the [/data](./data) directory will be available to the
spark cluster in `/usr/local/data/`

### Access the spark shell

```sh
# Access the spark shell in the running docker image
> docker exec -it \
     spark_cluster_master_1 \
     spark-2.4.4/bin/spark-shell \
     --packages org.apache.spark:spark-avro_2.11:2.4.4

# Prints Hello, world
scala> val myList: List[String] = List("Hello", "world")
scala> println(myList.mkString(", "))

```

To paste large blocks of text, use `:paste`. Example:

```sh
scala> :paste

// Entering paste mode (ctrl-D to finish)

val foo = "bar"

// Exiting paste mode, now interpreting.

```

### Read data

Copy the following into the scala shell:

```scala
import org.apache.spark.sql.types._
import spark.implicits._
import org.apache.spark.sql.Dataset

val schema: StructType = StructType(
    List(
        StructField("id", IntegerType, nullable = false),
        StructField("favColor", StringType, nullable = false),
        StructField("favComedyType", StringType, nullable = false),
        StructField("luckyNumber", IntegerType, nullable = false),
        StructField("liedAboutFavColor", BooleanType, nullable = false)
    )
)

case class User(
    id: Integer,
    favColor: String,
    favComedyType: String,
    luckyNumber: Integer,
    liedAboutFavColor: Boolean
)


val userDf: Dataset[User] = spark.read
    .format("csv")
    .option("mode", "FAILFAST")
    .option("header", "false")
    .option("sep", ",")
    .schema(schema)
    .load("/usr/local/data/example.csv")
    .as[User]
```

Use `show` to dump the data to the terminal:

```sh
scala> userDf.show
```