Apache Spark, Scala, Storm Training

Module 1 – Understanding Architecture of Storm

• Bayesian Law
• Hadoop Distributed Computing
• Big Data features
• Legacy Architecture of Real Time System
• Storm vs. Hadoop
• Logical Dynamic and Components in Storm
• Storm Topology
• Execution Components in Storm
• Stream Grouping
• Tuple
• Spout
• Bolt-normalization bolt

Module 2 – Installation of Apache storm

• Installing Apache Storm

Module 3 – Grouping

• Different types of Grouping
• Reliable and unreliable messaging
• Fetching data – Direct connection and En-queued message
• Bolt Lifecycle

Module 4 – Overview of Trident

• Trident Spouts and its types
• Components  and Interface of Trident spout
• Trident Function, Filter & Aggregator

Module 5 – Boot Stripping

• Twitter Boot Stripping
• Detailed learning on Boot Stripping
• Concepts of Storm
• Storm Development Environment

Projects

Real-time Project on Storm

The Project Bolt Blue Print

Apache Spark

Module 1-Why Spark? Explain Spark and Hadoop Distributed File System

• What is Spark
• Comparison with Hadoop
• Components of Spark

Module 2-Spark Components, Common Spark Algorithms-Iterative Algorithms, Graph Analysis, Machine Learning

• Apache Spark- Introduction, Consistency, Availability, Partition
• Unified Stack Spark
• Spark Components
• Comparison with Hadoop – Scalding example, mahout, storm, graph

Module 3-Running Spark on a Cluster, Writing Spark Applications using Python, Java, Scala

• Explain python example
• Show installing a spark
• Explain driver program
• Explaining spark context with example
• Define weakly typed variable
• Combine scala and java seamlessly.
• Explain concurrency and distribution.
• Explain what is trait.
• Explain higher order function with example.
• Define OFI scheduler.
• Advantages of Spark
• Example of Lamda using spark
• Explain Mapreduce with example

Module 4-RDD and its operation

• Difference between RISC and CISC
• Define Apache Mesos
• Cartesian product between two RDD
• Define count
• Define Filter
• Define Fold
• Define API Operations
• Define Factors

Module 5-Spark, Hadoop, and the Enterprise Data Centre, Common Spark Algorithms

• How hadoop cluster is different from spark
• Define writing data
• Explain sequence file and its usefulness
• Define protocol buffers
• Define text file, CSV, Object Files and File System
• Define sparse metrics
• Explain RDD and Compression
• Explain data stores and its usefulness

Module 6-Spark Streaming

• Define Elastic Search
• Explain Streaming and its usefulness
• Apache bookeeper
• Define Dstream
• Define mapreduce word count
• Explain Paraquet
• Scala ORM
• Define Mlib
• Explain multi graphix and its usefulness
• Define property graph

Module 7-Spark Persistence in Spark 

• Persistence
• Motivation
• Example
• Transformation
• Scala and Python
• Examples – K-means
• Latent Dirichlet Allocation (LDA)

Module 8-Broadcast and accumulator

• Motivation
• Broadcast Variables
• Example: Join
• Alternative if one table is small
• Better version with broadcast
• How to create a Broadcast
• Accumulators motivation
• Example: Join
• Accumulator Rules
• Custom accumulators
• Another common use
• Creating an accumulator using spark context object

Module 9-Spark SQL and RDD

• Introduction
• Spark SQL main capabilities
• Spark SQL usage diagram
• Spark SQL
• Important topics in Spark SQL- Data frames
• Twitter language analysis

Mini Projects    

Project 1. List the items

Project 2. Sorting of Records

Project 3. Show a histogram of date vs users created. Optionally, use a rich visualization like

Project 4. Prepare a map of tags vs # of questions in each tag and display it.

Major Projects 

Project 1 Movie Recommendation

Project 2 Twitter API Integration for tweet Analysis

Project 3 Data Exploration Using Spark SQL – Wikipedia dataset

Scala

Module 1-Introduction of Scala

• Scala Overview

Module 2-Pattern Matching

• Advantages of Scala
• REPL (Read Evaluate print loop)
• Language Features
• Type Interface
• Higher order function
• Option
• Pattern Matching
• Collection
• Currying
• Traits
• Application Space

Module 3-Executing the Scala code

• Uses of scala interpreter
• Example of static object timer in scala
• Testing of String equality in scala
• Implicit classes in scala with examples.
• Recursion in scala
• Currying in scala with examples.
• Classes in scala

Module 4-Classes concept in Scala

• Constructor
• Constructor overloading
• Properties
• Abstract classes
• Type hierarchy in Scala
• Object equality
• Val and var methods

Module 5-Case classes and pattern matching

• Sealed traits
• Case classes
• Constant pattern in case classes
• Wild card pattern
• Variable pattern
• Constructor pattern
• Tuple pattern

Module 6-Concepts of traits with example

• Java equivalents
• Advantages of traits
• Avoiding boilerplate code
• Linearization of traits
• Modelling a real world example

Module 7-Scala java Interoperability

• How traits are implemented in scala and java
• How extending multiple traits is handled

Module 8-Scala collections

• Classification of scala collections
• Iterable
• Iterator and iterable
• List sequence example in scala

Module 9-Mutable collections vs. Immutable collections

• Array in scala
• List in scala
• Difference between list and list buffer
• Array buffer
• Queue in scala
• Dequeue in scala
• Mutable queue in scala
• Stacks in scala
• Sets and maps in scala
• Tuples

Module 10-Use Case bobsrockets package         

• Different import types
• Selective imports
• Testing-Assertions
• Scala test case- scala test fun. Suite
• Junit test in scala
• Interface for Junit via Junit 3 suite in scala test
• SBT
• Directory structure for packaging scala application