What is VORA and How it helps to Bridge the gap between Enterprise data and Big Data

Before getting into the Topic of VORA first we lets try to understand what is Enterprise data, Big Data, HADOOP, SPARK.

What is Enterprise Data – Data that comes from Day today business transactions eg. Sales order, Purchase Order, etc.

What is Big Data – Data that comes from information-sensing mobile devices, aerial (remote sensing), software logs, cameras, microphones, radio-frequency identification (RFID) readers and wireless sensor networks, Social Media and Archived Data.

Optimising HANA Query push-down from Apache Spark

As you start using it with larger Tables or Views in HANA, you may notice that the query that is pushed down into HANA is not optimised efficiently. SUM & GROUP BY Clauses are NOT pushed down into HANA,  this may cause a large granular result set to be move across the network, to only be Aggregated in Spark.  That is certainly a waste of HANA's powerful query engine.
In this blog I will demonstrate the problem and show several ways to help get around it, using Apache Spark.

Using the same dataset from the earlier blog  we can see how more complex Spark SQL ( executing against a test table in HANA - "RDATA") is actually pushed down into HANA.

Calling HANA Views from Apache Spark

Open Source Apache Spark  is fast becoming the de facto standard for Big Data processing and analytics. It’s an ‘in-memory’ data processing engine, utilising the distributed computing power of 10’s or even 1000’s of logical linked host machines (cluster). It’s able to crunch through vast quantities of both structured and unstructured data. You can easily scale out your cluster as your data appetite grows.

In addition to this it can also be used as a data federation layer spanning both traditional databases as well as other popular big data platforms, such as Hadoop HDFS, Hadoop Hbase, Cassandra, Amazon Redshift and S3, to name a few.

Vora 1.2 Modeling Tool

SAP HANA Vora provides an in-memory processing engine which can scale up to thousands of nodes, both on premise and in cloud. Vora fits into the Hadoop Ecosystem and extends the Spark execution framework.

Following image shows you where Hadoop fits in the Hadoop ecosystem:

Introducing SAP HANA Vora1.2

SAP HANA Vora 1.2 was released recently and with this new version we have added several new features to the product. Some of the key ones I want to highlight in this blog are

• Support for MapR Hadoop distro
• Introducing new “OLAP” modeler to build hierarchical data models on Vora data
• Discovery service using open source Consul – to register Vora services automatically
• New Catalog to replace Zookeper as metadatstore
• Native persistency for metadata catalog using Distributed shared log
• Thriftserver for client access thru jdbc-spark connectivity

The new installer for Vora in ver1.2 extends the simplified installer to be able to use Hadoop Management tools like MapR Control System to deploy Vora on all the Hadoop/Spark nodes. This is an addition to what was provided in ver1.0 for Cloudera Manager and Ambari admin tools.

[SAP HANA Academy] Configure the SAP HANA Spark Controller to Read SAP HANA Vora Tables

In another part of the SAP HANA Academy's SAP HANA Vora series Tahir Hussain Babar (Bob) walks through how to configure the SAP HANA Spark Controller in nine tutorial videos. With the SAP HANA Spark Controller you will be able to read your SAP HANA Vora tables from SAP HANA.

Each and every script that Bob uses through out this nine part series can be found here on GitHub.

How to Install Hive and Load Data