it's theCUBE,

covering Spark Summit 2017.

Brought to you by Databricks.

>> Welcome back we're here at theCube at Spark Summit 2017.

I'm David Goad here with George Gilbert, George.

>> Good to be here.

>> Thanks for hanging with us.

Well here's the other man of the hour here.

We just talked with Ali, the CEO at Databricks

and now we have the Chief Architect

and co-founder at Databricks, Reynold Xin.

Reynold, how are you?

>> I'm good. How are you doing?

>> David: Awesome.

Enjoying yourself here at the show?

>> Absolutely, it's fantastic.

It's the largest Summit.

It's a lot interesting things,

a lot of interesting people with who I meet.

>> Well I know you're a really humble guy

but I had to ask Ali what should I ask Reynold

when he gets up here.

Reynold is one of the biggest contributors to Spark.

And you've been with us for a long time right?

>> Yes, I've been contributing for Spark

for about five or six years

and that's probably the most number

of commits to the project

and lately more I'm working with other people

to help design the roadmap

for both Spark and Databricks with them.

>> Well let's get started talking about some

of the new developments that you want

maybe our audience at theCUBE hasn't heard

here in the keynote this morning.

What are some of the most exciting new developments?

>> So, I think in general if we look at Spark,

there are three directions I would say we doubling down.

One the first direction is the deep learning.

Deep learning is extremely hot and it's very capable

but as we alluded to earlier in a blog post,

deep learning has reached sort of a mass produced point

in which it shows tremendous potential but the tools

are very difficult to use.

And we are hoping to democratize deep learning

and do what Spark did to big data, to deep learning

with this new library called deep learning pipelines.

What it does, it integrates different

deep learning libraries directly in Spark

and can actually expose models in sequel.

So, even the business analysts

are capable of leveraging that.

So, that one area, deep learning.

The second area is streaming.

Streaming, again, I think that a lot of customers

have aspirations to actually shorten the latency

and increase the throughput in streaming.

So, the structured streaming effort is going to be

generally available and last month alone

on Databricks platform,

I think out customers processed three trillion records,

last month alone using structured streaming.

And we also have a new effort to actually push down

the latency all the way to some millisecond range.

So, you can really do blazingly fast streaming analytics.

And last but not least is the SEQUEL Data Warehousing area,

Data warehousing I think that it's a very mature area

from the outset of big data point of view,

but from a big data one it's still pretty new

and there's a lot of use cases that's popping up there.

And Spark with approaches like the CBO and also impact here

in the database runtime with DBIO,

we're actually substantially improving the performance

and the capabilities of data warehousing futures.

>> We're going to dig in to some of those technologies here

in just a second with George.

But have you heard anything here so far from anyone

that's changed your mind maybe about what to focus on next?

So, one thing I've heard from a few customers

is actually visibility and debugability

of the big data jobs.

So many of them are fairly technical engineers

and some of them are less sophisticated engineers

and they have written jobs and sometimes the job runs slow.

And so the performance engineer in me would think

so how do I make the job run fast?

The different way to actually solve that problem

is how can we expose the right information

so the customer can actually understand

and figure it out themselves.

This is why my job is slow and this how I can tweak it

to make it faster.

Rather than giving people the fish,

you actually give them the tools to fish.

>> If you can call that bugability.

>> Reynold: Yeah, Debugability.

>> Debugability.

>> Reynold: And visibility, yeah.

>> Alright, awesome, George.

>> So, let's go back and unpack some of those

kind of juicy areas that you identified,

on deep learning you were able to distribute,

if I understand things right, the predictions.

You could put models out on a cluster

but the really hard part, the compute intensive stuff,

was training across a cluster.

And so Deep Learning, 4J and I think Intel's BigDL,

they were written for Spark to do that.

But with all the excitement over some of the new frameworks,

are they now at the point where they are as good citizens

on Spark as they are on their native environments?

>> Yeah so, this is a very interesting question,

obviously a lot of other frameworks

are becoming more and more popular,

such as TensorFlow, MXNet, Theano, Keras and Office.

What the Deep Learning Pipeline library does,

is actually exposes all these single note

Deep Learning tools as highly optimized

for say even GPUs or CPUs, to be available as a estimator

or like a module in a pipeline of the machine learning

pipeline library in spark.

So, now users can actually leverage Spark's capability to,

for example, do hyper parameter churning.

So, when you're building a machine learning model,

it's fairly rare that you just run something once

and you're good with it.

Usually have to fiddle with a lot of the parameters.

For example, you might run over a hundred experiments

to actually figure out what is the best model I can get.

This is where actually Spark really shines.

When you combine Spark with some deep learning library

be it BigDL or be it MXNet, be it TensorFlow,

you could be using Spark to distribute that training

and then do cross validation on it.

So you can actually find the best model very quickly.

And Spark takes care of all the job scheduling,

all the tolerance properties and how do you read data

in from different data sources.

>> And without my dropping too much in the weeds,

there was a version of that where Spark wouldn't take care

of all the communications.

It would maybe distribute the models and then do some

of the averaging of what was done out on the cluster.

Are you saying that all that now can be managed by Spark?

>> In that library, Spark will be able to actually

take care of picking the best model out of it.

And there are different ways you an design

how do you define the best.

The best could be some average of some different models.

The best could be just pick one out of this.

The best could be maybe there's a tree of models

that you classify it on.

>> George: And that's a hyper parameter

configuration choice?

>> So that is actually building functionality

in Sparks machine learning pipeline.

And now what we're doing is now you can actually

plug all those deep learning libraries directly into that

as part of the pipeline to be used.

Another maybe just to add,

>> Yeah, yeah,

>> Another really cool functionality

of the deep learning pipeline is transfer learning.

So as you said, deep learning takes a very long time,

it's very computationally demanding.

And it takes a lot of resources, expertise to train.

But with transfer learning what we allow the customers to do

is they can take an existing deep learning model

as well train in a different domain and they we'd retrain it

on a very small amount of data very quickly

and they can adapt it to a different domain.

That's how sort of the demo on the James Bond car.

So there is a general image classifier that we train it on

probably just a few thousand images.

And now we can actually detect whether a car

is James Bond's car or not.

>> Oh, and the implications there are huge,

which is you don't have to have huge training data sets

for modifying a model of a similar situation.

I want to, in the time we have,

there's always been this debate

about whether Sparks should manage state,

whether it's database, key value store.

Tell us how the thinking about that has evolved

and then how the integration interfaces

for achieving that have evolved.

>> One of the, I would say, advantages of Spark is that

it's unbiased and works with a variety of storage systems,

be it Cassandra, be it Edgebase, be it HDFS, be is S3.

There is a metadata management functionality in Spark

which is the catalog of tables that customers can define.

But the actual storage sits somewhere else.

And I don't think that will change in the near future

because we do see that the storage systems

have matured significantly in the last few years

and I just wrote blog post last week about the advantage

of S3 over HDFS for example.

The storage price is being driven down

by almost a factor of 10X when you go to the cloud.

I just don't think it makes sense at this point

to be building storage systems for analytics.

That said, I think there's a lot of building

on top of existing storage system.

There's actually a lot of opportunities for optimization

on how you can leverage the specific properties

of the underlying storage system

to get to maximum performance.

For example, how are you doing intelligent caching,

how do you start thinking about building indexes

actually against the data

that's stored for scanned workloads.

>> With Tungsten's, you take advantage of the latest hardware

and where we get more memory intensive systems

and now that the Catalyst Optimizer

has a cost based optimizer or will be, and large memory.

Can you change how you go about knowing

what data you're managing in the underlying system

and therefore,

achieve a tremendous acceleration in performance?

>> This is actually one area we invested in the DBIO module

as part of Databricks Runtime,

and what DBIO does, a lot of this are still in progress,

but for example, we're adding some form

of indexing capability to add to the system

so we can quickly skip and prune out all the irrelevant data

when the user is doing simple point look-ups.

Or if the user is doing a scan heavy workload

with some predicates.

That actually has to do with how we think

about the underlying data structure.

The storage system is still the same storage system,

like S3, but were adding actually

indexing functionalities on top of it as part of DBIO.

>> And so what would be the application profiles?

Is it just for the analytic queries

or can you do the point look-ups and updates

in that sort of scenario too?

>> So it's interesting you're talking about updates.

Updates is another thing that we've got a lot

of future requests on.

We're actively thinking about how

we will support update workload.

Now, that said, I just want to emphasize for both use case

of doing point look-ups and updates,