Top 9 Search-Driven Analytics Evaluation Criteria

9TOP SEARCH-DRIVEN ANALYTICS
Evaluation Criteria

It’s been said that data is the new oil. An explosion of
data sources and new technologies for capturing them
are creating massive opportunities for companies. But in
this new quest for insights, the last mile of data access
remains the biggest obstacle.
In our personal lives, search has transformed how we
access information. Google, Facebook and Amazon have
raised our expectations for how we want to access data
at work. Finally, after a decade of failed promises and
misguided approaches in the enterprise, search is making
a comeback.
The hype is behind us. It’s now time to evaluate today’s
search-driven analytics vendors on what matters most
to creating new insights: ease of use, data volumes, user
scale, and whether you will need an army of consultants
to integrate these new technologies into your existing BI
environment.
In this book, we present nine different criteria that you
can use to evaluate search-driven analytics products -
everything from training time to search intelligence, data
modeling, and total cost of ownership.
Introduction
Page 2

1 Training Time
2 Search Experience
3 Search Intelligence
4 Chart Creation
5 Speed at Scale
6 Data Modeling
7 Data Environment
8 Data Security & Governance
9 Cost
Table of Contents
Page 3

TRAINING TIME
Despite $69B spent annually on BI software and services,
there’s only 22% adoption in the enterprise.
Traditional BI products require you to take multi-day
classes or get certifications before you can use them.
Meanwhile, over a billion people use Google every day. Do
you remember going to your first Google training class?
Average duration of a beginner
BI training class.
1 Page 4
3days

THE LESS TRAINING NEEDED, THE MORE ADOPTION GROWS
TRAINING TIME
1
Most BI products are designed for business analysts who need to
go to a week-long training class to become productive. Even IT
teams need training to support these products effectively. This
training requirement and the continuous need to stay on top of
technical skills is why the BI industry is plagued by such a terrible
adoption problem (22%).
In contrast, today’s most popular consumer tech services that are
driven by a search interface don’t require any training. Google,
Yelp, Uber, Mint, Amazon, and many others rely on search to drive
their user experience - no manual required. If you had to go to
a training class to use those products their adoption would be
terrible, too.
This is the reason consumer companies measure their adoption in
millions, while enterprise technologies measure in thousands.
Ask vendors for the length of a typical training session for
non-technical users, business analysts, and IT and BI teams.
Page 5
64% of business users are
confused by legacy BI interfaces.
“

You use search every day on consumer websites such as Google,
Amazon and Facebook. All three are similar, but work slightly
differently. Google returns lists of web pages, Facebook lists of
friends and events, and Amazon lists of products.
Most BI products have search boxes designed similarly to return
ranked lists of pre-built reports of dashboards.
But for search to reach the next level in BI, a fundamentally
different approach is required. If you type “revenue last year
in California”, you don’t want a list of ranked reports and
dashboards. You want a single number. This requires a new kind
of search experience designed for numbers that is very different
from the search engines powering the consumer web.
searches per day on Google
SEARCH EXPERIENCE
Page 6
3.5B
2

NOT ALL SEARCH IS CREATED EQUAL
SEARCH EXPERIENCE
2
Many BI products advertise a search box. It is important to understand
how each of them work. Does it only search pre-built reports and
dashboards? Does it only look at metadata? Does it merely return a list
of matches? Does it use any guesswork in estimating results? Or does
it provide a single answer?
Some approaches, like natural language processing (NLP),
rely on programmable algorithms that interpret what the user is asking
and provide error-prone estimates for answers. Others modeled after
web search return a long list of ranked search results of pre-built
reports that the user has to wade through.
Meanwhile, the newest breed of search-driven analytics engines search
through all the underlying raw data, compute results, and then present
charts and numbers based on those real-time calculations.
“Search” has many flavors - document, metadata, dashboards,
or numbers. Determine which best meets your needs.
Source: Gartner
Page 7
Data discovery will continue to displace IT
authored static reporting as the dominant BI
and analytics user interaction paradigm.“

Google changed consumer search forever when it invented the
PageRank algorithm that ranked pages by how many other pages
link to them. This was different from how Facebook grew using
graph search for social networks, or how Amazon’s faceted search
made it easy to browse large catalogs.
Search technologies in the BI world today mostly equate to a BI
analyst either setting up a database of pre-defined search terms
and answers for a business user to “discover”, or providing search-
based access to saved reports and dashboards.
What is more rare but more useful is a search engine designed
for numbers, one that can look directly at raw data and compute
results on-the-fly with 100% accuracy.
percentage of users who click
on the first Google link.
SEARCH INTELLIGENCE
Page 8
33%
3

ACCURACY BUILDS TRUST. TRUST DRIVES ADOPTION.
SEARCH INTELLIGENCE
3
Business users need to be able to trust the numbers they get from a
BI solution. A search-driven analytics engine should provide a single
consistent and reliable answer - always. Some methods such as NLP
provide probabilistic results based on programmed algorithms that must
be constantly refined. Even after months of tuning, they still have a 10-20%
error rate.
Most users don’t understand how all their data relates to each other, or
which schema represents the underlying tables, or which joins are needed
to find an answer. A smart search-driven analytics engine should hide all
such complexity away from the user.
Users need a search experience that recognizes patterns, understands
synonyms, has spell check, and offers suggestions as they type based on
other users’ activity - similar to Google’s type-ahead feature.
Ask if search results are calculated on the fly or retrieved from
pre-calculated aggregate tables. Are the results accurate or estimates?
Page 9
It’s also critical for a user to easily analyze results at different time resolutions (daily,
monthly) without waiting for the BI team to create new cubes or aggregate tables.
Search-driven analytics solutions should do this automatically and compute results across
billions of rows of data in under a second.
Finally, a good search-driven analytics experience should provide a way to verify how
results were calculated, without requiring users to learn SQL or other programming
languages.

Isn’t it amazing when you type “wea” into Google and
before you’ve finished typing “weather” you instantly
get current and forecasted conditions for the city
you’re in along with a “card” visual showing you a
picture of a sun or cloud? It’s like the app knows what
you’re looking for before you do and presents the
information in an easy way to consume it.
Contrast that with legacy BI products: after days of
training, you still need to remember how to click eleven
times in order to build a chart and then decide if it has
the information you seek.
hours per day saved by Google
Instant users.
CHART CREATION
Page 10
950K
4

THE BEST VISUALIZATIONS CREATE THEMSELVES
CHART CREATION
4
In today’s world where search pervades our consumer experience,
search and speed have become synonymous. If a search-driven
analytics product is to be adopted widely, it needs to cut down
any unnecessary wait time between the user’s query and the
visualized results.
An important part of this process is to decide intelligently the
best chart type for the user’s query and instantly return a visual
along with an answer. But data is complicated. Picking axis and
chart types is hard. This is a situation in which machines trump
humans. Any assistance a user can get goes a long way toward
adoption and insight. Then if the user wants to change the
chosen chart type, they should always have the option.
Count the number of clicks it takes to create a chart.
Page 11
Only 23% of current BI users
are comfortable creating charts
& graphs.
“Source: TDWI

The power of Google is that it delivers the one-two punch of
a simple search experience done at massive scale. Using a
search bar is simple and intuitive, but the most powerful part
of Google is its ability to search everything across the web.
If Google was restricted to the files on your local machine
it would be significantly less useful. Yet in the BI world, so
many products offer restricted views into your data, that do
not scale across the enterprise, across thousands of users, or
across large volumes of data and data sources.
percentage of people who abandon
a website that takes more than
3 seconds to load.
SPEED AT SCALE
Page 12
40
5 %

SPEED AT SCALE IS THE SECRET TO SEARCH-DRIVEN INSIGHTS
SPEED AT SCALE
5
Mid-to-large size enterprises have hundreds of tables, billions of rows,
and thousands of users. The key to providing insights is delivering a
simple search experience at scale and still returning answers to the
user in less than a second.
Studies have shown that if a user doesn’t get a result from Google in
less than three seconds, they abandon the page. Compare that statistic
to waiting overnight for a big report to run in a legacy BI product and,
again, it’s not surprising that there’s an adoption issue in the industry.
Meanwhile, some of today’s faster more popular data visualization
tools are desktop products that can’t handle data sets larger than
a few gigabytes. With hundreds of gigabytes created quarterly by
the average enterprise, BI teams are faced with the challenge of
determining which datasets are most important for different types of
users. It’s a continuous task that always leaves users wanting more.
If the technology doesn’t scale with speed, your BI project is destined
for problems.
Ask how much data the product can handle. And how
many users it can support simultaneously.
Page 13
62% of enterprises store more than
100TB of data.
“Source: Microsoft

IT teams spend too much time modeling data.
Data modeling headaches are the reason enterprises spend
nearly 3 times more on BI software services than on software
licences. It’s why entirely new careers like “data wrangling” have
emerged.
Creating cubes and aggregate tables for individual lines of
business is not the best use of time for BI teams, especially
when tactical dashboards may not have the answer an end
business user needs
Consumer search technologies have enabled untrained users to
search through complex product catalogs, network graphs, and
any type of document imaginable on the web. Why can’t the
enterprise user do the same with their data?
percentage of time a data scientist
spends modeling and preparing
data for analysis.
DATA MODELING
Page 14
80%
6

MINIMIZE MODELING TO REDUCE PROFESSIONAL SERVICES SPEND
DATA MODELING
6
A traditional BI environment takes months of modeling - building OLAP cubes
or aggregate tables, and significant database tuning before any results can be
exposed through a search interface. On an ongoing basis, these databases need
maintenance and care, which sucks up even more time and resources.
Other systems based on NLP techniques require a significant professional
services spend to build semantic search models for each implementation. Then,
even after months of tuning from the world’s top experts, they only yield 80-
90% accuracy.
Meanwhile, some search-driven analytics products are schema-aware and able
to remove a significant amount of modeling complexity. Schema-awareness
means the search engine understands the relationships between different
sources of data and it is able to relate them together automatically.
A complicated product typically comes with an expensive professional services
engagement in order to get it to work. Better products will free up BI teams to
focus on higher value problems like data governance and data quality.
Find out how long a typical implementation takes before
you can start using the product.
Page 15
Source: Gartner
Through 2016, 90% of self-service BI initiatives will suffer from
data governance inconsistencies.
“

%
When it comes to data access within the enterprise, the
last mile is always the hardest, even more so when the
data is split across several sources requiring different data
integration tools.
The entire process of getting useful data into the hands of
business users can take months, which no company can
afford to waste.
Businesses need to gather insights from external data
sources just as easily as they would from their internal
systems. Google compiles search results from a variety of
sources. Why should enterprise BI tools be any different?
Search-driven analytics should accomplish this with the
same ease of use we expect from consumer technology.
average number of applications
used by enterprises today.
DATA ENVIRONMENT
Page 16
500+
7

SEARCH SHOULD ANALYZE ANY SOURCE
DATA ENVIRONMENT
7
Ensure the product can search through data from any
source you might need to analyze.
Source: TDWI
Page 17
Speed of insight and breadth of data
sources are the critical factors to help
stand out in the marketplace.“
The ability to search data at scale from a variety of sources
is essential to a productive business user. In the same way
Google combines search results from across the entire
web, search-driven analytics solutions should be capable
of analyzing search results across tables from different
databases, applications, spreadsheets, or Hadoop clusters.
For this to happen, the search-driven analytics solution has to
be compatible with your existing data environment - different
types of data sources, as well as different data integration or
ETL technologies.
Instead of learning to use different BI products for different
types of data sources, one search-driven experience for all
data sources lowers the bar for business users and makes
significant adoption more likely.

%
8
Securing data within the enterprise is a solved problem. The
best BI vendors already offer that. But packing all of those
security requirements into a sophisticated search bar? Now
that’s a different story.
How do you ensure that even the search suggestions obey
security restrictions? In other words, how do you secure the
search intelligence at a user level?
This is a unique challenge in the enterprise that even the likes
of Google haven’t had to tackle for consumer search.
percentage of IT professionals that
say data security is a top concern.
DATA SECURITY &
GOVERNANCE
Page 18
90%

SECURITY SHOULD BE BUILT INTO THE RESULTS & SEARCH BOX
DATA SECURITY & GOVERNANCE
8
A good search interface needs to be able to access all data across the
enterprise, while limiting access to only what each user is supposed
to see. It should be able to integrate easily into the existing directory
services through LDAP or similar protocols.
The underlying data needs to be secured at a row, column, and table level.
An employee table might have a compensation column that is visible
only to select users. A sales table might have rows of sales information
by region that can be seen only by reps in that region. And table level
protection should ensure that departments can see only their own tables.
An enterprise-class search-driven analytics experience needs to honor
access privileges, while accessing billions of rows of data, and returning
results in under a second.
Verify that both the search box and search results obey your
access rules and users see only what they are allowed to see.
Source: Gartner
Page 19
More than 80% of organizations will fail to develop a
consolidated data security policy across silos.“

Business users today often wait months to get access to new
BI products thanks to lengthy deployment cycles. Cobbling
together different pieces of infrastructure to get your BI
environment up and running is a nightmare for most IT
organizations. There’s a huge cost to implementing and an
arguably even greater opportunity cost to waiting for insights.
Best-of-breed BI solutions should work right out of the
box,with minimal implementation headaches - just like your
Mac computer or favorite consumer app.
percentage of BI dollars spent on
services to make the software work.
COST
Page 20
80%
9

UNDERSTAND THE TRUE COST OF DEMOCRATIZATION
COST
9
Time to value is the first thing to evaluate. Will the product take months
to deploy? Weeks? By eliminating data modeling, cube building, semantic
modeling, and hardware tuning, new search-driven analytics products can be
up-and-running in a matter of hours.
Beyond implementation and licensing, the true cost of many BI solutions
include hardware, tuning and storage costs, training costs, IT maintenance
and support, and user training costs.These occur after the initial
implementation and can have a major impact on ROI. Modern search-driven
products radically reduce these costs.
Then there’s the financial impact of user adoption. For many BI products
today, more than half of the usage is attributed to simple report and
dashboard viewing. This means the user logins are simply replacing emailed
PDF reports - thereby making the cost of those licenses hard to justify.
A modern, well-designed search experience should go far beyond scheduled
reports and give business users the ability to answer ad hoc questions on the
fly. It should be addictive and spread quickly within an enterprise.
As adoption builds, it’s important to evaluate the per user costs and not
artificially penalize new users. When software works well, adoption should be
both contagious and economically beneficial.
Understand hidden implementation and maintenance costs. Ensure
that wide adoption is not gated by high per-user license costs.
Page 21
Understand the cost of adoption
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
???
? ??? ?
? ??? ?? ??? ??? ?
? ??? ?? ??? ??? ?
??? ?? ??? ?? ?
?? ?? ??? ??
?

If software is eating the world, search is clearly eating
software. Search has infiltrated every aspect of our
consumer tech lives and is now making bold new
strides into enterprise software. Products that offer
search-driven analytics are poised for rapid growth
because they bring both speed (instant results)
and scale (billions of rows) to business intelligence.
With so many approaches, it is critical to understand
the differences between vendors before making a
significant investment. We hope this framework proves
useful as you begin delivering instant answers to every
business user in your company.
Conclusion

ThoughtSpot has built the world’s first search-driven
data analytics solution for the enterprise. Anyone can
use ThoughtSpot with zero training to ask questions,
analyze company data, and build reports and
dashboards - all in seconds - using a browser-based
search interface. ThoughtSpot’s Analytical Search
Appliance combines data from on-premise, cloud
and desktop data sources, can scale up to terabytes
of data, and can be deployed in under an hour. The
company’s founding team has previously built market-
defining search and analytics technologies at Google,
Amazon, Oracle and Microsoft.
For more information, please visit thoughtspot.com
Page 23

DON’T BI. JUST SEARCH.
01N0812

Top 9 Search-Driven Analytics Evaluation Criteria

More Related Content

What's hot (20)

Similar to Top 9 Search-Driven Analytics Evaluation Criteria (20)

Recently uploaded (20)

Top 9 Search-Driven Analytics Evaluation Criteria