Dec 17, 2015 · 22m · mad
10 Commandments for BI in Big Data, Shant Hovsepian, Arcadia Data (Data Driven NYC / FirstMark)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this presentation at Data Driven NYC #42, Arcadia Data co-founder and CTO Shant Hovsepian presents '10 Commandments' for building native Business Intelligence on scale-out Big Data platforms. He outlines how enterprises can unlock real business value by eliminating data movement, preserving native security, and replacing legacy static reporting with interactive visual applications.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 3.5% of the talking time here. How this is scored →
Matt as informed peer 0.3 Guest teaching 0.3 Guest disagreement 0.3 Matt pushing back 0.3
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 18:07 Playful banter on presentation time overruns
Shant playfully remarks that Matt will extract payback during Q&A after Shant poked fun at Matt in his opening slides.
Hardest push from Matt ▶ 19:18 Probing on pricing model specificsMatt pushes Shant to articulate how Arcadia Data avoids the pricing traps and scaling fears that plague legacy BI vendors.
Biggest teaching moment ▶ 19:31 Detailing concurrent node subscription billingShant explains to the audience and host how legacy named-seat and per-gigabyte licensing models fail, contrasting them with node-aligned concurrent billing.
Matt holds his own ▶ 18:07 Matt calling out the talk length and re-centering the product pitchMatt humorously calls out Shant for exceeding his time limit before steering the conversation directly into Arcadia Data's unique value proposition.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Arcadia Data Overview and Framing the 10 Commandments | 0 | 0 | 1 | 0 | This is a solo presentation monologue by Shant setting up his talk and framing the 10 Commandments for BI on Big Data. The host does not speak during this segment. | |
| Commandment 1: Thou Shalt Not Move Big Data | 0 | 0 | 0 | 0 | Shant continues his presentation on Commandment 1 regarding keeping computation close to data rather than moving big data. As a pure monologue segment, host metrics remain zero. | |
| Commandment 2: Respect Corporate Security Policy | 0 | 0 | 0 | 0 | Shant outlines Commandments 2 and 3 regarding corporate security compliance and avoiding predatory per-gigabyte pricing models. The host is not involved in this presentation segment. | |
| Commandment 4: Prioritize Collaborative Exploration | 0 | 0 | 0 | 0 | Shant explains the need for collaborative exploration and analyzing data in native formats like JSON and FIX. The segment is an uninterrupted solo presentation. | |
| Commandment 6: Demand Fast Query Results | 0 | 0 | 0 | 0 | Shant delivers Commandments 6 and 7, emphasizing rapid query execution and building interactive applications over static reports. Host participation is zero during the slide talk. | |
| Commandment 8: Leverage Intelligent BI Features | 2 | 2 | 1 | 2 | Matt opens the Q&A session by playfully teasing Shant about going over time before asking insightful questions about Arcadia Data's business model and pricing structure. The dynamic is warm and collaborative. |
Statements from this episode (13)
Extracting and moving big data into separate data marts creates severe overhead
“It's also a huge maintenance problem. So, a lot of customers we talk to don't want to, you know, forget about the network performance or the CPU computation, right? It's just a management overhead of now there are two copies of something as logically the same.”
YARN and Mesos enable modern on-cluster business intelligence
“On cluster BI is something that's possible now. Thanks to things like Yarn, Mesos, we're seeing a whole new resurgence of operating systems.”
Reimplementing security across storage, database, and BI layers causes data loss
“If you have to sort of re-implement your whole security model, you know, once in your storage layer, once your database layer, and once in your application BI layer, you're not going to do it, right? You're going to lose information.”
MongoDB security architecture enables enterprise application integration and data lineage
“Mongo has this amazing security architecture now that your applications can plug into, propagate that user information all the way up to the application layer, and enforce, you know, this visualization and the data lineage associated with it along the way.”
Business intelligence pricing models that penalize increased user adoption are dangerous
“Pricing models that penalize you for increased adoption are kind of dangerous, right?”
Arcadia Data saw enterprise clients scale from 12 to 600 active users
“We've had multiple customers who, within a couple of months, have had deployments go from, you know, I'm gonna say, like, tens of billions of entries to hundreds of billions, right? They went from having 12 active users on the system to 600, right?”
Big data analytics sharing should mirror GitHub's clone and fork model
“When I think of sharing, I think of the GitHub model, right, where it's not, here's your final published product, but here is, clone it, fork it, and this is how I derive at those insights, so other people can learn from those insights, and that's really impor…”
Effective BI tools must analyze big data natively without ETL transformation
“You want to find the BI solution that won't tell you, hey, sorry, please transform your data into a pretty table. You want BI solutions that can really analyze the data in its native form because there is value for having data in that form.”
Big data sampling must occur below blocking operations to ensure valid visualizations
“Certain types of analytics require blocking operation, right? At certain points in the analysis, I need to stop and count everything. So you need to be able to push that sampling down below that and not above it. Otherwise, the only things you'll be able to sa…”
D3 succeeded by decoupling data from visualization and DOM element mapping
“So D three became awesome because not that it created this great visual interface in the JavaScript library, but it decoupled the relationship between the data and how it's visualized and how the actual elements in the DOM and how they map.”
Analytics users prefer direct visual interaction over traditional menus and parameters
“So now people don't want to click on menus and play with parameters. They want to actually interact with the visual elements to get the results they want.”
Telling business users to run R functions is the wrong BI approach
“And, you know, you can run this R function isn't the right answer. The answer is what matters for business users, correlation, forecasting, these things, making them very easy to use.”
Arcadia Data charges based on concurrency rather than named user seats
“We don't penalize you for named users. We don't penalize you for embedding, but concurrent users, right? So the things, the functionality and the features that are sort of a technological burden on us, concurrency is a big deal, so we try to sort of bill, ah, …”