Ben Rogojan (Seattle Data Guy) illustrates the overlap between data engineering and machine learning engineering tools.
Prediction Not checkable as stated
Rogojan: Apache Iceberg is becoming the de facto open table format standard
“Iceberg as the de facto, like if we're going to use an open, open table format, that's going to be likely the one people will, I assume, pick since everyone's now supporting it, right?”
Prediction Not checkable as stated
Rogojan: SQL will remain essential as LLM raw data-dumping approaches fail
“The next one I said SQL isn't going anywhere, and I kind of said that for more than one reason. One, I feel like someone's going to tell us it's time to revamp data lake v one again, where we'll just dump everything in there and we'll let an LLM figure it out …”
Insight
Ben Rogojan: Data teams bear primary responsibility for bridging business communication gaps
“I think in general, I found that if you're the data team, you're going to probably have to do a little more of the lift just because it's, Maybe a little easier for you to make that leap.”
Prediction Not checkable as stated
Rogojan: Fully replacing data engineers with AI is still years away
“So I think we're a few years away from just like replacing data engineers, which has been the goal for what it feels like a decade or plus now.”
Insight
Rogojan: Databricks and Snowflake are fighting for workflows, not contract lock-in
“It's really been a white fight for workflows versus a fight for like contracts. Cause in the past it was like, okay, you sign a ten million dollar contract for Teradata. You're kind of stuck with it, right? Like there's no choice now for the next three to five…”
Opinion
Rogojan: Databricks is better suited for machine learning than data warehousing
“I think Databricks makes sense to me still more from an ML standpoint. I know they like the data warehousing side. I know they, that's where they're going to pitch themselves”