Prediction Not checkable as stated
Rogojan: Apache Iceberg is becoming the de facto open table format standard
“Iceberg as the de facto, like if we're going to use an open, open table format, that's going to be likely the one people will, I assume, pick since everyone's now supporting it, right?”
Prediction Not checkable as stated
Rogojan: SQL will remain essential as LLM raw data-dumping approaches fail
“The next one I said SQL isn't going anywhere, and I kind of said that for more than one reason. One, I feel like someone's going to tell us it's time to revamp data lake v one again, where we'll just dump everything in there and we'll let an LLM figure it out …”
Insight
Ben Rogojan: Data teams bear primary responsibility for bridging business communication gaps
“I think in general, I found that if you're the data team, you're going to probably have to do a little more of the lift just because it's, Maybe a little easier for you to make that leap.”
Prediction Not checkable as stated
Rogojan: Fully replacing data engineers with AI is still years away
“So I think we're a few years away from just like replacing data engineers, which has been the goal for what it feels like a decade or plus now.”
Insight
Rogojan: Databricks and Snowflake are fighting for workflows, not contract lock-in
“It's really been a white fight for workflows versus a fight for like contracts. Cause in the past it was like, okay, you sign a ten million dollar contract for Teradata. You're kind of stuck with it, right? Like there's no choice now for the next three to five…”
Opinion
Rogojan: Databricks is better suited for machine learning than data warehousing
“I think Databricks makes sense to me still more from an ML standpoint. I know they like the data warehousing side. I know they, that's where they're going to pitch themselves”
Prediction Not checkable as stated
Ben Rogojan: Natural language-to-SQL AI will not take over anytime soon
“I don't think we're going to see that take over anytime soon, unless you have your data in such a well-formatted way that, like, yeah, of course it's easy to do.”
Insight
Rogojan: Organizations repeat foundational data mistakes in every AI hype cycle
“People Kind of go through the same iterations. New people come to the space or possibly maybe it's more the business side get really excited by the prospect of what AI or data science or neural networks, whatever iteration we're in can do. And then we kind of …”
Assertion Not checkable as stated
Ben Rogojan observed ML engineers at Facebook accessing data directly from warehouses
“Some companies do have maybe the ML engineers access data more directly from maybe the data warehouse, which was things I saw like at Facebook.”
Insight
Ben Rogojan: Non-Big Tech data engineers must manage their own pipeline tooling
“At Facebook, you essentially have your internal airflow that a different team runs, and you can just push files to it and it picks up the data pipelines, you know, automatically. Whereas in most other companies, you might be the person that has to even manage,…”
Insight
Rogojan: Understanding data flow takes longer than learning SQL syntax
“SQL you can learn quickly. What's going to take time is just getting a sense for how data Operates and flows and data sets in general.”
Prediction Not checkable as stated
Rogojan: Enterprise migration away from Spark and Kafka will take years
“I think initially just picking up Kafka, Spark, those are still so heavily in use that even if they go out in the next three to five years, it's going to take time to migrate and things of that nature.”
Insight
Ben Rogojan: API connections are usually the hardest part of data extraction
“Cause that's usually the big thing. It's like the API connection tends to be the one that is the hardest.”
Opinion
Rogojan blames excessive VC funding for the bloated data engineering tool ecosystem
“I think the reason you see so many tools is like, I mean, VC funding was pretty high, so you can blame yourselves.”
Opinion
Rogojan: Snowflake is difficult to beat for lean, resource-constrained data teams
“Snowflake makes it really easy, and I'd say BigQuery does too, but Snowflake makes it really easy. Like, if you do not have a ton of people to manage things, I think it's hard to go against it.”
Prediction Not checkable as stated
Rogojan: Companies will keep chasing data holy grails like self-service analytics
“I think we'll continue chasing the same holy grails. I feel like we want to see new things, but I think We're still trying to figure out some things like self-service analytics or some of the other holy grails, like even being like data driven kind of feels li…”
Insight
Rogojan: Data teams get trampled by fickle business demands without strong leaders
“If you don't have a strong data leader that can either push back or make sure that you're doing things in a way that makes sense you kind of can get trampled by the business side and what they need and their kind of fickle nature where it's like, okay, now we'…”
Prediction Not checkable as stated
Rogojan predicts a rise in vertical-specific data solutions for domain data
“Yeah, I think we'll probably start seeing some vertical specific data solutions come out whether that's like tooling or maybe just one example I gave in, in my article was like Tuva Health, which is trying to make it easier to just standardize healthcare speci…”
Assertion Supported
Riot Games deployed machine learning models using Apache Airflow
“One of the teams at Riot Games was like deploying their ML models and they were using Airflow.”
Assertion Not checkable as stated
Ben Rogojan: Most data engineers transition laterally from analyst or developer roles
“I see a lot of people move laterally into data engineering, often from either data analyst or software engineer. Data analyst, because I think there's just the sheer quantity of data analytics jobs are More.”
Opinion
Rogojan prefers AWS and GCP due to Azure usability issues
“I have my own qualms with Azure every time I use it. So I just like AWS and GCP slightly better.”
Assertion Supported
Rogojan: Legacy data software like Informatica cost six to seven figures
“Before it's like, okay, you want Informatica, you know, let's talk six, seven figures in, in contract costs.”
Opinion
Ben Rogojan is not a fan of Amazon Redshift
“Not a big Redshift fan.”
Assertion Not checkable as stated
Ben Rogojan: Data type issues still routinely break data pipelines
“But to this day, people are still having like a data type issue break a pipeline that then causes everyone to have to kind of spend a little bit of time fixing them.”
