DocETL

“The reason it was a hundred dollars, if I ran the optimizer with GPT-Foro mini as the LLMs, it would be 10 dollars. But we use GPT four. Oh, just because I think we did this at a time where many hadn't come out yet.”

“So we're redoing our evaluation to be on data sets where we actually have ground truth from human annotators, but that's just not a practical setting. Like, nobody's coming to doc ETL with the ground truth.”

“I'm of a slightly different take. I feel like we do need some set of seed human labeled data.”

“Sometimes it's beneficial to chunk and sometimes you should not chunk. And we have observed this in a number of workloads and the insight that we've gained is that we will never know what it's all task specific and data specific. And we are so further convince…”

“But the problem is, is that for fairly complex tasks and data, LLM outputs for what we wanted to do is fairly inaccurate.”

“Running the optimizer, right, they used the optimizer. I don't know how many plans the optimizer generated, but it cost approximately a hundred dollars and less than half an hour. Just go get lunch and you come back and you get your optimized pipeline. And the…”

“Essentially, what this is trying to do is it's trying to take database, database concepts and pandas data frame concepts and try to apply them to shapeless documents.”

“This is very different from traditional rag or Q&A or document processing for a chatbot. Like, the kinds of queries that people are, people want to use .etl for can be expressed as etl style sweep and harvest, kind of, I want to look at my entire dataset. I wo…”