GitHub, every mention

Best of 2024 in Agents (from #1 on SWE-Bench Full, Prof. Graham Neubig of OpenHands/AllHands) Dec 25, 2024 · 22 mentions

• ▶ 4:58 Graham Neubig And here, um, I, I want to open up specifically that GitHub repo.
• ▶ 10:34 Graham Neubig Um, the agents are super good at using the GitHub API also, so they can do, you know, things on GitHub, like finding all of the, you know, comments on your issues or checking GitHub actions and stuff.
• ▶ 10:34 Graham Neubig Um, the agents are super good at using the GitHub API also, so they can do, you know, things on GitHub, like finding all of the, you know, comments on your issues or checking GitHub actions and stuff.
• ▶ 12:26 Graham Neubig We have a GitHub plugin for tagging and resolving issues.
• ▶ 15:53 Graham Neubig And so for solving GitHub issues, you can kind of have, uh, an overall plan.
• ▶ 18:43 Graham Neubig So just to give an example, um, we fix GitHub actions when GitHub actions are failing, and we do that over and over and over again.
• ▶ 23:12 Graham Neubig Benchmark that comes from real world pull requests on GitHub.
• ▶ 26:33 Graham Neubig Um, right now we have 53% or 55% on sweet bench verified, which is real world GitHub PRS. 3 times in the scene
• ▶ 39:48 Graham Neubig It's like, um, if you detect the word GitHub anywhere, you get instructions about how to interact with GitHub, like use the API and don't browse. 6 times in the scene
• ▶ 43:31 Graham Neubig It will sometimes fail to fix a GitHub workflow because it will not look at the GitHub workflow and understand what the GitHub workflow is doing before it solves the problem.
• ▶ 48:38 Graham Neubig My preferred solution to this at the moment is GitHub, like, fine-grained authentication tokens. 5 times in the scene