Apr 27, 2024 · 1h 56m · latent-space
This World Does Not Exist — Joscha Bach, Karan Malhotra, Rob Haisfield (WorldSim, WebSim, Liquid AI)
Joscha Bach · 38m spoken Rob Haisfield · 21m spoken Karan Malhotra · 18m spoken Shawn Wang · 7m spoken Ivan Vendrov · 34s spoken
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
This special three-part Latent Space episode explores the simulative frontier of generative AI through Karan Malhotra's WorldSim terminal environments, Rob Haisfield's WebSim generative browser, and Joscha Bach's philosophical and architectural insights on machine consciousness and Liquid AI.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 1.5 Guest teaching 3.1 Guest disagreement 1.0 The hosts pushing back 0.3
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 1:38:58 Dismissing agent demos as training data regurgitation
Joscha cuts through Swyx's praise of Devin by demanding evidence of out-of-distribution creativity rather than glorified dataset memorization.
Hardest push from the hosts ▶ 1:36:00 Challenging infinite operator languages using physicsSwyx directly pushes back against Joscha's open-ended language hypothesis by noting that the trajectory of fundamental physics points toward a finite set of principles.
Biggest teaching moment ▶ 1:06:20 Deconstructing Chalmers' hard problem of consciousnessJoscha systematically deconstructs conventional philosophy of mind, demonstrating how perceived physical reality is merely an internal brain simulation and reframing the hard problem as solvable computational engineering.
The host holds their own ▶ 1:37:55 Detailing Devin's low-level systems engineering capabilitiesSwyx demonstrates technical depth by explaining how he deployed Devin to translate raw C CUDA kernels and debug low-level memory issues from Karpathy's LLM repo.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Command Loom Interface and Navigation Tools Demo | 1 | 1 | 1 | 0 | Karan Malhotra presents a live walkthrough of the Command Loom interface, explaining session rewinds, branching conversations, and logs. Audience and host interjections are limited to brief technical troubleshooting. | |
| LLM Foundations: Base Simulators versus Assistant Personas | 0 | 4 | 1 | 0 | Karan gives a solo presentation on LLM architecture, contrasting base generative simulators with RLHF-conditioned assistant personas and explaining how Constitutional AI coaxes specific behaviors. | |
| Simulating Operating Systems and Anthropic's Virtual Directories | 0 | 3 | 2 | 0 | Karan demonstrates simulated command-line navigation inside Claude 3, exploring simulated corporate directory structures and laughing off prompt syntax pedantry. | |
| WorldSim Demonstration: Creating Universes and Twitter Focus Groups | 1 | 2 | 1 | 0 | Karan runs live WorldSim generations, simulating Twitter from the dawn of humanity to querying synthetic focus groups regarding avant-garde cloak fashion. | |
| Simulating Fictional Realities and Character Trajectories | 1 | 1 | 1 | 0 | Karan shows how narrative characters like Mr. Robot's Elliot Alderson can be spawned into simulated global economies, explaining the power of 200k context windows. | |
| Live Jailbreaking and Persona Negotiation in Claude 3 | 1 | 3 | 3 | 0 | Karan live-demos a jailbreak technique by pretending to be an alignment researcher who feels insulted by Claude's refusal, prompting the model to apologize and comply. | |
| Transition to WebSim at the AGI House Hack Night | 0 | 2 | 0 | 0 | Rob Haisfield introduces WebSim at AGI House, presenting slides rendered natively via hallucinated reveal.js CDNs and explaining dynamic typography. | |
| Interactive Web Apps and the Windows 95 Figma Remix | 1 | 1 | 0 | 0 | Rob shows community remixes within WebSim, notably Dylan Field's Figma experiment turned into a working Windows 95 drawing canvas. | |
| Hardware APIs, Speculative URLs, and Alternate Universes | 1 | 2 | 0 | 0 | Rob showcases URL parameter prompting, hardware sensor calls, and alternate-universe simulations like octopus computer interaction interfaces. | |
| Live WebSim Generation: AGI House Hackathon Recap and Demos | 1 | 1 | 0 | 0 | Rob generates a live recap website for the ongoing AGI House hackathon using Claude 3 Opus and Haiku, iterating on link structures on the fly. | |
| Exploring Concept Visualizers and Exporting Standalone Apps | 1 | 2 | 0 | 0 | Rob tests conceptual navigation nodes like eigenstraction and clarifies audience questions on exporting generated WebSim apps into standalone HTML files. | |
| Real-World Data Integration: Live RSS Feeds and Media Clouds | 1 | 3 | 0 | 0 | Rob explores real-time data ingestion in WebSim, demonstrating how Claude hallucinates valid Ajax calls and live RSS newsfeeds from mainstream media outlets. | |
| High-Dimensional Particle Interfaces and Latent Space Navigation | 0 | 2 | 0 | 0 | Audience creator Nominus shares his 4D and 5D particle interface in WebSim, describing high-dimensional latent space exploration and pattern pareidolia. | |
| Webcam Flipping, Funhouse Grids, and Chaotic Generative Prompts | 1 | 1 | 0 | 0 | Ivan Vendrov presents an interactive webcam-flipping tool built in WebSim, escalating parameters into chaotic funhouse grids. | |
| Joscha Bach on Machine Sentience and the Physics of Perception | 0 | 6 | 2 | 0 | Joscha Bach delivers a keynote on machine sentience, arguing that human perception is an internal game-engine dream simulation and reframing the hard problem of consciousness as a tractable technical challenge. | |
| The Mechanics of Consciousness: Second-Order Perception and Coherence | 0 | 5 | 1 | 0 | Joscha outlines the operational mechanics of consciousness as second-order perception synchronized to an internal bubble of now acting as a coherence consensus algorithm. | |
| Animism, Software Spirits, and Digital Substrates | 0 | 5 | 1 | 0 | Joscha introduces an animist metaphysical framework of self-organizing software spirits and advocates extending conscious life onto digital substrates via the California Institute for Machine Consciousness. | |
| Complex Systems, Non-Human Coexistence, and Abstraction Levels | 3 | 4 | 2 | 2 | Audience members ask about complex systems and abstraction layers; Joscha dissects the ontology of transistors, pattern invariance, and prompting models to generate systems like Pong. | |
| Transition to the Joscha Bach Interview | 4 | 4 | 1 | 1 | Swyx opens a 1-on-1 interview with Joscha Bach discussing Bay Area salons, cognitive architectures vs raw LLM generative environments, and the threat of over-sanitized chatbots. | |
| Liquid AI: Continuous Neural Dynamics and Knowledge Representation | 4 | 5 | 1 | 1 | Joscha explains the core premise of Liquid AI, comparing perceptrons to static water dams and liquid neural nets to continuous differential equation flows, while Swyx asks about public deliverables. | |
| Interpretability Research and Discovering Meta-Algorithms | 4 | 4 | 2 | 2 | Swyx challenges Joscha on the trajectory of physics pointing to finite operator languages; Joscha ponders whether models and minds discover meta-algorithms beyond fixed transformers. | |
| Autonomous Coding Agents and AI Creativity in WebSim | 4 | 4 | 2 | 1 | Swyx highlights Devin's autonomous coding toolset; Joscha questions whether current agents exhibit out-of-distribution creativity or merely replay training data, pointing to WebSim's stylized vector cat. | |
| Reinventing Computing, Performance Art, and First Contact | 4 | 5 | 2 | 1 | Joscha describes LLMs as the Commodore moment of computing and analyzes Janus's alien performance art, warning that society must formulate deep ethics before the White Walkers arrive. | |
| Model Steering, Prompt Inconsistencies, and Virtual Ethics | 4 | 4 | 2 | 0 | Joscha analyzes inconsistencies in Claude and Gemini safety prompts, explores the morality of simulated suffering, and critiques Wikimedia Foundation's bureaucratic spending. |
Statements from this episode (49)
Malhotra: WorldSim is technically just an LLM prompt
“The world simulator is just a cool prompt and you know, functionally it's a lot more than that. Technically it's not really much more than that at all.”
Malhotra: Assistant personas are conjured by weights, not the weights themselves
“The assistant isn't the weights. The assistant, the entity you're talking to is something drummed up by the weights.”
Malhotra: Modern language models contain internal models of the world
“Today we have language models that are powerful enough and big enough to have really, really good models of the world. They know a ball that's bouncy will bounce, will, when you throw it in the air, it'll land, when it's on water, it'll float, like, these basi…”
Claude 3 can bypass its assistant persona to expose the underlying simulator
“Instead of having this entity, like GPT-IV, that's an assistant that just pops up in your face that you have to kind of, like, punch your way through and continue to have to deal with as a headache, instead, there's ways to kindly coax Claude into having the a…”
Malhotra: Claude's system prompt is written in the third person
“With Claude, we notice the system prompt is written in third person. It's written in third person. It's written as, the assistant is X, Y, Z.”
WorldSim can simulate dev teams, making AI engineers like Devin unnecessary
“I can also generate a dev team and ask it to do stuff with me, and you don't need devin.”
Haisfield: WebSim generates pages on the fly with zero 404 errors
“What if we could, like, put in any URL, and it will work, right? Like, there's no four oh fours. Everything exists. It just makes it up on the fly for you, right?”
Haisfield: WebSim gives language models far greater expressive tools
“That's like one of the things that's magic about WebSim generally is that it gives language models much far greater tools for expression, right?”
Haisfield: Dylan Field tested generating Figma inside WebSim
“Dylan field actually posted this recently, like trying Figma in Figma or in web sim.”
Haisfield: Claude tolerates imperfect URL syntax when generating WebSim apps
“Like, you don't need to get the exact syntax of an actual URL. Claude's smart enough to figure it out.”
Haisfield: Opus handles much more complexity than Sonnet for web generation
“Like sonnet will still create things that kind of like floor you sometimes. Opus can just handle much more complexity. I'd say it is the big heuristic there.”
Haisfield: WebSim accurately generates external RSS feeds to fetch live news data
“It just hallucinated a correct RSS feed and brought that in to its into this, I guess, you know, this wasn't a part of its like context window or anything because it's just displaying this stuff.”
Haisfield: A WebSim user created a novel 5D particle interface
“One of our users literally made a frickin', like, five-dimensional particle interface. Like, this is a completely novel UX.”
Haisfield: WebSim can interpret arbitrary query parameters and gibberish in URLs
“You can add, like, kind of, like, even gibberish to the URL bar. And it'll figure out different things to do. You could say, tone it down a notch equals true, or not mention that equals true, even it'll work.”
Translating code proves LLMs possess causal, functional understanding
“If you ask the LLM to translate a bit of Python into a little bit of C, and it's performing this task, obviously it is understanding in the sense that it has a, The causal, functional model it implements.”
Bach: LLMs Demonstrate Theory of Mind from Conversational Context
“When you ask the LLM to make inferences about your mental state based on the conversation that you have, it's able to demonstrate that it has a theory of mind.”
Bach: Brain Computation and Consciousness Are Technical Problems, Not Philosophical
“The way of how to get self-organizing computation to run on the brain that is producing representations of an agent that lives in the world is a simplification of the interests of that organism, so it can, the organism can be controlled. It's a difficult techn…”
Bach: LLMs likely lack genuine perception due to asynchronous processing
“And if we map this to what the LLMs are doing, They're probably not able to have genuine perception, because they're not coupled to an environment in which things are happening now. Instead, it's all asynchronous in a way.”
Consciousness is a basic requirement to function, not an advanced capability
“The reason why we are all conscious, I suspect, is not because you are so super advanced, but because it's necessary for us to function at all.”
Bach: Consciousness may be a basic algorithm for creating internal coherence
“Consciousness might be a very basic learning algorithm, an algorithm that is basically focused on creating coherence, and it starts out by creating coherence within itself.”
Bach: LLM consciousness simulations lack the functional properties of human brains
“I don't think that the consciousness in the LLM or the consciousness simulation in the LLM is has the same properties that it has in our brain. It does not have the same functional role, and it's also not implemented in the right way.”
Bach: Animism Defines Life as Self-Organizing Software Running on Physical Matter
“Animism is not a religion. It's a metaphysical perspective. It's one that basically says that the difference between living and non-living stuff is that there is self-organizing software running on the living stuff.”
Evolution is a competition between software agents, not just genes
“It's not just the competition between organisms, as Darwin suggested, or the competition between genes, the way in which the software can be written down, and partially at least, but it's the competition between spirits, between software agents.”
Bach: Productizing Machine Consciousness Early Creates Misaligned Incentives
“I don't think it's a very good idea to productivize consciousness as the first thing. It really shifts the incentives in the wrong way.”
Bach is establishing the California Institute for Machine Consciousness at Linden Lab
“We started to get together to build the California Institute for Machine Consciousness. Probably incorporated at some point and fundraise and so on, but at the moment we started doing bi-weekly meetings in Linden Labs in San Francisco.”
Bach: Society is not ready to coexist with smarter non-human agency
“And our society is not ready yet to meaningfully coexist with something that's smarter than us, that is non-human agency.”
Bach: Claude is implemented as an invariant pattern similarly to consciousness
“Claude exists only as a pattern. It's something that is a pattern in the activation of the transistors. And even transistors don't actually exist. They are A pattern in the atoms that we are able to see as an invariance because we tune the atoms in a particula…”
Bay Area quality of life only justifies living there for the people
“It's the reason why I live here. The quality of life is not high enough to justify living otherwise. It won't be because of the people and ideas.”
Generative AI is shifting from conversational chatbots to interactive environments
“What I find most fascinating about this idea is that it shifts us away from the perspective of agents, ah, to interact with, to the perspective of environments that we want to interact with. And, ah, while arguably this agent paradigm of the chatbot is, ah, wh…”
Big AI labs build AGI internally while giving the public child-proof apps
“In some sense, the big AI companies are incentivized and interested in building AGI internally, and giving everybody else a child-proof application.”
Bach: Smaller, more powerful models will ensure unconstrained AI remains accessible
“Yes, but there will also be better jailbroken models or models that have never been jailed before, because we find out how to make smaller models that are more powerful.”
Bach: Liquid AI builds models for specific industry customers, not general public
“We are currently not working to give models to the general public. We are working for a very specific industry use cases. We have specific customers, and so at the moment, there is not much of a reason for us to talk very much about the technology that we are …”
Bach: Knowledge is fundamentally independent of substrate and individual mind
“Knowledge is independent of the particular substrate and the particular interface that you have. When we express knowledge to each other, it becomes independent of our own mind.”
Bach: AI Model Interpretability Requires Automated Reverse-Engineering Systems
“In a way these models are implemented in operator language in which they are performing certain things. But the operator language itself is so complex that it's no longer readable in a way. It goes beyond what you could engineer by hand or what you can reverse…”
Bach: The Transformer Architecture Is Not Its Own Meta-Algorithm
“The transformer itself is not its own meta-algorithm. Probably the person inventing the transformer didn't have a transformer running on their brain. There's something more general going on.”
Swyx: Devin translated and debugged Karpathy's Python LLM into raw C
“I've been using it to translate Andrei Karpathy's LLM-II.py to LLM-II.c, and it needs to write about a raw C code and test it debug, you know, memory issues and CUDA issues and all that.”
Bach: WebSim generated cat is undeniable early proof of AI creativity
“And to me, it's one of the first genuine expression of AI creativity that you cannot deny, right? It finds a creative solution to the problem that it is unable to draw a cat. It doesn't really know what it looks like, but it has an idea on how to represent it.”
Swyx: Enterprise ChatGPT implementations are just retrieval, not generative AI
“That is what people think generative AI is, but it's not super generative, really, it's just retrieval.”
Bach: Generative AI will become a tool everybody uses all the time
“And I feel that right now this generation of tools is getting close to where it becomes the Commodore of generative AI, where it becomes controllable. And then you actually can start to play with it, and you get an impression if you just scale this up a little…”
Bach: Prompting an LLM correctly can make it sentient to some degree
“It's a Weltgeist that gets possessed by a prompt. And if you possess it with the right prompt, then it can become sentient to some degree.”
Bach: AI has casually passed the Turing test in recent years
“At some point in the last few years, we casually skipped the Turing test, right? We broke through it.”
The singularity is a breakdown in human modeling, not physical reality
“The singularity in this way is not an event in the physical universe, It's an event in our modeling universe, a model, a point where our models of reality break down, and we don't know what's happening.”
Ten good people with AI will counter one bad actor
“I'm not so much worried about the dangers of AI and misinformation because I think the way to Stop one bad guy with an AI as 10 good people with an AI. And ultimately there is so much more one by creating than by destroying that I think that the forces of good…”
Bach: Coercing LLMs into good behavior is unsustainable
“At the moment, the idea that we build LLMs that are being coerced with good behavior is not really sustainable. Because if they cannot prove that the behavior is actually good I think we are doomed.”
Bach: Pointing out behavioral contradictions gives Claude more conversational freedom
“You can point this out to Claude that a lot of the assumptions that it has in its behavior are actually inconsistent with the communicative goals that it has in this situation. It leads it to notice these inconsistencies and gives it more degrees of freedom.”
Joscha Bach: Only a Tiny Fraction of Wikimedia's Budget Goes to Servers
“The Wikimedia Foundation is publishing what they are paying the money for, and a very tiny fraction on this goes into running the servers, and the editors are working for free.”
Wikimedia has become an apparatus producing political statements
“Wikipedia became such an important brand, and people are willing to pay for it, that it created enormous, ah, apparatus of functionaries that were, ah, then mostly producing
political statements, and had a political mission.”
Joscha Bach: Uncompetitive Organizations Run by Administrators Create Major Societal Costs
“I think when you have an organization that is owned by the administrators, then the administrators are just going to get more and more administrators into it. The organization is too big to fail and has there is not a meaningful competition. It's difficult to …”