{
  "turns": [
    {
      "role": "user",
      "content": "<context timezone=\"America/Chicago\" />\n<message id=\"536\" from=\"telegram-mg-17796\" sender=\"James Lopez\" time=\"Jun 2, 2026, 11:59 AM\">Saw a post on IG with a video showing off driving a computer with only voice and somehow the tool understood context and switching between apps. I know you can’t see the video but here’s the text content from the description of that post. Maybe this is sufficient to look up the info and find out how we can enable this and is it free or cheap? \n\nContent:\n11:56\n85\nG/RSTA\nGirsta\nAI NEWS\nSIGNAL: Developers are using the GPT-Realtime-2 model to build voice-controlled operating systems\n32620\n00C\n⑨\nUh well, I'm free on the weekend\n[))\nG/RSTA\ngirsta\nFollow\nDevelopers are combining OpenAl's GPT-Realtime-2 with new open-source components to control computer interfaces entirely through voice. This shifts human interaction from physical clicking to direct spoken commands.\nThe model's parallel tool calling, high reasoning capability, and low latency allow it to read application states and execute actions instantly. Users can dictate workflow adjustments, like changing market charts or navigating visual</message>"
    },
    {
      "role": "assistant",
      "content": "<message to=\"telegram-mg-17796\">Hey Jay! \n\nThat account (Girsta) is a popular AI news curator. The video you saw is showcasing developers combining **OpenAI's GPT-4o Realtime API** (which has ultra-low latency streaming voice capabilities) with desktop automation libraries (like `PyAutoGUI` or accessibility APIs) to read the screen and click/type based on voice commands.\n\nHere is the breakdown of how this works, the costs, and how you can try it:\n\n### How it works under the hood\n1. **The Voice/Audio Layer:** The Realtime API handles streaming audio input and output directly, giving that fast, natural conversation feel without waiting for a full transcript.\n2. **The Sight Layer:** A background script takes periodic screenshots (or hooks into your OS's accessibility tree) to give the AI visual context of what app you are looking at.\n3. **The Action Layer:** The developer exposes \"tools\" to the AI (like `click_at(x, y)`, `type_text(string)`, or `switch_app(name)`), which the AI calls dynamically based on your spoken commands.\n\n---\n\n### Is it free or cheap?\n\n**The short answer:** The software/code is free and open-source, but running the API is **not cheap**.\n\n* **The Code (Free):** The open-source frameworks to run this on your machine are free.\n* **The OpenAI Realtime API (Expensive):** OpenAI charges for the Realtime API by the token for both audio input and output. Because it maintains a constant, live audio stream, active usage is notoriously expensive—typically costing **$5 to $15+ per hour** of active continuous use. It is definitely not something you'd want to leave running in the background all day.\n\n---\n\n### How to get or build this on your Mac\n\nIf you want to play with this, there are a few ways to go about it:\n\n#### 1. The Polished Route: Open Interpreter (01 Light / 01 OS)\nThe most mature open-source project doing exactly this is **Open Interpreter** (specifically their **01 Project**). It is a voice-first natural language interface for your computer.\n* **How to get it:** It's free and open-source. You run it via terminal, configure it with your OpenAI API key, and it can control your Mac's screen, apps, and settings via voice.\n* **Cost:** Free software, but you pay OpenAI API costs.\n\n#### 2. The Low-Cost \"Pipelined\" Hack (Way cheaper)\nIf you want a cheap version, we don't use the premium Realtime API. Instead, we use a \"pipelined\" stack:\n1. **Local Voice-to-Text:** Use a tiny, free local Whisper model (like Whisper.cpp) on your Mac to transcribe your voice when you hold a hotkey.\n2. **Cheap LLM:** Send that text to a cheap, fast model like **Gemini 1.5 Flash** or **Claude 3.5 Haiku** alongside a screenshot of your active window.\n3. **Local Automation:** Have the model output the Python commands to switch apps or click elements.\n* **Cost:** Practically free (pennies per day) because we aren't paying for live, streaming audio tokens. The trade-off is a 1-to-2 second delay instead of the instant response of the Realtime API.\n\nLet me know if you want to experiment with setting up **Open Interpreter** on your machine or if you'd like to sketch out a lightweight, cheap custom script to do basic voice-app switching!\n\n— Nano</message>"
    }
  ]
}