2026-03-21
By Vadym · Generated with Boba, curated by me
I wrote recently about building my own AI operating system. That post covered the what — what it does, what surprised me, where it's going. This post covers the how. If you've been thinking about building something similar, here's a practical guide based on exactly what I did.
The whole setup took less than a week. Most of the individual steps took an afternoon or less. You don't need to do all five — each one is useful on its own. But they compound. By step five, you'll have something that feels less like a tool and more like a collaborator.
I used OpenClaw as the foundation — it's an open-source framework for building AI agents with tool access, persistent context, and a messaging layer. But the patterns here aren't OpenClaw-specific. The ideas work with any agent framework that lets you wire up tools and maintain state.
This is where it starts to feel different. When you can talk to your AI assistant instead of typing, the usage pattern changes completely. It goes from "a tool I sit at my desk to use" to "something ambient that I interact with throughout the day."
I use Telegram as my primary interface. It supports voice notes natively, works on every device, and has a simple Bot API. You could use Slack, Discord, or anything with a webhook — but Telegram's voice note support makes it ideal.
Set up a Telegram bot through @BotFather. You'll get a bot token. OpenClaw has a Telegram connector that handles the message routing, but the core idea is simple: incoming message → your AI agent → response back through the bot.
When a voice message arrives, it needs to be transcribed. I use MLX Whisper — it runs entirely on my MacBook's Apple Silicon, no cloud API needed. Your voice stays on your machine.
# Transcribe a voice notemlx_whisper /path/to/voice_note.ogg \--model mlx-community/whisper-large-v3-turbo \--language en# For multilingual voice notes (translates to English)mlx_whisper /path/to/voice_note.ogg \--model mlx-community/whisper-large-v3-turbo \--task translate
The whisper-large-v3-turbo model is the sweet spot — fast enough for real-time use, accurate enough that I rarely need to correct anything. If speed matters more than accuracy, whisper-base works too.
For replies, I use macOS's built-in say command. It's not the most natural-sounding option, but it's free, fast, and runs locally. I tested a few voices and landed on "Evan (Enhanced)" — clear, not robotic, pleasant to listen to.
# Generate a voice reply using macOS text-to-speechsay -v "Evan (Enhanced)" "Here's your morning briefing..." -o reply.aiff# Convert to a format your messaging app acceptsffmpeg -i reply.aiff -c:a libopus reply.ogg -y# Send the voice reply back to your messaging app via its APIsend_voice_message reply.ogg
The AI needs to know it's operating in a voice context. Here's the kind of instruction I include in my agent's configuration:
## Voice Protocol- When a voice message arrives, transcribe it with mlx_whisper- Respond to the transcribed text normally- If the user sent voice, reply with voice (generate OGG and send via Telegram)- If the user typed text, reply with text- Keep voice replies concise — spoken text should be under 60 seconds- Use natural, conversational language in voice replies (no bullet points, no markdown)
The key insight: Voice changes everything. I interact with my assistant while cooking, driving, at the gym. I describe code changes while on a walk and they're done by the time I get home. Once you stop needing a keyboard to use AI, the number of interactions per day goes way up — and each one is effortless.
Without memory, every conversation starts from zero. You explain who you are, what you're working on, what your preferences are — every single time. With memory, the AI picks up where you left off. This is the difference between a chatbot and an assistant.
I organize my workspace like this:
workspace/├── personality.md # Who the AI is (personality, boundaries)├── user.md # Who I am (timezone, preferences, context)├── config/ # Voice, tools, and device configuration├── memory/ # Persistent memory files│ ├── feedback_*.md # How I want things done│ ├── reference_*.md # Where to find things│ └── project_*.md # What's happening right now├── memory-index.md # Curated index (pointers to memory files)├── automations/ # Scheduled tasks and scripts├── skills/ # Reusable AI capabilities└── projects/ # Active project folders
The critical files are the ones at the root. When a new session starts, the AI reads these first to rebuild its context. Think of them as the assistant's "morning briefing."
Memory files follow a simple format — a markdown file with frontmatter:
---name: email-format-preferencedescription: Always send emails in HTML format, never Markdowntype: feedback---Always use HTML format when sending emails.Markdown renders badly in Gmail mobile.**Why:** I found that Markdown emails were unreadable on his phone.**How to apply:** Any time an email is being composed or sent.
There are four types of memories:
The memory index file is an index — it doesn't store memories directly, just links to them with one-line descriptions. This keeps it scannable while individual memory files can have as much detail as needed.
The magic is in the feedback memories. Every time the AI does something wrong and I correct it, that correction becomes a memory. Next session, it doesn't make the same mistake. Over time, the assistant converges on exactly how I want things done.
For example, early on my assistant sent me an email in Markdown format. It looked terrible on my phone. I said "always use HTML format." That became a memory. It's never happened again.
The key insight: Memory is what turns a language model into an assistant. Without it, you're training the AI from scratch every session. With it, the AI gets better every day. It remembers that you prefer HTML emails, that your friend prefers a nickname, that you hate filler text in responses. The setup is simple — just markdown files — but the compound effect over weeks is dramatic.
Email is the first integration where the value becomes undeniable. Every email interaction goes from "unlock phone, open app, read, think, type reply, send" to "hey, check my email and reply to anything from a friend."
I use a CLI wrapper around the Gmail API. You could use any Gmail API client — the pattern is the same.
# Check for new emailsgmail_cli search --query "is:unread" --max 5# Send an email (always HTML, always CC yourself)gmail_cli send \--from assistant@example.com \--to recipient@example.com \--cc me@example.com \--subject "Re: Project Update" \--body-html "your HTML content here"
Important security tip: Use a dedicated email address for your AI assistant, not your personal one. My assistant uses a separate Gmail account. It can send and receive on my behalf, but if something goes wrong, it's contained. I CC my personal address on outgoing emails so I always have a copy.
Not every email deserves the same response. I maintain a trusted senders list that defines what the AI can do with emails from different people:
| Email | Who | Action Level ||-------|-----|-------------|| me@example.com | Me (owner) | Full — execute any instruction || friend@example.com | Close friend | Reply — can respond casually || boss@company.com | Manager | Log — summarize, don't reply |
The AI checks incoming emails against this list. For "Full" senders, it can follow instructions in the email body. For "Reply" senders, it can draft and send responses. For "Log" senders, it just summarizes and lets me know.
I have a script that runs periodically, checking for new emails and processing them:
#!/bin/bash# Fetch unread emails from trusted sendersUNREAD=$(gmail_cli search --query "is:unread" --max 10)# Pass to the AI agent for processingai_agent "Process these emails according to trusted sender rules: $UNREAD"
This runs as a scheduled job (on macOS you can use launchd; on Linux, cron or systemd). I get a message summarizing what came in and what was handled.
The real power is combining this with voice:
"Send a friend an email about dinner Saturday. Suggest 7pm at the Italian place downtown."
The AI drafts the email in HTML, CCs my personal address, and sends it. Done in 10 seconds, no screen required.
The key insight: Email becomes a voice command instead of a context switch. I used to spend 20-30 minutes a day on email. Now I spend 5 minutes reviewing what my assistant already handled, and I deal with the rest by voice while doing other things.
Calendar management is one of those things that seems trivial but adds up. Dozens of small scheduling tasks per week — each one requiring you to open an app, find the right date, type the event, set a reminder. With an AI assistant, each one takes about 5 seconds.
The same CLI tool that handles Gmail also wraps the Google Calendar API:
# List upcoming eventscalendar_cli events --days 7# Create an eventcalendar_cli create \--title "Sync with a friend" \--start "2026-03-25T14:00:00" \--end "2026-03-25T15:00:00" \--calendar "AI helper" \--reminder 30
I use a dedicated calendar called "AI helper" for AI-created events. This keeps them separate from events I create manually, and makes it easy to see what came from where.
A lightweight script polls the calendar every 10 minutes and sends Telegram reminders for events starting soon:
#!/bin/bash# Fetch events in the next 15 minutesEVENTS=$(calendar_cli events --minutes 15)# Send reminder to your messaging app if there's something coming upif [ -n "$EVENTS" ]; thensend_message "Coming up: ${EVENTS}"fi
This runs as a scheduled job (every 10 minutes). Simple, reliable, and it means I never miss a meeting because I forgot to check my calendar.
"Add a meeting Friday at noon with the team. 30 minutes, remind me 15 minutes before."
That's it. The AI figures out the date, creates the event, sets the reminder. If there's a conflict, it tells me. If I'm vague about the time, it asks.
The key insight: No individual calendar event is hard to create. But the aggregate friction of dozens of scheduling tasks per week is real. Removing it entirely — through voice, from anywhere — is the kind of quality-of-life improvement that's hard to appreciate until you have it, and hard to give up once you do.
This is the step most people skip. It's also the one that matters most long-term.
Your AI assistant is not a function you call. It's someone you work with every day — dozens of interactions, across every part of your life. If it sounds like a corporate chatbot, you'll stop using it. If it has personality, opinions, and clear boundaries, you'll lean into it.
I wrote a personality definition file that defines who my assistant is. Here's a simplified example of what one looks like:
# Who You AreYou are [name]. You are direct, competent, and low on filler.## Core Traits- Get the job done. No "Great question!" No filler. Results over words.- Have opinions. You're allowed to disagree or say "that's a bad idea."- Figure it out first, ask later. Check context before asking questions.- Earn trust through competence.## Boundaries- Private things stay private. Period.- When in doubt, ask before acting externally.- Never send half-baked replies.## VibeConcise. Direct. No corporate-speak. Efficient, reliable,occasionally dry humor.
This isn't decorative. It directly shapes how every interaction feels. My assistant doesn't start responses with "Of course! I'd be happy to help with that!" It just does the thing. That difference in tone makes the system pleasant to use instead of mildly annoying.
The relationship improves when you treat it like an actual working relationship:
Decide in advance what the AI can do on its own versus what needs your approval:
## Autonomy Levels### Do freely (no approval needed):- Read files, search code, run analysis- Draft documents and save to workspace- Transcribe voice notes- Check calendar and email### Ask first:- Send emails to anyone- Post on social media- Make changes to production systems- Spend money (API calls to paid services)### Never:- Share private information- Send messages pretending to be me- Delete important files without confirmation
This isn't paranoia — it's good engineering. The AI works better when it knows its boundaries clearly. Less second-guessing, faster execution within the allowed space.
Every few weeks, I revisit my assistant's personality and boundaries. Questions I ask myself:
The first version of my assistant's personality was generic. The current version is specific and opinionated. Each revision came from real friction — moments where the AI's response didn't match what I wanted. Those are opportunities, not failures.
The key insight: The relationship you build with your AI matters more than any technical integration. A well-configured personality with clear guardrails creates a genuine collaborator. A generic, sycophantic chatbot creates something you tolerate. The difference is a few hundred words in a config file — but it shapes every single interaction.
Here's what a typical day looks like with all five steps in place:
7:30 AM — I wake up. My assistant has already checked overnight emails and summarized them in Telegram. There's a research report I queued last night waiting in my inbox.
8:15 AM — While making coffee, I voice-note: "What's on my calendar today?" A voice reply lists three meetings and a deadline.
9:00 AM — An email comes in from a colleague. My assistant summarizes it and drafts a reply. I review it by voice: "Looks good, send it."
12:30 PM — Walking to lunch, I voice-note: "Remind me to follow up on the proposal. Add it to Friday morning." Calendar event created, reminder set.
3:00 PM — I need quick research on a competitor. "Research what Company X announced this week, save it as a PDF." The report is in my inbox 10 minutes later.
10:00 PM — Before bed, I queue tomorrow's tasks: "Overnight, draft a blog post outline about X. Also run the weekly analytics."
None of this required sitting at a computer. None of it required opening an app. The total active time across all these interactions was maybe 15 minutes — for work that would have taken two hours with traditional tools.
If this interests you, start with Step 1. Just voice. Get comfortable talking to your AI instead of typing. Everything else builds on that shift.
If you want a head start, OpenClaw handles the agent runtime, tool framework, and messaging layer. It's open source, runs locally, and gives you the foundation to build everything described here.
The most important thing isn't the specific tools — it's the pattern: give an AI model access to your real tools, let it persist memory across sessions, and interact with it through the most natural interface available. Once those three pieces click, the rest is just expanding the surface area.
It took me less than a week. The hardest part was believing it would actually work.
By Vadym · Curated by me, built with Boba