clef - a musical MCP - 8.14.26

I've spent the past few weeks working on a new project of mine I'm calling clef.

I've been looking into the Model Context Protocol recently and thought it would be cool to implement an MCP server of my own as a way to learn a little more.

The idea is that the server wraps a custom audio library that indexes local audio files and provides playback capability. The server's endpoints would then allow an agent to see your local music, make playlists based on user requests, and play songs accordingly. All of this would be built in Rust.

I'm trying to go alongside the principle of three innovation tokens for each project. In other words, I wanted to limit myself to 3 new things to learn. In this case, those would be building a larger scale project with Rust, the MCP spec, and developing a coding agent workflow that actually helps instead of just oneshotting and calling it a day.

What was originally a weekend project spun into something much (much) more indepth and time consuming, but I think I've learned a lot. I want to use this post to document that.

ideation vs building

The bulk of the work on this project ended up being clef-core, the main audio core of the project. I wanted to own control over the queueing and playback controls, so I decided to wrap rodio, a Rust audio decoding library, with my own queue and playback logic. I also wanted to keep everything synchronous, mirroring rodio.

Seems easy enough, right?

This is when I fell into the system design rabbit hole. When dealing with systems with an increased level of complexity, ie handling multiple control, handler, and audio threads, the process of evaluating tradeoffs and formulating system designs can be all-consuming. Here are just a few of the manic scribblings I spit out over 2 days, likely scaring everyone in the immediate surroundings of my coffee shop.

In the end, here's a brief description of how my system works.

Rodio encodes and plays audio on a struct called a MixerDeviceSink, which contains an audio stream from cpal, the base audio encoder. This mixer has playback controls and can hold a queue of raw audio streams, but here's the kicker: those audio streams have no information outside of the raw audio sample data. This gave me two options: either keep a seperate queue with metadata synced with rodio's audio queue (bad) or keep my queue the source of truth and update the mixer accordingly (better?).

This queue would still need to receive updates from the mixer for auto advancement upon a track ending, which led me to organizing a message handling system.

The track player abstraction is made of two main structs: Player and PlayerMessageHandler. The Player struct is responsible for storing the sink reference, which can't be sent across threads, and handling outside playback commands. It creates a channel with a sender and receiver, and instantiates the PlayerMessageHandler. The PlayerMessageHandler loops in a thread spawned by Player, and is responsible for taking the messages received from the Player sender and updating the rodio audio backend. This receiver also gets messages sent through a callback in the rodio mixer when tracks end, signaling the next track to start.


flowchart TD
   subgraph Caller["caller thread"]
     P["Player"]
   end
   subgraph Policy["handler thread"]
     H["PlayerMessageHandler"]
     R["rodio::Player"]
   end
   subgraph Audio["audio thread"]
     S["Sink: cpal::Stream"]
   end
   P -->|player command| H
   H -->|playback function| R
   R -->|audio samples| S
   S -->|sink error| H
   R -->|track finished| H

Got it?

As I said, this project really showed me how easy it can be to fall into a loop of iterating, thinking, and re-thinking about the layout of your system. At a certain point, you just have to start getting text in an editor and see if it works, rather than trying to optimize what's not even there yet.

vertical slices really do matter

It's crazy how many times you can hear an age-old adage and still forget to put it into practice. In my case, I conveniently forgot how important vertical slices are for personal motivation.

After a few days working on the intricacies of my audio core, I kind of got sick of the project. I did all this work, and still didn't have any of the main, user facing functionality of the system I wanted to build. Only after realizing this did I start on building the TUI for the project, just to see some resemblance of my desired experience. Even just this simple action of making a text interface that controlled the core revitalized the way I felt about the thing I was making. Feedback is important, and if you don't get material results quickly you'll lose motivation.

ai workflows are weird

I've found that using AI in any capacity for coding is an incredibly slippery slope. Again, if I really wanted to, I probably could have vibeslopped this project together in a day and come up with an equally good solution. But what's the point of that? I do this because I enjoy it. If it wasn't my job, I'd still be doing it. Programming (typing into my computer!) is fun.

Even so, there are cases where having an agent is incredibly helpful. If you take a look at the AGENTS.md files in my different directories you can see some of the methodologies I try to follow when working with an agent:

  1. Don't make any changes without asking

    this one should be obvious

  2. Point me towards concrete sources and references, summarize rather than recommend

    agents are excellent search engines, use them to explore a problem space and find sources

  3. When something has a bug, point me towards it and explain the issue rather than fixing directly

    bug fixing is annoying, but it's still part of learning

I'm still working on my best practices for working with clankers, but I think I'm slowly getting to a more stable workflow. Who knows though.

Concluding

I'm really proud of what I have so far. There's still a ton more work I want to do on the usability of the TUI, the context efficiency of the MCP, and the stability of the audio player, but this is one of the most comprehensive and quality projects I've worked on. I wanted to document some of my thoughts while working, and will probably have more updates in the future.

For now though, I gotta start leetcode grinding. damn it

more to come.