GrayHaze - Live content on the ATmosphere

GrayHaze | GitHub | My Blog | Me

Introduction

I've been on a 10 day long programming bender working on my most recent project, I know burnout is inevitable if I keep going at this rate, so I figured I'll take a break... by writing a blog post about the development of it. 😅

GrayHaze is a live streaming service built atop ATProto. It's the protocol that powers BlueSky (where you probably came from), it's the protocol that powers this blog posting service, and now it's the protocol that powers GrayHaze. The key point of ATProto is that you decide who hosts your content, and you have ability to change your mind whenever you want. It's quite difficult to do this at the moment, but the fact that it's possible at all even with some amount of trouble is really cool.

Overview

I'm operating on the assumption that you're aware of how live streaming works, or are at least familiar with services like Twitch or YouTube Live, I found out yesterday that not everyone knows what "live streaming" is! Fundamentally GrayHaze is the same, you go live, people can pull up your channel and watch the video, or the archived VOD of it if they miss the stream.

My GrayHaze channel

What makes it the same as other platforms isn't interesting though, let's talk about what makes it different. For one, while I host the frontend and AppView, all the media, the live stream, the chat messages you send, the titles, the thumbnails, those are all on your PDS, and relayed through to the end user. You own your content. This is a major shift from other streaming platforms, where unless you're also recording the broadcast, the VOD is entirely in the hands of the site you're streaming on. This isn't the most impressive as it comes about due to the nature of ATProto, but as far as I can tell GrayHaze is the first to do it with live content.

Screenshot of pdsls.dev. The list of chat records from my friend. She makes a good suggestion in this highlighted record!

Intricacies

Sending the Stream

GrayHaze is a complicated beast, but at its heart is Blobs. If you're not familiar with ATProto terminology, that sounds hilarious, and if you are, it probably still sounds hilarious, but it's the truth! Live content is just a bunch of blobs. Blobs are media that doesn't fit the convention of a record (text, json, that kind of stuff). Photos, video, ZIP archives, a linux ISO anything that has a binary format probably belongs as a blob. That blob then has to be attached to a record of some kind, otherwise it expires.

Bearing all of that in mind, let's talk about HLS. HTTP Live Streaming (HLS) is a protocol developed by Apple in 2009. It consists of 2 components; chunks of video called segments usually about 1-10 seconds long, and a playlist file containing some metadata and a link to some (or all) of the segments. I hope you picked up on this but HLS and ATProto live streaming were a match made in heaven. All we have to do is watch the playlist for new segments, upload them as blobs, and then create or update a record with all of the playlist metadata.

Like magic, in just 2 paragraphs we have come up with Blobify, my bodge of a solution for getting the live content off of my computer and onto my PDS. For the moment it works really well, but it's not the most user friendly solution in the world and I would like to provide a way for the blob upload and playlist record management to be handled by the AppView in a stable & safe manner.

Receiving the Stream

Now that we've gotten the content to the PDS we have to consider how we're going to get it from the PDS and to users eyes. If we did everything right, then the record should easily be able to be reversed into a HLS playlist (m3u8 format):

Another screenshot of pdsls.dev. The start of an hls playlist, with the first 3 BlobRef segments

We store 4 things from the HLS playlist file, and that is enough to recreate it accurately:

1. The version number - just for consistency's sake, GrayHaze doesn't really do anything with it but it's probably helpful for the browser when rendering the video.
2. The media sequence number - HLS playlists can be incomplete, this number represents the amount of segments that have been discarded since the first segment. Because blobify goes out of its way to retain the full list of segments, this is always zero, but another 'blobify-like' client could come along, so the option is open for said client to not retain it. In that case they'd have to increment that number for each discarded segment.
3. The end boolean - Whether or not this stream has ended. This actually stands in for 2 values in the playlist metadata, the first declares whether the playlist is a live "event", or a past "VOD", and the second is a value that's tacked onto the end of the playlist to declare that there are no more segments coming.
4. Finally, the sequence array - A list of blobrefs, and their duration in microseconds. HLS uses seconds but they're floating point values that appear to be precise to the microsecond. Records don't accept floating point values for reasons, so we multiply the value before putting it on the record.

On top of all of this we add our own metadata to a different record that references this HLS record, a title, thumbnail, and tags. That is what we present to the end user.

Chatting

Fundamentally sending a chat message is easy, it's just a matter of creating a record. But what about the process of delivering that record (that is on your PDS repository), out to other viewers of the stream? To answer this I looked at how BlueSky does this, and the answer was a solution similar to the firehose, a stream of every record creation/modification/deletion that occurs on a PDS. The firehose is an ATProto lexicon type called a "subscription". As an aside, "records" are also a primary lexicon type so you've already been familiarized with lexicons if you weren't previously aware.

This "subscription" record type is perfect for a stream chat because it uses a WebSocket. WebSocket connections function as pipes to clients, anything you send through it gets to them without them needing to make a request to get that information. Most chat systems you're familiar with (Discord, Twitch/Youtube live chat) use WebSockets to relay this information out. In our situation we have to create a special subscription that takes in what you're watching and filter chat records out from the fire hose based on that. Fortunately, since I've been using typescript this whole project there exists the @atproto collection of packages, one of them being xrpc-server which bundles helper functions to create a subscription, and functions that interface with the fire hose.

Screenshot from my most recent VOD wherein chat was rambling (note: grayhaze does not steal passwords, it uses session tokens that are validated and can be revoked by your PDS)

Additionally you can see I quickly scrapped together a ban functionality, which also gets brought in through the subscription. However instead of placing a message in chat, all clients filter through and delete messages that that user has sent, and the firehose behind the chat subscription does not notify the clients of any further records from the chatter.

Conclusion

I held my first major test stream of the platform yesterday, and while overall I would consider it a success it had some major issues that I will be addressing in the future.

First off, the stream crashed due to an unknown maximum record size, meaning that blobify wasn't able to attach the blobs it was making to the hls playlist. That's an easy fix, another record will be made and linked to the first when the sequence list reaches a certain size. It was quite funny that this wasn't something I considered, I'll have to be more careful of it in the future.

The next issue is one that's not technical, but going to take a bit more thinking about. The bandwidth used was quite high, in the 100s of Megabit per second, just for one stream with 3 or so viewers. If I am going to make GrayHaze the platform I want it to be I'm going to need tens of hundreds of Gigabit per second, and presumably way more than the 3 or so servers I'm working with. That boils down to money, money I don't have. I'm not going to get on a soap box and preach about how my budding platform is the future of live streaming and ask for you to reach into your wallet and donate. Rather, I think it would be more effective for me to spend some time putting together a pitch for development grants, and maybe if I really see a potential with this project, move forward and accept seed funding. I understand this as being risky though, I'm not a business guy, I just like writing code that makes people happy. With that, I really worry about slipping up and accidentally losing the project resulting in it falling in the hands of some greedy investor or something. Maybe that's irrational since I'm already well aware that it's something that happens and will already be rallying against it, but I would hate to be wrong.

Moreover, other platforms like Twitch, Youtube Live, Rumble, Kick, etc. are all operated by companies that already had money from other sources, and are operating the streaming services at a loss. Live streaming services are more than an unsolved problem of profitability, they're an unsolved problem of breaking even. I don't have a solution right now and realistically it's not something I should be concerned about until I have something that looks and feels more than like someone made it in their garage. But that most recent stream made me acutely aware that if I want to move forward with the project I'm going to need an income/investment.

Anyways, I've spent pretty much the whole day pondering and writing out this article, I'm going to get back to work on the project. I have so much more that I want to do with it, between making moderation much more than just a ban button, emotes and emote collections, profiles and username coloring, I've got my work cut out for me! As always, thanks for reading!

~ hugeblank