About SoB
Summer of Bitcoin (SoB) is a program intended for university students who want to get to know Bitcoin and contribute to open source projects in the Bitcoin space. It's pretty much like Google Summer of Code in terms of the admission process, but the difference is that it only lists Bitcoin related projects. You can check out the organizations and the projects for 2022 here.
And this year, I was accepted as a student contributor in SoB in talaia-labs organization, where I worked on a feature in The Eye of Satoshi (TEOS) watchtower.
What I was working on
The feature I worked on was adding an additional interface for the watchtower's public API. This new interface speaks Lightning. Lightning (besides being the name of the Lightning Network) is a transport protocol that lightning nodes understand. It's defined in the Basis of Lightning Technology (or BOLTs for short), specifically in BOLTs 1 and 8. It's a simple TCP connection with messages encoded in a specific way.
Since we have the draft BOLT13 for watchtowers that uses the Lightning transport protocol, It was obvious that we should add a Lightning interface for the watchtower. Using Lightningdevkit (LDK), which TEOS already is built on top of, we can add BOLT13 messages as custom messages for now and when further progress is made in BOLT13 and it gets merged, these messages should be no more custom.
LDK abstracts the hard bits about lightning and its messaging protocol in simple and easy to use constructs. It also supports adding custom messages to your application/node. LDK is a library that one can use to build a lightning node (like this one) or application, but not is a node itself. Being a library makes it very extensible and tailorable to each project's interest.
For the custom messages part, LDK exposes some Rust traits that you can import and implement for your types to inject your application specific custom messages logic inside LDK constructs.
Phase 1
For phase one of SoB, I defined the custom messages that both clients (lightning applications) and servers (watchtowers) will use, based on BOLT13 and with some help from the already implemented HTTP version of the public API. Also, implemented the proper Lightning (de)serialization for these messages based on BOLT1. These definitions and (de)serialization implementations can be found in teos-common crate (introduced in this PR), which is a Rust crate that contains the common parts that both the watchtower and clients might use.
The toughest problem I faced in this phase was dealing with TLV fields found in BOLT13. Encoding and decoding TLVs is quite tricky and involves the weird BigSize integers and other special reader types. Also, LDK doesn't help in (de)serializing TLVs in the custom messages as of now (although some TLV related structs were made public recently). But it has a private macro-based (de)serialization framework for Lightning messages (de)serialization (TLVs included). I first tried to make this serialization framework public in this PR, but the LDK team didn't have the same interest in exposing some specific macros as this might create greater responsibility upon them.
So I then made a mini macro-based TLV serialization framework (mostly taken from LDKs private macro framework with some tweaks) based on LDK serialization traits. The framework cuts a lot of boilerplate and allows for easy updates in the messages' fields in the future. I have learnt a lot while working on this mini framework and had to understand Rust's macro system, which turn out to be pretty powerful. The framework consisted of the parts we needed from LDKs framework plus other things I added to support (de)serialization for other types like Strings and Vec<T>s.
Phase 2
In phase 2, I worked on adding the Lightning interface (server) to the public API. I mainly followed LDK docs and sample applications/nodes made with LDK while working on this part (like ldk-sample).
I started with writing a custom message handler for TEOS that will receive BOLT13 custom messages, deserialize them, convert them to gRPC messages, and pass them to the tower's public gRPC API. Thus reusing the same tower interface that the HTTP API already uses. This means that there is no critical difference between the HTTP and the Lightning versions of the public API and that they can be used interchangeably by clients.
Then I created a basic lightning server using the lightning-net-tokio crate. This crate has methods to connect our PeerManager (which manages everything with peers/clients) to other Lightning peers and also manage, read from, write to, and terminate connections with them. The peer manager in the heart of the Lightning server doesn't understand Channel and Routing messages as of now. It replies with an error or ignores the message when it's a non-BOLT13 message. We might extend it later and support channel and routing messages to be able to receive watchtower subscription fees when TEOS lands support for non-altruistic watchtowers.
The hardest part in this phase, which I previously believed would be the simplest, was actually creating a Lightning server using lightning-net-tokio. I faced an issue similar to this one, which I had a very hard time dealing with since I was new to async and tokio at that time. I had about a week-long journey of trial and error trying to understand how tokio works and how threads affect each other. I made a little crate to simulate the same environment in TEOS and test every single combination I can use tokio with to accomplish the weird thing I needed (to call sync code which blocks on async code from inside async code :D). I was so proud when I finally found a solution, but not for so long, as the solution I found worked only in the test crate I worked on but not in TEOS. Turns out the environment I made wasn't that identical after all *. The solution I found was close to the final solution though. Being desperate, I revisited devrandom's issue in LDK and found that he had a similar issue in his repo as well, and also sensei. It was before my eyes all that time O_O. But learning how tokio works was worth it anyways.
* I simulated an async gRPC call with a tokio sleep, which aren't equivalent because an async gRPC call from gRPC client calls on another tokio thread (the gRPC server) and waits on it. But a tokio sleep simply doesn't.
Post SoB
My feature PRs haven't been merged yet, so I will stick around until we merge them successfully. Plus, since the next month is going to be some free time for me before college starts, I will have enough time to help in TEOS and other bitcoin projects in general.
My experience in SoB
I was so lucky to have the chance to work with my mentor, Sergi in talaia-labs. He is a very helpful and thoughtful person. He takes his time to explain things and is always ready to have discussions in search for the best approaches.
TEOS is a super interesting project. It's like blockchain detective that busts the bad guys cheating. Knowing about TEOS had been made possible by Summer of Bitcoin, a program that changed my understanding of money and economy. It put me on the path to what is now my favorite career.
I Would love to have this experience again <3.