Once upon a time in 2013, there was a tool called etcd which was a really lightweight database written
around the Raft consensus algorithm. This tool was
originally written in 2013 for a bullshit unsuccessful project called CoreOS Container Linux that was
EOL'd several years ago, but that doesn't really matter --- etcd was greater than its original use-case. Etcd
provided a convenient and simple set of primitives (set a key, get a key, set-only-if-unchanged, watch-for-changes) with
a drop-dead simple HTTP API on top of them. I have built a number of tools using etcd as a lightweight consensus store
behind them and it's absolutely a pleasure to work with.
- The word "rant" is right up there in the tags line. This is not meant to be a persuasive argument to the secret cabal that controls API design or a nuanced technical comparison article. It's just some off-the-cuff thoughts. Chillax.
- If this didn't come across clearly enough in the article: I think etcd is great! I have written a bunch of tools and applications on top of it! I think it's a fantastic little dæmon and its API, even the new janky v3 API, is still a million times better than ZooKeeper
Okay, then. Read on.
In 2015, an unrelated tool called Kubernetes was released by Google (but, really, by Xooglers). I would go so far as to say that Kubernetes (or, as the "cool kids" say, k8s) is the worst thing to happen to system administration since systemd. It's a comprehensive suite that promises to simplify operating clusters of software and give something like the experience of Google's borg cluster manager. What it really does is:
- Add hundreds of new failure modes to your software
- Move you from writing portable software configuration to writing thousands of lines of k8s-specific YAML
- Ensnare you in a mesh of questionably-good1 patterns like containerization and software defined networking
If you are running a truly enormous system and want to have off-the-shelf orchestration for it, Kubernetes may be the tool for you. For 99.9% of people out there, it's just an extra layer of complexity that adds almost nothing of value.
I digress, though; this is a story about etcd. And, unfortunately, our stories come together because Kubernetes was quickly changed to use etcd as its state store. Thus began the rapid decline of etcd.
With the massive influx of Kubernetes users came, of course, a large number of Xooglers who decided to infect etcd with Google technologies, as is their way[^infection]2. Etcd's simple HTTP API was replaced by a "gRPC"3 version; the simple internal data model was replaced by a dense and non-orthogonal data model with different types for leases, locks, transactions, and plain-old-keys. etcd 3.2 added back a tiny subset of the HTTP API through the "gRPC Gateway", but not enough to implement any of the rich applications built on top of the original API. The v2 API lives on for now, but upstream threatens to remove it in every new version and there will surely come a time when it'll be removed entirely.
That's it. That's the story. Popular modern technology is taken over by expats from a megacorp and made worse in the service of a hyper-specialized (and just plain over-hyped) orchestration platform. That's the world today. Anything that has a simple and elegant feature-set ends up coöpted by people who just want to build big ungainly architecture and ends up inheriting features from whatever megacorp the coöpters came from4. The software development world would prefer to use their multi-gigabyte IDEs running on ElectronJS to build thousand-dependency Java applications targeting ungainly APIs on hard-to-operate systems than support something simpler and better. Quality is, alas, a dying art.
Read: "not good"
I've worked with a lot of Xooglers in my career (heck, I worked there myself). I now consider it to be a serious negative on someone's resume to have worked at Google. The many ex-Google coworkers I've had have (even when they've been otherwise brilliant) been uniformly less capable of working on non-Google systems than their much more junior equivalents with other backgrounds. All big companies have their own proprietary technology stacks, but the degree to which Googlers never learn how to do anything without involving protocol buffers, bigtable, and a mile-high stack of other proprietary tools is frankly remarkable. And they spread this to everything new they touch. Any open-source project will inevitably get pull requests to switch from JSON or BSON to Protocol Buffers; every web server now needs to support the cancerous user-hostile protocols of HTTP/26 and HTTP/37 that were passed in a mockery of IETF procedures.
Yes, I know, gRPC was added to etcd by Xiang Li, the original author of the project. That doesn't give them a pass for being influenced by the bad ideas coming out of Mountain View, or by their project's newfound popularity in the land of the Xooglers.
gRPC is Protocol Buffers8 running over HTTP/26. It's a got a "g" at the beginning of the name to remind you that the only time it's acceptable to use is when you are actually working for Google inside a Google-owned building eating Google-branded food and breathing Google-branded air.
Protocol Buffers or protobuf is Google's very bad serialization format
HTTP/2 a.k.a. SPDY is a comically bloated Layer 5/6/7 mega-combo protocol designed to replace HTTP. It takes something simple and (most importantly!) comprehensible and debuggable for junior programmers and replaces it with an insanely over-complicated9 system that requires tens of thousands of lines of code to implement the most minimal version of, but which slightly reduces page load time and server costs once you reach the point of doing millions of requests per second. I am filled with rage just thinking about how we took a fundamental part of the Internet, simple enough that anyone can implement an HTTP server, and replaced it with this garbage protocol pushed by big megacorps that doesn't solve any real problems but will completely cut out future generations from system programming for the web.
My favorite HTTP/2 interaction has been finding and reporting this bug in haproxy. The compression scheme in HTTP/2 is so shitty that the "compression table" in RFC 7541 § Aa is just a list of the 61 most popular headers from Google properties.
HTTP/3 is all the badness of HTTP/2, but run over a worse layer 4 protocol named QUIC that totally fucks up networking for everybody in order to get a tiny bit more optimization for Google. That's all it does. It makes the Internet strictly worse for everybody but slightly better for the hugest of huge web properties. Nobody out here in the real Internet gives the slightest shit about head-of-line blocking from TCP, and lots of people want TCP state-aware firewalls and load-balancers to work.
I talk a lot of shit about Google, but Facebook and Microsoft are nearly as bad at turning out legions of ex-employees who can't be left alone in the room with a keyboard lest they attempt to recreate their previous employer's technology stack, poorly.
Want to comment on this? How about we talk on Mastodon instead?
Share on Mastodon