Werd I/O

As an experiment, I’m trying out Kagi as my default search engine, switching from Google (no link required; they’re probably behind you right now).

I like the idea of an ad-free experience: a paid-for engine puts the incentives in the right place. But it’s got to be about more than ideology. Because search is such an important part of my working life (and most knowledge workers’ working lives) it’s important that the results are actually better than Google’s.

For a while, I tried to use DuckDuckGo, which uses Bing’s search engine behind the scenes. It was just fine for most things and markedly worse for a few, so I had to switch back, even though I love its privacy focus.

Kagi uses a mix of third-party engines and its own to provide its results. So far, they seem pretty good, but the proof will be in intensive use.

What I already know I love: they have a StumbleUpon-like site for discovering small websites, and surface blogs in some search results when they’re relevant. That’s something I want from every search engine.

We will see! I’ll give it a month and then report back.

It’s no surprise to anyone that I prefer reading peoples’ long-form thoughts to tweets or pithy social media posts. Microblogging is interesting for quick, in-the-now status updates, but I find myself craving more nuance and depth.

Luckily, Blogging is enjoying a resurgence off the back of movements like the Indieweb (at one end of the spectrum) and platforms like Substack (at the other), and far more people are writing in public on their own sites than they were ten years ago. Hooray! This is great for me, but how do I find all those sites to read?

I figured that the people I’m connected to on Mastodon would probably be the most likely to be writing on their own sites, so I wondered if it was possible to subscribe to all the blogs of the people I followed.

I had a few criteria:

1. I only wanted to subscribe to blogs. (No feeds of updates from GitHub, for example, or posts in forums.)
2. I didn’t want to have to authenticate with the Mastodon API to get this done. This felt like a job for a scraper — and Mastodon’s API is designed in such a way that you need to make several API calls to figure out each user’s profile links, which I didn’t want to do.
3. I wanted to write it in an hour or two on Sunday morning. This wasn’t going to be a sophisticated project. I was going to take my son to the children’s museum in the afternoon, which was a far more important task.

On Mastodon, people can list a small number of external links as part of their profile, with any label they choose. Some people are kind enough to use the label blog, which is fairly determinative, but lots don’t. So I decided that I wanted to take a look at every link people I follow on Mastodon added to their profiles, figure out if it’s a blog I can subscribe to or not, and then add the reasonably-bloggy sites to an OPML file that I could then add to an RSS reader.

Here’s the very quick-and-dirty command line tool I wrote yesterday.

Mastodon helpfully produces a CSV file that lists all the accounts you follow. I decided to use that as an index rather than crawling my instance.

Then it converts those account usernames to URLs and downloads the HTML for each profile. While Mastodon has latterly started using JavaScript to render its UI — which means the actual profile links aren’t there in the HTML to parse — it turns out that it includes profile links as rel=“me” metatags in the page header, so my script finds end extracts those using the indieweb link-rel parser to create the list of websites to crawl.

Once it has the list of websites, it excludes any that don’t look like they’re probably blogs, using some imperfect-but-probably-good-enough heuristics that include:

1. Known silo URLs (Facebook, Soundcloud, etc) are excluded.
2. If the URL contains /article, /product, and so on, it’s probably a link to an individual page rather than a blog.
3. Long links are probably articles or resources, not blogs.
4. Pages with long URL query strings are probably search results, not blogs.
5. Links to other Mastodon profiles (or Pixelfed, Firefish, and so on) disappear.

The script goes through the remaining list and attempts to find the feed for each page. If it doesn’t find a feed I can subscribe to, it just moves on. Any feeds that look like feeds of comments are discarded. Then, because the first feed listed is usually the best one, the script chooses the first remaining feed in the list for the page.

Once it’s gone through every website, it spits out a CSV and an OPML file.

After a few runs, I pushed the OPML file into Newsblur, my feed reader of choice. It was able to subscribe to a little over a thousand new feeds. Given that I’d written the script in a little over an hour and that it was using some questionable tactics, I wasn’t sure how high-quality the sites would be, so I organized them all into a new “Mastodon follows” folder that I could unsubscribe to quickly if I needed to.

But actually, it was pretty great! A few erroneous feeds did make it through: a few regional newspapers (I follow a lot of journalists), some updates to self-hosted Git repositories, and some Lemmy feeds. I learned quickly that I don’t care for most Tumblr content — which is usually reposted images — and I found myself wishing I’d excluded it. Finally, I removed some non-English feeds that I simply couldn’t read (although I wish my feed reader had an auto-translate function so that I could).

The upshot is that I’ve got a lot more blogs to read from people I’ve already expressed interest in. Is the script anything close to perfect? Absolutely not. It it shippable? Not really. But it did what I needed it to, and I’m perfectly happy.

Tracy Durnell celebrates 20 years of blogging:

A blog is a much nicer place to publish than social media, sparking fewer but more meaningful interactions. Blogging allows writers a more forgiving pace with slower conversation. On their blog, people can be themselves instead of playing to an audience and feeling judged — a place to escape the pressures of one-upmanship and signaling, the noise of the ever-demanding attention economy, and the stress of hustle culture.

It’s a huge achievement, to be sure, and I couldn’t agree more with Tracy’s sentiment here. Congratulations, Tracy!

I’m a little jealous that she can pinpoint an anniversary date. For me, it depends on how you judge: I had a hand-rolled blog of sorts when I went to university in 1998, but was it really a blog? I definitely had a public Livejournal in 2001, but was that a blog? How about blog I used to keep on Elgg dot net (now a domain squatter, may it rest in peace)? My old domain, benwerd.com, dates back to 2006, and my current one, werd.io, only goes back to 2013. It’s a bit of a messy history, with stops and false starts.

On the other hand, I know people who have posted to the same domain for almost as long as they’ve been online. I don’t know if I can match that sort of dedication - or a commitment to even having a continuous identity for all that time. Am I the same person I was 20+ years ago? A little bit yes, but mostly not really. The idea of joining up my life online on a long-term basis is actually quite daunting.

Tracy links to Mandy Brown’s piece on writers vs talkers, which also deeply resonates: I’m a writer. I hate being drawn into making decisions in ad hoc meetings. I want to write my thoughts down, structure them, and then come to a conclusion after getting feedback and iterating. Perhaps that’s why blogging early appeals to me so much: I can put out ideas and very quickly engage in conversations about them that pushes my thinking along.

Blogging might seem like a solitary activity, but it’s very, very social. Even the name — a pun derived from weblog = we blog — is about community. Writing for 20 years also means building community for that long.

Here’s to the next 20!

I’ve been neck-deep in a long-form first draft for months; at this point I’m many tens of thousands of words in. Every time I look back at my writing from tens of thousands of words ago, it’s a horrible mistake that opens up floodgates of self-questioning. How could I possibly have thought that I could do this? Who on earth would want to read this? Amateur! Go back to whatever it is you do for your day job. (Do you even know? I thought you wrote software? When was the last time you actually wrote software, you hack?)

But I’m determined. The only thing I can say for sure is that, eventually, I will have a manuscript. I have professional mentors who will read and critique it once I’ve iterated on it a few times. Beyond that, I can’t say. Perhaps, if I’m lucky, someone will like it. But perhaps it really is doomed to sit on my hard drive, unloved.

The deeper I get into it, the more I’m comfortable with the idea of failure. I think I started with the idea that I might be intentionally writing something that a lot of people might enjoy, but at this point it’s for me. The more I pour in of myself, and the ideas I have about the world (and the future of technology, because that’s the kind of book this is), the more I feel comfortable with it. Even if nobody loves it, it’ll be representative of me: a genuine work of self-expression hooked onto a plot that I continue to think is really interesting. And the feedback I get will help me learn to write the next one.

It turns out that the thing which most motivates me to write is my sense of humor. If it’s too self-serious, I stall. (Honestly, I expect readers would, too.) On the other hand, if I’m amusing myself, undercutting my serious points with irony or adding notes about things from the real world that I think are ridiculous, I can go forever. That’s probably something worth knowing about myself: I thrive on irreverence. I cut my teeth on Douglas Adams, Terry Pratchett, and Charlie Brooker’s early stuff, so that’s probably not surprising. I could probably use more of that here, too.

Anyway. It’s like pulling teeth, but joyously. A gleeful festival of unpleasant monotony wherein I make myself laugh while disgusting myself with my own ineptitude. And maybe, if I’m really, really lucky, something will even come of it.

It’s been interesting to watch all of the articles celebrating the death of NFTs lately. For years, they were the harbinger of the next big thing, hawked by A-list celebrities. Behind the scenes, some of the biggest tech companies in the world spawned NFT strategies, even as critics noted that valuations were partially driven by money laundering and wash trading.

Cut to 2023, and surprise, surprise: 95% of NFTs are now completely worthless.

If you missed the craze: while most digital data can be infinitely replicated for almost no cost, Non Fungible Tokens, or NFTs, were a way to ensure there was only one of an item using blockchains. NFTs were often attached to digital art — for example, these hideous apes — and because they were both scarce and tradable, for a while each one was going for the equivalent of thousands of dollars. Of course, it couldn’t last, and NFTs turned out to be the digital equivalent of investing in Beanie Babies or tulips(pick your proverbial market collapse).

It’s now controversial to say that crypto isn’t completely useless, but if you look beyond the brazen grift, international crimes, and planet-destroying environmental impact, I do think there are a few things to celebrate about the trend. The crypto community deployed the most widely-used ever implementation of identity in the browser, for one: people who installed software like Metamask could choose to identify themselves to a website with a single click. In some countries, digital currencies also gave citizens an accessible safe haven when their own local currency tumbled. And finally, it introduced a much wider audience to the concept of decentralization, where a large-scale internet system is run co-operatively by all of its users instead of a giant megacorp.

Although the rampant speculation and wildly inflated prices are gone, there are some technical outcomes that will likely be with us for some time. And some of those are positive and useful.

This is exactly how the hype cycle works. A technology breakthrough kicks things off and gets people all excited. The market works itself into a frenzy over the technology, and lots of people imagine that it can do all kinds of amazing things. Those inevitably don’t actually pan out, and people lose hope and interest. But it turns out that the technology is useful for something, and eventually, it finds a mainstream use.

Crypto is very much in the trough of disillusionment right now; eventually some aspects of the technology (maybe identity in the browser, maybe something else) will find a use.

Meanwhile, AI? AI is right at the top of that hype curve.

There are people out there who believe we’re building a new kind of higher consciousness, and that our goal as humans should be to support and spread that consciousness to the stars. A galaxy full of stochastic parrots is an inherently funny, Douglas Adams-esque idea, but naturally, they’re serious, partially because they feel this idea absolves them of dealing with the truth that there are actual human beings living on a dying planet who need help and assistance right now. In erasing the needs of vulnerable communities, AI supremacy (officially called effective accelerationism) is the new white supremacy (sitting comfortably alongside the old white supremacy, which is still going strong).

There are also people who think AI will replace poets, artists, neurosurgeons, and political leaders. AI systems will farm for us, tend to our children, and imagine whole new societies that we wouldn’t otherwise be capable of envisioning. They will write great literature and invent wholly new, never-ending dramatic entertainment for us to sit and consume.

It’s horseshit. The technology can’t do any of those things well. It’s best thought of us a really advanced version of auto-complete, and everyone who claims it’s something more is trying to sell you something.

Which isn’t to say it’s not useful. I’ve certainly used it as a utility in my writing — not to do the writing itself (it produces mediocre porridge-writing), but to prompt for different angles or approaches. I’ve used it to suggest ways to code a function. And I’ve certainly used it, again and again, as a quick way to autocomplete a line of code or an English sentence.

What’s going to happen is this: in a few years, AI will come crashing down as everyone realizes it’s not going to be an evolution of human consciousness, and some other new technology will take its place. Valuations of AI companies will fall and some will go out of business. Then, some of the actual uses of the technology will become apparent and it’ll be a mainstream, but not dominant, part of the technology landscape.

The hype cycle is well-understood. What surprises me, again and again, is how thoroughly people follow it. Across industries, CEOs are right now thinking, “holy shit, if we don’t jump on AI, we’re going to be completely left behind. This is a paradigm shift.” It’s kind of the equivalent of a bunch of soccer players chasing the ball — It’s over here! No, it’s over here! Let’s run towards it! — which is how three-year-olds play soccer. A more strategic approach (let’s call it thinking for yourself) will be more productive for most businesses.

There will absolutely be uses for AI tools. The important thing is to take a step back and think: what are my needs? What are the needs of my customers or my community? Given the actual demonstrated capabilities of the software, does it help me meet any of them in a reliable way? If I do use it, am I holding true to my values and keeping my customers and community safe? If the answer is yes to all of these things, then great! Otherwise it might be worth taking a step back and letting the dust settle.

Keep me honest: if AI doesn’t enter a trough of disillusionment and just keeps growing and growing exponentially, call me on it. But I think it’s a pretty safe bet that it won’t.

The way my link blog works is like this:

I save an article, website, or book I thought was interesting to a database in Notion using the web clipper, together with a description and a high-level category. (These are Technology, Society, Democracy, and so on.) I also have a checkbox that designates whether the link is something I’d consider business-friendly.

Zapier watches for new links. When it finds one, it publishes it to my website using the micropub protocol. (My website then tries to send a webmention to that site to let it know I’ve linked to them.)

Then, it publishes the link to my Mastodon profile using the top-level category as a hashtag. If the link is to a book, it also adds the bookstodon hashtag.

Following that, it publishes to all my other social networks via Buffer, without the hashtag. (The exception is my Bluesky profile, which I had to write some custom API code for). If the business-friendly box was checked, that includes publishing to my LinkedIn profile.

If I’m feeling particularly motivated, I’ll copy and paste the link to my Threads profile, but because there’s no API, it’s a fully manual process. Which means I usually don’t.

Very occasionally, Zapier will pick up a link before the Notion entry has fully saved, which means that links post without a description or a category. Then I either shrug my shoulders and accept that I have some weird posts on my timeline, or I go back and edit or repost each and every one.

Because of this bug, I’ve thought about writing my own code to do all of the above on my server. It would work the exact way I want it to be. It would be cheaper, too: I pay for Zapier every month, and the cost adds up.

But while I could do this, and the up-front cost would certainly be lower, what if something goes wrong? Let’s say LinkedIn changes the way their API works. If I wrote the connection myself, I would need to keep my code up to date every time this happened — and, in turn, stay on top of codebase changes for every single social media platform I used.

And the truth is: I’m tired, friends. I want to be really careful about the amount of code I set myself up to maintain. It might seem like a simple script now, but over time I build up more and more simple scripts and, cumulatively, I end up buried in code.

As I get older, I find myself optimizing that cost more and more. I’d much rather pay something up-front that saves me a ton of time and cognitive overhead, because both of these things are at such an enormous premium for me.

I could also just not post to those social media accounts, or do it fully-manually, but there’s something really satisfying about publishing once and syndicating everywhere I’m connected to people. I could save my links straight to something like Buffer, but I also like having my categorized database of everything I’ve shared. And Notion makes it easy to save links across my devices (I’m sometimes on my phone, sometimes on my laptop, sometimes on my desktop).

So I’m keeping Zapier, at least for now. I like keeping my links, and I like sharing them. And, more than anything else, I like not having to maintain the code that does it.