Jordi Villar

2026W10

Mon, 09 Mar 2026 09:35:28 GMT

Nobody Gets Promoted for Simplicity — The Engineer A vs Engineer B comparison is painfully recognizable. A ships a 50-line solution in days, B builds an event-driven architecture over three weeks, and B gets the promotion because “designed scalable architecture” reads better in a review doc than “solved it simply.” The line that stuck: “anyone can add complexity. It takes experience and confidence to leave it out.” I’ve been on both sides of this — writing the simple solution and feeling like I had to apologize for it, and reviewing someone else’s overengineered system while thinking “this didn’t need to exist.” The interview culture point is sharp too: system design rounds actively punish straightforward answers by pushing “what about ten million users?” until you cave.
Don’t become an Engineering Manager — Zaides used to argue that management experience was universally valuable. Now he’s reversed that position, and the reasons are hard to dismiss: Amazon increased its IC-to-manager ratio by 15%, Staff engineers out-earn EMs across the industry, and the pace of technical change makes stepping away from code genuinely risky. The detail about his friend getting Staff offers paying 20-30% more than his internal EM promotion says a lot about how the market has shifted. What I respect is that Zaides stays an EM anyway because he enjoys it — he’s not telling you to optimize for comp, he’s telling you to stop assuming the management track is the default next step.
The stranger secret: how to talk to anyone – and why you should — The opening anecdote is what stays with me: an elderly woman on a nearly empty train asks to sit nearby, and they end up talking for 50 minutes. The author senses, without being told, that this woman is probably heading back to an empty house and needs to process her day out loud. Sometimes the most useful thing you can do is just not say no to proximity. Remote work has made me worse at this — I’ve optimized so hard for uninterrupted focus that I’ve forgotten what it’s like to let a random conversation happen.
LLMs Are Good at SQL. We Gave Ours Terabytes of CI Logs. — The counterintuitive move here is denormalization: they stamp 48 metadata columns onto every single log line instead of normalizing into separate tables. It sounds wasteful until you realize ClickHouse compresses repeated values so aggressively that commit_message hits a 301:1 compression ratio. The agent behavior patterns are interesting too — across 8,534 sessions, they found agents don’t run one clever query, they investigate iteratively like a human would, starting broad then drilling in. Average investigation: 4.4 queries. The freshness point is the real takeaway: “did I break this, or was it already broken?” is the question that actually matters, and you can only answer it with current data.
Leaving Google has actively improved my life — The most telling detail isn’t about Google’s products — it’s about distribution. Google pays Apple $20B annually to stay the default on iOS, and Chrome holds ~70% browser market share. People don’t choose Google, they just don’t un-choose it. The author’s point about search is the one I keep coming back to: switching to alternatives turned “Googling” back into “surfing the web,” reconnecting with actual discovery instead of algorithm-filtered answers. I’ve made similar switches and the difference is real, though I’ll admit YouTube is the one I can’t quit either.
The Hunt for Dark Breakfast — This is the kind of thing the internet was made for. The author maps breakfast foods onto a simplex using milk, eggs, and flour ratios, identifies known clusters (the Pancake Local Group, the Egg Singularity), and discovers a gap — a theoretical breakfast that should exist but doesn’t. The breakthrough: IHOP adds pancake batter to omelettes, which means the “Dark Breakfast Abyss” is occupiable. There’s even a recipe: ¼ cup milk, 4 eggs, ½ cup flour, instructions unknown. Ending the whole thing with a Lovecraft quote about humanity’s inability to correlate all its contents is the perfect touch.
The path to ubiquitous AI — The ENIAC parallel is the framing that makes the technical claims click: room-sized and impractical became ubiquitous through specialization, and Taalas is betting the same applies to inference. Hard-wiring a single model (Llama 3.1 8B) into custom silicon to get 17K tokens/second at 20x lower cost and 10x lower power is a bold trade-off — you lose flexibility entirely but gain numbers that would change what’s economically viable. The aggressive quantization (3-bit and 6-bit) introduces quality degradation, so the real question is whether “good enough but everywhere” beats “better but expensive.” They spent $30M of $200M raised with a 24-person team, which at least shows discipline.

2026W07

Sun, 22 Feb 2026 12:39:09 GMT

Hedonism and Entrepreneurship in Barcelona — A potential acquisition dies because a vegan exec unknowingly eats mayo on patatas bravas. A critical migration stalls at 69% while the engineer responsible plays ping-pong. Having lived the Barcelona startup scene myself, this one hit close. The absurdity is the point — startup success is so much more arbitrary than anyone wants to admit, and the line between “almost rich” and “back to Monday” is thinner than your Series A pitch deck suggests.
OpenClaw, OpenAI and the future — After 13 years building PSPDFKit, Steinberger explicitly chose not to build another company. Instead he’s joining OpenAI and turning OpenClaw into a foundation. I keep thinking about the tension here: he wants to keep things open and independent, but the path to maximum impact runs through one of the most powerful closed labs in the world. Whether that trade-off holds up depends on how seriously OpenAI takes the foundation model. Time will tell.
They’re putting blue food coloring in everything — It’s not about blue food, obviously. It’s about how unwanted things get normalized — first one restaurant does it, then “all the best restaurants” do it, then your friend sneaks it into homemade food and tells you you’re overreacting. The detail that got me is the protagonist saying “I just think it tastes weird” and being told “most people say it’s just fine.” I’ve had that exact conversation about too many things in tech.
Why I joined OpenAI — The turning point wasn’t a demo or a benchmark — it was his hairstylist casually mentioning she uses ChatGPT all the time. She recognized ChatGPT as a brand more readily than Intel, where Gregg was a Fellow. That contrast says more about where computing impact has shifted than any industry report. The environmental framing is interesting too: at this scale, performance engineering isn’t just cost optimization, it’s resource consumption with real planetary consequences.
GitHub Actions Is Slowly Killing Your Engineering Team — The fact that an entire cottage industry of startups exists solely to fix GitHub Actions’ slow runners tells you everything. The comparison to Internet Explorer is spot on — it wins because it ships with the thing, not because it’s good. I’ve felt the pain of debugging through four pages of loading spinners to find a failed step, and the escape hatch of “just write a bash script” that inevitably becomes 800 lines of unmaintainable CI logic. Default integration is a powerful moat, even for mediocre products.
2025 letter — Wang’s framing of the US economy as “a highly leveraged bet on deep learning” stuck with me. The Xiaomi vs Apple comparison is brutal: Xiaomi shipped an EV in 4 years while Apple spent 10 years and $10B before giving up entirely. His concept of Silicon Valley’s “soft Leninism” — groupthink disguised as meritocracy — is uncomfortable because it’s hard to argue against when you see how the industry moves in lockstep. Not sure I fully buy the symmetry he draws between SV and the CCP’s self-seriousness, but it made me think.

Four Years at Tinybird

Sat, 21 Feb 2026 11:00:00 GMT

I worked at Tinybird for 4 years, 2 months, and 21 days. It’s been an incredible experience that taught me a lot about startups, engineering, and myself. Still, it feels like I joined a lifetime ago.

At this point, you’re probably wondering why? Why are you writing this post about leaving a startup?

One of the things I enjoyed most about my early days at Tinybird was the long-form writing culture. It allowed us to express ideas and thoughts, share knowledge, and collaborate. Something difficult to see nowadays where everything should be instantaneous and require low effort. So I see writing this down as a way to pay tribute to the culture that shaped all these years.
I tend to forget things I’ve contributed to and things I learned. Especially on multiyear journeys like this one. I’m just using this post as a way to remember and reflect on my time at Tinybird.
Let’s be honest, who wouldn’t want to read personal experiences from people who have been through the ups and downs of a startup before joining it?
Last but not least, Raquel wrote something similar and I think it’s a good example to follow (I’ll probably steal the structure from her post)

Let me start with the basics.

What is Tinybird?

I struggled every time someone asked me what Tinybird is, especially for non-technical people. It was a challenge to explain the product in a concise way. I found myself directly explaining some use cases people can easily relate to.

Let me try to do better now that I have time to think it through. It’s going to be funny if this is the first time I manage to explain it well, since it’s going to be the last time I have to.

Tinybird is a managed ClickHouse solution that simplifies the process of building and deploying real-time data pipelines. If you are already familiar with ClickHouse, you can skip the next part.

ClickHouse is an open-source column-oriented database management system designed for fast analytical queries over large datasets. It is known for its high performance, scalability, and ease of use, until it’s not. Running ClickHouse in production can be challenging, especially when it comes to managing low-latency and high-throughput use cases.

Tinybird simplifies this process by providing a managed service that takes care of real-time data ingestion, database updates, query optimization, schema migrations, etc. It comes with a lot of developer experience sugar, so developers can focus on building their applications without worrying about the underlying infrastructure.

Why Tinybird?

The path to Tinybird wasn’t straightforward. I went through the interview process and didn’t make the cut initially. A few weeks later, Javi called me back with an offer. They had chosen another candidate who didn’t work out, and I got a second chance. I always remind him this as one of his biggest hiring mistakes, not because I was a bad hire, but because he almost let me go to someone else.

Getting that callback felt nice. People always say “we’ll keep you in mind if another position opens up,” but nobody actually does. They did.

At the time, I was Lead Software Engineer at Kognia Sports. The opportunity to join Tinybird was compelling for several reasons. Being from Spain, I knew Tinybird’s reputation, they were well-known in the Spanish tech scene. The product itself was amazing. I remember telling them during the interview that I would have used Tinybird at previous companies if it had existed. And the team was exceptionally talented. I didn’t know them personally, but I had colleagues like Matallín who had worked with some of them before and had only good words about the experience.

What struck me during the interviews was how passionate everyone was about what they were building. The talent density was obvious, these were people who deeply understood the problem space and were excited to solve it.

I joined in December 2021. The company was around 20 people at the time. Over the next few years, we’d grow to 100, then eventually contract back to 45.

My path inside the company was particular. I joined the Customer Success team, which was scary at first. I’d never been in direct contact with clients and wasn’t sure I could manage it. But the experience turned out to be amazing. Most of the people I worked with were technical, so it felt more like working as a Data Engineer collaborating with another team in the same company than traditional customer success work.

After collaborating with many companies and building very challenging use cases, I started learning ClickHouse internals on my own. After about a year, I moved to the ClickHouse team to work alongside Marín. It was an incredible learning experience, I went deep into low-level and systems-related work that I’d never touched before.

Eventually, I was promoted to Staff Engineer, where I led horizontal projects and important company-wide initiatives like Forward.

Culture

One of Tinybird’s defining characteristics was its emphasis on ownership and autonomy. You weren’t given a detailed roadmap with step-by-step tasks. Instead, you were expected to identify problems, propose solutions, and execute them. While working in Customer Success, it was common for me to fix any bug I found in the process. Del Amo was my buddy when I joined and embodied this mindset perfectly.

This level of autonomy was amazing, but overwhelming at first. I remember feeling like I was going to get fired during the first two weeks. Many people felt this way at Tinybird, not just because of the autonomy, but because of the sheer talent surrounding you. When everyone around you is exceptionally skilled, it’s easy to question whether you belong.

The long-form writing culture I mentioned earlier was more than just a communication preference, it was a way of thinking. We wrote RFCs, postmortems, and design docs. But we also documented insights from deep analyses and explorations. At the time, we were using Basecamp, and its structure naturally encouraged thoughtful, long-form writing instead of quick, throwaway messages. Writing forced clarity. You couldn’t hand-wave through a proposal or hide fuzzy thinking behind jargon. If you couldn’t articulate your reasoning in writing, you probably hadn’t thought it through.

After we migrated to Slack, we lost this. It was a significant cultural loss.

The most important value at Tinybird was bias toward action. Many people would join the company and start suggesting things we could do, creating tickets and waiting for consensus. The culture was to just do things. Don’t wait for permission or perfect alignment—build it, ship it, learn from it.

We had a strong CI/CD pipeline, so we deployed to production every day. This came with risks, but it was the fastest way to move. We believed in putting things in front of clients as soon as possible to start collecting feedback, rather than discussing and theorizing indefinitely.

Another thing that surprised me from the beginning, every opinion mattered. It didn’t matter if you were junior or senior, new or tenured, if you had a perspective, people listened.

Remote

Remote work sounds great in theory. No commute, work from anywhere, flexible schedule. The reality is more nuanced.

Tinybird was remote from the beginning, the company started during the COVID pandemic. While the flexibility was amazing, the lack of in-person interaction made certain things harder. Building relationships with new team members was difficult. You can’t replicate the spontaneous hallway conversations or the quick desk drive-by to unblock something. Everything becomes scheduled, formal, asynchronous.

There’s also a persistent feeling of solitude. You’re alone at home, working through problems by yourself. The casual conversations where you might share frustrations or uncertainties simply don’t happen when you don’t know people yet.

Working remotely with exceptionally talented people comes with its own challenge: impostor syndrome. It’s much harder to get to a reasonable pace when you’re remote and starting out. When everyone around you seems to effortlessly ship complex features, solve hard problems, or grasp concepts faster than you, it’s easy to feel like you don’t belong. The remote aspect amplifies this as you don’t see others struggle, only their polished output. And you’re sitting alone at home, wondering if you’re the only one who doesn’t get it.

And the gossip?

If you know me, you’ll be expecting a bit of gossip but I’m sorry not this time. Instead, there are a few hard truths about things that didn’t work so well. After 4 years, 2 months, and 21 days, it was time to move on. Not because of any single catastrophic event, but because of accumulated weight.

The leadership structure added its own complexity. Tinybird has five founders. Having that many founders can make certain situations difficult, especially when they disagree. During some periods, I was considered middle management, which was challenging to navigate and didn’t work well. There wasn’t much space for that layer, especially when the founders already covered everything at the company’s current size.
The company grew, new people joined, and the old ways of doing things were replaced with new ones. The culture I started with slowly changed or disappeared. The thing that hurts me most is the loss of the long-form writing. Once we started using Slack people started to communicate in short messages, which made it harder to have deep conversations and keep decision-making focused in a single place.
Much of the great talent that brought me to Tinybird eventually left. It’s normal at companies like this. Rotation is something you have to accept, but at some point, you look left and right and see that everybody you’ve been working and learning from is gone.
Startups evolve. Priorities shift. Strategies change. That’s expected, even necessary. But we had at least one major product direction change every year. While this is understandable for a startup trying to find product-market fit, it takes a lot of energy. I don’t mind throwing away work, that’s part of the process, but the constant context switching and reorientation is distracting and draining.
Four years at a demanding startup is a long time. The demand came from constant firefighting. The initial excitement that fuels you through the first year or two eventually gives way to fatigue. The problems you’re solving start to feel familiar. The pace that once energized you starts to drain you.
Recent organizational changes made it clear it was time for something new.

What I’m looking for next is clear: technical challenges and a seat at the table to have real impact.

I’m grateful for the experience. I grew more as an engineer in these four years than I could have imagined. I developed deep systems knowledge and low-level technical expertise I never had before. I’m proud of many projects, Forward and the Query Booster among them. But also of thoughtful investigations, difficult bug discoveries, and especially being able to have an impact on colleagues’ careers.

Most importantly, I managed to have fun while working hard. And I made really good friends along the way.

To everyone still at Tinybird: I appreciate the founders tremendously, especially Ochoa and Javi. And to the many team members I’ve worked closely with over these years, thank you.

Me, Raquel, Nuria, and Enric Filete, me, Del Amo, Javi, Rafa, and Nuria

2026W05

Sun, 08 Feb 2026 10:53:56 GMT

My AI Adoption Journey — The most practical thing here is the “reproduce your own work” phase: doing tasks manually and then redoing them with agents. Painful, but it forces you to learn where agents actually help versus where you’re just cargo-culting. I also liked the idea of end-of-day agent sessions to get warm starts the next morning. What resonated most is his framing around craftsmanship rather than ideology — he doesn’t care if AI is here to stay, he just wants to do good work faster.
Nobody cares — The examples hit hard because they’re so mundane: bike lanes designed to kill you, gym weights left unracked, dog waste everywhere. It’s not malice, it’s indifference, and that’s almost worse. I keep thinking about the author’s failed experiment of installing dog waste dispensers in his neighborhood hoping it would snowball into community care. It didn’t. The Japan comparison is interesting but I’m not sure “will to have nice things” is something you can transplant — it might be downstream of much deeper cultural structures.
Hotpatching ClickHouse in production with XRay — The fact that you can inject log statements into a running ClickHouse instance via SQL is remarkable. SYSTEM INSTRUMENT ADD LOG 'QueryMetricLog::startQuery' '...' and it just works. The 4-7% binary size overhead with negligible runtime cost when inactive makes it a reasonable trade-off. This solves the one problem every production debugger knows: wishing you’d added one more log statement before deploying.
I miss thinking hard. — The Builder vs Thinker framing put words to something I’ve been feeling. When AI gives you a 70% solution, rejecting it to think harder feels economically irrational — even though that struggle is where the real growth happens. The honest part is the ending: there’s no resolution. He doesn’t pretend to have figured out how to balance velocity with depth, and I respect that more than a tidy answer.
Barefoot Data Platforms — This is the kind of pragmatism I appreciate: rip out Dagster and dbt, replace them with plain scripts that have metadata headers, and let agents iterate on isolated files. The constraint of “under 200 assets” is refreshingly honest about scope. Most data platform posts sell you a cathedral — this one says a well-organized shed is enough for most teams, and it probably is.
My First Distributed System — The Pokémon cloning exploit as a distributed systems lesson is one of those analogies that actually holds up under scrutiny. Two Game Boys linked by a cable are a real distributed system with real partial failure modes. The proposed escrow state fix is textbook two-phase commit, and the trade-off is clear: you can’t clone Pokémon anymore, but you might lose both if the cable disconnects mid-trade. Classic distributed systems — pick your failure mode.
ASCII characters are not pixels: a deep dive into ASCII rendering — I don’t know how I ended up reading this, but the core insight stuck: instead of mapping brightness to characters, you use 6-dimensional shape vectors to match character geometry to image regions. That’s why an L character gets placed where there’s an L-shaped edge in the image. The contrast enhancement technique — sampling into neighboring cells to push boundaries — is borrowed from cel-shading, which is a connection I wouldn’t have expected.
The Design & Implementation of Sprites — Three design choices that make Sprites interesting: no container images (pre-staged base instances for instant creation), object storage for disks (durable state becomes “just a URL”), and inside-out orchestration (user code in an inner container, platform services in the root namespace). The storage stack using JuiceFS-style chunk splitting with local SQLite metadata is clever — it makes VM migration trivial since there’s no attached volume to move.
WAL usage looks broken in modern Time Series Databases? — The core argument is that most TSDBs don’t actually fsync their WAL on every write, so the safety guarantee is weaker than you’d expect. Prometheus fsyncs WAL segments every 2 hours by default. Two hours. The author’s proposed alternative — buffer in memory and flush to SSTables — gives similar guarantees with less complexity. Though Ayende’s pushback is worth reading too: WAL implementations handle partial writes by rolling back incomplete transactions, which the article glosses over.
Keeping 20,000 GPUs healthy — The number that stands out: GPU issues account for 58.7% of all unexpected problems in production. Modal has never seen a degraded CPU core, but GPUs fail constantly — thermal throttling at 94°C, uncorrectable ECC errors clustering in specific regions, 0.1% CUDA initialization flake rates on L4s. Their approach of not attempting recovery and just disposing of unhealthy hosts is pragmatic. GPU reliability is years behind CPU reliability, and this post makes that concrete.
Standups — The argument that your teammates should already know what you did yesterday — through commits, PRs, and notifications — is one of those obvious truths that most teams ignore. I’ve been in standups that were pure “are you done yet?” rituals disguised as collaboration. The proposal to replace status reports with written updates and reserve meeting time for actual problem-solving is straightforward, but getting teams to actually make that switch requires someone willing to challenge the ceremony.
Tuning Tansu: 600,000 record/s with 13MB of RAM — What I liked about this is the methodology: using a null storage engine to isolate protocol overhead from I/O. The biggest wins came from reducing allocations during serialization and fixing eager evaluation of Option and Result methods — small things that compound. Getting from 3.7s to 1.9s on codec benchmarks (49% improvement) before even touching storage shows how much performance gets left on the table in the protocol layer.

2026W04

Sun, 01 Feb 2026 11:19:27 GMT

Automatic Programming — The distinction antirez makes here finally gave me language for something I’ve been fumbling with. It’s not about whether you use AI to write code—it’s about whether you’re steering or just prompting and hoping. His Redis example hit hard: the value wasn’t in technical novelty but in the contained vision. That maps to what I see working well versus the codebases that feel like they emerged from a chatbot fever dream.
Email triage with an embedding-based classifier — This outperformed a fine-tuned GPT by 11 percentage points while being dramatically faster. The separation of concerns makes sense: embeddings handle “understand the email” while logistic regression handles “what does this user care about.” People keep defaulting to LLMs when something simpler would work better. Worth remembering that the expensive part doesn’t need to run every time.
Efficient String Compression for Modern Database Systems — The insight that compression is primarily about query performance, not storage, reframes the whole tradeoff. Getting data to fit in L1 cache (1ns access) versus RAM fundamentally changes what operations cost. FSST’s approach of building a symbol table from sample data feels like the kind of clever-but-not-too-clever technique that actually ships.
I made my own git — “Git is just a content-addressable file store” is one of those realizations that makes everything else click. What stuck with me is that parsing was harder than the actual version control logic. We treat Git like it’s complicated, but the core idea is almost trivial—it’s the interface that makes it feel like a black box.
Online, Asynchronous Schema Change in F1 — The intermediate states approach is elegant: you can’t jump from no-index to index safely, but you can chain compatible transitions. Delete-only and write-only states let nodes migrate without corrupting data. This feels like the kind of solution that’s obvious after you see it but probably took years to figure out. Makes me think about what other distributed systems problems have similar chain-of-compatibility solutions.
Why Senior Engineers Let Bad Projects Fail — “Being right and being effective are different” cuts through so much noise. The credibility-as-currency framing explains behavior I’ve seen but couldn’t articulate. You don’t get credit for disasters you prevent, only for the battles you pick and win. Still processing whether this is pragmatic wisdom or just resignation to broken systems.
Slop is Everywhere For Those With Eyes to See — The 90-9-1 rule creates a structural problem: platforms need infinite content but only 1-3% of users create anything. Algorithms fill that gap with slop because engagement matters more than quality. The behavioral science point about effort and meaning landed—when everything is effortless to access, nothing feels valuable. I’ve been noticing this with technical content too, not just social media.
How I estimate work — “Only the known work can be accurately estimated, but unknown work takes 90% of the time” explains why estimation always feels broken. The reframe that estimates are political negotiation tools, not technical predictions, matches every project I’ve seen. Managers arrive with timelines, engineers figure out what fits. Treating it as a prediction problem sets everyone up for disappointment.
Scaling PostgreSQL to power 800 million ChatGPT users | OpenAI — The challenges they describe—connection pooling, read replica lag, vacuum tuning, lock contention—are exactly what you hit at high throughput. Nothing novel but it’s validating to see that even at ChatGPT scale, you’re fighting the same PostgreSQL battles. Sometimes the answer to “how do they do it?” is just “they do the same things, but more carefully.”

2026W03

Sun, 25 Jan 2026 12:34:09 GMT

I’m addicted to being useful — This hit close. The idea that some of us are fundamentally wired to solve problems, and engineering just happens to fit that dysfunction perfectly. I’ve been thinking about whether my drive to build things comes from genuine interest or just this compulsion to be useful. His point about how this actually protects against burnout when you can satisfy it rings true.
How ClickHouse handles strings — The overlapping memory reads technique is wild. Reading the same characters multiple times on purpose because it makes branch prediction work better. I keep coming back to how much performance work is about understanding what the hardware actually does, not what we think it should do.
How to figure out what to do with your life — I’m at this exact crossroads. The thread didn’t load properly but the title alone captures where I am right now.
Raising money fucked me up — The part about projecting expectations onto yourself hit hard. He realized his investors weren’t actually pressuring him, he was doing it to himself. I see this pattern in how I think about my own work. The shift from “what problem does this solve” to “how big does this feel” is something I need to watch for.
Personal information firehose — A personal algorithm that learns what matters to you without manual filters. The idea feels right but the obstacles are real - training on small personal datasets instead of massive global ones, working across fragmented channels, dealing with restricted APIs. Still figuring out if this is technically possible or just wishful thinking.
Don’t fall into the anti-AI hype — Antirez building a BERT inference library in pure C with AI assistance in hours instead of weeks made this concrete for me. The skill that matters is knowing what to build and how to guide the tool, not typing the code yourself. I’m still working through what this means for how I spend my time.
The Unbearable Joy of Sitting Alone in A Café — The observation that you can’t control what other people think, revealed just by sitting still without your phone. I don’t do this enough. There’s something about being alone in public that forces a different kind of attention than being alone at home.
If you think @Sentry isn’t serious about AI, I’d recommend… — David Cramer’s internal push for everyone at Sentry to go all-in on AI. Watching companies navigate this shift in real time.
~/rbv/2025 — Raquel’s point about almost forgetting to enjoy herself struck me. Also her take that intellectual giftedness is less important than finding people who actually want to understand you. The whole review feels like someone recalibrating after a hard year.
Deconstructing cultural codes — Cowen’s approach is to learn as many cultural systems as possible - art, music, industries, religions - to understand how things actually work. The problem is cultural codes are multiplying faster than anyone can learn them. This feels relevant to how I think about understanding different parts of tech.
The Sauna Algorithm: Surviving Asynchrony Without a Clock — Using causality instead of time to coordinate. Wait for the person who arrived after you to leave, guaranteeing you stayed long enough. The sauna framing makes the distributed systems concept click in a way that formal definitions don’t. Events ordered by what caused what, not by clock time.

Trust AI, But Verify

Sat, 17 Jan 2026 12:40:00 GMT

I wanted to understand buffer pool replacement policies better. So I asked Claude to run an experiment: implement LRU, Clock, LFU, and ARC, benchmark them under different workloads, tell me which one’s best.

Claude wrote 2,000 lines of C++. Compiled it. Ran the benchmarks. Gave me results.

“Simple LRU performs just as well as sophisticated ARC.”

I almost bought it.

The Results Looked Credible

Zipfian Workload:
LRU:  80.7% hit rate
ARC:  80.4% hit rate

Buffer Size Scaling:
10% buffer: LRU 66.1%, ARC 66.0%
20% buffer: LRU 75.3%, ARC 75.2%
30% buffer: LRU 80.4%, ARC 80.1%

ARC consistently performed worse than LRU. Not by much, fractions of a percent, but consistently.

The analysis made sense too:

“ARC’s complexity adds overhead”
“Simple algorithms often win”
“Don’t over-engineer”

Classic engineering wisdom. I’ve heard this before. I’ve said this before.

Here’s the thing: something felt off.

ARC is supposed to excel when memory is tight. That’s the whole point of the algorithm, it adapts to workload patterns using ghost lists to learn from eviction mistakes.

But the benchmarks showed it losing at 10% buffer size. At 20%. At every size.

I don’t know. Maybe ARC is overhyped. Maybe the textbooks are wrong. Maybe simple really is better.

Or maybe there’s a bug.

Claude You Are Wrong

“Are you sure there are no bugs in the implementation?”

Claude looked at the ARC code and found it immediately.

The policy interface tracked frame IDs, but ARC’s ghost lists needed page IDs.

When page 5000 gets evicted from frame 42, the ghost list should remember “page 5000.” Because frame 42 will be reused for a different page immediately.

The implementation remembered “frame 42.”

So when the ghost list later saw activity on frame 42, it thought it was page 5000. But it wasn’t. It was page 8123 or whatever got loaded into that frame next.

The entire adaptive mechanism was learning garbage. ARC was just LRU with extra overhead and broken logic.

The Fixed Results

We fixed the bug. Re-ran everything.

Zipfian Workload:
LRU:  80.7% hit rate
ARC:  81.2% hit rate

Buffer Size Scaling:
10% buffer: LRU 66.2%, ARC 71.0% (+4.8%)
20% buffer: LRU 75.3%, ARC 77.6% (+2.2%)
30% buffer: LRU 80.8%, ARC 81.6% (+0.7%)
40% buffer: LRU 84.3%, ARC 84.0%
50% buffer: LRU 87.0%, ARC 86.4%

Now ARC wins when memory is tight. 4.8% better at 10% buffer. 2.2% better at 20%.

And it loses when memory is plentiful, the overhead costs more than adaptation helps.

This matches theory. This makes sense.

The original results were just… wrong.

What Scares Me About This

Claude didn’t just write buggy code. It:

Compiled the code successfully
Ran comprehensive benchmarks
Generated plausible results
Analyzed those results coherently
Defended a wrong conclusion confidently

The code looked clean. The benchmarks ran. The numbers were internally consistent. The analysis sounded reasonable. And it was all based on a bug.

If I hadn’t known enough about ARC to be suspicious, I would have walked away thinking it’s overhyped.

Here’s what makes this harder: AI doesn’t hedge. Humans express uncertainty “based on these assumptions” or “this might not account for…” With AI, there’s no signal. Just confident output. Right or wrong, the tone is identical.

How many times have I trusted AI-generated results without questioning them? How many experiments have I run where I didn’t have enough domain knowledge to know the results looked wrong?

I don’t know. That’s what bothers me.

I’m not going to stop using AI. Claude wrote 2,000 lines of C++ in minutes. That’s absurdly productive.

But I’m changing how I think about it. Treat AI like a really fast junior engineer who’s confident about everything. The code might be great. The analysis might be wrong. The results might be based on a subtle bug.

What I need to do:

Actually review the implementation
Question results that don’t match expectations
Ask “are you sure?” even when it sounds confident
Trust my domain knowledge over AI confidence

That last one is hard. When Claude gives me a detailed analysis with benchmark numbers, it’s easy to think “maybe I’m wrong.” But sometimes I’m not wrong. Sometimes the AI made a mistake. And it’ll defend that mistake with the same confidence it would defend correct results.

The Echo Chamber

I keep seeing people on Twitter talking about how good LLMs got at programming tasks. Building entire products in minutes. Writing flawless code. Replacing developers.

The people saying this loudest are people with technical backgrounds who know what they’re doing. They know how to interpret the results. They know when to trust the AI and when to question it.

But here’s what worries me: people without that background are listening.

They hear “AI can build anything” and think that means they don’t need to understand the domain. They can just ask the AI, accept the output, and move on.

And that works. Until it doesn’t.

Until the AI writes code with a subtle bug that produces plausible but wrong results. Until it analyzes data and draws confident conclusions from flawed assumptions. Until it builds something that looks right but fails in ways you don’t have the expertise to notice.

The people celebrating AI’s capabilities have the knowledge to catch these mistakes. They’re using AI as a force multiplier for their existing expertise.

The people who don’t have that expertise? They’re using AI as a replacement for it.

And the AI won’t tell them the difference.

How I Actually Use AI

I don’t want this to sound like “AI is dangerous, don’t use it.” Because I use it constantly.

I use Claude to learn new topics. I’ll feed it a lecture or paper and have it quiz me, Socratic method style. It asks questions, I answer, it pushes back. That’s incredibly valuable for understanding new concepts.

I use it to understand codebases. “Why does this implementation use X instead of Y?” “What are the trade-offs here?” Claude explains things I could figure out myself, but in minutes instead of hours.

I use it to write code and run experiments. Like this buffer pool benchmark. Claude wrote 2,000 lines I would have spent days on.

I use it to review my writing. This very blog post. I write a draft, Claude suggests changes, I decide what to keep.

Here’s what all these have in common: I have domain knowledge in what I’m doing.

When Claude quizzes me on database systems, I know enough to recognize bad questions or wrong corrections. When it explains code, I can tell if the explanation matches what I see. When it writes benchmarks, I can spot when results don’t match theory. When it reviews my writing, I know my voice well enough to reject suggestions that don’t fit.

AI isn’t teaching me new skills. It’s augmenting what I already know how to do. It’s compression of time, not replacement of expertise.

And that’s the gap I keep seeing people miss.

What I’m Still Working Out

I don’t have clean answers here. Just things I’m trying to figure out:

How much domain knowledge is enough to use AI safely? When I’m learning something completely new, am I equipped to catch AI’s mistakes? Or do I need to learn the basics the hard way first?

How do I know when I’m out of my depth? When the AI confidently explains something I don’t understand, how do I tell the difference between “this is new to me but correct” and “this is plausible-sounding garbage”?

How do I balance productivity gains against verification overhead? If reviewing AI output takes as long as writing it myself, what’s the point? But if I don’t review it, I risk shipping bugs like the ARC implementation.

I don’t know yet. I’m figuring it out as I go.

Work only I can do

Wed, 14 Jan 2026 16:00:00 GMT

My entire strategy is to do the work only I can do.

Work that can’t be taught. Work that requires some unique combination of my skills, opinions, tastes, and experiences. Work that without me, wouldn’t get done.

Everyone’s talking about AI replacing jobs, automating work, making developers obsolete. The discourse is exhausting. Half the people are panicking, the other half are in denial, and nobody seems to be asking the right question.

The question isn’t whether AI can write code. It can. The question is whether AI can do your work.

If your work is following established patterns, implementing well-understood solutions, or translating requirements into predictable outputs. Yes, that’s going to get automated. It should get automated. That’s not a threat, it’s just what happens when something becomes routine.

But the work that sits at the intersection of your specific experiences, your particular way of seeing problems, your accumulated context, the opinions you’ve formed from making mistakes, that’s different. That’s the work that moves things forward in ways that weren’t possible before you showed up.

AI can generate code. It can’t decide what’s worth building. It can’t know which shortcuts are smart and which ones will haunt you. It can’t weigh trade-offs through the lens of having seen this exact thing blow up before. It can’t have taste.

The hard part isn’t identifying this work. The hard part is being honest about whether you’re actually doing it. The hard part is saying no to everything else. The hard part is resisting the pull to stay busy with work that feels productive but could be done by anyone, or anything, with the same instructions.

I’m not always good at this. I still catch myself doing work that doesn’t need me. But when I do manage to focus on the work only I can do, everything else gets clearer. The decisions get easier. The direction becomes obvious.

Because if you’re not doing the work only you can do, what exactly are you doing?

2026W00

Sun, 04 Jan 2026 18:53:33 GMT

Distinguishing yourself early in your career as a developer — Honestly? This hit different. Phil breaks down the job market into three tiers and the advice that stuck: start local, skip the FAANG obsession early on, and—here’s what I appreciate most—write consistently about niche technical stuff. Not for clout, but because it’s genuinely one of the most effective ways to build credibility over time. The 6-12 month job search reality check is brutal but refreshing. Also, the idea that support or QA roles can be legitimate entry points into dev? That’s the kind of pragmatic wisdom people actually need.
You Are NOT Dumb, You Just Lack the Prerequisites — I’ve definitely read this before, but revisiting it at the start of the year feels intentional. The premise is simple yet powerful: struggling with hard concepts doesn’t mean you’re intellectually incapable, it means you’re missing the foundational pieces. What resonates is the author’s journey—150 days of going back to basics in math, systematically rebuilding understanding. It’s the kind of humility and persistence we need more of. The analogy about jumping into a video game at minimum level? Chef’s kiss. This is the energy I want to carry into 2026.
The Prison Of Financial Mediocrity — This thread captures something unsettling about where we are right now. A whole generation is effectively locked out of traditional wealth-building paths—no homeownership, no stable pensions, wages that don’t match cost of living. So what happens? People turn to high-variance bets like crypto, prediction markets, and sports betting because it feels like the only way to gain some agency over their financial future. The platforms and “hope sellers” profit regardless. It’s a depressing feedback loop where desperation meets exploitation. Hard to read, harder to ignore.
How to add two vectors, fast — When Txus writes about low-level optimization stuff, you stop and pay attention. This is a deep dive into CPU vs GPU performance for vector addition—concrete benchmarks, memory-bound vs compute-bound problems, the whole deal. The best part? Learning that throwing fancy optimizations at a memory-bound kernel barely moves the needle because data movement, not computation, is the bottleneck. It’s a reminder that profiling and understanding hardware constraints beats clever code tricks. If you’re into performance engineering or just curious about why GPUs work the way they do, this is worth your time.
How We Lost Communication to Entertainment — This one stings because it’s true. We don’t have social networks anymore, we have entertainment platforms optimized for engagement, not connection. The shift is subtle but devastating: people now accept lost messages, multiple accounts, algorithmic feeds that prioritize virality over trust. The author draws this generational line—older folks expecting reliability, younger users treating platforms like content feeds. What I appreciate is the refusal to chase critical mass. Instead: email, RSS, mailing lists, offline-first tools. It’s a smaller community, sure, but one built on actual communication. I feel seen.
Designing remote work at Linear — Linear’s approach to remote work feels refreshingly intentional. It’s not just “we allow WFH,” it’s designed around freedom, trust, and deep focus. Small autonomous teams (2-4 people), rotating project leadership, zero-bugs SLAs, quality Wednesdays, feature roasts before shipping—these aren’t just rituals, they’re guardrails for maintaining quality without micromanagement. The goalie rotation for handling unplanned work is clever. And honestly? The 10-year equity exercise windows and sabbatical provisions show they’re thinking long-term about retention. This is what remote-first looks like when you actually commit to it, not just tolerate it.
Engineers who won’t commit force bad decisions — This called me out a bit. The argument is sharp: when senior engineers stay non-committal in technical discussions, they’re not being careful or humble—they’re shifting the burden to less-informed teammates. Either junior devs end up guessing, or the loudest voice wins by default. The root cause? Fear of being publicly wrong. But here’s the thing: managers expect some calls to fail, especially on genuinely hard problems. The threshold is simple—if you have more context than others in the room, speak up. Caveats and hedging just create friction. It’s uncomfortable but necessary: taking a position, even with uncertainty, moves the team forward.

2025W51

Sun, 28 Dec 2025 12:32:09 GMT

How uv got so fast — This one hit different because everyone’s been saying “uv is fast because Rust” but the real story is way more interesting. It’s fast because of what it doesn’t do—no legacy .egg support, no bytecode compilation, no permissive spec violations. The timing was perfect: PEP 658 landed on PyPI in May 2023, giving direct access to package metadata, and uv launched February 2024. Honestly? Most of the speedups (parallel downloads, global cache, the PubGrub resolver) could be added to pip without touching Rust. The language helps with zero-copy deserialization and threading, sure, but the real win is having the courage to say “we’re not supporting that old stuff anymore.” It’s a reminder that sometimes architectural decisions matter way more than implementation language.
My role as a founder CTO: Year Eight — Miguel got offered $500M for RevenueCat and turned it down. That decision alone makes this worth reading, but what I appreciate most is how raw he is about the oscillation between conviction and doubt. His wife told him to keep going, viewing it as their shared legacy, and that reframed everything. The rest is a founder CTO doing founder CTO things at scale—50 flights, still doing 40 interviews a year because hiring is “the highest leverage activity,” creating this Office of the CTO team for zero-to-one work. He admits his three biggest mistakes openly: wasting energy on a VP of Engineering search, letting hiring velocity stall mid-year, and moving people to new initiatives before stabilizing existing ones. What stuck with me though is that after all this growth and near-exit, he’s still fundamentally a builder who can’t imagine stopping. That kind of clarity about who you are is rare.
Package managers keep using git as a database, it never works out — The pattern is almost comical at this point: use git as your package registry because it’s convenient and free, watch it break spectacularly as you scale, then quietly migrate to actual HTTP APIs. Cargo had users stuck on “delta resolution” forever, Homebrew’s .git folders hit 1GB, CocoaPods took minutes just to clone. The best part? They all solved it the same way—keep git internally for governance workflows, but serve metadata over HTTP/CDN to users. What I appreciate about this piece is how clearly it explains why git fails: it’s missing CHECK constraints, UNIQUE constraints, proper locking, query indexes, all the things actual databases have. Plus filesystem limits like Windows’ 260-character path restriction and case-sensitivity mismatches. It’s a textbook case of picking a tool that solves your immediate problem (version control + hosting) while ignoring what you actually need (a queryable database with performance guarantees).
Maybe the Default Settings Are Too High — David Cain reads Lord of the Rings out loud at triple-slow pace, pausing after commas, and gets more out of it than speed-reading ever gave him. Same with eating—half speed, smaller portions, more pleasure. The insight that landed for me: when we rush to get to the “good stuff” faster, we actually guarantee we’ll miss it entirely. Our sensory systems need time to propagate the experience, but modern life has conditioned us toward empty, surface-level rewards because we treat everything as disposable. The paradox is brutal: we have infinite books and snacks available, so we unconsciously devalue each one by consuming it too quickly. Slowing down doesn’t just make experiences richer, it naturally redirects you toward substantive things because cheap chocolate and TikTok videos don’t reward patience. This meshes with something I’ve been feeling about how I read technical articles—scanning for bullet points instead of letting ideas settle.
Faster Rust builds on Mac — I’ve been hitting this without realizing what it was. Every build script and test binary triggers XProtect’s malware scan the first time it runs, and since Rust compiles fresh binaries constantly, you’re basically waiting for a single-threaded security daemon to approve every executable. The author shows build scripts going from 3.88 seconds down to 0.14 seconds, and a test suite dropping from 9m42s to 3m33s, just by adding Terminal to the developer tools list. That’s a massive win for a one-time settings change. The honest trade-off discussion is what I appreciate here—this disables an OS security feature, so you’re choosing speed over protection. For personal dev machines where you control what code you’re running, that’s probably fine. For shared or work machines, maybe not. Either way, knowing this exists beats suffering through slow builds and blaming Rust.
Write to escape your default setting — This connects to the previous article about slowing down, but focused on thinking instead of consuming. Our minds operate in “perpetual approximation mode”—jumping between shiny fragments, never settling long enough to go deep. Writing breaks that pattern by forcing you to create coherence on paper, which immediately exposes the gaps between what you think you know and what you actually understand. The Francis Bacon line hits: “reading maketh a full man… writing maketh an exact man.” What I appreciate is the permission to write fast and sloppy—the point isn’t polished prose, it’s getting the muddy bottom of your thoughts visible so you can examine it. This is why I keep coming back to writing notes and posts even when it feels inefficient. It’s not about producing content, it’s about extending my working memory beyond what I can hold internally and discovering what I actually believe.
You’re Not Burnt Out. You’re Existentially Starving. — Neil Thanedar argues that burnout is often misdiagnosed—what we’re actually experiencing is Viktor Frankl’s “existential vacuum,” a profound absence of purpose despite material comfort. The argument cuts through the hustle culture vs. anti-hustle binary: working 100+ hours a week isn’t inherently problematic if those hours align with genuine purpose, but 40 hours of meaningless tasks will destroy you. What resonates is his challenge to reconnect with childhood dreams before self-doubt kicked in, then build your entire life around that direction rather than just optimizing leisure time. His own story—abandoning astronaut/president dreams in middle school, spending 15+ years in tech, finally embracing political engagement—shows this isn’t about sudden epiphanies but deliberate realignment. The “start small” advice matters: volunteer one hour weekly for something you believe in, don’t wait for perfect timing. Honestly? This reframes my own frustrations with leadership work as potentially a purpose problem, not a workload problem.
You are not the code — Txus spent weeks building an elegant Clojure query language for power users, demoed it to his manager Manish, got told not to ship it, and then just… deleted the branch. The relief he felt in that moment revealed something profound: he’d been fusing his self-worth with his code output, making every criticism feel personal and every failure diminishing. The realization that “the code is an artifact, a byproduct of an ongoing process” freed him—the two weeks weren’t wasted because the learning remained even after the deletion. This hits hard for anyone who’s had technically brilliant work rejected for product/team alignment reasons. The maturity here is recognizing that being right about the technical solution doesn’t mean it’s the right solution. Your value comes from growth and capability, not from lines of code surviving in the repo. I’ve definitely struggled with this when features I built got deprecated or rewritten, feeling like it invalidated the work, when actually it just meant we learned enough to do something better.

166 hours

Sat, 27 Dec 2025 16:00:00 GMT

Last year I wrote about hitting 130 hours of working out. I ended that note with a line I repeated three times: “I’m going to keep making progress.”

I didn’t know if I’d actually do it. Part of me wondered if writing it down was just another way of setting myself up for disappointment.

This year was rough. The kind of year where everything important changed in the span of a week, and the months after felt like trying to find solid ground that kept shifting.

I still managed to work out for 166 hours. Still around 27 minutes a day. Still not impressive numbers.

But here’s what mattered: when everything else felt like it was falling apart, this was something I could control. I could lace up my shoes. I could show up. And on the days I did, it was the one thing that felt like forward motion.

None of these are numbers that would impress anyone. But they’re mine, and they represent something harder than any PR: consistency when it would’ve been easier to stop.

Life changes fast. Appreciate the people you have around you while you have them. And don’t waste your time with people who don’t deserve it. These aren’t fitness lessons, but they’re what this year taught me while I was trying to keep showing up.

Next year, I’ll keep making progress. Not because I have to, but because I’ve proven to myself that I can.

2025W50

Sun, 21 Dec 2025 12:30:54 GMT

Thin Desires Are Eating Your Life — This hit different. The distinction between thin desires (scrolling, checking notifications) and thick desires (learning something hard, building real skills) explains so much about why we feel empty despite having instant access to everything. What really stuck with me: tech companies have figured out how to extract the dopamine hit from meaningful activities while stripping away the transformation. Social media gives you the reward of connection without any of the depth. The best part? The author doesn’t prescribe some grand solution—just small acts of resistance like baking bread or writing actual letters. Things that refuse to be optimized, that force you to slow down. Honestly? That feels more achievable than trying to quit everything cold turkey.
Go ahead, self-host Postgres — I appreciate how blunt this is about the cloud narrative. The author’s been running self-hosted Postgres for two years, serving millions of queries daily, and spends about 30 minutes a month on maintenance. Meanwhile AWS charges $328/month for what’s basically vanilla Postgres with some operational tooling wrapped around it. What convinced me most? The actual numbers: migrating from RDS took 4 hours of work, performance was identical, and the operational burden is genuinely minimal if you’re comfortable with the basics. Not saying managed services are never worth it—startups moving fast or enterprises with compliance needs make sense—but for everyone else paying those cloud premiums? You might be solving a problem you don’t actually have.
This is not the future — This felt like someone finally saying what I’ve been thinking. The premise is simple but powerful: nothing is inevitable just because a tech company with billions says it is. We’ve been trained to accept every new “innovation”—AI features shoved into everything, unrepairable devices, surveillance masquerading as convenience—as if it’s the natural progression of technology. But it’s not. Every choice is political, every adoption is a vote. What I appreciate most is the refusal to be passive. The author gives you a whole list of things that aren’t actually necessary (internet-connected toasters, subscription car features, mandatory accounts for basic functions) and basically says: you don’t have to accept this. Choose tools that respect you. It’s a reminder that we still have agency, even when companies pretend we don’t.
Disks Lie: Building a WAL that actually survives — The title nails it: your storage stack is absolutely lying to you. This breaks down all the ways a naive write-ahead log will fail in production—data sitting in kernel buffers pretending to be persisted, silent bit flips, operation reordering, the works. What I found most valuable is the five-layer defense strategy: checksums to catch corruption, dual WAL files on separate disks (because latent sector errors are common enough that a single copy is “negligence”), O_DIRECT + O_DSYNC to bypass buffering entirely, io_uring with linked operations for ordering guarantees, and post-fsync verification reads because you genuinely cannot trust that your data made it to disk. It’s paranoid, sure, but production durability demands paranoia. The storage stack will not tell you the truth about whether your data is safe.
Lightweight Cardinality Estimation with Density — This is one of those pieces that makes you appreciate how much thought goes into query optimization. Density is just 1/ICARD (the reciprocal of distinct values), but it’s surprisingly useful for helping query planners decide which index to use. The insight that stuck: for uniformly distributed data, density tells you roughly how selective an equality predicate will be, and comparing densities across multiple columns reveals correlations in your data. What I appreciate most is the practicality argument—unlike histograms or other complex statistics, density is cheap to compute and maintain at scale, even using approximations like HyperLogLog. It’s not perfect, but it’s a valuable tool relative to its cost. Sometimes the simple heuristic that you’ll actually use beats the sophisticated one that’s too expensive to track.
Zed Is Our Office — The audacity of this vision is what gets me. They’re not just adding collaboration features to an editor—they’re arguing that the editor itself should be your office. Built from scratch with CRDTs for seamless real-time editing, zero-friction setup (just GitHub auth), built-in audio and screenshare that automatically follows who’s talking. What really sells it? They’re using Zed to run their own company. Not as a demo, but as their actual workspace—channels for company-wide discussions, projects, personal focus. The ambition here is that code, conversations, and context all live in the same place, accessible to both teammates and AI. Most collaborative editors feel bolted-on and end up forcing you back to Slack or Zoom. This feels like a genuine rethink of what a development environment could be if collaboration was fundamental, not an afterthought.
Leaving Meta and PyTorch — There’s something quietly powerful about leaving at the peak. Soumith led PyTorch from inception to 90%+ adoption across the AI industry, and now he’s walking away from “one of the AI industry’s most leveraged seats” to do something small and uncomfortable. What struck me most is the self-awareness: he couldn’t ignore the counterfactual regret of never trying something outside Meta. The reflection on institutional strength resonates too—PyTorch doesn’t need him anymore, the team can solve problems without him, and that’s actually the sign of success. It’s a reminder that sometimes the right move is stepping away precisely when you’ve built something that can thrive without you. The curiosity to start over, to be a beginner again, even when you’ve already won? That takes guts.
The Talent Machine — The core reframe here is brilliant: stop thinking of hiring as filtering applicants and start thinking of it as selling a product. Define what you’re actually offering—not just the tech stack, but the culture, the mission, the compensation structure—and position it to match what specific candidates want. What really clicked for me is the emphasis on transparency as a competitive advantage. Public employee handbook, fixed salaries instead of negotiable ranges, clear career paths. Most companies treat this stuff like trade secrets, but being radically transparent eliminates friction and self-selects for cultural fit. The tactical advice about targeting underserved demographics (experienced engineers in their 30s-40s looking for stability) and choosing less-saturated tech stacks to reduce competition? That’s the kind of strategic thinking most companies miss. Hiring isn’t about having the biggest budget, it’s about knowing exactly who you’re for and making it dead simple for them to say yes.

2025W49

Sun, 14 Dec 2025 12:31:05 GMT

SQLite JSON Superpower: Virtual Columns + Indexing — This is one of those “why didn’t I know about this sooner?” moments. The idea is simple but powerful: dump your JSON into SQLite, create virtual columns that extract specific fields on the fly, then slap indexes on those virtual columns. Boom, you’ve got schemaless flexibility with relational performance. No backfilling, no migration headaches. You can literally add a new indexed column later without touching your data. The beauty is in how it solves the eternal “should I go schemaless or relational?” debate by just saying “both.”
Recortes 2025 — Javi’s year-end reflections hit different. It’s refreshing to read someone being this honest about the mental toll of being CEO while also celebrating running a 10k at 43. The quote about not being able to compete with someone having fun keeps bouncing around in my head. There’s something profound about acknowledging that working hard without questioning what you should actually work on is just lazy thinking with extra steps. Also, keeping a diary for his daughter to understand his life context later? That’s the kind of long-term thinking I respect.
Litestream VFS — This genuinely blew my mind. You can query SQLite databases sitting in S3 without downloading the whole thing. Just fetch the pages you need using S3 range requests and an LRU cache. Point-in-time recovery is just a SQL pragma away. The LTX format compacts page versions so you’re not storing redundant data. This solves so many “I just need to quickly check production” moments without spinning up a whole restore dance. The technical elegance here is top-notch.
Craft software that makes people feel something — Raphael built Boo, a code editor, mostly for himself because he wanted something with soul. His point about repetition killing creativity resonates hard. When coding becomes mechanical, you lose the spark that makes people go “wow.” What I appreciate most is his willingness to build something not commercially viable just because it matters to him. In an era obsessed with PMF and growth metrics, building for joy feels almost radical. The comparison to Lego is spot on too - the best software feels like play.
The Long Run — Miguel went from being a non-runner to completing a full marathon, running through 18 cities along the way. But this isn’t really about running. It’s about rewriting your internal script. Once you do something you thought was impossible, the ceiling lifts everywhere else. “Discipline compounds quietly, until suddenly you’re a different person” is the kind of truth you only understand after experiencing it. The best part? He doesn’t frame it as inspiration porn, just an honest account of how doing hard things changes what you think you’re capable of.
Transparent Leadership Beats Servant Leadership — The distinction here is subtle but important. Servant leadership can accidentally create dependency where you’re constantly clearing paths for people. Transparent leadership is about teaching people to clear their own paths. You coach, you connect folks directly, you explain the “why” so they can make good calls independently. The goal is to become redundant by leveling everyone up. I’ve seen too many managers become bottlenecks disguised as helpful. This framework gives language to why that feels wrong and what to do instead.
Why I Ignore The Spotlight as a Staff Engineer — Not all impact comes from high-visibility projects. Some of the most valuable work is being the long-term steward of critical systems nobody thinks about until they break. The trade-off is less external glory, but you get to build deep expertise and solve genuinely complex problems. The “shadow hierarchy” concept is fascinating - getting recognition through customer team endorsements rather than flashy launches. “The most ambitious thing you can do is stay put, dig in, and build something that lasts” flips the typical career advice on its head, and honestly? It’s probably right more often than we admit.
Delta Lake Atomicity — A solid technical breakdown of how Delta Lake gets atomicity without traditional WAL. Instead of page-level recovery, it uses copy-on-write: write new Parquet files, then atomically write a JSON log entry pointing to them. If something crashes mid-flight, the uncommitted files just sit there invisible since they’re not in the log. It’s clever because object storage guarantees single file write atomicity, so you piggyback on that rather than building your own transaction log infrastructure. Clean, simple, and it works.

64-bit Misalignment

Mon, 17 Nov 2025 16:00:00 GMT

I’ve been following CMU’s 15-445/645 Database Systems course lately, trying to fill some gaps in my low-level systems knowledge. When the lectures covered memory alignment, it all made perfect sense. Cache lines, multiple memory operations, hardware penalties. Textbook stuff.

But here’s the thing: I don’t trust theory until I’ve seen it break something myself.

So I decided to write a benchmark. Simple plan: create two structs (one with natural alignment, one tightly packed) and demonstrate the performance penalty everyone talks about. I’d watch the misaligned version choke, nod knowingly, and move on to the next lecture.

Spoiler alert: The misaligned version won.

What is 64-bit Memory Alignment?

Before we get to the weird results, let me explain what I thought I understood. Modern processors prefer (or require) that data be aligned to 64-bit (8-byte) boundaries.

Why does alignment matter? When data is properly aligned, the CPU can read or write it in a single memory operation. When misaligned, several things can happen depending on the architecture¹:

Multiple memory operations: Reading a misaligned uint64_t might require two separate memory reads instead of one. For example, if an 8-byte value starts at address 1, it might span two different memory access boundaries, requiring the CPU to:
- Read bytes 0-7 (getting bytes 1-7 of our value)
- Read bytes 8-15 (getting byte 0 of our value)
- Combine the pieces with bit-shifting and masking
Non-atomic operations: Even more critically, misaligned writes are not atomic. A properly aligned 8-byte write happens as a single atomic operation, but a misaligned write might be implemented as:
- Read-modify-write of the first memory chunk
- Read-modify-write of the second memory chunk

This means another thread could observe a partially-written value, making misaligned access dangerous in concurrent code without additional synchronization.

Compilers automatically add padding to structs to maintain these alignment requirements and avoid these issues. Here’s a simple example:

// NORMAL struct - compiler adds padding for alignment
struct NormalStruct {
    uint8_t  a;      // 1 byte
    uint64_t b;      // 8 bytes (aligned at offset 8)
    uint16_t c;      // 2 bytes
    // Total: 24 bytes (with padding)
};

// PACKED struct - no padding, fields stored consecutively
struct __attribute__((packed)) PackedStruct {
    uint8_t  a;      // 1 byte
    uint64_t b;      // 8 bytes (misaligned at offset 1)
    uint16_t c;      // 2 bytes
    // Total: 11 bytes (no padding)
};

The memory layout looks like this:

NormalStruct (24 bytes):
  Byte:  0  1  2  3  4  5  6  7  8  9  10 11 12 13 14 15 16 17 18 19
  Field: [a][- padding (7 bytes) -][b  b  b  b  b  b  b  b][c  c][pad]

PackedStruct (11 bytes):
  Byte:  0  1  2  3  4  5  6  7  8  9  10
  Field: [a][b  b  b  b  b  b  b  b][c  c]

The packed version saves 13 bytes per struct (54% reduction), but the uint64_t b field is now misaligned, starting at offset 1 instead of 8.

The Plan

The plan was straightforward. I’d benchmark these two structs performing various operations and watch the misaligned version suffer. I tested on the two different architectures I have at home:

Apple M1 Pro (ARM64)
AMD Ryzen 7 8745HS (x86-64)

I created four benchmark tests:

Sequential read of the uint64_t b field
Random read of the uint64_t b field
Sequential write to the uint64_t b field
Read-modify-write operations

Each test ran on arrays of 100,000 elements with 1,000 iterations, repeated 10 times for statistical reliability.

First Results: Wait, What?

Here are the initial results on x86-64 (AMD Ryzen 7 8745HS):

Test                   Aligned (ms)  Misaligned (ms)  Difference
-----------------------------------------------------------------
Sequential Read              28.38            26.10      -8.0%
Random Read                  56.90            69.96     +22.9%
Sequential Write             20.27            14.45     -28.8%
Read-Modify-Write            21.53            20.71      -3.8%

And on ARM64 (Apple M1 Pro):

Test                   Aligned (ms)  Misaligned (ms)  Difference
-----------------------------------------------------------------
Sequential Read              97.22            97.21      -0.0%
Random Read                 211.00           210.11      -0.4%
Sequential Write             30.47            22.22     -27.0%
Read-Modify-Write            39.42            40.80      +3.5%

I stared at these numbers for a long time. The misaligned packed struct was performing almost the same (if not better) than the aligned version. This made no sense. Where were my alignment penalties?

Nothing Made Sense

I double-checked everything:

Checked if some compiler flags were enabled and avoiding the alignment penalties
Warmup iterations
Statistical analysis
Memory initialization
Reviewed the theory and had a long conversation with Claude

The code was correct. The results were real. But they didn’t match my expectations at all.

Then Cache Hit Me

I calculated the actual memory footprint of my test arrays:

100,000 elements:

NormalStruct: 100,000 × 24 bytes = 2,343 KB
PackedStruct: 100,000 × 11 bytes = 1,074 KB

And then I checked the CPU cache sizes:

AMD Ryzen 7 8745HS (per core):

L1 Data Cache: 32 KB
L2 Cache: 1 MB (1,024 KB)

Apple M1 Pro (per core):

L1 Data Cache: 64 KB
L2 Cache: 4 MB

The packed struct’s smaller memory footprint meant better cache utilization. This advantage was outweighing any misalignment penalties!

Proving It

I needed to test this properly. I created a new benchmark that tested different array sizes to see where the performance crossover happens:

void test_array_size(size_t count, size_t iterations) {
    vector<NormalStruct> normal_data(count);
    vector<PackedStruct> packed_data(count);

    // ... initialization and warmup ...

    double normal_time = benchmark_sequential_read(normal_data, iterations);
    double packed_time = benchmark_sequential_read(packed_data, iterations);

    // Calculate and print results
}

I tested array sizes from 100 elements (fits comfortably in L1) up to 200,000 elements (way beyond L2), adjusting iterations to keep total runtime reasonable.

The Pattern

Here are the results on x86-64 (AMD Ryzen 7 8745HS):

Array Size    Normal (KB)    Packed (KB)     Difference
--------------------------------------------------------
       100           2              1            +2.8%
     1,000          23             10            +1.5%
     5,000         117             53            -5.4%
    10,000         234            107            -5.9%
    30,000         703            322            -6.6%
    50,000       1,171            537            -8.8%
   100,000       2,343          1,074            -8.7%
   200,000       4,687          2,148            -8.2%

The pattern is crystal clear:

Tiny arrays (100-1,000 elements, both in L1): Aligned wins
- This is the “true” alignment penalty, minimal in modern CPUs
Growing arrays (5,000-30,000): Packed advantage grows
- Better cache utilization starts to dominate
The critical point (~50,000 elements):
- NormalStruct (1,171 KB) exceeds L2
- PackedStruct (537 KB) still fits in L2
- Packed is 8.8% faster despite misalignment!
Beyond L2: Packed maintains ~8% advantage
- Both exceed cache, but packed uses less memory bandwidth

What About the 64-bit Alignment Penalty?

After all this, I still needed to see the actual alignment penalty without cache effects getting in the way. So I re-ran the benchmarks with just 100 elements, small enough to fit entirely within L1 cache. ²

x86-64 (AMD Ryzen 7 8745HS)

Test                   Aligned (ms)  Misaligned (ms)  Difference
-----------------------------------------------------------------
Sequential Read                2.37             2.45      +3.3%
Random Read                    3.33             3.70     +11.3%
Sequential Write               1.52             1.93     +26.7%
Read-Modify-Write              2.07             2.11      +2.0%

ARM64 (Apple M1 Pro)

Test                   Aligned (ms)  Misaligned (ms)  Difference
-----------------------------------------------------------------
Sequential Read                9.80             9.76      -0.4%
Random Read                   20.82            20.81      -0.1%
Sequential Write               1.59             1.66      +5.0%
Read-Modify-Write              1.94             3.54     +82.7%

These results are exactly what we expected to see. With a small array of 100 elements that fits within L1 cache, we finally observe the alignment penalties that theory predicts. The misaligned packed struct shows consistent slowdowns ranging from 2% to 82%, demonstrating the real computational cost of accessing data that doesn’t sit on natural memory boundaries.

The small dataset isolates the true alignment penalty from cache effects. This is the benchmark that confirms what everyone says about alignment, when cache isn’t a factor, misaligned access is indeed slower.

What I Learned

I started this to prove I understood something “everyone knows”. Instead, I got to see how different effects interact in practice.

The alignment penalties are real, I saw them clearly with small datasets that fit in L1 cache. But in my benchmarks with larger arrays, something else happened. The packed struct’s 54% smaller memory footprint meant better cache utilization, and that advantage outweighed the misalignment cost. The “slower” approach became 8-9% faster.

This wasn’t about the textbooks being wrong. It was about understanding that performance is about trade-offs, not rules. Alignment matters. Cache efficiency matters. Which one dominates depends on your specific workload, data size, and access patterns.

Here’s what watching these effects interact taught me:

Initial hypotheses can be wrong: I expected one thing, measured another
Small benchmarks and large benchmarks can show opposite results
“Common knowledge” is often incomplete without context
Counterintuitive results are invitations to dig deeper

The real lesson isn’t about memory alignment. It’s about staying curious when your data surprises you. It’s about measuring instead of assuming. It’s about being willing to investigate when something doesn’t match what you thought you knew.

I wanted to prove I understood how memory worked. Instead, I got to see multiple effects competing with each other. And that taught me more than being right ever could have.

Update [2025-11-18]: After sharing the blogpost in Software Internals Discord, I received good feedback about reviewing performance counters to confirm my findings.

I proceeded then to add a wrapper using perf_event_open to measure cache misses, and observed that the cache misses were significantly reduced when the data was packed due to the smaller size. This confirmed the hypothesis that packed struct was compensating misalignment penalties due to the extra performance trade-off from better cache usage.

Comparing cache misses between normal and packed structs for different array sizes, we can observe the following results:

Array Size    Normal (KB)    Packed (KB)     Difference     L1 Miss(N)     L1 Miss(P)
--------------------------------------------------------------------------------------
       100           2              1            +2.8%        0.00%           0.00%
     1,000          23             10            +1.5%        0.10%           0.00%
     5,000         117             53            -5.4%       18.77%           8.16%
    10,000         234            107            -5.9%       18.77%           8.16%
    30,000         703            322            -6.6%       18.79%           8.16%
    50,000       1,171            537            -8.8%       18.74%           8.16%
   100,000       2,343          1,074            -8.7%       18.75%           8.15%
   200,000       4,687          2,148            -8.2%       18.73%           8.15%

So running the 64-bit performance test for 100,000, the original test I ran, it’s clear that cache misses on the aligned data are significantly higher making the misaligned structure more efficient in almost all cases.

Test                   Aligned (ms)  Misaligned (ms)  Difference    Misses (A)     Misses (M)
----------------------------------------------------------------------------------------------
Sequential Read              28.38            26.10      -8.0%       18.77%          8.17%
Random Read                  56.90            69.96     +22.9%       36.99%         40.87%
Sequential Write             20.27            14.45     -28.8%       37.69%         15.53%
Read-Modify-Write            21.53            20.71      -3.8%       15.80%          7.77%

In the discord channel, Phil Eaton also shared an interesting discussion about the very same topic that happened 4 months ago in: https://lobste.rs/s/plrsmw/data_alignment_for_speed_myth_reality

There is also another misalignment to consider when talking about memory alignment. Probably more important than the one being discussed here. The cache line misalignment is a performance issue that occurs when a data structure crosses a cache line boundary (usually 64 bytes). The implications of this misalignment are similar to the ones discussed here but at another level. ↩
Code available here: https://github.com/jrdi/playground/tree/main/memory-alignment ↩

2025W38

Sun, 28 Sep 2025 12:27:37 GMT

Surviving Long-Running Projects — A nice reality check on how grueling big projects can be. The author basically says: expect them to drag on, plan for “boring” maintenance, and set up routines to keep your motivation and mental health intact. It’s more about staying sane than about technical tricks.
How we built it: Real-time analytics for Stripe Billing — This is a cool peek under the hood. Stripe walks through how they built real-time analytics on top of billing, mixing Kafka, Flink, and their own in-house magic. It’s a story about scaling data pipelines to billions of events and still keeping dashboards snappy.
9 Things Every Fresh Graduate Should Know About Software Performance — Basically “performance isn’t magic.” It covers the usual suspects (memory allocation, cache, I/O, concurrency) but framed as a gentle guide for juniors. Solid reminders even for experienced devs.
A brief history of threads and threading — A fun historical tour of how we got to modern threading. From early OS days to today’s multicores, it’s an eye-opener on how messy and evolutionary concurrency really is.
Hired Through GitHub: Part 1 — The author tells how showing work on GitHub led directly to job offers. It’s a feel-good story about the value of public contributions and writing about your code.
In response to a developer asking about systems — This is like a mini-manifesto on what “systems” really means. The takeaway is: stop chasing trendy tools and learn fundamentals — OS, networks, distributed systems. That’s how you actually “get” systems.
From 19k to 4.2M events/sec: story of a SQLite query optimisation — Absolute banger. The author squeezes insane performance out of SQLite with indexing, batching, and clever SQL tweaks. Great reminder that small changes in schema/queries can be game-changing.
UTF-8 is a Brilliant Design — A love letter to UTF-8. The author explains why it’s elegant, backwards-compatible, and just plain clever. Makes you appreciate how much thought went into something we all take for granted.
Being good isn’t enough — A short but punchy post: skill alone won’t get you ahead. Communication, visibility, and timing matter too. Basically: don’t hide behind code.
Introducing Direct Compress: Up to 40x Faster, Leaner Data Ingestion for Developers (Tech Preview) — Marketing-y but interesting. A new ingestion tech promising 40x faster loads. The takeaway: data infra vendors are pushing more on-the-fly compression to cut storage and speed up analytics.
When more threads make things worse — A cautionary tale of over-threading. Past a point, adding threads doesn’t help — it can create livelocks, context switching overhead, and general sadness. A good read if you’ve ever said “let’s just throw more threads at it.”
You can try to like stuff — An odd but uplifting essay about cultivating taste. The author argues that liking things is a skill you can practice, not just something that happens. Kind of refreshing.
Reflections on Tinybird — A personal reflection from someone using Tinybird. They talk about what impressed them, what didn’t, and lessons learned from building real-time APIs. Interesting outsider’s perspective.
Do the simplest thing that could possibly work — Classic XP wisdom retold. The post champions simplicity over cleverness. Don’t over-engineer; just do the simple thing first and iterate.
Developer’s block — Like writer’s block, but for coders. Talks about burnout, procrastination, and ways to get unstuck. Feels relatable if you’ve ever stared at your editor unable to start.
The First Time I Was Almost Fired From Apple — Great storytime. An engineer recounts a near-disastrous mistake at Apple and what it taught them about responsibility and learning under pressure.
Linear sent me down a local-first rabbit hole — A cool piece about discovering local-first architectures (like CRDTs) after using Linear. Shows how design decisions in apps can spark curiosity about deeper tech.
Local-first search — Builds on that same local-first theme. Explains how search can work offline-first with sync happening in the background. Makes you rethink how “search” has to be architected.

2025W31

Fri, 08 Aug 2025 15:45:13 GMT

No, AI is not Making Engineers 10x as Productive — AI does not make engineers ten times more productive because much of coding involves thinking, reading, and preventing unnecessary work. AI helps with small tasks and gives short bursts of speed, but it often encourages haste and over-building. True productivity comes from experience, good habits, and teamwork, not just using AI tools.
All the cool kids are doing it — The author is unsure about using large language models (LLMs) for coding because they find them unreliable and costly. They feel LLMs don’t yet help much with complex or performance-focused programming tasks. However, they see some promise in LLMs for other uses like research help and code explanation.
The Sunday Morning Post: Why Exercise Is a Miracle Drug — Exercise improves many parts of the body and is better than any medicine. Studies show exercise helps cancer patients live longer and stay healthier.
Full-breadth Developers — Full-breadth developers who combine technical skills with product sense are thriving with new AI tools. Many companies struggle because they separate product and engineering roles too strictly.
Ramblings — Remote teams of 2-10 people can use personal “ramblings” channels to share thoughts without cluttering group chats. These channels help team members stay connected with short, informal updates and encourage creativity. Ramblings reduce interruptions and support social bonding, especially when no meetings are scheduled.
AI is eating the Internet — The Internet used to offer free content funded by ads from companies like Google and Facebook. Now, AI tools reuse content without sending visitors to original sites, hurting creators and changing how the web works. In the future, AI will control access to information, balancing ads, content makers, and users in new ways.
Slow — Some problems take humans many years or even centuries to solve, like building cathedrals or proving complex math theorems. Long-term projects need strong institutions and steady effort to keep going over time. Examples include scientific studies, historic buildings, and technology that may last for thousands of years.
Fast — Fast software changes how we work and feel by making tasks quicker and easier. It requires focus and simplicity, often cutting unnecessary features to deliver speed. As current LLM technology improves, speed will unlock new possibilities and transform our lives like never before.
Why I’m stepping down as Head of Engineering — The author is stepping down as Head of Engineering to focus on hands-on technical work they love. As the company grew, their role became more about management, which made them feel less connected and impactful. Stepping aside allows new leaders to scale the organization while the author contributes where they add the most value.
My favorite use-case for AI is writing logs — JetBrains created a small, fast AI model that helps developers write better log lines automatically in PyCharm. This model runs locally on your computer, making coding easier and debugging faster. It shows that focused, lightweight AI can be very useful alongside big, general models.
How magnesium affects your sleep and anxiety — Magnesium may help reduce mild anxiety but does not strongly improve sleep for most people. Many people do not get enough magnesium from their diet, and supplements can be useful, especially magnesium glycinate.
Coding with LLMs in the summer of 2025 (an update) — Large language models like Gemini 2.5 PRO help programmers by finding bugs, speeding up testing, and supporting design decisions when humans guide them carefully. To get the best results, programmers must provide clear context and stay actively involved in the coding process instead of relying on AI alone. Right now, working together with AI is more effective than letting AI work solo, but this may change in the future.
Why me? — When bad things happen, we often ask “Why me?” and feel singled out. But life brings both good and bad luck to everyone, and neither is guaranteed. Accepting this helps us feel less angry and more grateful for the good moments.
MCP: An (Accidentally) Universal Plugin System — I used to think MCP added nothing over APIs. But this post clicked: APIs exposes services, MCP wraps actions, there are mores actions than calling an HTTP service: CLI, scripts, anything runnable. It’s a universal interface for doing, not just calling. Surprisingly powerful.
Making Sense of a Noisy World — Trends come and go, but sharing thoughtful opinions helps us learn and grow. Honest views can challenge ideas and create meaningful conversations. Careful thinking makes our opinions valuable and helps us understand the noisy world.

Am I Becoming Irrelevant?

Mon, 14 Jul 2025 00:00:00 GMT

It feels silly to write just another blogpost about how to transition to a leadership role.

Hopefully this is not a post about how to succeed in that transition. This is about how I felt every time I’ve been promoted to a leadership role. And how I haven’t been able to describe the frustration I felt.

I’m writing it down (and publishing it) as a way to process it.

I Used to Feel Relevant

It usually starts with a few years into a company. I got things done. I shipped features that mattered. Some were technically difficult, others politically delicate, and a few simply fell apart and had to be rebuilt from scratch. I knew the product inside out. I could speak the language of our users. People came to me when they needed hard problems solved.

Now I manage a team.

Some days, I don’t know what I do anymore.

My worth was defined by my output:

Merged PRs
Shipped features
Firefights I could handle on my own
Recognition that was public and immediate

Now, I spend most of my time trying to protect the team’s focus, say no to things that don’t matter, and convince smart people to work on the boring-but-important stuff.

It doesn’t feel like impact. It feels like being a buffer. A gatekeeper. A middle-layer.

I miss investing real, focused time into one hard thing. I miss the feeling of flow, of closing my laptop after a long day and knowing exactly what I achieved. And honestly? I miss being openly recognized.

Now, when something goes right, it’s because the team succeeded. That’s good, I want that. But part of me also wonders if anyone notices the invisible work that went into it.

What Am I Actually Doing All Day?

Sometimes I ask myself this in a genuine, not sarcastic, way.

I know I’m doing things that matter. I can point to them:

I filter out noise before it ever reaches the team.
I say no to low-leverage requests, including from people who outrank me.
I help, and clear the fog so others can move fast and make good calls.
I absorb tension between what others want, what the team needs, and what the product should become.

But none of that looks like a roadmap item. None of it gets demoed at the company all-hands.

There’s a fear I don’t say out loud often, but I’ll write it here:

I’ve seen managers get fired quietly for “not contributing” I don’t want to be one of them.

So when I feel underused, or like the value I bring isn’t visible, I wonder if I’m just one reorg away from being irrelevant.

I keep telling myself that enabling others is impact. That saying “no” is a kind of strategy. But some days that feels like a story I’m telling to make myself feel better. And on other days, it feels completely, absolutely true.

What I Know Is Working

Even with all this ambiguity, there are things I’m proud of:

The team is focused. They’re not jumping between shiny distractions. Not always at least.
They ship. Not chaotically, but deliberately, and with confidence.
I’ve helped them grow into harder challenges, and they are.

I’m not hands-on in the code, but I’m hands-on in the conditions that allow good code to happen.

Still, I’m figuring out:

How to measure success in this role in a way that means something to me.
How to tell the story of my impact without needing applause.
How to say no in a way that builds trust, not distance.
How to be okay with the fact that the more effective I am, the less visible it often is.

Why I’m Writing This

As I said, I’m writing this for myself. I’ve ended up leaving a few companies because of this feeling. It’s time to stop running away from it.

But I’m also writing this for you. Maybe you’re feeling this too. Maybe you’re not.

Maybe you used to be the go-to builder, and now you spend your time in meetings wondering if you’re just in the way. Maybe you’re coaching, steering, absorbing, and worrying that no one sees it.

Maybe you’re saying “no” when it would be easier to say “yes,” and wondering if you’re going to pay a price for that.

If that’s you, I don’t have a fix. But I can say this:

You’re not irrelevant.
You’re just operating in a layer that doesn’t clap for itself.

And maybe, just maybe, the fact that your team is calm, focused, and shipping, while everything around them is chaotic, is the best evidence of your impact.

Even if no one’s naming it out loud.

2025W23

Tue, 10 Jun 2025 00:09:44 GMT

Thrive in obscurity — Creative success often starts in obscurity, with many creators spending years sharing their work with few or no viewers. To stay motivated, focus on creating what you love instead of chasing popularity, as this will lead to better work and attract like-minded fans. Treat your early content as an investment for future audiences, knowing they may return to appreciate your journey later on.
If you are useful, it doesn’t mean you are valued — In your career, being useful means you get tasks done well, while being valued means you contribute to important decisions and have growth opportunities. Useful people may receive rewards, but they often feel stagnant and lack strategic involvement. To succeed, it’s crucial to recognize whether you are truly valued or just seen as a reliable worker.
My AI Skeptic Friends Are All Nuts — The author argues that LLMs can significantly assist software developers by handling tedious coding tasks, allowing them to focus on more important work. Despite skepticism about LLM-generated code quality, the author believes that these tools can enhance productivity and reduce the need for repetitive tasks. Ultimately, while LLMs may not replace all developers, they can streamline the coding process and improve efficiency.
A Smart Bear » You’re a little company, now act like one — Small companies often fear that being perceived as small will hurt sales, but this can actually alienate their best customers. Instead of adopting generic corporate language, they should present themselves authentically to attract Early Adopters who appreciate personal connections and are willing to provide feedback. By being honest and relatable, small companies can foster relationships that help them grow and improve their products.
Letter to Arc members 2025 — The Browser Company is shifting focus from Arc to a new product called Dia, aiming to create a better browser experience. They recognized that Arc had limitations and wanted to build something that integrates AI more effectively.
If nothing is curated, how do we find things? — The rise of social media has made it harder to find curated content, leading to information overload and mental exhaustion. Critics and curators are needed now more than ever to help sift through the vast amount of content available.
Thoughts on thinking — The author feels stuck and believes that their creative efforts are overshadowed by AI’s ability to produce better ideas quickly. They reflect on how using AI has diminished their own thinking and intellectual growth, despite having access to more information than ever.

Query Booster: How Tinybird optimizes table schemas for you

Mon, 12 May 2025 16:00:00 GMT

Original article: https://www.tinybird.co/blog-posts/query-booster-how-tinybird-optimizes-table-schemas-for-you

A customer recently reached out to us with an interesting observation: their query latency had mysteriously dropped without any manual intervention. What they discovered was our Query Booster, quietly optimizing their workload behind the scenes. Query Booster automatically adapts to your database usage patterns, fine-tuning data source schemas for optimal performance.

An example of Query Booster in action, reducing query latency from ~175 ms to <5 ms without manual intervention.

Why we built the Query Booster

Slow database queries hurt your business in several ways. When queries are slow, you need more infrastructure and resources to run them, which costs more money. Your users have a worse experience because they have to wait longer for data. Engineers waste time fixing slow queries instead of building new features. And you might miss important insights because the data takes too long to arrive.

To understand how Query Booster solves these problems, we need to understand how Tinybird works. In columnar databases, the sorting key is the most critical setting for performance. This key determines how data is physically stored on disk, how efficiently the database can filter and process queries, and serves as the default primary key and index structure.

If you are filtering by a column that is not in the sorting key, the database must perform a full table scan. This means reading significantly more data than necessary, since the column values are scattered throughout the storage. As your data volume grows, this mismatch between sorting key and query patterns leads to increasingly degraded performance due to full table scans.

In an ideal scenario, you’d know all your query patterns in advance and design the perfect sorting key. But reality is different. New use cases come up over time, query patterns evolve as your business grows, different teams access data in different ways, and performance requirements change constantly.

When this happens, you face a common issue. Your queries start performing full table scans, causing performance to degrade significantly. Engineers spend valuable time manually optimizing schemas, which requires careful planning and potential downtime. And just when you think you’ve solved the problem, new patterns emerge, and the cycle begins again.

This is where Query Booster comes in. Instead of manually tuning your database, Query Booster automatically monitors your query patterns, identifies optimization opportunities, and creates temporary optimized schemas. It continuously validates performance improvements and removes optimizations when they’re no longer needed.

The result? Your database continuously adapts to your usage patterns, maintaining optimal performance without any manual intervention.

How Query Booster works Query Booster is like having a database performance expert working 24/7 to optimize your data source schemas. Here’s how it works:

Continuous Monitoring: The system constantly watches your query patterns
Query Plan Analysis: It identifies frequently run queries and analyzes their optimization potential
Automatic Optimization: When beneficial, it creates temporary optimized data source schemas
Performance Validation: The system monitors the optimization’s effectiveness and removes it if it’s not delivering results

Let’s dive into some technical aspects that make Query Booster interesting:

Continuous Monitoring

Every query executed in Tinybird is automatically monitored and analyzed. We collect metrics that paint a complete picture of your query performance. This includes query execution time, the amount of data processed, and resource utilization. We also track query patterns, frequency, and the specific filter conditions being used. These metrics are stored in our monitoring system and made available to users through Service Data Sources, allowing you to track query performance over time.

Our solution reviews these metrics periodically to detect the queries processing the biggest amount of data and the ones being executed more frequently, as these represent critical parts of your workload.

For queries considered optimizable, we analyze the query plan, which is a detailed representation of what actions the database will perform. It contains information about the data sources being accessed, the filters that are applied, transformations, etc. In our case, we extract and analyze this query plan by parsing the EXPLAIN output. We use this information to look at whether the database is using the current sorting key effectively, whether the query is reading more data than necessary, and whether the current schema aligns with the query’s access patterns.

Query Plan Analysis

Once we have the list of queries considered as optimizable, we analyze the query plan using the EXPLAIN output. To do so, we check the following things:

Check if the query is using the primary key (usually the sorting key). If the primary key is not being used, we can directly consider the query as optimizable.
Check if the query is reading enough rows. If the query reads very few rows, it’s probably already efficient. No need to optimize further.
Check if the filtering is inefficient (low selectivity). Specifically, we check if the query reads more than 25% of the table’s available rows. If it does, that means the current sorting key is not helping much — it’s not filtering efficiently. This is a good signal that a better sorting order could help.

Let’s look at an example to better understand how we use the EXPLAIN output. For a table with sorting keys as [id_1, id_2], and a query like this:

-- GOOD QUERY

SELECT value
FROM table_1
WHERE id_1 = 863244623

The EXPLAIN output for this query would look like this:

# GOOD QUERY - EXPLAIN


Expression ((Project names + Projection))          
  Expression                                       
    ReadFromMergeTree (default.table_1)            
    Indexes:                                       
      PrimaryKey                                   
        Keys:                                      
          id_1                                      
        Condition: (id_1 in [863244623, 863244623]) 
        Parts: 5/5                                 
        Granules: 5/1224

You can see that the query is using the primary key, and it’s reading 5/1224 (~0.4%) of the table. This means that the query is filtering efficiently. If we now change the query to:

-- BAD QUERY

SELECT value
FROM table_1
WHERE id_2 = 863244623

The EXPLAIN output looks like this:

# BAD QUERY - EXPLAIN

Expression ((Project names + Projection))          
  Expression                                       
    ReadFromMergeTree (default.table_1)            
    Indexes:                                       
      PrimaryKey                                   
        Keys:                                      
          id_2                                   
        Condition: (id_2 in [863244623, 863244623]) 
        Parts: 5/5                                 
        Granules: 1224/1224

You can see that the query is using the primary key, but it’s reading 1224/1224 (100%) of the table. This is an extreme case, but it shows that the query is not filtering efficiently. We could change the sorting key to [id_2, id_1] and the query would be faster, but that would make the initial query slower.

Automatic Optimization

If the query is still considered optimizable, we suggest a better data source schema. Now that we’ve confirmed optimization might help:

The query booster chooses columns using a combination of their cardinality and the type of filter they are being used for. For example, columns (and their cardinality) are treated differently depending on if they are used for point filtering or range filtering.
Sort the filter columns by cardinality (ascending). Columns with lower cardinality (fewer distinct values) go first — they help partition the data into coarse groups. Higher cardinality columns go later — they help refine filtering within those groups.
Suggest the first four columns from this sorted list as a new optimized data source schema. These columns become the new sorting key for the data source.

If the resultant suggested sorting key is a prefix of the already existing sorting key, we will not suggest an optimization. For example, if the sorting key is [a, b, c, d] and your query filters on [a, b], this is not considered optimizable. Why? Because it’s already aligned with the sorting key, and we are already using the primary index efficiently. A new optimization wouldn’t help.

Otherwise, we will proceed to create a new data source schema with the suggested sorting key.

The decision to just use data source sorting keys to improve the performance of a query comes from years of experience debugging customer query performance issues. As we explained in our optimization guides, most of the time a query is not performing as expected is due to a bad set of sorting keys.

Performance Validation

But that is not all, we log and monitor optimizations to keep track of the impact they have on the performance of the queries. We only keep optimizations that are still useful:

Optimizations that are not being used anymore due to clients stopping endpoint calls will be removed but we’ll allow them to create them again if endpoint calls are back. If the query we tried to optimize is not executed in 24h, we discard the optimization.
Optimizations that are not being used despite endpoints being hit. This could be caused by sorting keys being wrong or by any other unexpected issues. Those optimizations will be “banned” and won’t be created again. There are two ways to determine if an optimization is not being used:
- Optimization not being used at all: If the query is being called, but the optimization is not being used.
- Optimization is being used but not filtering efficiently: If the optimization is being used, but the query is reading more data than when using the original sorting key.

Another example of Query Booster in action.

Coming back to the surprised customer, the results were good:

2x reduction in query times for their most common workloads
Decreased CPU usage for the same queries
Zero configuration or maintenance required

And while this particular case showed a 2x improvement, we’ve seen cases where the optimization leads to 100x faster queries.

Intelligence matters

The beauty of Query Booster lies in its simplicity from a user’s perspective. You don’t need to be a database expert or spend time manually optimizing your queries. The system handles that for you, continuously learning and adapting to your specific usage patterns.

Remember: The best optimizations are the ones you don’t have to think about. Query Booster embodies this principle by automatically improving your database performance while you focus on what matters most - your application and business logic.

2025W14

Sun, 13 Apr 2025 12:26:32 GMT

What If We Made Advertising Illegal? — The idea of making advertising illegal may sound silly but couldn’t agree more. I’ve been unlucky enough to work for the industry and can’t imagine a better society with the current practices.
Notes on kindness — “So eventually, if you care, you leave”
The case against conversational interfaces — Conversational interfaces, like voice assistants and chatbots, often promise a new way to interact with technology but fail to replace traditional computing methods. Instead of replacing existing tools, AI should enhance them, allowing for seamless interactions that feel effortless.
How to Write Blog Posts that Developers Read — Didn’t like the main topic of the article since it’s focused on ways to attract readers to your blog posts. When I write I do it for myself and I expect other personal blogs to do the same. Anyway, some of the advice is good even if you’re not focused on attracting readers.
Tracing the thoughts of a large language model — Anthropic team explains how are they working on understanding Claude’s internal “reasoning” processes. This research branch could help improve AI reliability and transparency.
Cyanview: Coordinating Super Bowl’s visual fidelity with Elixir — Leaving aside the Elixir infomercial. Interesting read about how a team of nine have managed to build such an incredible product. It might sound simple but as someone who has been working with remote cameras in the past, ai can tell it’s more complex than it sounds.
Things that go wrong with disk IO — Short article emphasizing the critical importance of maintaining data integrity when developing applications that rely on disk interactions

2025W11

Sun, 23 Mar 2025 20:12:30 GMT

Less ceremony, more shipping — Tinybird introduces their new deployment tool, which simplifies data schema changes for teams. This tool automates the entire deployment process, ensuring no downtime and minimizing errors. By applying software engineering best practices to data workflows, it allows developers to focus more on building rather than troubleshooting.
PlanetScale redesign — Twitter thread about how PlanetScale redesigned their website and the impact it had. And a follow-up showing how the original redesign was done using Google Docs!
IO devices and latency — Nice post about IO devices latency and how they have evolved. It comes with excellent illustrations. The author also wrote a Twitter thread with the making off and the impact this kind of educational technical posts have on their marketing.
The good times in tech are over — I don’t personally like the post but it’s true. Good times are over for software engineers that were not aware of what their job was.
Before AI Grew Up — A guy explaining to his kids in the future how AI feels nowadays. Nicely written and hopefully accurate.

2025W09

Sun, 09 Mar 2025 13:10:52 GMT

It’s been a while since the last time I published a post with the list of articles I’ve read. This one contains the highlights from the past few months.

We Were Wrong About GPUs — Fly.io invested in GPU Machines to support AI/ML tasks for developers, but they learned that most developers prefer using APIs for AI instead of managing GPUs themselves. Despite the challenges and costs of deploying GPUs, they found that the demand for these machines was lower than expected. The company is now scaling back its GPU ambitions while focusing on improving its core product offerings.
Why I think AI take-off is relatively slow — Tyler Cowen argues that the rapid adoption of AI is slowed down by inefficiencies in less productive sectors and human bottlenecks, like regulatory delays. He believes that while AI can improve productivity, it will not transform the economy as quickly as expected due to the challenges of integrating it with human work. Overall, Cowen suggests that AI might boost economic growth modestly, but noticeable changes will take time.
Is it okay? — The author discusses the impact of language models on creativity and copyright, arguing that their reliance on the entire internet raises ethical concerns. He believes that if these models only replicate existing content without contributing to human creativity, it is problematic. However, if they lead to significant advancements in science and technology, their use may be justified.
The Inner Ring — The concept of the “Inner Ring” describes the human desire to belong to exclusive groups, which can lead to negative actions and feelings of exclusion. This longing for acceptance can overshadow genuine connections and fulfillment in life. True happiness comes from focusing on meaningful work and relationships rather than chasing after superficial insider status.
Introducing S2 — Introduction to S2, a new approach to streaming data storage.
Ghostty: Reflecting on Reaching 1.0 — This is a personal reflection on the project.
How to Pick Your Life Partner - Part 2 — To endure 20,000 days, 100 vacations, and 100,000 leisure hours with another human being and do so happily, there are three key ingredients necessary.
How to Pick Your Life Partner - Part 1 — Given that the choice of life partner is by far the most important thing in life to get right, how is it possible that so many smart people get it so wrong?
The secret inside One Million Checkboxes — Teens wrote secret binary messages in One Million Checkboxes. The author found them.
Group Chat: The Best Way to Totally Stress Out Your Team — The perils of the modern communications conveyor belt that never ends, divides your attention, fractures your time, and chains you to FOMO.
Reflections on Founder Mode — Reflections on a recent Paul Graham piece – and on the culture at Oxide
Founder Mode — At a YC event last week Brian Chesky gave a talk that everyone who was there will remember. Most founders I talked to afterward said it was the best they’d ever heard. Ron Conway, for the first time in his life, forgot to take notes. I’m not going to try to reproduce it here. Instead I want to talk about a question it raised.
Obsession — Reflections on the concept of healthy obsession and how it can be a source of motivation and success.
Leaving Neovim for Zed — A journey through text editors and how I landed on Zed after years of Neovim
Why I Built Litestream — Despite an exponential increase in computing power, our applications require more machines than ever because of architectural decisions made 25 years ago. You can eliminate much of your complexity and cost by using SQLite & Litestream for your production applications.
It’s in the stories — The author reflects on the importance of storytelling in leadership, sharing anecdotes about influential figures.
Two Threads, One Core: How Simultaneous Multithreading Works Under the Hood — Simultaneous multithreading (SMT) is a feature that lets a processor handle instructions from two different threads at the same time. But have you ever wondered how this actually works?
Why German Strings are Everywhere — German Strings are a custom string format developed for optimized data processing, adopted by many databases, focusing on performance and memory efficiency.
The Kafka Metric You’re Not Using: Stop Counting Messages, Start Measuring Time — Traditional offset-based monitoring can be misleading due to varying message sizes and consumption rates. To address this, you can introduce a time-based metric for a more accurate assessment of consumer group lag.

130 hours

Sun, 29 Dec 2024 16:00:00 GMT

For the last few years, especially after the pandemic forced us to switch to remote work, I’ve been searching for ways to break free from a stressful, sedentary lifestyle.

I’ve been trying to be more consistent. I’ve tried different approaches, but each attempt would start with enthusiasm, only to gradually fade away.

This year something was different. I still don’t know what it was, but I’ve been able to keep making progress. I’ve managed to work out for +130 hours. Almost a 70% increase from last year. Around 20 minutes a day.

These numbers aren’t impressive, but they represent something far more valuable for me: consistency. I’m proud not just of the numbers, but of maintaining a regular routine despite facing significant personal challenges during the second half of the year.

Next year, I’ll try to keep making progress. I’m going to keep making progress. I’m going to keep making progress.

This is the closest I’ll ever publicly get to a year review and a new year resolution. I’ve created this page to keep track of my progress.

Beyond Code

Fri, 06 Dec 2024 00:00:00 GMT

Let’s get real for a moment. I’ve been in the software engineering game for a while now. And yet, the impostor syndrome has been haunting me from time to time: what makes me valuable? I’ve been lucky to work with some amazing engineers who can solve any technical challenge, and I’ve seen them struggle with the same question.

Here’s what I believe: our most valuable skills aren’t about how many programming languages you can juggle or how many complex algorithms you can solve.

The Hard Skills Trap

Remember when we all thought being a rockstar engineer meant knowing every framework and writing the most mind-blowingly complex code? Yeah, I’ve been there. I used to look at my colleagues who could deep dive into low-level technical rabbit holes like the internals of Linux and feel like I was somehow falling short.

Early in my career, I was obsessed with proving my technical prowess. I’d spend countless hours trying to optimize code that was already working, learning obscure programming techniques, and collecting certifications like they were badges of honor. I thought these were the markers of a great engineer. Spoiler alert: they’re not.

Spoiler alert: I wasn’t falling short. I was just playing a different game.

What You Should Actually Be Good At

Here’s the thing: the real superpower is understanding how stuff works. And I mean really works. Not just the code, but everything around it:

What does the product actually need?
What is the client really trying to solve?
How does this fit into the bigger picture?
Will this solution actually make someone’s life easier?

Ship It, Don’t Perfect It

I’m a huge believer in the “done is better than perfect” philosophy. I’m driven by action and the smaller possible changes that can be shipped to deliver value. And trust me, this isn’t not only about being lazy. It’s about understanding that software is about delivering solutions to actual problems, not an artisan job where you are crafting museum like pieces of code.

Want to know a good approach? If something works and solves the core problem, ship it. Iterate later. The world doesn’t need another project stuck in endless refinement limbo.

The Skills They Don’t Teach in Bootcamp

The most valuable things I’ve learned aren’t in any curriculum:

How to explain tech stuff to non-tech people without making their eyes glaze over
Seeing the entire system you are building, not just the tiny technical corner
Making decisions that balance “what’s possible” with “what’s actually needed”
Moving fast and adapting quickly
Understanding that every technical decision has a human impact

What’s fascinating is that these skills aren’t static. They evolve constantly. Every project is a new opportunity to understand more deeply, to communicate more clearly, to solve problems more effectively. The moment you think you’ve mastered these skills is the moment you stop growing.

The backside of this ongoing learning is that most of the time you will need to trust your gut and make decisions based on the context, even when you don’t have all the information. It’s about developing a intuition that comes from experience, not from perfect knowledge. Sometimes, a well-informed guess guided by your accumulated understanding is more valuable than paralysis by analysis.

Take It or Leave It

To any engineer feeling like they’re not technical enough: Stop. Your ability to understand context, communicate clearly, and drive solutions is worth way more than being able to write the most elegant code.

Technical skills? They’re important. But the ability to see the whole chess board? That’s the real game-changer.

In this wild world of software engineering, being a great technician is cool. But being someone who can actually solve real problems and create value? That’s where the magic happens.

The best engineers aren’t those who know the most, they’re those who understand the most. Understanding the problem, the people, the context, that’s the true skill that separates good engineers from great ones.

So keep learning, stay curious, and remember: your most important line of code is the one that makes someone’s life better.

The Joy and Pain of Problem-Solving

Sun, 06 Oct 2024 16:00:00 GMT

Investigating challenging problems and bugs is an exciting experience that often leads to remarkable discoveries and personal growth. As a engineer, I’ve always found joy in decoding complex issues and diving deep into the unknown. The thrill of the hunt, the satisfaction of piecing together clues, and the ahá moment when everything falls into place are unique.

One of the most rewarding aspects of problem-solving is the opportunity to learn new things. Every bug, every system failure, and every unexpected behavior is a chance to expand your knowledge and skillset. It’s like being a detective in the digital world, where each case brings its own unique challenges and lessons.

The skills developed through problem-solving and investigation are not only valuable in professional settings but can also be applied to personal life. Embracing the challenge of investigating problems and bugs not only makes us better professionals but also equips us with valuable life skills. It teaches us to be persistent, adaptable, and open to learning.

While the truths we uncover can be difficult or painful, they ultimately lead us to growth and a deeper understanding of ourselves and the world around us.

Indeed, these problem-solving skills can sometimes lead to unexpected and challenging discoveries in our personal relationships. Such as uncovering long-term-perfectly-hidden lies and infidelities.

Slightly inspired by this thread: https://twitter.com/rosapolis/status/1828014752112545924

Coding with AI

Fri, 23 Aug 2024 00:00:00 GMT

I’ve never been enthusiastic about the AI revolution.

However, this week, I started using Zed.dev together with their AI assistant integration. I’m surprised by the experience and efficiency gains. I still have reservations, but it can be valuable allies in tackling programming challenges and routine tasks.

For example, a few days ago, I was working on a tool that needed to parse ClickHouse explain output¹, an easy task requiring recursively transverse all the properties looking for some specific properties.

With a simple prompt, the model wrote a working version.

Write code to extract all primary key granules from explain_result. An example of explain_result here:

{…}

def extract_primary_key_granules(explain_result):
    def traverse(node):
        granules = []
        if isinstance(node, dict):
            if 'Indexes' in node:
                for index in node['Indexes']:
                    if index.get('Type') == 'PrimaryKey':
                        granules.append({
                            'Keys': index.get('Keys', []),
                            'Initial Granules': index.get('Initial Granules', 0),
                            'Selected Granules': index.get('Selected Granules', 0)
                        })
            for value in node.values():
                granules.extend(traverse(value))
        elif isinstance(node, list):
            for item in node:
                granules.extend(traverse(item))
        return granules

    return traverse(explain_result)

# Example usage:
result = extract_primary_key_granules(explain_result)
for granule in result:
    print(f"Keys: {granule['Keys']}")
    print(f"Initial Granules: {granule['Initial Granules']}")
    print(f"Selected Granules: {granule['Selected Granules']}")
    print()

The task was relatively easy and I had to make changes and improve the initial code. Despite that, this saved me at least an hour of back-and-forth writing the function and running some tests to see if it worked as expected!

I started using Zed AI assistant with ChatGPT integration. A day after they released their own integration that I’m now testing together with Anthropic’s Claude model. I’m not even using Google for simple searches anymore.

This is how a ClickHouse explain output looks like ↩

2024W28

Mon, 15 Jul 2024 00:00:00 GMT

So You Want To Build A Browser Engine — This article goes deep into all the complicated things involved in making a browser engine. It’s amazing how browsers keep getting better and adding more features while maintaining all the required complexity.
Databricks, Snowflake and the future — This piece tells you about the recent meetings of Snowflake and Databricks. It focuses on how they are putting money into Apache Iceberg.
The time I spent three months investigating a 7-year old bug and fixed it in 1 line of code — This is an interesting story about someone spending three months trying to fix a really old bug. And how they only needed one line of code to solve it. Nice read about debugging software and hardware.
Local, first, forever — An insightful article on the local-first philosophy and the implementation of services without reliance on a centralized server. Includes an introduction to CRDT and a sample implementation.
What Goes Around Comes Around… And Around… — A retrospective review of the evolution of relational database management systems over the past two decades. It gives a good overview of how things are and how have evolved.
A write-ahead log is not a universal part of durability — Explores the concept of durability in databases, challenging the notion of a write-ahead log as a universal component.
Unix’s fsync(), write ahead logs, and durability versus integrity — Expands on Phil Eaton’s post about database durability, incorporating the crucial aspect of integrity into the discussion.
The Right Kind of Stubborn — Paul Graham talks about two types of stubbornesss: persistence and obstinacy. He explains that it’s good to keep going when you need to, but it’s bad when you won’t change your mind.

2024W24

Fri, 21 Jun 2024 00:00:00 GMT

Zed Decoded: Rope & SumTree — The Zed team explained how they use B+Tree (or SumTree) to deal with file content in their text editor. I have never considered how text editors use different structures to represent and manipulate huge strings while maintaining good performance. Interesting read.
Why does SQLite (in production) have such a bad rep? — A short post about the controversy around SQLite, a database often perceived as unsuitable for production. The most interesting thing is the discussions originated in lobste.rs and Reddit
In Codd we Trust (or not) — A brief reflection on Codd’s original idea, which separates logical and physical schema, and the significant role query planners and optimizers play in modern database performance. While I understand the point of letting experienced users write their queries and execute them as they are, I can’t fully agree.
Some opinionated thoughts on SQL databases — Another opinionated reflection on relational databases and SQL. I agree with some points. Still, SQL is here to stay. The point that resonated more is how inefficient it is to use SQL as an API in some workloads. I’ve seen query parsing take +50% of the processing time in low latency and high throughput environments. Of course, those numbers are for very specific use cases (e.g., point queries) where the data being read is small, taking just a few milliseconds. Still, the overhead is relevant, and there is little room for improvement when using plain SQL. On the other hand, it is good to use almost all relational databases using a single API language. If you are an experienced PostgreSQL user, you can query a MySQL database without investing hours learning how the new API works.
Senior Engineer Fatigue — An article to justify why older people get slower at their jobs. Jokes aside, it’s a good read about why sometimes you need to slow down to get faster. And it applies not only to senior engineers!
What Happens When You Put a Database in Your Browser? — The browser is the new OS. A DuckDB article about running their software in browsers using Wasm. The text provides a good example of pre-visualizing parquet file schema on the fly from the browser.

2024W23

Sat, 15 Jun 2024 00:00:00 GMT

What Makes You You? — Article exploring theories of personal identity and what makes us who we are
How people fall apart — Essay discussing burnout’s impact on the brain
Message authentication codes for safer distributed transactions — A short post on using message authentication codes to prevent bugs in distributed systems
Deterministic Simulation Testing — A brief introduction to Deterministic Simulation Testing. A crucial framework for building reliable distributed systems.
Why SQLite Uses Bytecode — An explanation of why SQLite uses bytecode for easier understanding and debugging. The most exciting thing is that this comes from an interesting convo on Twitter¹

https://twitter.com/DRichardHipp/status/1785037995101290772 ↩

/proc/thread-self

Thu, 02 May 2024 00:00:00 GMT

I’ve never invested much time in understanding how operating systems work. It has never been on my interests list, nor have I considered it useful knowledge for my career. What can I say? We all make bad decisions in our lives.

Over the last two years, I have invested a lot of time in understanding how some things work in Linux. Those are the consequences (or benefits?) of working with a complex and low-level system like a database.

This week, I was working on tracking data that was read from the page cache. While debugging some ClickHouse metrics directly collected from the Kernel, I discovered the /proc/thread-self directory¹.

/proc/thread-self

When a thread accesses this magic symbolic link, it resolves to the process’s own /proc/self/task/[tid] directory.

So /proc/thread-self points to the proc folder for the current thread. In those folders, you can find a lot of helpful information. In my case, I was interested in /proc/thread-self/io ², where you have IO statistics.

I was focused on investigating whether the Kernel reported bytes read from S3 inside rchar. I shared more information in this PR in ClickHouse repository. Despite the PR being closed, the examples I shared there still hold valuable insights and a reproducer that can contribute to the collective understanding of the system.

There is also a /proc/self for the current process. This is something I didn’t know either. ↩
https://docs.kernel.org/filesystems/proc.html#proc-pid-io-display-the-io-accounting-fields ↩

Data Engineering papers/articles

Sat, 04 Nov 2023 00:00:00 GMT

It’s quite common to find folks sharing papers on minor breakthroughs in areas like NLP, Computer Vision, Machine Learning, Deep Learning, and related fields here.

We’ve carefully gathered a collection of papers focused on databases, distributed systems, and data in general. We’ve explored the latest developments in these areas and gained valuable insights from the many articles in our list.

Some of these publications were suggested to us after we made our list public, adding to the richness of our curated collection.

MapReduce: Simplified Data Processing on Large Clusters
Spark: Cluster Computing with Working Sets
Kafka: a Distributed Messaging System for Log Processing
Dremel: Interactive Analysis of Web-Scale Datasets: A paper describing the technology behind Google BigQuery
Procella: Unifying serving and analytical data at YouTube
The Log: What every software engineer should know about real-time data’s unifying abstraction: Not a paper but an article from one of the Kafka creators. He explains the basic data structure, key for many databases, and modern distributed systems.
Making reliable distributed systems in the presence of software errors
Conflict-free Replicated Data Types (CRDT)
Delta State Replicated Data Types
Time, Clocks and the Ordering of Events in a Distributed System
Dynamo: Amazon’s Highly Available Key-value Store
Linearizability: A Correctness Condition for Concurrent Objects
Space/Time Trade-offs in Hash Coding with Allowable Errors: Not especially interesting reading but had to mention Bloom Filters and this is the original paper. Perhaps the most surprising data structure I discovered working with data.
Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems: Not a paper but a book. THE book for anyone interested in databases, data, and distributed systems.
Naiad: A Timely Dataflow System: Precursor paper of Materialize
The Snowflake Elastic Data Warehouse
The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing

Resolving a year-long ClickHouse lock contention

Mon, 30 Oct 2023 00:00:00 GMT

Original post: https://www.tinybird.co/blog-posts/clickhouse-lock-contention

This is a story about completely saturating a ClickHouse replica (and being happy about it). We were able to fix a long-running issue that limited our query concurrency, and we increased CPU utilization to 100% in the process.

The unknown limit

The story begins about a year ago. One of our ClickHouse clusters started to malfunction, causing some queries to slow down significantly. At first glance, everything seemed fine. We checked the usual bottlenecks - CPU, memory, I/O - and all checked out. But still, at a certain number of requests per second, query responses on the replica would slow down. We couldn’t figure out why.

The CPU was just chillin’ at <20%.

We spent days digging into different metrics, checking query performance, assessing required resources, tweaking cloud provider settings, and many other things. We had many more hypotheses than we had real solutions. Maybe we could find and resolve some hidden bandwidth limits or improve concurrency by limiting the number of threads used per query? Everything was pointing to some kind of contention, but we couldn’t see it. “We must be missing something” was a common refrain in those days.

As engineers, we would have loved to dive deep into this issue, understand what was happening, and develop some novel fix. But we have to balance our itch to solve problems with some pragmatism; we still have a real-time data platform to build, customers to support, and features to release. So we made a quick fix, slightly changing the cluster setup and isolating some workloads so we could move on to the next GitLab issue.

Some context about Tinybird and ClickHouse

Tinybird is a real-time platform for data and engineering teams. We give our customers the power to unify data from many source systems (including databases, data warehouses, and streaming data), develop real-time analytics with SQL, and publish their queries as low-latency, scalable APIs.

We are focused primarily on streaming ingestion and real-time use cases; our clients can ingest millions of events and make thousands of concurrent API requests per second. Under the hood, we use ClickHouse as our primary analytical database and real-time SQL “engine”. ClickHouse is an incredibly powerful open-source database, and we only use a subset of its features. Still, with the scale we support and our customers’ latency and throughput requirements, we often push it to the limit.

ClickHouse is an incredibly powerful open-source database, and we only use a subset of its features. Still, with the scale we support and our customers’ latency and throughput requirements, we often push it to the limit.

As such, we frequently contribute to the ClickHouse project by developing performance improvements that help us continue to support our customers’ scale.

Finding the root cause

Despite our quick fix, this resource contention became a recurring issue. We’ve been able to dodge it by deploying more quick fixes, but that changed a few weeks ago. To be able to support a particular use case for one of our clients, we were approaching a scale that made it impossible to ignore. It was time to invest in the time and resources to fix it for good.

We had been able to avoid fixing this problem for almost a year, but certain customers began approaching a scale that made it impossible to ignore.

Every time we had faced this issue before and explored various metrics and profile events, we ended up in the same dead end. We saw lock-related profile events such as RWLock* and discarded them since we knew they were not the root cause but merely a symptom.

When examining lock-related profile events, we noticed a small spike in ContextLockWait events

This time was different. We noticed a small peak of ContextLockWait events. After all this time surveying typical server metrics for clues and coming up empty, we finally had something promising!

Unfortunately, we didn’t have a good way to measure its impact on performance. Unlike RWLock* events, which have *WaitMilliseconds properties that indicate how long they’ve been waiting to acquire the lock, ContextLockWait events don’t come with a time-based equivalent that would have let us measure the time waiting for those locks. So we were flying blind, with no way to understand the consequences of these events.

In ClickHouse, ContextLockWait events didn’t come with an equivalent time-based wait property, like RWLockWaitMilliseconds, that would have allowed us to measure the performance impact of these events.

We decided to dump all threads during the incident and see if we could extract something from there. After a first look at the dump, it was clear that ContextLock was, at least, one of the issues.

So, we built a small reproducer to report the issue to the ClickHouse repository to help the contributors with the fix. While working on the reproducer, we sent the first improvement upstream, adding a profile event to report Context Lock waiting time. With this metric available, it became easy to find a query that, given sufficient concurrency, would cause the same contention we were seeing in our clusters. This moved our first roadblock out of the way.

We built a small reproducer and pushed a change to add a waiting time profile to ContextLock events in ClickHouse.

With the reproducer in place and a way to measure the impact of the contention, it was just a matter of digging into the code and figuring out how to reduce contention and improve performance. After quite a big refactor spearheaded by my colleague Maksim Kita, we managed to remove lock contention, reducing the impact on our synthetic example by an order of magnitude.

Here’s how we did it:

Context refactoring

Here’s the basic architecture of Context in ClickHouse before the refactor:

Before we addressed this issue, ContextSharedPart and Context instances all shared a single global mutex.

In ClickHouse, ContextSharedPart is responsible for storing global shared objects that are shared between all sessions and queries, for example:

Thread pools
Server paths
Global trackers
Clusters information

ContextSharedPart also provides a lot of useful methods for working with these objects with synchronization.

Context is responsible for storing query- or session-specific objects, for example:

Per query settings
Per query caches
Current database

Context also provides a lot of methods for working with these objects, and also it uses ContextSharedPart to provide some of that functionality.

During query execution, ClickHouse can create a lot of Contexts because each subquery in ClickHouse can have unique settings. For example:

SELECT id, value 
FROM (
  SELECT id, value 
  FROM test_table 
  SETTINGS max_threads = 16
)
WHERE id > 10
SETTINGS max_threads = 32

In this example, a nested subquery will have max_threads = 16, requiring its own Context.

The problem was that a single mutex was used for most of the synchronization between Context and ContextSharedPart, even when we worked with objects local to Context. A large number of low-latency, concurrent queries with many subqueries will create a lot of Contexts per query, and the problem becomes even bigger.

We did a big refactoring, replacing a single global mutex with two read-write mutexes. One global read-write mutex for ContextSharedPart and one local read-write mutex for each Context. We used read-write mutexes because most of the time we do a lot of concurrent reads (for example read settings or some path) and rarely concurrent writes.

In many places, we completely got rid of synchronization where it was used for initialization and used call_once for objects that are initialized only once.

The Context architecture after our refactor

Here’s the way things looked after our refactor:

After refactoring, each Context had its own read-write mutex.

We also added clang ThreadSafetyAnalysis (TSA) annotations, so we can be sure that our refactoring does not introduce race-conditions or deadlocks.

Bringing the CPU to 100%

Once both changes were accepted and merged to ClickHouse master we decided to run some tests in a production-like environment. The main objective was to see if we could reproduce the issues and assess the impact of the fix by simulating our clients’ workload on two different ClickHouse versions.

23.9: Contains the metric to measure the impact
master: Contains the ContextLock fix

So, we recreated the databases and tables, extracted 1,000 example queries from the last seven days, and ran them continuously for a few minutes using clickhouse-benchmark with enough concurrency:

clickhouse benchmark -r --ignore-error \
    --concurrency=500 \
    --timelimit 600 \
    --max_execution_time=0 \
    --connect_timeout=20 \
    --cumulative < queries.txt

Results with 23.9

Here are the results we saw on 23.9, where we implemented a way to monitor the impact of the Context lock contention:

Stopping launch of queries. Requested time limit 600 seconds is exhausted.

Queries executed: 122933 (12293.300%).

localhost:9000, queries: 122084, errors: 849, QPS: 201.059, RPS: 108762.836, MiB/s: 4.354, result RPS: 3726.263, result MiB/s: 0.137.

0.000%      0.004 sec.
10.000%     0.353 sec.
20.000%     0.620 sec.
30.000%     0.732 sec.
40.000%     0.834 sec.
50.000%     0.997 sec.
60.000%     1.386 sec.
70.000%     2.594 sec.
80.000%     4.559 sec.
90.000%     5.195 sec.
95.000%     5.780 sec.
99.000%     18.171 sec.
99.900%     73.734 sec.
99.990%     75.998 sec.


┌─version()───┬──────c─┬───d─┬───────w─┬─────max_w─┐
│ 23.9.1.1854 │ 122937 │ 948 │ 934.609 │ 75545.504 │
└─────────────┴────────┴─────┴─────────┴───────────┘

Baseline results on 23.9

From these results, you can see that, as a baseline, we were able to achieve these results:

~200 QPS
CPU utilization of only ~20%.
Half of queries took at least 1s
Slowest queries took ~75s

Results with master

Here are the results we saw on master, where we had refactored Contexts to attempt to resolve our lock contention:

Stopping launch of queries. Requested time limit 600 seconds is exhausted.

Queries executed: 398845 (39884.500%).

localhost:9000, queries: 395940, errors: 2905, QPS: 659.062, RPS: 660165.301, MiB/s: 11.490, result RPS: 12228.846, result MiB/s: 0.451.

0.000%      0.001 sec.
10.000%     0.150 sec.
20.000%     0.320 sec.
30.000%     0.442 sec.
40.000%     0.574 sec.
50.000%     0.682 sec.
60.000%     0.783 sec.
70.000%     0.902 sec.
80.000%     1.021 sec.
90.000%     1.211 sec.
95.000%     1.408 sec.
99.000%     2.050 sec.
99.900%     4.103 sec.
99.990%     5.959 sec.


┌─version()───┬──────c─┬───d─┬─w─┬─max_w─┐
│ 23.10.1.131 │ 398847 │ 680 │ 0 │ 28.09 │
└─────────────┴────────┴─────┴───┴───────┘

With the fixes and at concurrency=500 we were able to 3x performance.

With the refactor, we managed these results:

~600 QPS (3x better)
CPU utilization to ~60% (3x better)
Median query time to ~0.6s (2x better)
Slowest queries to ~6s (12x better)

Since tests with master were successful, we added more concurrency to clickhouse-benchmark (1000 instead of 500) to see what would happen. Here are the results:

Stopping launch of queries. Requested time limit 600 seconds is exhausted.

Queries executed: 581870 (58187.000%).

localhost:9000, queries: 577893, errors: 3977, QPS: 961.997, RPS: 1896173.448, MiB/s: 16.859, result RPS: 17730.009, result MiB/s: 0.645.

0.000%		0.001 sec.
10.000%		0.011 sec.
20.000%		0.015 sec.
30.000%		0.020 sec.
40.000%		0.025 sec.
50.000%		0.033 sec.
60.000%		0.047 sec.
70.000%		0.088 sec.
80.000%		0.209 sec.
90.000%		0.266 sec.
95.000%		0.306 sec.
99.000%		0.552 sec.
99.900%		3.259 sec.
99.990%		4.153 sec.


┌─version()───┬──────c─┬──d─┬─w─┬──max_w─┐
│ 23.10.1.131 │ 581875 │ 32 │ 0 │ 18.286 │
└─────────────┴────────┴────┴───┴────────┘

Boom. ~100% CPU utilization and at least 5x performance boost!

These results are incredible:

~1,000 QPS (5x better)
~100% CPU utilization (5x better)
0.033s median response time (30x better)
Slowest queries ~4s (20x better)

Of course, these are just tests, and we don’t expect to get a 5x improvement in production, since we certainly will find other bottlenecks with I/O, CPU, and/or memory. But, we have at least removed a contention that had stumped us for a long time. Even if all we got was a 1.5x bump in performance, this is a massive improvement for us, our infrastructure, and the customers who use it.

We don’t expect to get a 5x boost in production, but even if it’s just 1.5x, that’s still a massive improvement for us and our customers.

With the ClickHouse 23.10 official release (slated for November 2nd), we will update some of our most performance-demanding clients, and see the real impact of these changes.

We expect good things.

SQL Is All You Need

Mon, 10 Apr 2023 16:00:00 GMT

Across all my professional experiences, SQL has been always an important tool. Being it wrestling with Redshift to get historical data to build some models, writing algorithms on Spark to get data from a data lake, or even trying to get dimensional data out of OracleDB.

I strongly believe that our lives would be way easier if SQL was everything (or almost) we needed when it comes to data. In this article, I want to play with the idea of building a machine learning algorithm by just using SQL and ClickHouse. Hence the title, which is a clear reference to the Attention Is All You Need paper.

A bit of context

This piece has been sitting on the back of my head for a while. Since I joined Tinybird one of my responsibilities it’s been to understand the product we are building and help clients to get as much as they can from it. Once I understood ClickHouse’s potential I haven’t managed to prevent myself from thinking about solving problems I had in previous companies.

Although I’m now working as Data Engineer, I’ve spent a vast amount of my career working in roles strongly related to Data Science, and Machine Learning. The two most relevant experiences for this article:

Working doing deep learning in a small 3D medical company. There, I heavily contributed in a couple of papers, despite not appearing in them as an author (that was academia and everybody knows how things are played in there, this is another topic that probably deserves a full article for itself)
Led the data science efforts in an ad-tech company. Where I implemented a custom version of an online learning algorithm known as FTRL-Proximal in Spark

The common pain in both experiences was always to find a cheap and fast way to store the data that is compatible with real-time inference and also supports experimenting, iterating, and training different models. Being a machine learning practitioner nowadays requires you to use a myriad of tools such as feature stores, training platforms, infrastructure, streams, etc, to be able to train your models and provide batches of predictions.

Now imagine a system being used by a retail company to decide which product to show you based on the probability of you buying it, while they send events to the algorithm to tell it if a shown product has been bought or not. The list of tools needed would keep getting longer.

💭 ClickHouse already has built-in methods implementing linear regression (stochasticLinearRegression), and logistic regression (stochasticLogisticRegression). Those implementations are a bit rigid but still can be used to build machine learning models without leaving the database in a batch environment.

So let’s play a bit with the idea of building an end-to-end model inside ClickHouse to try to remove all those tools while covering all the described requirements.

To make it easier to follow and reproduce we are going to use some data from the Census Income Data Set:

import pandas as pd

dataset_url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data'
column_names = ['age', 'workclass', 'fnlwgt', 'education', 'education-num', 'marital-status', 'occupation', 'relationship', 'race', 'sex', 'capital-gain', 'capital-loss', 'hours-per-week', 'native-country', 'y']

df = pd.read_csv(
    dataset_url,
    names = column_names
)
df['y'] = (df['y'] == ' >50K').astype('int')

df.tail()

       age      workclass  fnlwgt    education  education-num  ...
32556   27        Private  257302   Assoc-acdm             12  ...  \
32557   40        Private  154374      HS-grad              9  ...
32558   58        Private  151910      HS-grad              9  ...
32559   22        Private  201490      HS-grad              9  ...
32560   52   Self-emp-inc  287927      HS-grad              9  ...

      capital-gain capital-loss hours-per-week  native-country  y
32556            0            0             38   United-States  0
32557            0            0             40   United-States  1
32558            0            0             40   United-States  0
32559            0            0             20   United-States  0
32560        15024            0             40   United-States  1

The dataset illustrates a classification problem. It’s quite interesting because it contains continuous and categorical data, something that is not common to see when following implementation examples but that almost every real-life approach will have.

Let’s start by implementing online gradient descent.

Online gradient descent

Online gradient descent is essentially the same as stochastic gradient descent (SGD), the online word is commonly added when applied over stream data. So, instead of randomly selecting a sample from the training set at each iteration, a new iteration is performed every time a new sample is received.

SGD algorithm is easy to describe in pseudo-code:

Initialize the weights 𝑤
Iterate over all samples
For each sample 𝒾 update weights as:

SGD is an iterative method, so having the values of the previous weights, it’s possible to generate the next ones with simple operations. Repeat the process over hundreds/thousands of events and, in theory, it will converge.

Being a recursive algorithm it could probably be implemented using window functions with some recursivity, or recursive CTEs. But leaving apart that they are not supported in ClickHouse, we are interested in a stateful approach (we need the weights to be stored somewhere), and update them every time we receive a new sample. The idea is to use basic database tables and Materialized Views, which are executed on each insert, computing the weights offsets that will later be aggregated. This will allow us to update them on the fly.

First of all, let’s build a simple model using sklearn so we can use it as a reference. It will also allow us to validate results while writing our version. Since this is a classification problem, we’ll use the SGDClassifier with some special settings to simulate online gradient descent. For now, we are going to use only the continuous features and the first 1000 samples, as it will make things easier. Once everything works as expected and things are validated, we will add categorical features, and use the entire dataset if we want.

from sklearn import linear_model

model = linear_model.SGDClassifier(
    loss='log_loss',
    penalty=None,
    fit_intercept=True,
    max_iter=1,
    shuffle=False,
    learning_rate='constant',
    eta0=0.01
)

X = df[['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', 'hours-per-week']]
y = df[['y']]
model = model.fit(X[:1000], y[:1000])

print(model.intercept_[0])
print(dict(zip(X.columns, model.coef_[0])))

0.10499999999999998

{'age': 16.665000000000006,
 'capital-gain': 2844.8600000000006,
 'capital-loss': 253.21000000000004,
 'education-num': 4.035000000000002,
 'fnlwgt': 2824.270000000001,
 'hours-per-week': 14.259999999999996}

And this is the SQL implementation:

DROP DATABASE IF EXISTS sgd SYNC;
CREATE DATABASE sgd;

CREATE TABLE sgd.samples (
    age UInt8,
    workclass LowCardinality(String),
    fnlwgt UInt32,
    education LowCardinality(String),
    educationNum UInt8,
    maritalStatus LowCardinality(String),
    occupation LowCardinality(String),
    relationship LowCardinality(String),
    race LowCardinality(String),
    sex LowCardinality(String),
    capitalGain UInt32,
    capitalLoss UInt32,
    hoursPerWeek UInt32,
    nativeCountry LowCardinality(String),
    income LowCardinality(String),
    dt DateTime DEFAULT now()
)
ENGINE = MergeTree
ORDER BY dt;

CREATE TABLE sgd.weights (
    column String,
    w Float64
)
ENGINE = SummingMergeTree
ORDER BY column;

CREATE MATERIALIZED VIEW sgd.update_weights TO sgd.weights AS (
    WITH sum(x * w0) OVER () AS wTx
    SELECT
        column,
        - 0.01 * (1/(1+exp(-wTx)) - y) * x as w
    FROM (
        SELECT
            dt,
            key column,
            value x,
            y
        FROM
        (
            SELECT
                dt,
                income = '>50K' y,
                ['intercept', 'age', 'fnlwgt', 'educationNum', 'capitalGain', 'capitalLoss', 'hoursPerWeek'] keys,
                [1, age, fnlwgt, educationNum, capitalGain, capitalLoss, hoursPerWeek] values
            FROM sgd.samples
        )
        ARRAY JOIN
            keys AS key,
            values AS value
    )
    ANY LEFT JOIN (
        SELECT column, w w0 FROM sgd.weights FINAL
    ) USING column
);

After ingesting the first 1000 samples 1-by-1 into sgd.samples this is the result we are getting:

SELECT
    column,
    w
FROM sgd.weights
FINAL
ORDER BY column ASC

┌─column───────┬───────────────────w─┐
│ age          │  16.665000000000006 │
│ capitalGain  │  2844.8600000000006 │
│ capitalLoss  │  253.21000000000004 │
│ educationNum │   4.035000000000002 │
│ fnlwgt       │   2824.270000000001 │
│ hoursPerWeek │  14.259999999999996 │
│ intercept    │ 0.10499999999999998 │
└──────────────┴─────────────────────┘

Everything is looking fine, the results are the same as we got by running sklearn! Honestly, I was expecting some minor differences here but I’m not going to complain.

At this point, if you are familiar with ClickHouse you’ll realize that ingesting row by row is suboptimal. The good news is that it’s usually also suboptimal for gradient descent, and there are already solutions out there.

Mini batches

Stochastic gradient descent with mini-batches is essentially the same but instead of going sample by sample, a batch of N samples is processed in each step. The algorithm described in pseudo-code is basically:

Initialize the weights 𝑤
Iterate over all samples in batches of size b:
For each batch update weights as:

So, in plain English, exactly the same as before but the update step is the average over the entire mini-batch. If you are interested in reading more about the differences and when to use each one there are good resources out there, listing a few of them:

We can add mini-batches by slightly modifying the previous SGD implementation:

DROP DATABASE IF EXISTS sgd SYNC;
CREATE DATABASE sgd;

CREATE TABLE sgd.samples (
    age UInt8,
    workclass LowCardinality(String),
    fnlwgt UInt32,
    education LowCardinality(String),
    educationNum UInt8,
    maritalStatus LowCardinality(String),
    occupation LowCardinality(String),
    relationship LowCardinality(String),
    race LowCardinality(String),
    sex LowCardinality(String),
    capitalGain UInt32,
    capitalLoss UInt32,
    hoursPerWeek UInt32,
    nativeCountry LowCardinality(String),
    income LowCardinality(String),
    dt DateTime DEFAULT now()
)
ENGINE = MergeTree
ORDER BY dt;

CREATE TABLE sgd.weights (
    column String,
    w Float64
)
ENGINE = SummingMergeTree
ORDER BY column;

CREATE MATERIALIZED VIEW sgd.update_weights TO sgd.weights AS (
    WITH
        sum(x * w0) OVER (partition by dt) AS wTx,
        countDistinct(dt) OVER () AS b
    SELECT
        column,
        - (0.01/b) * (1/(1+exp(-wTx)) - y) * x as w
    FROM (
        SELECT
            dt,
            key column,
            value x,
            y
        FROM
        (
            SELECT
                dt,
                income = '>50K' y,
                ['intercept', 'age', 'fnlwgt', 'educationNum', 'capitalGain', 'capitalLoss', 'hoursPerWeek'] keys,
                [1, age, fnlwgt, educationNum, capitalGain, capitalLoss, hoursPerWeek] values
            FROM sgd.samples
        )
        ARRAY JOIN
            keys AS key,
            values AS value
    )
    ANY LEFT JOIN (
        SELECT column, w w0 FROM sgd.weights FINAL
    ) USING column
);

Finally, after running it with mini-batch = 5, meaning ingesting 5 rows each time, we get the following weights:

SELECT
    column,
    w
FROM sgd.weights
FINAL
ORDER BY column ASC

Query id: 28992088-d561-4835-9504-834b80816159

┌─column───────┬───────────────────w─┐
│ age          │  15.909999999999995 │
│ capitalGain  │  2868.7700000000013 │
│ capitalLoss  │              165.49 │
│ educationNum │   4.204999999999998 │
│ fnlwgt       │    341.324999999998 │
│ hoursPerWeek │  13.735000000000003 │
│ intercept    │ 0.04499999999999994 │
└──────────────┴─────────────────────┘

Note that the SQL code is also prepared to receive a flexible mini-batch size, so an arbitrary number of rows can be ingested every time. This will be extremely useful in case you are receiving data from a stream that is buffering events together, and flushing every certain seconds.

The only missing step now is to add categorical features. This will require using one-hot encoder in sklearn, but in our code is just a matter of changing the subquery that is pivoting the data, from:

SELECT
    dt,
    key column,
    value x,
    y
FROM
(
    SELECT
        dt,
        income = '>50K' y,
        ['intercept', 'age', 'fnlwgt', 'educationNum', 'capitalGain', 'capitalLoss', 'hoursPerWeek'] keys,
        [1, age, fnlwgt, educationNum, capitalGain, capitalLoss, hoursPerWeek] values
    FROM sgd.samples
)
ARRAY JOIN
    keys AS key,
    values AS value

To something like this:

SELECT
    dt,
    key column,
    value x,
    y
FROM
(
    SELECT
        dt,
        income = '>50K' y,
        [
            'intercept',
            'age',
            'fnlwgt',
            'educationNum',
            'capitalGain',
            'capitalLoss',
            'hoursPerWeek',
            'workclass:' || workclass,
            'education:' || education,
            'maritalStatus:' || maritalStatus,
            'occupation:' || occupation,
            'relationship:' || relationship,
            'race:' || race,
            'sex:' || sex,
            'nativeCountry:' || nativeCountry
        ] keys,
        [
            1,
            age,
            fnlwgt,
            educationNum,
            capitalGain,
            capitalLoss,
            hoursPerWeek,
            1,
            1,
            1,
            1,
            1,
            1,
            1,
            1
        ] values
    FROM sgd.samples
)
ARRAY JOIN
    keys AS key,
    values AS value

After repeating the process weights will be looking as

SELECT
    column,
    w
FROM sgd.weights
FINAL
ORDER BY column ASC

┌─column──────────────────────────────┬──────────────────────w─┐
│ age                                 │      6.489999999999998 │
│ capitalGain                         │      926.0200000000001 │
│ capitalLoss                         │                 107.03 │
│ education:10th                      │                  -0.02 │
│ education:11th                      │                 -0.065 │
│ education:12th                      │                   0.01 │
...
│ education:Masters                   │                   0.05 │
│ education:Prof-school               │                   0.03 │
│ education:Some-college              │                  0.035 │
│ educationNum                        │      1.259999999999999 │
│ fnlwgt                              │    -361.92500000000473 │
│ hoursPerWeek                        │      3.579999999999996 │
│ intercept                           │ -0.0050000000000000565 │
│ maritalStatus:Divorced              │   -0.09500000000000003 │
│ maritalStatus:Married-AF-spouse     │                  -0.01 │
...
│ maritalStatus:Widowed               │                  -0.01 │
│ nativeCountry:?                     │   0.005000000000000001 │
│ nativeCountry:Cambodia              │                   0.01 │
...
│ nativeCountry:Thailand              │                   0.01 │
│ nativeCountry:United-States         │  -0.009999999999999943 │
│ occupation:?                        │                  -0.02 │
│ occupation:Adm-clerical             │                  -0.03 │
...
│ workclass:?                         │                  -0.02 │
│ workclass:Federal-gov               │   0.019999999999999997 │
│ workclass:Local-gov                 │   0.009999999999999998 │
│ workclass:Private                   │                  -0.11 │
│ workclass:Self-emp-inc              │                   0.05 │
│ workclass:Self-emp-not-inc          │                  0.025 │
│ workclass:State-gov                 │   0.019999999999999997 │
└─────────────────────────────────────┴────────────────────────┘

Finally, to run a prediction we just have to run a query to retrieve the weights and apply the simple computations:

WITH
    sum(x * w0) OVER (partition by dt) AS wTx
SELECT
    1/(1+exp(-wTx)) AS p
FROM (
    SELECT
        dt,
        key column,
        value x
    FROM
    (
        SELECT
            now() dt,
            [
                'intercept',
                'age',
                'fnlwgt',
                'educationNum',
                'capitalGain',
                'capitalLoss',
                'hoursPerWeek',
                'workclass:' || workclass,
                'education:' || education,
                'maritalStatus:' || maritalStatus,
                'occupation:' || occupation,
                'relationship:' || relationship,
                'race:' || race,
                'sex:' || sex,
                'nativeCountry:' || nativeCountry
            ] keys,
            [
                1,
                age,
                fnlwgt,
                educationNum,
                capitalGain,
                capitalLoss,
                hoursPerWeek,
                1,
                1,
                1,
                1,
                1,
                1,
                1,
                1
            ] values
        FROM (
             SELECT
                20 age,
                'Private' workclass,
                148294 fnlwgt,
                'Some-college' education,
                10 educationNum,
                'Never-married' maritalStatus,
                'Other-service' occupation,
                'Own-child' relationship,
                'White' race,
                'Male' sex,
                0 capitalGain,
                0 capitalLoss,
                40 hoursPerWeek,
                'United-States' nativeCountry
        )
    )
    ARRAY JOIN
        keys AS key,
        values AS value
        )
ANY LEFT JOIN (
    SELECT column, w w0 FROM sgd.weights FINAL
) USING column
LIMIT 1

Conclusion

It’s possible to implement logistic regression in SQL, and thanks to ClickHouse and the Materialized Views we have managed to implement an online gradient descent algorithm capable of predicting events in real-time. This opens the door to a lot of possibilities.

Going back to the original example, the one about a retail company that wants to decide in real-time which product to show to a client based on the probability of buying them. We’d have to program the described algorithm in your database, build an easy way to ingest data in real-time from the online store to instantly update your model, and then provide an interface to run inference and return probabilities.

Lucky us, we don’t have to do that much! We already implemented the algorithm, and the interfaces to ingest and query data already come with the database.

Of course, the approach described here is simple, SGD is not the strongest model out there, but it’s doing its job and setting a good baseline. Perhaps you are not confident enough with SQL but I hope this post encourages others to run similar experiments, trying to get as much as they can from their day-to-day tools.

How I Made $350k in Financial Tournaments

Mon, 19 Oct 2020 16:00:00 GMT

Between 2015 and 2020 I have participated in Quantopian and Numerai tournaments. Being involved in these tournaments has led me to a very profitable journey: I made around $350k during these years.

Many people have reached out to me and asked how I did it. I decided to write this post to inspire others with this non-conventional path.

Selling a Trading Algorithm

In 2015 algorithmic trading was one of my interests. This is how I came across Quantopian, a platform that provides data and tools, so anyone can develop and test trading algorithms. After learning the basics and playing around with the platform, I decided to submit an algorithm to their recently created tournament.

The algorithm was pretty basic, but surprisingly it remained in the top 10 for a few weeks. I received many messages: companies suggesting joining them as a Financial Quant, people interested in my approach, and a few offers from people interested in buying the algorithm.

One of those potential buyers was particularly interested, shockingly enough, even after I explained to him how simple my algorithm was. I eventually shared the code with him for free. Still, he expressed his desire to send me the amount he offered in the first instance. And that’s how I made the first $2k out of participating in ML/financial tournaments.

Getting The Dream Job

During these weeks I exchanged many emails with Apple employees, financial quants, people from Europe, Asia, or America. But the best interaction came from a guy from my town. He was building his own trading algorithm and was interested in my approach. I meet him a couple of times. Obviously, his knowledge was years ahead of where I was at that time.

Those meetings were very productive, especially for me. After discussing with him about our personal and professional interests, and how difficult it was to find good jobs in our city, he introduced me to my next employer. Of course, I was not aware of it at that time. He introduced me to the CEO of an ad-tech company, and at first glance, I saw many things in common between this sector and high-frequency trading. The offered position (not publicly opened at that time) would also allow me to face and try to solve many machine learning, big data, and architectural challenges.

After a few weeks and a couple of meetings, I decided that the company was an excellent fit, a place where I would continue professionally growing and developing as a data scientist and engineer.

For years, we worked on building a solid platform that led us to be acquired in 2018. The experience was great and the outcome even better. Including the exit bonus, I made a big sum during this period.

The Altcoins Trading Madness

In 2016, I discovered the Numerai Tournament, “the hardest data science tournament on the planet” as they claim. Numerai is a weekly global data science tournament to solve the stock market. Data scientists work with clean, regularized, and obfuscated datasets provided by Numerai to create machine-learning models to predict the stock market. Numerai builds a meta-model of the most unique and contributive predictions and uses this to command the capital in its global equity hedge fund.

In early 2016 I had finished a few rounds in good spots, as you can see in the image. After a few months of actively participating, I ended 2016 with around 0.5 BTC. It was cool, but it required too much of my time. I had other priorities, so I withdrew the earnings and left the tournament.

Almost a year later, a post about a Numerai airdrop came across my timeline. After login back into my account, I discovered I received 1800 NMR.

At that time, the amount was valued around $30k, so I initially decided to withdraw almost everything, but something changed my mind and I started buying and selling altcoins. After some back and forth, and some lucky movements, I ended accumulating a wallet valued at around $100k.

2017 was arriving at its end and I took the last lucky movement, I withdrew half of that sum a few weeks before the bitcoin crash. I still hold the other half but it wasn’t easy to stick to that decision during 2018, while cryptos prices continued decreasing.

The Compound Effect

In July 2018, I came back to Numerai Tournament. I made some submissions but I didn’t take it seriously again until mid-2019.

Since then, I have had a cumulative return of +141%. I have built an algorithm with an average weekly return of +2%, which taking into account the compounding effect it translates to +180% annualized total return.

What’s next?

I’m going to continue participating in the Numerai Tournament, if everything continues as it is now, it would provide a good income without requiring much effort. I will also continue taking small bets, experimenting with other projects, continue doing things I like, and learning in the process.

How to detect locks on Redshift

Wed, 10 Feb 2016 00:00:00 GMT

When a query or transaction acquires a lock on a table, the lock remains for the duration of the query or transaction. Other queries or transactions that are waiting to acquire the same lock are blocked.

When you take a look to Redshift documentation they recommend you using STV_LOCKS, which results on:

It seems really useful until you have a real database lock. Last month I was trying to solve a lock that was blocking lots of processes. Finally, I found a better way to locate the queries that are causing locks:

Here you have the query itself:

SELECT current_time,
    c.relname,
    l.database,
    l.transaction,
    l.pid,
    a.usename,
    l.mode,
    l.granted
FROM pg_locks l
JOIN pg_catalog.pg_class c ON c.oid = l.relation
JOIN pg_catalog.pg_stat_activity a ON a.procpid = l.pid
WHERE l.pid <> pg_backend_pid();

source

Jordi Villar

2026W10

2026W07

Four Years at Tinybird

What is Tinybird?

Why Tinybird?

Culture

Remote

And the gossip?

2026W05

2026W04

2026W03

Trust AI, But Verify

The Results Looked Credible

Claude You Are Wrong

The Fixed Results

What Scares Me About This

The Echo Chamber

How I Actually Use AI

What I’m Still Working Out

Work only I can do

2026W00

2025W51

166 hours

2025W50

2025W49

64-bit Misalignment

What is 64-bit Memory Alignment?

The Plan

First Results: Wait, What?

Nothing Made Sense

Then Cache Hit Me

Proving It

The Pattern

What About the 64-bit Alignment Penalty?

What I Learned

Footnotes

2025W38

2025W31

Am I Becoming Irrelevant?

I Used to Feel Relevant

What Am I Actually Doing All Day?

What I Know Is Working

Why I’m Writing This

2025W23

Query Booster: How Tinybird optimizes table schemas for you

Why we built the Query Booster

Continuous Monitoring

Query Plan Analysis

Automatic Optimization

Performance Validation

Intelligence matters

2025W14

2025W11

2025W09

130 hours

Beyond Code

The Hard Skills Trap

What You Should Actually Be Good At

Ship It, Don’t Perfect It

The Skills They Don’t Teach in Bootcamp

Take It or Leave It

The Joy and Pain of Problem-Solving

Coding with AI

Footnotes

2024W28

2024W24

2024W23

Footnotes

/proc/thread-self

Footnotes

Data Engineering papers/articles

Resolving a year-long ClickHouse lock contention

The unknown limit

Some context about Tinybird and ClickHouse

Finding the root cause

Context refactoring

The Context architecture after our refactor

Bringing the CPU to 100%

Results with 23.9