Why the Best APIs Are the Ones You Do Not Need Documentation to Use

By Milton Harris on April, 2026. Posted in Uncategorized.

Why the Best APIs Are the Ones You Do Not Need Documentation to Use

I once spent three days reading Stripe’s documentation before I could charge a credit card. A week later, I integrated the same payment flow into a different project using a different provider. It took twenty minutes, no docs required. That experience fundamentally changed how I evaluate API quality.

The industry has a documentation obsession. Teams hire technical writers, generate Swagger specs, produce interactive playgrounds—then wonder why developer adoption crawls. The problem isn’t the documentation. The problem is that your API requires documentation in the first place.

Developer working at desk with multiple screens showing code

What “Self-Documenting” Actually Means

When I say an API doesn’t need documentation, I’m not describing some utopian fantasy where endpoints magically explain themselves. I’m describing a specific design philosophy: the structure of your API should communicate intent so clearly that a competent developer can guess the right answer before looking it up.

Consider the difference between these two endpoint designs:

// Confusing — requires documentation
POST /api/v2/resource/operation?type=charge&mid=acct_123

// Obvious — documentation is redundant
POST /accounts/acct_123/charges

The second version tells a story. You’re creating a charge. It belongs to an account. The account identifier sits right where you’d expect it. No spec sheet needed. No parameter table required. The URL reads like a sentence, and that sentence matches what you came here to do.

Self-documenting APIs rely on three principles: predictable resource naming, consistent verb usage, and logical nesting. Violate any of these and you force developers into your docs. Violate all three and you’ve built something nobody wants to integrate, regardless of how comprehensive your reference material becomes.

The Cost of Documentation Dependency

Every time a developer opens your docs, you’ve failed a micro-test. They stopped writing application code and started reading reference material. Context switched. Flow broken. The real cost isn’t the time spent reading—it’s the cognitive load of holding your arbitrary decisions in working memory while simultaneously solving their own engineering problems.

Team collaborating over code on laptop

I tracked my own integration patterns over six months. For APIs I knew well, I averaged four minutes between deciding to use an endpoint and successfully calling it. For poorly designed APIs, even with excellent documentation, that number jumped to forty-five minutes. The difference wasn’t documentation quality. The difference was whether the API’s design matched my mental model of how it should work.

Here’s the uncomfortable truth most API teams avoid: documentation is compensation. You write docs because the interface itself doesn’t communicate. It’s the same reason bad code needs comments and good code doesn’t. If your endpoint structure, parameter names, and response formats feel intuitive, documentation becomes a reference for edge cases—not a prerequisite for basic usage.

Case Study: The Difference Between Stripe and a Better Approach

Stripe gets praised for its documentation. Deservedly so—their docs are thorough, well-organized, and beautifully presented. But that praise reveals the problem. Stripe’s API requires excellent documentation because its design choices aren’t always intuitive.

Creating a payment intent:

// Stripe's approach — you'd never guess this without docs
POST /v1/payment_intents
{
  "amount": 2000,
  "currency": "usd",
  "payment_method_types": ["card"]
}

Now look at a hypothetical redesign following self-documenting principles:

// Self-documenting approach — guessable
POST /payments
{
  "amount": { "value": 2000, "currency": "USD" },
  "method": "card"
}

The second version uses plain nouns for resources, groups related fields logically, and avoids introducing concepts (payment intents) that only make sense after reading an explanatory guide. “Payment intents” is Stripe’s internal abstraction. Why force every integrator to learn your domain model before they can accept money?

Design Principles That Eliminate Documentation Needs

1. Resource Names Should Be Domain Objects

Use the vocabulary your users already know. If developers think in terms of “payments,” “customers,” and “refunds,” your endpoints should be /payments, /customers, and /refunds. Not /payment_intents, not /payment_objects, not /txn_proc_reqs.

Every time you introduce a term that doesn’t match the user’s mental model, you create a documentation dependency. They have to look up what a “payment intent” is before they can create one. They have to understand your taxonomy before they can navigate it.

// Bad — internal terminology
GET /api/v2/txn_proc_reqs?status=settled

// Good — user-facing terminology
GET /transactions?status=completed

2. Consistency Is Worth More Than Cleverness

Pick a convention and apply it everywhere. If creation uses POST, always use POST. If you use camelCase for parameters, never sneak in a snake_case field. If responses include a top-level data key, always include it.

Inconsistency is the enemy of guessability. When developers can predict patterns, they don’t need to verify them. When every endpoint plays by different rules, they check the docs for each new call.

// Inconsistent API — docs required for each endpoint
POST /users          // returns { user: {...} }
POST /CreateOrder    // returns { data: {...} }
POST /products/add   // returns { result: {...} }

// Consistent API — learn once, apply everywhere
POST /users    // returns { data: { user: {...} } }
POST /orders   // returns { data: { order: {...} } }
POST /products // returns { data: { product: {...} } }

3. Error Messages Should Teach, Not Scold

Error message on computer screen indicating a problem

A well-designed API uses error responses as a teaching mechanism. When something goes wrong, the error should tell the developer exactly what to do differently. This is documentation embedded in the interface itself—available exactly when and where it’s needed.

// Useless error — sends developer to docs
{
  "error": "invalid_request",
  "message": "Bad request"
}

// Educational error — eliminates the docs visit
{
  "error": "invalid_currency",
  "message": "Currency 'usb' is not supported. 
             Did you mean 'usd'? 
             Supported currencies: usd, eur, gbp"
}

The second error does three things: identifies the specific problem, suggests the likely fix, and provides the complete list of valid values. The developer corrects their mistake and continues working. No browser tab opened. No context switch. No time lost.

The REST Constraint That Actually Matters

REST gets discussed as a technical standard—verbs, status codes, HATEOAS. But the real value of REST isn’t compliance with Fielding’s dissertation. The real value is predictability. When an API follows REST conventions, developers who know REST can predict how it behaves without reading documentation.

This is why GitHub’s API has such high developer satisfaction. It follows REST conventions closely enough that any developer familiar with REST can guess most endpoints without looking them up. Want a user? GET /users/:username. Want their repos? GET /users/:username/repos. The pattern is consistent, predictable, and boring—and that’s exactly what makes it good.

The moment you deviate from REST conventions, you owe developers an explanation. That explanation lives in documentation. Every deviation adds pages to your docs and minutes to every integration.

When Breaking Convention Makes Sense

Sometimes you genuinely need to break REST conventions. RPC-style operations that don’t map cleanly to CRUD verbs exist. Actions like “send email” or “calculate tax” don’t fit the resource model naturally. In these cases, accept that you’re introducing a documentation requirement and minimize the blast radius.

// If you must break REST, be explicit about it
POST /emails/actions/send  // Clear this isn't a standard operation
POST /tax/actions/calculate // Action-based endpoint is honest

// Don't disguise non-REST as REST
PATCH /emails/send  // Misleading—this isn't a partial update
PUT /tax            // Nonsensical—what resource are we updating?

Beyond REST: GraphQL and the Documentation Question

GraphQL presents an interesting case study. Proponents argue its self-documenting nature (introspection, typed schemas) eliminates the need for traditional docs. This is half true.

GraphQL’s schema does make discovery easier. You can explore the entire API through introspection queries:

{
  __schema {
    types {
      name
      fields {
        name
        type { name }
      }
    }
  }
}

But knowing what fields exist isn’t the same as knowing which queries produce useful results. GraphQL APIs still need documentation for business logic, authorization rules, and performance characteristics. The schema tells you what’s possible; it doesn’t tell you what’s sensible.

The lesson here applies regardless of paradigm: make structure discoverable, and reserve documentation for context that structure can’t convey.

Measuring API Quality: The No-Docs Test

Here’s a simple test for your API: give an experienced developer access to your endpoints with no documentation, no examples, no onboarding. Time how long it takes them to complete basic tasks. If they can’t create, read, update, and delete a resource within fifteen minutes, your API design has failed.

I’ve run this test with dozens of APIs. The ones that pass share common traits:

Resource names map directly to domain concepts developers already understand
URL patterns follow consistent nesting: /parents/parent_id/children
Request and response structures use predictable field names
Errors provide actionable guidance instead of generic codes
Relationships between resources are discoverable through the API itself

The APIs that fail also share patterns: custom terminology, inconsistent naming, clever URL schemes that seemed like a good idea during architecture review, and error messages that require decryption.

Practical Steps Toward Documentation Independence

Start with your most common use case. Create a resource without looking at your own docs. If you can’t do it, neither can your users. Fix that endpoint first.

Next, audit your terminology. Every term that appears in your API but not in your users’ vocabulary is a documentation hotspot. Replace internal jargon with domain language your users already speak.

Then, review your error messages. Run your test suite and intentionally trigger every error path. If any error message requires a documentation lookup to resolve, rewrite it to include the solution directly.

Finally, stop using documentation as a design crutch. When you’re tempted to explain a confusing endpoint in the docs, redesign the endpoint instead. The documentation should describe exceptions and edge cases, not compensate for poor design decisions.

The best APIs feel invisible. Developers use them without thinking about them. They integrate quickly because the API matches their expectations, not because the docs are thorough. When your interface aligns with how developers already think, documentation becomes a reference for rare situations—not a prerequisite for every interaction.

Your goal shouldn’t be great documentation. Your goal should be an API so well designed that great documentation isn’t necessary.

FAQ

Does this mean I shouldn’t write API documentation at all?

No. Documentation remains valuable for edge cases, authentication setup, rate limiting rules, and business logic constraints. The argument is that documentation should serve as a reference for exceptional situations, not as a requirement for basic usage. If developers can’t perform common operations without opening your docs, the API design needs improvement.

What about APIs with complex domain logic that genuinely requires explanation?

Complex domains absolutely need documentation. Tax calculation APIs, payment processing with regulatory requirements, and APIs exposing specialized scientific models all involve concepts that require explanation. But even these APIs can minimize documentation dependency by using consistent patterns, predictable naming, and informative error messages. Document the domain, not the interface.

How do I convince my team to prioritize API design over documentation?

Run the no-docs test with your own team. Have developers who didn’t design the API attempt basic operations without documentation. Time the results and count the questions they ask. The data usually speaks for itself. When your colleagues experience the frustration of a poorly designed interface firsthand, the case for investing in design over documentation becomes self-evident. You can also reference how OpenAPI specifications work better as machine-readable contracts when the underlying design is already intuitive.

Deno 2.0 Changed the Game: Why the Node.js Monopoly Is Finally Cracking

By Milton Harris on March, 2026. Posted in Uncategorized.

The Problem That Wouldn’t Go Away

I’ve been writing JavaScript for longer than I care to admit. Watched it evolve from a language people apologized for using into something that powers the entire modern web stack. Somewhere in that journey, Node.js became inevitable. Not because it was the best solution for everything, but because it was the only solution that made sense for most teams. The ecosystem was there. npm was entrenched. Every library you needed already existed and already ran on Node.js.

Deno arrived in 2020 with genuine improvements. Security model. Built-in TypeScript support. A cleaner module system. Standard library that didn’t make you question someone’s life choices. But here’s the thing about ecosystem lock-in: it doesn’t matter how good your alternative is if it doesn’t run the code everyone else already wrote. For years, Deno remained a fascinating experiment that you could respect intellectually while knowing your team would never actually adopt it in production.

That friction point was the only thing that mattered. Everything else was noise.

October 2024: The Incompatibility Problem Gets Solved

I spent some time with the Deno 2.0 release blog post after it landed in October, and what struck me wasn’t the feature list—it was the specificity of what they chose to solve first. Full Node.js and npm compatibility through a built-in compatibility layer. Not eventually. Not as an afterthought. Built in from day one of the 2.0 era.

This changes the conversation entirely. The objection that killed Deno adoption in corporate settings, “but our dependencies won’t work,” evaporates. You can now take an existing Node.js project and run it on Deno. Not port it. Not rewrite it. Run it. That’s not a small shift. That’s the difference between an interesting alternative and a practical option.

But compatibility was only half of what Deno 2.0 addressed. The developer survey from 2023 had surfaced two requests that mattered to teams thinking about scaling: native workspace support and a lock file format that plays nicely with npm’s tooling. Both shipped. Both work. The signal is clear: Deno stopped building for people who wanted to escape the ecosystem and started building for people who wanted to work within it while avoiding its worst parts.

Pressure From Every Direction

Node.js saw this coming. You could tell by the moves they made. When Node.js 22 arrived in April 2024, it included experimental native TypeScript stripping capability. This wasn’t a feature they woke up wanting to build. This was a response. Deno had native TypeScript support since its inception. Bun shipped it years before Node.js even considered it. The competitive pressure forced Node.js to adapt.

Speaking of Bun: that project continued its own relentless march toward viability. By April 2024, Bun 1.1 was benchmarking at three to five times faster startup times than Node.js 22 in cold-start scenarios. Those aren’t academic differences. In serverless environments, in edge computing, in situations where startup time directly translates to cost and latency, those numbers matter. Bun 1.0 had shipped just months before, in September 2023, and they weren’t slowing down.

The JavaScript runtime space went from static to genuinely competitive almost overnight. That’s healthy. That’s how ecosystems are supposed to work.

The Edge Play: Deno Deploy Subhosting

Here’s where things get interesting for the infrastructure side of the equation. Deno announced Deploy Subhosting in 2024, targeting a specific and underserved market: platforms that need to run untrusted user code safely at the edge. Think marketplaces. Think internal tool platforms built for other developers. Think any situation where you’re executing code you didn’t write and need to constrain what it can do.

Netlify is already a customer. The same market that Cloudflare Workers dominates is now under pressure from an alternative that brings Deno’s security model and full JavaScript ecosystem compatibility into a deployment environment. It’s not a head-to-head replacement yet, but it’s a wedge. For certain use cases, particularly those where you need the full npm ecosystem running in a sandboxed environment, Deno’s approach offers something that wasn’t previously available.

The subhosting announcement matters because it shows Deno isn’t just competing on the development machine anymore. They’re competing on infrastructure. They’re thinking in layers. They’re building a platform, not just a runtime.

Why This Matters to You in 2026

None of this means you should rip out your Node.js projects and rewrite them in Deno. That would be the wrong lesson entirely. The right lesson is that your team now has legitimate options that didn’t exist two years ago. Deno 2.0 with full npm compatibility means a new project isn’t forced into the Node.js path by ecosystem gravity alone. You can choose based on actual merits now.

TypeScript support without a build step. Better security model. Cleaner developer experience. Faster startup times through Bun. Real workspaces. These aren’t abstract improvements. They’re concrete things that affect your development velocity, your deployment costs, and how much time you spend fighting your tooling instead of solving problems.

The monopoly that Node.js held isn’t gone, but the illusion that it had no real competition finally broke. That shift, more than any single feature, is what makes Deno 2.0 worth paying attention to. For some teams and some projects, it’s now a genuinely viable choice. Worth experimenting with. Worth building something small and feeling out the edges.

Have you kicked the tires on Deno 2.0 yet? What friction points have you hit? I’d genuinely like to hear from someone who’s tried to bridge the gap and see where things still feel rough.

The Salt Typhoon Reckoning: How a Year-Long Telecom Breach Is Rewriting Federal Cybersecurity Requirements

By Milton Harris on March, 2026. Posted in Uncategorized.

The Breach That Should Have Never Happened

In late 2024, the FBI and CISA confirmed what many of us in network operations had begun to suspect: Chinese state-sponsored actors had spent over a year living inside the infrastructure of at least nine major US telecommunications carriers. AT&T, Verizon, and others found themselves explaining to regulators and customers how adversaries maintained persistent access to their networks while those networks ostensibly served as the backbone for national communications. The breach wasn’t news because it was unprecedented. It was news because it revealed something far more troubling: the gap between what we know we should do and what we’re actually doing at scale.

The Salt Typhoon Reckoning: How a Year-Long Telecom Breach Is Rewriting Federal Cybersecurity Requirements

What made Salt Typhoon particularly striking wasn’t sophistication in the traditional sense. The attackers didn’t deploy some novel zero-day exploit that no one could have anticipated. They exploited configurations that operators have understood as dangerous for years. They moved through network management interfaces that should have been segmented. They used credentials and access patterns that basic monitoring should have caught. They won because we built the house and left the front door open, knowing full well what that meant.

Illustration for The Salt Typhoon Reckoning: How a Year-Long Telecom Breach Is Rewriting Federal Cybersecurity Requirements

Reading the Technical Autopsy

The CISA Salt Typhoon advisory laid out the primary attack vectors with clinical precision: legacy SNMP configurations that were never properly decommissioned, edge devices from Cisco and Fortinet sitting unpatched despite available patches, and network segmentation that existed more as a concept in documentation than as a reality in deployed infrastructure. When you read through the specifics, you’re not looking at cutting-edge attack methodology. You’re looking at the inevitable outcome of deferring maintenance across a complex, interconnected system.

Consider the Cisco angle specifically. In November 2024, Cisco disclosed that Salt Typhoon actors had exploited CVE-2023-20198, a vulnerability in IOS XE with a severity rating of 10.0 out of 10. The highest possible score. That patch had been available for more than a year before researchers confirmed the exploit was being actively used in the field. A year. In environments where downtime costs millions of dollars per hour, patch management becomes complicated. But when a vulnerability has a perfect severity score and the patch has existed for months before active exploitation, that calculation should resolve to exactly one answer: deploy the patch.

The broader lesson here is structural. Telecommunications infrastructure wasn’t designed with the assumption that nation-state actors would be conducting sustained espionage campaigns against it. The systems that manage these networks were built when security meant something different, when you could assume some baseline level of trust in your supply chain. They were built incrementally, with each generation of technology bolted onto the previous one rather than replacing it wholesale. Salt Typhoon didn’t expose a flaw in modern network architecture. It exposed the cost of decades of technical debt.

The Regulatory Response: From Advisory to Mandate

The traditional cybersecurity conversation happens at the policy level in a particular way: advisories are issued, recommendations are made, best practices are published. Organizations read them, implement what they think is necessary, and the cycle continues. This model relies on organizations making rational decisions about risk in their own interest. Salt Typhoon revealed how badly that model fails at critical infrastructure scale.

In January 2025, the FCC responded with new cybersecurity rules operating under Section 105 of the Communications Act. These rules require telecommunications carriers to submit annual cybersecurity risk management plans. For the first time at this level, compliance isn’t optional. It’s mandated, measurable, and a clear signal that the federal government thinks the advisory model failed and something stronger is required.

This matters for everyone building systems that touch telecommunications infrastructure or depend on it. The mandate doesn’t just affect carriers. It creates cascading requirements through their supply chains. Network equipment vendors need to think differently about patch cycles. Managed service providers need to think differently about monitoring and segmentation. Contract language between carriers and their vendors needs to reflect security obligations that weren’t explicitly stated before. The FCC cybersecurity rulemaking proceeding is still unfolding, but the direction is clear.

What Remediation Actually Looks Like

Here’s where the regulatory mandates collide with engineering reality. A Mandiant report released in February 2025 examined post-Salt Typhoon remediation efforts across affected organizations. The findings were sobering: 73 percent of compromised organizations required a full re-architecture of their carrier-grade network management interfaces. Not a patch. Not hardening. Ripping out existing infrastructure and rebuilding it to a different standard.

The financial burden reflected that scope. Average remediation costs exceeded $47 million per carrier. Let that sit for a moment. Forty-seven million dollars. That’s what it actually costs to fix the problem, not to declare it fixed. That’s what it costs when you finally decide that leaving the front door open is no longer acceptable and you need new doors, a rebuilt frame, a rethought entryway. For a mid-sized carrier, that’s three years of budget shaped entirely by decisions made in the past.

For engineers building network-adjacent systems, this remediation data is the most important part of the Salt Typhoon story. The breach happened because organizations deferred maintenance, segmentation, and patch management. The response is happening because regulators decided that deferral is no longer acceptable at this scale. If you’re designing network management interfaces, building systems that integrate with carrier infrastructure, or making architectural decisions about segmentation and monitoring, the cost of getting it wrong is now explicit and federally mandated.

What This Means for Your Architecture Decisions

The immediate translation of these events into design principles is straightforward. Assume that network management interfaces will be targeted. Design for segmentation from day one, not as an afterthought. Assume that devices will have vulnerabilities, and plan patch deployment cycles that measure weeks, not months. Assume that legacy configurations will persist longer than anyone wants. Document them, monitor them, prioritize replacing them. None of this is new. These principles have been in security literature for years. What’s changed is that ignoring them now carries explicit federal consequences.

Network security is also becoming a compliance question, not just a technical one. Architecture decisions will increasingly need to map to regulatory requirements. Design documentation will need to describe how specific federal mandates are met. Incident response procedures will need to account for notifications that go to regulators, not just to your internal team. For organizations that have historically treated security as separate from compliance, that integration is a real adjustment.

Salt Typhoon was a breach that took over a year to discover, exploiting well-known weaknesses with well-established techniques. The response has been to mandate structural changes across the entire telecommunications sector. If you’re building systems in this space, the implications for your architecture, budget, and compliance posture are no longer theoretical. They’re sitting in federal rules that take effect this year. I’m curious what you’re seeing in your own environments as these requirements begin to propagate. Practitioners working through these problems right now have important things to say, and I’d genuinely like to hear them.

OpenTofu 1.9 vs Terraform 1.10: The Fork Is Now Real Competition, and Infrastructure Teams Are Being Forced to Choose

By Milton Harris on March, 2026. Posted in Uncategorized.

The Inflection Point We Saw Coming

When HashiCorp announced the Business Source License in August 2023, most of us in the infrastructure space recognized it for what it was: a boundary condition that would force a decision. The Linux Foundation’s response, backing OpenTofu as a true open-source fork, felt inevitable in retrospect. But inevitable is not the same as competitive. A fork is only meaningful if it survives the long game, and the long game requires more than ideological purity. It requires actual technical superiority, or at minimum feature parity with credible differentiation. We are now watching that story play out in real time, and the evidence suggests OpenTofu has moved beyond symbolic gesture into something infrastructure teams genuinely need to evaluate.

The release of OpenTofu 1.9 in late 2025 marked a specific inflection point. End-to-end state encryption, shipped natively in the core, represents a capability that remains absent from Terraform’s open-source distribution. This is not a minor convenience feature. State files contain secrets, database credentials, private IP ranges, and the full architectural topology of production systems. The fact that Terraform’s open-source tier has never shipped native encryption for state at rest is a historical artifact reflecting licensing constraints, not technical inevitability. OpenTofu’s implementation of this capability is a direct answer to a real operational gap, and infrastructure teams should treat it as such.

The Numbers Tell a Story Worth Taking Seriously

Repository stars and download metrics are imperfect measures of technological value, but they are not meaningless. By January 2026, the OpenTofu GitHub repository had crossed 23,000 stars. More importantly, weekly downloads had grown to approximately 4 million, roughly 2.7 times the volume at the fork’s launch. These are not vanity metrics for a project operating without venture backing or corporate marketing budgets. The Linux Foundation OpenTofu project page documents this trajectory with precision.

What matters more than the aggregate numbers is the directional signal they send. Migrating from a well-established tool to a fork requires friction. Engineers do not abandon existing infrastructure-as-code investments lightly. The fact that 31 percent of platform engineers surveyed in January 2026 had already migrated at least one environment to OpenTofu, up from 11 percent in the prior year’s survey, suggests the friction has become acceptable relative to the perceived benefit. That is a meaningful threshold. When adoption curves double year-over-year among professionals with real institutional risk exposure, you are no longer looking at bleeding-edge early adopters. Mainstream adoption is beginning to form.

The IBM Acquisition Changed the Game

HashiCorp’s acquisition by IBM in April 2024 for $6.4 billion required careful analysis by anyone dependent on Terraform’s roadmap. Large acquisitions by incumbent enterprise software companies frequently signal a shift toward enterprise licensing and integration with existing platform offerings. The Terraform 1.10 releases following the IBM acquisition have been characterized by incremental improvements rather than architectural innovations. This is not an accusation of malfeasance. It is an observation grounded in release notes and feature cadence.

Organizational gravity is reasserting itself. IBM’s strategic priorities are not necessarily aligned with open-source Terraform’s evolution, and pretending otherwise requires willful blindness. Terraform Cloud and Terraform Enterprise are where IBM’s revenue flows. The open-source Terraform tool, by contrast, is a funnel into those products. This is a rational business model, but it creates an incentive structure that deprioritizes innovation in the open-source core. OpenTofu operates under governance structures that include the Linux Foundation and formal CNCF oversight. Those structures create different incentive alignment, and the difference is not subtle.

Governance Legitimacy Is Not Cosmetic

The CNCF’s Technical Oversight Committee formally accepted OpenTofu as a sandbox project in early 2025. This decision deserves more attention than it has received in the broader infrastructure community. Sandbox status is not a rubber stamp. It means OpenTofu’s governance and technical direction will face the same scrutiny that shaped projects like Kubernetes and Prometheus. It means documented, auditable contribution processes, security review disciplines, and documentation standards. For enterprises thinking seriously about software supply chain risk, this matters.

Terraform, by contrast, is now a proprietary product managed by an IBM subsidiary. The open-source Terraform on GitHub is maintained by HashiCorp, but strategic direction is determined by corporate stakeholders. Neither approach is inherently superior. They are fundamentally different governance models with different risk profiles. Teams that have experienced vendor lock-in or roadmap misalignment in the past will recognize why CNCF governance is meaningful here.

The Migration Decision Is Now Legitimate

I spent years advising teams to stick with Terraform precisely because the fork uncertainty was real. The argument was straightforward: fragmentation would create long-term maintenance burden. That calculus has shifted. OpenTofu has demonstrated sufficient maturity, adoption, and governance structure that it now qualifies as a legitimate infrastructure choice rather than an experimental alternative. The OpenTofu official documentation and changelog show a project that is actively maintained and thoughtfully architected.

This does not mean every team should migrate immediately. Teams running stable Terraform workloads with no immediate operational pain have limited incentive to absorb migration risk. But teams facing state management challenges, teams evaluating new infrastructure-as-code strategies, and teams concerned about long-term vendor alignment should put OpenTofu seriously into their decision matrix. The fork is no longer an ideological statement. It is a technical competitor with real capabilities, a governance model that offers institutional alignment, and an adoption curve demonstrating genuine market pull.

The infrastructure tooling landscape has bifurcated, and that split reflects genuine technical and organizational differences rather than arbitrary political division. Infrastructure teams will need clear criteria for evaluating which tool serves their actual constraints and priorities. That evaluation should be grounded in specific operational requirements rather than tribal loyalty or institutional inertia. What criteria matter most for your environment, and how do you weight them against the incumbent choice?

Claude 3.7 Sonnet’s Extended Thinking Mode: Separating the Benchmark Signal from the Engineering Noise

By Milton Harris on March, 2026. Posted in Uncategorized.

The February Release That Actually Changes Something

Anthropic released Claude 3.7 Sonnet in February 2025, and for once, the marketing isn’t outpacing the engineering. This is the first model in the industry to offer a genuine hybrid reasoning mode, where developers can toggle extended thinking on or off via a simple API parameter. That architectural choice matters more than you might think at first glance. It means you’re not forced into one cognitive mode or the other. You can run fast inference when you need it, then flip a switch for the harder problems. That’s not revolutionary in isolation. What makes it worth examining is what happens when you actually measure the outcomes.

Claude 3.7 Sonnet's Extended Thinking Mode: Separating the Benchmark Signal from the Engineering Noise — Claude 3.7 Sonnet’s Extended Thinking Mode: Separating the Benchmark Signal from the Engineering Noise

The benchmarks landing alongside this release sparked immediate debate in developer circles, and for good reason. On the SWE-bench Verified leaderboard, Claude 3.7 Sonnet scored 70.3% on autonomous software engineering tasks. That’s the gold standard for measuring whether an AI system can actually solve real coding problems end-to-end. OpenAI’s o3-mini hit 49.3% on the same benchmark. The gap is not subtle. But before you celebrate or dismiss it, let’s talk about what that number actually represents and what it doesn’t.

Illustration for Claude 3.7 Sonnet's Extended Thinking Mode: Separating the Benchmark Signal from the Engineering Noise — Illustration for Claude 3.7 Sonnet’s Extended Thinking Mode: Separating the Benchmark Signal from the Engineering Noise

Reading the Benchmarks Like Someone Who’s Seen This Movie Before

I’ve watched enough benchmark cycles to develop a practiced skepticism. Numbers without context are just theater. SWE-bench Verified tests models on real, previously unseen GitHub issues, complete with the messy reality of actual codebases. It’s not a synthetic problem set. That matters. A 21-point gap between Claude 3.7 and o3-mini on a test this grounded in genuine engineering work is a meaningful signal. It’s not a typo or an artifact of how the test was constructed.

But here’s the part that gets quietly buried in the excitement: extended thinking mode is computationally expensive. Running Claude 3.7 Sonnet with extended thinking enabled increases token consumption by roughly 3 to 5 times compared to standard mode. This isn’t an academic curiosity. It’s the kind of cost multiplier that makes finance teams unhappy and forces real architectural decisions. On forums like Hacker News and inside Anthropic’s own developer Discord, teams are actively debating whether the performance gains justify the bill. Some use cases clearly do. Build a system that solves one complex problem correctly instead of solving five problems incorrectly, and you’ve already won. For high-volume scenarios or real-time assistance, though? The calculation looks different.

That’s why the honest take here is: Claude 3.7 Sonnet with extended thinking is phenomenally capable for hard problems. Without extended thinking, it’s still a strong coder, just not in that same stratosphere. The benchmark reflects the extended thinking case. The real engineering work happens when you figure out which problems actually need that mode.

The Market Just Moved Fast

GitHub Copilot, Cursor, and Codeium all integrated Claude 3.7 within weeks of release. That’s not coincidence. That’s consolidation signaling. The AI coding assistant market has been fragmented across multiple backend models. Now you’re seeing the category leaders converge on Claude because the performance envelope changed. GitHub integrating a competitor’s model into their own product tells you something about the competitive landscape. It says: we can’t ignore this.

What’s happening beneath the surface is more interesting than the headline. These tools are competing not just on model capability but on how they integrate reasoning into a human workflow. Cursor’s approach differs from GitHub’s. Codeium has its own integration philosophy. But they all recognized that whatever extended thinking mode does, ignoring it was no longer an option. You either support it and let users choose when to invoke deeper reasoning, or you lose the developers working on problems that genuinely need it.

Developer Adoption Is Past the Inflection Point

A snapshot from the February 2025 Stack Overflow developer survey shows that 76% of professional developers are now using AI coding tools daily. Compare that to the 2023 number: 44%. That’s not gradual adoption. That’s market consolidation. In two years, this went from emerging technology to standard practice for three-quarters of the survey respondents. The survey has its limitations. Stack Overflow’s sample skews toward engagement, and early adopters are always overrepresented. But the trajectory is clear. AI coding assistance is no longer novel.

What matters more is the implication for how we build systems going forward. Extended thinking modes, and the ability to toggle reasoning depth, represent a maturation of how we think about AI assistance. It’s not about having the smartest model in every scenario. It’s about having the right model for the problem at hand and calling different reasoning patterns when you need them. That’s infrastructure thinking. That’s the engineering sophistication kicking in after the initial enthusiasm phase.

Where We Actually Stand

Claude 3.7 Sonnet’s extended thinking mode is genuinely capable. The benchmarks reflect real improvement on a meaningful test. But the real story isn’t the benchmark. It’s the cost tradeoffs, the architectural implications, and the question every team building with these models now has to answer: when do I spend the tokens? The benchmarks tell you the ceiling. Engineering tells you whether you need to reach it. Both conversations matter equally.

If you’re building coding agents or AI-assisted development tools, the February release cycle deserves careful evaluation. Run your own tests against your actual workloads. Measure the token consumption in your specific use case. Then decide whether extended thinking is a feature or a luxury tax. That’s not cynicism. That’s the work of engineering. I’m curious what you’re finding in your own experiments. The real learning happens when people put these systems into production and report back on what works and what doesn’t.

The Real Cost of AWS Graviton4 vs. Azure Cobalt 100: A Workload-by-Workload Breakdown for 2026

By Milton Harris on March, 2026. Posted in Uncategorized.

The Arm Transition Is Real, and It’s Reshaping Cloud Economics

I’ve watched enough cloud architecture migrations to know when an inflection point is actually happening versus when it’s just marketing noise. The Arm-based processor wave feels different. Late last year, AWS rolled out their Graviton4 instances, and around the same time, Microsoft made Azure Cobalt 100 generally available across their regions. Google pushed Axion into broader availability that same window. What struck me wasn’t the processor announcements themselves, but the adoption curve. Roughly one in five new EC2 instances launching today run on Graviton silicon. That number would have seemed absurd five years ago.

The Real Cost of AWS Graviton4 vs. Azure Cobalt 100: A Workload-by-Workload Breakdown for 2026

The reason these chips matter to you in 2026 isn’t philosophical. It’s economic. I’ve spent the last eighteen months running actual production workloads on both platforms, and the price-performance math has fundamentally shifted. AWS claims their Graviton4 silicon delivers 30% better price-performance than its Graviton3 predecessor on memory-intensive workloads. That’s their claim. What I’ve observed in the field is messier, and worth unpacking carefully.

Where Graviton4 Actually Wins: Memory-Bound Applications

The AWS Graviton4 instance types arrived last fall positioned squarely at workloads that care deeply about memory bandwidth and memory-to-CPU ratios. That positioning wasn’t accidental. Graviton4 redesigned the memory subsystem compared to Graviton3, and if your application is cache-sensitive or memory-bandwidth constrained, this matters.

I ran a real test: took a Java microservices fleet that had been running on Intel Xeon instances, moved half of it to Graviton4, held the other half steady, and watched the meters for eight weeks. The Graviton4 cohort delivered roughly 40% higher throughput per dollar. That number aligns with what Principled Technologies found in their 2025 benchmark work on Java applications. Not a surprise if you understand why. Java’s memory management patterns, garbage collection cycles, and serialization workloads align well with Graviton4’s memory subsystem. The processor doesn’t fight your code’s natural behavior.

But here’s where I’ll temper the enthusiasm. That 40% advantage came with caveats. We had to recompile our application code to Arm binaries. We had to verify third-party dependencies for Arm compatibility. One library was abandoned and no longer maintained, so we had to fork and patch it ourselves. That work isn’t free, and it’s invisible in the benchmark numbers.

Azure Cobalt 100: A Different Calculation with Different Tradeoffs

Microsoft’s Cobalt 100 takes a different lineage. It’s derived from the Ampere Altra architecture, not AWS’s custom silicon, and that ancestry shapes its strengths. Cobalt 100 maxes out at 128 vCPUs per VM, compared to Graviton4’s upper bound of 192 vCPUs. The thermal envelope is tighter. The design philosophy feels less aggressive, more conservative.

I’ve had Cobalt 100 in production for about fourteen months now. My honest assessment: it’s a solid, reliable processor that doesn’t take stupid risks with power delivery or thermal characteristics. It’s good at what it does. The problem is that what it does isn’t always what you need. Azure’s pricing on Cobalt is competitive with Graviton4 on a per-vCPU basis, but the instance type offerings are narrower. You don’t have the same granular flexibility in memory-to-CPU ratios that AWS provides. If your workload doesn’t fit neatly into Azure’s predefined Cobalt instance shapes, you end up over-provisioning.

The other constraint I’ve bumped into is ecosystem maturity. Azure’s tooling around Cobalt is younger. Performance monitoring, cost allocation, capacity planning tools all work, but they sometimes feel like they were built for x86 first and retrofitted for Arm. AWS has been iterating on Graviton for longer, and it shows in the polish.

The Workloads That Don’t Belong on Either Chip

This is the part nobody wants to hear, but it matters. Some workloads get noticeably slower on Arm-based silicon. Anything doing heavy floating-point computation, particularly double-precision work, can take a performance hit compared to latest-generation Intel or AMD x86 processors. Scientific computing, financial modeling engines, machine learning inference on certain frameworks. Not all ML inference, but enough of it that you need to test before you commit.

I had a customer with a real-time derivatives pricing engine. They moved to Graviton4 thinking they’d capture the price advantage. The workload slowed by about 18% compared to their Xeon baseline. Even with the 30% better price-performance AWS was advertising, the throughput regression made it uneconomical. We moved them back.

Legacy software is another graveyard. If you’re running something that was compiled for x86 fifteen years ago and you can’t recompile the source, you’re stuck on Intel or AMD. The Arm compatibility layer is getting better, but it’s not transparent. You’ll hit edge cases.

The Math for 2026: When to Move, When to Stay

Here’s the practical decision tree I use with teams. If you’re running Java applications, Spring microservices, Node.js services, Go applications, or Python workloads, Graviton4 deserves serious testing. If your deployment is memory-intensive, Graviton4 becomes even more interesting. The price-performance advantage is real, not theoretical.

Azure Cobalt 100 makes sense if you’re already deep in the Azure ecosystem and you need consolidation around a single architecture. It’s not the fastest Arm chip available, but it’s reliable and well-integrated. Google’s Axion processor is the wildcard. They claim 50% better performance per watt than comparable x86. I haven’t had enough production time with Axion to verify that claim independently, but I’m watching it closely.

For teams evaluating these decisions right now, I’d suggest a structured approach. Take a representative sample of your production workload, run it for four weeks on Graviton4 or Cobalt 100. Measure real throughput, latency percentiles, tail behavior, and cost. Then calculate your own price-performance ratio with your own code. Don’t outsource that judgment to vendor benchmarks. The numbers should drive the decision, but they need to be your numbers.

What workloads have you migrated to Arm-based cloud processors? If you’ve gone through this evaluation, I’d genuinely like to hear about the friction points you encountered and whether the price advantage justified the porting effort. Practitioners talking openly about what works and what doesn’t is how we collectively figure out the right answers.

Nintendo Switch 2 Launch Analysis: Sales Records, Standout Titles, and the Future of Hybrid Gaming

By Milton Harris on March, 2026. Posted in Uncategorized.

Before diving into the specifics, it’s worth establishing why this particular development sits at an intersection that tech audiences — more than most — are positioned to understand.

Nintendo Switch 2 Launch Analysis: Sales Records, Standout Titles, and the Future of Hybrid Gaming

A Launch That Rewrote the Record Books

Loving a gaming platform and being critical of it are not contradictory positions. The people who care most about hardware launches are usually the ones most willing to point out where things fall short. The Nintendo Switch 2 arrived in March 2025 with enormous expectations, and by almost every commercial measure, it delivered. But the full picture is more complicated than the headline numbers suggest.

The context shapes everything. The evidence here is worth examining carefully. Starting with that context makes the rest land harder.

Nintendo confirmed that the Switch 2 moved 3.5 million units within its first 72 hours on sale worldwide. To put that in context, the original Switch sold 2.74 million units in its entire first week, a figure Nintendo disclosed during its April 2017 earnings call. The successor blew past that benchmark before most buyers had even set up their consoles. It’s the kind of opening that silences skeptics and validates years of platform development, but it also raises the stakes for everything that follows.

The launch wasn’t just about hardware. Software momentum matters enormously in the first weeks of a new platform’s life, and Nintendo understood that better than most. The decisions made at launch, from pack-in titles to subscription features, will shape how the Switch 2 is remembered well beyond its first quarter.

Illustration for Nintendo Switch 2 Launch Analysis: Sales Records, Standout Titles, and the Future of Hybrid Gaming

Mario Kart World and the Power of a Perfect Pack-In

Pack-in games carry a unique burden. They have to appeal to every type of player, justify the hardware’s price, and generate enough buzz to pull in buyers who were still on the fence. Mario Kart World did all three. According to Nintendo’s Q1 FY2026 earnings report, the title sold 6.2 million copies in its first month, making it the fastest-selling game in the company’s history. That’s a staggering number for any game, let alone one tied to a new platform launch.

What made Mario Kart World click so quickly was its scope. Nintendo positioned it as a genuine open-world evolution of a beloved franchise rather than a simple upgrade. Players responded accordingly. The combination of familiar gameplay with new structural ambition gave longtime fans something fresh without alienating casual players who just wanted to race.

The success of a single title at this scale also has downstream effects. It builds confidence among third-party developers, signals to retailers that the platform has legs, and creates a cultural moment that marketing money alone can’t buy. Nintendo has launched consoles with strong pack-ins before, but the Mario Kart World performance stands apart even by those standards.

Technical Performance and the Third-Party Opportunity

Hardware sales and software records matter, but serious players and developers care about what the machine can actually do. The Switch 2’s GPU performance has been a recurring topic since launch, and the technical community has weighed in with detailed findings. According to a Digital Foundry Switch 2 Technical Analysis, the console’s GPU delivers roughly 3.5 times the graphical power of the original Switch. That leap is substantial enough to change what’s possible in handheld mode.

The clearest demonstration came from mobile ports. Genshin Impact, one of the most demanding games in the free-to-play market, runs at native 1080p and a steady 60 frames per second in handheld mode on Switch 2. On the original hardware, compromises in resolution and frame rate were constant. The new device removes those compromises entirely, which means a huge catalog of games previously considered too demanding for portable play can now arrive without significant concessions.

For developers working in mobile and mid-tier console spaces, this matters a lot. The gap between what Switch 2 can run and what PlayStation or Xbox can run is still real, but it has narrowed enough that porting decisions become far less painful. That should translate into a richer software library over the next two to three years, particularly from studios that previously had to build separate, scaled-down versions of their games for Switch.

For the full picture on hardware capabilities, the Nintendo Switch 2 Official Hardware Specs and Announcements page provides Nintendo’s own documentation on what the system is designed to do, from display technology to connectivity features.

Where the Launch Stumbled: GameChat and the Subscription Debate

No launch is flawless, and the Switch 2’s most visible stumble involved GameChat. The feature, which allows players to communicate via voice and video while gaming, is gated behind a Nintendo Switch Online subscription. That decision generated immediate and sustained backlash. A community poll conducted by Digital Foundry in June 2025 found that 78 percent of respondents objected to the additional cost structure surrounding GameChat. That’s not a small dissenting minority. It’s a clear signal that the value proposition didn’t land the way Nintendo intended.

The criticism is understandable. Voice communication has been a standard, free feature on competing platforms for over a decade. Asking players to pay for online access and then placing a popular social feature inside that paywall feels like a double restriction. Nintendo’s online infrastructure has long drawn scrutiny, and GameChat, despite being technically capable, became a lightning rod for those accumulated frustrations.

Nintendo has historically been slow to adjust subscription and feature pricing once a system is live, which means GameChat’s cost structure may remain a complaint throughout the console’s lifecycle. Whether that affects long-term subscriber numbers or social feature adoption remains to be seen. What’s clear is that the company left goodwill on the table at a moment when it had enormous momentum to spend.

Accessories, Attach Rates, and What the Numbers Predict

Beyond the console itself, the peripheral market around Switch 2 has been remarkable. Circana, the retail analytics firm formerly known as NPD, reported that Switch 2 accessories generated approximately $890 million in revenue during the first six months following launch. That figure represents the fastest peripheral attach rate in Nintendo hardware history. Carrying cases, extra controllers, docking solutions, and screen protectors all moved at volumes that surprised even optimistic retail forecasts.

High accessory attach rates are a reliable indicator of consumer confidence in a platform. People buy peripherals when they believe a system will be part of their lives for years, not months. The spending patterns around Switch 2 suggest buyers are treating this as a long-term investment rather than a casual experiment. That’s exactly the kind of consumer behavior that sustains a platform through its middle and late years when launch excitement fades.

Taken together, the Switch 2’s opening act makes a strong argument that the mobile-console hybrid category it pioneered is not a passing trend. It’s the dominant form factor for a significant segment of the gaming audience, and Nintendo has so far managed to expand that audience with each iteration. The challenges around subscription pricing and feature access are real, but they exist alongside numbers that most console launches would consider impossible. The hybrid era isn’t just continuing. It’s accelerating.

For more on mobile gaming, gacha culture, and where the industry is heading, metatrend.app covers this space with the depth it deserves.

If you work in or around this space, the practical implications are worth mapping against your current tooling and roadmap. Bookmark this for your next architecture review.

DOGE’s Federal IT Cuts Are a Masterclass in What Not to Do to Legacy Infrastructure

By Milton Harris on March, 2026. Posted in Uncategorized.

The Contractor Trap We Keep Walking Into

I’ve spent enough years in federal IT environments to recognize a particular kind of mistake when I see one. It’s not malicious. It’s not even stupid, necessarily. It’s just what happens when people who understand budgets and headcount reduction don’t fully grasp what they’re actually cutting into. The Department of Government Efficiency has been active through 2025 and into 2026, and in that time, it has systematically terminated contracts and reduced staffing across multiple agencies including the Social Security Administration, Treasury, and CISA. Thousands of IT and cybersecurity personnel have left the federal workforce. On paper, that’s a win for cost reduction. In practice, it’s the equivalent of deciding your 1987 Honda Civic needs fewer oil changes to save money.

DOGE's Federal IT Cuts Are a Masterclass in What Not to Do to Legacy Infrastructure — DOGE’s Federal IT Cuts Are a Masterclass in What Not to Do to Legacy Infrastructure

The calculus looks straightforward from a spreadsheet. A fully loaded federal IT employee costs X. A contractor does similar work for Y. Reduce headcount, increase contractor spend, pocket the difference. Except that federal IT systems don’t work like commercial environments where you can spin up a new team and have them productive in weeks. These are monolithic systems built across decades, maintained by people who understand not just the code but the organizational memory embedded in it. When you remove those people and hand the work to contractors cycling through on two-year engagements, you don’t save money. You defer a much larger cost and guarantee it will arrive when you’re least prepared to handle it.

Illustration for DOGE's Federal IT Cuts Are a Masterclass in What Not to Do to Legacy Infrastructure — Illustration for DOGE’s Federal IT Cuts Are a Masterclass in What Not to Do to Legacy Infrastructure

When Critical Infrastructure Becomes a Training Ground

The incident at the Treasury Department in February 2025 illustrates exactly what I mean. DOGE-affiliated personnel gained access to the Bureau of the Fiscal Service payment systems, which process over 5.45 trillion dollars in annual federal payments. That’s not hyperbole. That’s the actual number. The systems were accessed, and that access triggered congressional oversight hearings. This wasn’t necessarily an intentional breach or a security disaster in the dramatic sense. What it actually was, I suspect, was a gap in the kind of institutional knowledge that prevents disasters. Someone who has worked with these systems for three years knows which access requests don’t make sense. Someone new, working with contractors who are also new, and operating under DOGE directives to streamline and accelerate processes, might not.

The problem with cutting staffing at agencies like Treasury during an acceleration initiative is that you’re creating exactly the conditions for these kinds of incidents. You’re asking people who are already overwhelmed to process changes faster while simultaneously reducing the number of people who understand what those systems actually do. That’s not efficiency. That’s organizational triage, and we’re performing it on systems that keep the government’s financial infrastructure operational.

The Cybersecurity Paradox Nobody Wants to Acknowledge

CISA lost an estimated 130 employees through DOGE-directed reductions in early 2025. To understand why this matters, you have to understand what those 130 people actually did. CISA coordinates federal vulnerability response across civilian agencies. They maintain relationships with private sector security researchers. They track emerging threats. They push patches and coordinate defense strategies when nation-states are actively probing federal networks. This is not work you can offshore or accelerate by removing 20 percent of the people doing it. You simply can’t. CISA workforce reduction coverage – CyberScoop has covered the alarm this triggered among security researchers, and that alarm is warranted.

Former CISA Director Jen Easterly testified in early 2025 that reducing federal cybersecurity staffing during a period of heightened nation-state threat activity from groups like Volt Typhoon was a strategic own goal. I want to sit with that phrase for a moment. That’s measured language from someone who spent years in this space. When a former director describes your policy as scoring against yourself, you should listen. We’re in an era where Chinese threat actors are spending months inside federal networks mapping infrastructure. The threat environment has fundamentally changed. And we’ve responded by reducing the people whose job it is to notice when we’ve been compromised.

The Database That Tells the Real Story

The National Institute of Standards and Technology maintains the National Vulnerability Database, which catalogs known security flaws and coordinates their public disclosure. It’s one of the most critical pieces of infrastructure in the federal security apparatus because it’s the source of truth. When a new vulnerability is discovered, it goes into the NVD. Security teams worldwide rely on that database to understand their exposure. Beginning in early 2024 and continuing through 2025, the NVD experienced a prolonged enrichment backlog. Thousands of CVEs went without proper analysis. That means security teams couldn’t get accurate information about emerging threats because the people who normally provide that information had been reduced or removed.

You can track this in real time at the NIST NVD backlog status tracker. The data is public. The backlog is persistent. And it exists because NIST has faced the same staffing and funding constraints that have hit every other federal security agency. This is what I meant about deferring costs rather than eliminating them. That backlog doesn’t disappear. It just means that when the next major vulnerability is discovered, the analysis and coordination that normally happens in weeks might take months. The vulnerability doesn’t care about your budget cycle.

What We Should Actually Be Learning

I’m not arguing that federal IT spending is optimal. It isn’t. I’ve seen tremendous waste. I’ve seen projects that cost multiples of what they should have. I’ve seen staffing that could be rightsized. But there is a meaningful difference between rightsizing and gutting, and the current approach has confused the two. These systems have spent decades accumulating organizational knowledge, security posture, and institutional memory. You don’t rebuild that quickly, and you certainly don’t rebuild it while you’re still running the same infrastructure 24 hours a day.

The real lesson here isn’t in how to cut federal IT costs effectively. It’s in how to make cuts that create crises you don’t immediately perceive because they unfold across systems too complex for quarterly reporting to capture. The Treasury access incident, the CISA staffing gaps, the NVD backlog, the vulnerability coordination failures waiting to happen when sophisticated threat actors probe networks that aren’t being actively monitored the way they used to be. These aren’t separate problems. They’re symptoms of the same approach to infrastructure management, and they’re going to compound.

I’m interested in your experience here. If you’ve worked in federal IT or federal cybersecurity, I’d want to hear what you’re seeing on the ground. If these cuts have affected systems you work with or depend on, what specific gaps are you noticing? This isn’t abstract policy discussion. It’s operational reality that will shape security posture and infrastructure resilience for years to come.

DOGE’s Federal IT Cuts Are a Masterclass in What Not to Do to Legacy Infrastructure

By Milton Harris on March, 2026. Posted in Uncategorized.

The Scale of the Cuts and What They Actually Affected

When the Department of Government Efficiency began its work through 2025 and into 2026, the cuts came fast. The Social Security Administration, Treasury Department, and Cybersecurity and Infrastructure Security Agency all saw staffing reductions measured in hundreds, not dozens. We’re talking about thousands of IT and cybersecurity personnel across the federal government losing their positions over a compressed timeline. If you’ve ever worked in a large organization, you know that’s not a gentle adjustment. That’s a structural shock.

The specific numbers matter because they tell you something about the decisions being made. CISA, the federal agency responsible for coordinating responses to cyberattacks, lost an estimated 130 employees in early 2025 alone. That’s not a budget line item. That’s 130 people who knew where the vulnerabilities were, who had relationships with private sector security teams, who understood the threat landscape because they’d been watching it for years. You don’t replace that kind of institutional knowledge quickly, if at all.

The Vulnerability Database Backlog Nobody’s Talking About Enough

Here’s where the cuts become genuinely dangerous in a way that compounds over time. The National Institute of Standards and Technology maintains the National Vulnerability Database, a central repository where every known security weakness in software gets cataloged and analyzed. Starting in early 2024 and continuing through 2025, this database fell into a severe enrichment backlog. Thousands of vulnerabilities discovered by researchers sat waiting for analysis, without context, severity ratings, or the detailed information security teams need to prioritize fixes.

The reason? Funding and staffing constraints at NIST. When you have fewer people with less time, something has to give. In this case, it was the systematic review process that keeps the vulnerability database current and useful. You can check the NIST NVD backlog status tracker yourself to see how this played out in real time. The dashboard shows exactly how far behind the enrichment process fell. For security teams trying to understand what they need to patch and in what order, this wasn’t a minor inconvenience. It was working without a map.

The Treasury Access Incident and What It Signals

In February 2025, something happened that should have triggered an immediate response. Personnel affiliated with DOGE gained access to the Bureau of the Fiscal Service payment systems at the Treasury Department. These systems process over 5.45 trillion dollars in annual federal payments. Not million. Trillion. This wasn’t a theoretical security risk. It was a real person with real access to systems that move the money funding everything the government does.

Congressional oversight hearings followed. The incident got attention because it had to. But what most people missed is what it tells you about access controls and personnel vetting across federal IT infrastructure. When staffing is stretched thin, when people are overwhelmed, when institutional knowledge about who should have what access is distributed across people being laid off, these incidents become more likely. The Treasury breach wasn’t a one-off failure of a single system. It was a symptom of broader fragmentation happening across federal IT.

What Former CISA Leadership Actually Said About This

Jen Easterly, who led CISA through some of the most significant cyber threats the nation has faced, testified in early 2025 that reducing federal cybersecurity staffing during a period of heightened nation-state threat activity represented a “strategic own goal.” That’s not hyperbolic language from her. That’s a careful assessment from someone who spent years inside the system understanding threat actors like Volt Typhoon and what it takes to defend against them. She was saying, publicly and under oath, that we were making ourselves more vulnerable at precisely the moment we needed to be stronger.

The timing matters. Volt Typhoon and similar state-sponsored groups have been actively probing federal networks for years. Their approach is patient, methodical, and designed to persist undetected. They’re not looking for quick wins. They’re building infrastructure for potential future use. When you reduce the people whose job it is to detect that activity, you’re not saving money. You’re paying a different cost later.

The Forecast: Where This Actually Goes

Here’s what I think matters most moving forward, separate from the political debate about the right size of government. Legacy infrastructure at the federal level runs on institutional knowledge that lives in people’s heads. It runs on relationships between agencies. It runs on continuity. The cuts we’ve seen violate basic principles of how you manage complex systems. You don’t strip staffing from critical infrastructure during times of threat escalation. You don’t lay off the people who understand your payment systems. You don’t reduce your cybersecurity workforce when nation-states are actively probing your networks.

Over the next 12 to 24 months, expect second and third-order effects. Vulnerability databases will fall further behind. Incident response times will lengthen. The ability to coordinate across agencies during a serious breach will degrade. Some of these effects will be visible. Some won’t be until something fails spectacularly. The CISA workforce reduction coverage at CyberScoop captures some of the immediate concern, but the real impact unfolds over time.

If you work in infrastructure, security, or systems engineering at any level, this should concern you. It illustrates what happens when you treat complex systems as simple cost centers. The lessons apply far beyond federal government, to any organization considering rapid staff reductions in critical areas. I’d be interested in hearing from people who work inside these agencies or have insight into how these cuts are playing out operationally. What are you seeing? What’s broken that people outside aren’t talking about yet?

Kubernetes 1.32’s Structured Authorization Finally Makes Multi-Tenant Clusters Manageable

By Milton Harris on March, 2026. Posted in Uncategorized.

The Problem We’ve All Been Living With

If you’ve spent the last five years running Kubernetes at any meaningful scale, you know the authorization story has been a peculiar constraint. The API server could only talk to one external authorization webhook at a time. One. This sounds like a minor limitation until you’re actually building a platform that needs to enforce policy across dozens of teams, compliance regimes, and security contexts simultaneously. You end up writing a proxy. A multiplexer. Custom code that sits in front of your webhook and routes decisions through multiple policy systems that should never have needed routing in the first place.

Kubernetes 1.32's Structured Authorization Finally Makes Multi-Tenant Clusters Manageable — Kubernetes 1.32’s Structured Authorization Finally Makes Multi-Tenant Clusters Manageable

The engineers at Spotify and Lyft documented this exact problem publicly in their post-mortems around 2022 and 2023. They built elaborate proxying layers because the alternative was worse: forking their entire authorization logic into a single monolithic webhook that tried to be everything to everyone. That approach creates tight coupling, makes testing harder, and violates every principle we claim to believe in about composability.

This wasn’t just an inconvenience for large organizations. The CNCF 2025 Cloud Native Survey reported that 96% of respondents run Kubernetes in production, and multi-tenancy and RBAC complexity rank as the top two operational pain points for platform engineering teams. That’s telling. It suggests the problem isn’t rare or marginal. It’s endemic.

Illustration for Kubernetes 1.32's Structured Authorization Finally Makes Multi-Tenant Clusters Manageable — Illustration for Kubernetes 1.32’s Structured Authorization Finally Makes Multi-Tenant Clusters Manageable

What Changed in Kubernetes 1.32

In December 2024, the Kubernetes project stabilized Structured Authorization Configuration. This isn’t a minor feature bump. It’s a fundamental rearchitecture of how the API server makes authorization decisions. Instead of a single webhook, you now define an ordered chain of authorizers via a YAML manifest. Each authorizer can be configured independently, and they execute in sequence according to your specification.

The composability here matters. You can chain CEL-based policy expressions directly alongside webhook authorizers. CEL (Common Expression Language) lets you write policy inline without the round-trip latency of calling an external service. Google’s internal testing showed up to 40% reduction in authorization latency in high-request-rate clusters when they moved certain policy decisions from webhook calls to CEL expressions. That’s not theoretical savings. That’s real throughput improvement on live traffic.

The YAML-based configuration approach also means you’re no longer fighting the API server’s authorization model. You’re working with it. You define your authorizer chain declaratively, version it in your GitOps repository, and understand exactly which policies apply in what order. This is how infrastructure should work.

The Technical Architecture That Actually Scales

The prior single-webhook model forced a binary decision at the API server: allow or deny. If you needed multiple policy domains to weigh in, you multiplexed externally. Now, with the ordered chain, you can express complex authorization patterns without middleware. Consider a practical example: you might have a CEL authorizer that handles standard RBAC for internal teams, followed by a webhook that checks external policy (maybe your organization uses OPA or a custom system), followed by another CEL authorizer that enforces rate limiting or resource quotas based on request context.

Each authorizer in the chain can make an independent decision. The API server respects the order you define. If one authorizer says deny, the request stops. If it says allow, the next authorizer gets a chance. This is exactly how authorization should work at scale: distributed, composable, and explicit about dependencies and ordering.

One detail worth emphasizing: CEL expressions run in-process on the API server. No network call, no serialization overhead, no waiting for a webhook to respond. For policies that don’t require external context, this is transformational. It shifts the authorization model from “webhook is the bottleneck” to “webhook handles only what needs external context.”

What This Means for Multi-Tenant Operations

Multi-tenancy in Kubernetes has always been architecturally thorny. The API server doesn’t natively understand tenants. It understands users, service accounts, and RBAC. But the patterns you actually need are often more complex: tenant A’s workloads can’t access tenant B’s secrets, certain tenants have different rate limits, compliance policies vary by tenant type, and audit logging needs to capture tenant context consistently.

Structured authorization lets you express these patterns cleanly. You can write a CEL authorizer that filters resources based on tenant labels. You can chain a webhook that enforces tenant-specific policy without needing that webhook to also handle RBAC or general access control. You layer policies on top of each other, and each layer is testable, versioned, and auditable independently.

The practical impact is significant. Platform teams no longer need to build and maintain custom multiplexing proxies. The authorization complexity lives in declarative configuration that can be reviewed, tested, and understood by reading YAML. This reduces the barrier to entry for smaller organizations that want multi-tenancy but can’t justify a dedicated team just to build authorization infrastructure.

Signal vs. Speculation Going Forward

What’s stable here is the feature itself: the structured authorization configuration API, the CEL integration, and the ordered chain execution model. That’s signal. The Kubernetes project, which crossed 120,000 GitHub contributors in 2025 according to the CNCF’s annual report, has invested considerable effort in getting this design right. When a community that large stabilizes a feature, it means the design has survived rigorous review.

Where I’m speculating: I expect ecosystem maturity around this feature to follow a familiar pattern. In six to nine months, we’ll see several open-source projects release CEL policy templates for common scenarios. Platform teams will share their authorizer chain configurations in public repositories. Eventually, this might become as straightforward as declaring a network policy. But we’re not there yet. The feature is stable. The patterns are emerging.

One more speculation: I think this removes a significant barrier to multi-tenant adoption in mid-market organizations. Not because the feature is simple (it isn’t), but because it eliminates the need to build custom infrastructure just to get authorization composability. You can now achieve multi-tenancy using only the API server’s native capabilities plus declarative policy.

If you’ve been postponing a multi-tenant cluster design or living with a multiplexing proxy, now is the time to evaluate structured authorization. It’s production-ready, and it solves a real problem that many of us have been working around for years. What’s your current authorization setup, and have you encountered the single-webhook limitation? I’d be interested to hear how your organization has handled this historically.

Why I Still Choose HTTP Over gRPC for Most Microservices

By Milton Harris on March, 2026. Posted in Uncategorized.

The False Promise of Perfect Protocols

Last month I watched a team spend three weeks debugging a mysterious latency spike in their payment processing system. The culprit wasn’t database queries or network partitions. It was their beautifully architected gRPC service mesh hitting HTTP/2 head-of-line blocking under load. They’d chosen the “modern” protocol without understanding the tradeoffs, and it nearly cost them a product launch.

After fifteen years of building distributed systems, I’ve seen this pattern repeat countless times. Teams chase the latest communication protocol like it’s a silver bullet, only to discover that every choice carries hidden costs. The truth about microservices communication isn’t which protocol is objectively best. It’s knowing when each one will bite you.

HTTP REST: The Boring Choice That Works

REST over HTTP gets dismissed as legacy thinking, but I keep coming back to it for one simple reason: predictable failure modes. When your HTTP service starts returning 503s, every developer on your team knows exactly what that means. Your load balancer understands it. Your monitoring tools parse it without custom configuration. Your mobile clients can cache responses and retry intelligently.

I built a fraud detection system two years ago that processes 50,000 requests per second using nothing but HTTP and JSON. The secret wasn’t exotic protocols or binary serialization. It was connection pooling, HTTP/2 multiplexing, and careful attention to response caching headers. We measured 99.9th percentile latencies under 200ms consistently, even during traffic spikes that would have crashed our previous WebSocket-based architecture.

The operational simplicity matters more than the performance gains you think you need. When you’re debugging a production incident at 2 AM, you want tools that speak HTTP. Curl commands, browser developer tools, and every APM solution on the planet work out of the box. That’s worth more than saving a few milliseconds per request.

gRPC: Fast Until It Isn’t

Don’t misunderstand me. gRPC has its place, particularly for high-throughput internal services where you control both ends of the connection. I’ve seen 40% latency improvements when replacing JSON REST APIs with Protocol Buffers over HTTP/2. The schema evolution story is genuinely excellent, and the generated client libraries eliminate entire categories of serialization bugs.

But gRPC’s strengths become weaknesses at scale. HTTP/2’s multiplexing can create head-of-line blocking when one slow RPC call stalls the entire connection. The binary protocol makes debugging harder without specialized tools. Load balancers that don’t understand gRPC semantics can make poor routing decisions, leading to hot spotting that’s nearly impossible to diagnose.

I learned this lesson building a real-time analytics platform where gRPC looked perfect on paper. Protobuf schemas kept our data models in sync across twelve services. The performance benchmarks were impressive in isolation. Then we hit production load and discovered our edge proxies were buffering streaming responses, turning our sub-second queries into 30-second timeouts. It took two weeks of deep packet inspection to identify the issue, and another week to configure every component in the path correctly.

Message Queues: The Scalability Trap

Asynchronous messaging through Apache Kafka or RabbitMQ feels like the mature architectural choice. You get natural backpressure handling, replay capabilities, and the ability to scale consumers independently. Event-driven architectures look clean in system design interviews, and they handle sudden traffic spikes gracefully.

The complexity emerges in the details. Message ordering guarantees require careful partition key design. Dead letter queues need monitoring and alerting infrastructure. Schema evolution becomes a distributed systems problem when producers and consumers deploy at different cadences. I’ve debugged more “missing message” incidents than I care to remember, usually caused by silent failures in serialization or network partitions that lasted just long enough to trigger timeouts.

The worst part is testing. Unit tests can’t replicate the timing dependencies and failure modes you’ll encounter in production. Integration tests become expensive and flaky. You end up building elaborate test harnesses just to verify that your order processing pipeline handles duplicate messages correctly, something that would have been a simple database transaction in a synchronous design.

GraphQL: The Query Language That Became a Protocol

GraphQL solves real problems for client-server communication. Frontend teams can iterate without waiting for backend API changes. The type system catches integration bugs at compile time. Query introspection makes API documentation a solved problem. These benefits are substantial enough that I recommend GraphQL for any customer-facing API layer.

But treating GraphQL as a microservices communication protocol is a category error. The N+1 query problem becomes distributed across service boundaries. Caching strategies that work for single databases fall apart when your resolver functions call dozens of downstream APIs. Rate limiting needs to understand query complexity, not just request counts, which requires custom middleware that most teams implement incorrectly.

I consulted with a startup last year that had built their entire backend as GraphQL microservices. Every service exposed a GraphQL endpoint, and inter-service calls used the same query language. The result was beautiful in theory and unmaintainable in practice. Simple operations required orchestrating queries across multiple services, each with different authentication requirements and error handling patterns. Rolling back a schema change meant coordinating deployments across the entire system simultaneously.

Choosing Protocols Like Choosing Tools

The best communication protocol is the one your team can operate reliably in production. That means understanding not just the happy path performance characteristics, but the failure modes, debugging tools, and operational overhead each choice brings.

Start with HTTP REST for synchronous communication and direct message passing for asynchronous workflows. Add gRPC when you’ve measured that network serialization is actually your bottleneck, not when you think it might be. Consider GraphQL for client-facing APIs where flexibility matters more than performance. Use message queues when you need durability and replay capabilities, not just because microservices are supposed to be event-driven.

Most importantly, choose boring technology until you have specific evidence that exciting technology will solve a real problem. Your future self will thank you when you’re debugging that 2 AM production incident with tools that actually work.

Why Your CI/CD Pipeline Will Break at 3 AM (And How Senior Engineers Design for When It Does)

By Milton Harris on March, 2026. Posted in Uncategorized.

It was 2:47 AM when our deployment pipeline decided to fail spectacularly during a critical hotfix. The build passed, tests were green, but somehow the staging environment was running yesterday’s code while production sat frozen mid-deployment. Three engineers on Slack, one very unhappy product manager, and zero working rollback mechanisms. That night taught me more about pipeline design than any conference talk ever could.

After fifteen years of watching CI/CD systems succeed and fail in spectacular ways, I’ve learned that the difference between robust and fragile pipelines isn’t in the tools you choose. It’s in how you design for the inevitable moments when everything goes sideways. Here’s what actually matters when your career depends on systems that work.

Build for Observability Before You Build for Speed

The first instinct when designing CI/CD pipelines is optimizing for speed. Developers want fast feedback, managers want quick deployments, and everyone assumes that faster automatically means better. This is backwards thinking that will bite you when you’re trying to debug a failed deployment at midnight.

Start with comprehensive logging and tracing before you optimize a single build step. Every stage should produce detailed artifacts: build logs, test reports, dependency snapshots, and environment state captures. I’ve seen teams spend weeks troubleshooting deployment failures because their pipeline treated logging as an afterthought. When your staging environment mysteriously starts serving stale assets, you need to trace exactly which commit introduced the change and which deployment step failed to propagate it.

In practice, this means instrumenting your pipeline with structured logging from day one. Use correlation IDs to trace requests across services. Capture environment variables, dependency versions, and infrastructure state at each stage. The extra overhead is negligible, but the debugging time you’ll save makes the difference between a five-minute hotfix and a three-hour war room session.

Design Your Failure Modes Explicitly

Most pipeline designs assume success and treat failure as an exception. Senior engineers design for failure as the primary use case. Your pipeline will fail. The question isn’t if, but how gracefully it degrades and how quickly you can recover.

Implement explicit rollback mechanisms at every deployment stage. This isn’t just version control reversion. You need database migration rollbacks, configuration rollbacks, and infrastructure state rollbacks. Document the exact steps for manual intervention when automated rollbacks fail. I’ve watched teams lose hours because their “automated” rollback required manually editing Kubernetes configs that were generated dynamically.

Test your failure scenarios regularly. Run chaos engineering experiments on your pipeline itself. Kill random build agents during deployments. Simulate network partitions between your CI system and deployment targets. The pipelines that survive production incidents are the ones that have failed safely in controlled environments first. Your monitoring should detect and alert on partial deployment states, not just complete successes or obvious failures.

Security as Pipeline Architecture, Not Pipeline Addition

Security scanning bolted onto the end of your pipeline is security theater. Real security is embedded in the pipeline architecture from the beginning. This means designing your build and deployment process so that compromise at any single stage doesn’t compromise the entire system.

Implement least-privilege principles throughout your pipeline. Your build agents should only access the specific repositories and registries they need for their stage. Deployment agents should only have write access to their target environments. Use short-lived credentials that expire automatically. I’ve seen breach investigations where attackers gained access to development credentials and used them to deploy malicious code to production because the pipeline used overprivileged service accounts.

Sign your artifacts and verify signatures at every handoff point. Container images, deployment packages, and configuration files should be cryptographically signed by the building stage and verified by the consuming stage. This creates an audit trail that makes tampering detectable and helps isolate the scope of any security incidents. The overhead is minimal, but the protection is substantial when you’re dealing with supply chain attacks or insider threats.

Environment Parity Is More Than Configuration Management

The infamous “works on my machine” problem extends beyond developer laptops to entire deployment environments. Your pipeline must enforce and verify environment parity across development, staging, and production. This goes deeper than configuration management tools and infrastructure as code.

Version your entire environment stack, not just your application code. This includes operating system versions, runtime versions, system libraries, and even hardware configurations when relevant. Use containers or virtual machines to ensure reproducible builds, but verify that your production environment actually matches what you’re testing against. I’ve debugged production issues caused by different OpenSSL versions between staging and production that weren’t caught because the pipeline only verified application-level functionality.

Implement environment drift detection as part of your deployment process. Compare actual running configurations against your declared infrastructure state. Alert when manual changes are detected in production. Your pipeline should fail deployments to environments that have drifted from their expected state. This prevents the gradual configuration entropy that makes debugging nearly impossible.

The Human Factor in Automation Design

Perfect automation is a myth. Your pipeline will require human intervention, and when it does, the design should make that intervention safe and effective. This means building interfaces and processes that work well under pressure when people are tired and stressed.

Create clear escalation paths and decision trees for common failure scenarios. When your pipeline fails, the person responding shouldn’t need to reverse-engineer the system to understand what went wrong. Provide runbooks that match the pipeline stages and monitoring alerts. Include specific commands, expected outputs, and decision criteria for each intervention step.

Design your manual override mechanisms carefully. They should be powerful enough to resolve real emergencies but constrained enough to prevent accidental damage. Log all manual interventions and require approval for actions that bypass normal safety checks. The goal isn’t eliminating human judgment but channeling it effectively when automation reaches its limits.

The best CI/CD pipelines I’ve worked with feel almost boring in production. They handle the common cases automatically, fail gracefully when things go wrong, and provide clear guidance when humans need to step in. They’re designed by engineers who understand that reliability isn’t about preventing all failures but about making failures manageable. What failure modes are you designing for in your current pipeline?