When I zoom out on my over twenty years of professional experience in developing software, and running teams developing software, the one word that I center on is maintainability. The code you write has a primary job to “tell the computer what to do”, and it will do exactly that and nothing more. While many tools have been successful at launching “no-code” solutions, from WYSIWYG tools of the 1990s to more modern “vibe coding” (and hello, WordPress!), for the foreseeable future, humans will need to be able to read, understand, and maintain long-standing code whether or not they are the ones who wrote it in the first place.

The evolution of programming languages paints the history of making code more maintainable by humans, not always in a linear path. Ever since the discovery of using a transistor and binary code (1s and 0s) to instruct a a machine to “compute” and accomplish its assigned task, the goal has been to abstract that language into a way that humans can read, process, reason about and augment. Through layers of abstraction, from machine code to assembly to higher levels of abstraction (FORTRAN, Pascal, C) and into modern languages (Java, Python, Golang, Rust) and less-familiar paradigms like functional programming (LISP, Haskell, Elm) the goal has been to instruct the computer what to do in a way that other humans can evaluate it. Even today, when using modern AI coding tools like Cursor, Windsurf, or Warp, most of the examples shown these days produce the code that is being generated is this higher-level code and not directly machine code.

When this code is generated, it needs to be managed and ultimately versioned. Because this code needs human review, it needs to be maintained in a way that humans can reason about the differences. With this ability to reason, humans can set checkpoints at known-good points in history, and roll back if any changes are found objectionable. This also provides an opportunity to establish collaboration, as interoperable systems need a means for establishing a contract for that interoperability. If functionality changes, the ability to identify that change via a version number is crucial.

Source code management (SCM) tools have existed almost as long as programming languages themselves. Earliest tools were command-line based, and when graphical user interface (GUI) systems were introduced, graphical front-ends were added as well. There have been many different attempts at both centralized and distributed tools, ranging from the original SCCS, to CVS and Subversion (SVN), to ClearCase and SourceSafe, but the most popular tool today is git. Invented by Linus Torvalds, the original author and lead maintainer of of the Linux kernel that drives the operating system by the same name, which is used extensively across modern computing devices.

For the remainder of this post, we’ll presume that readers are already familiar with the basics of git. We’re going to get deep here, so if you’re not up on your git-fu, I’d suggest using git’s own gittutorial or one of these others from W3 or Atlassian before proceeding.

Read more

Mobile applications have become ubiquitous and essential over the past two decades, particularly for direct-to-consumer products. While Steve Jobs initially believed smartphones didn't need third-party apps, it's now clear that installable apps are indispensable as part of the marketing, engagement and retainment strategy for companies large and small. The mobile landscape has simplified to just two platforms—iOS and Android (farewell to Blackberry, Palm, and Windows Phone, in addition to some also-rans)—yet most apps still need to maintain a presence on both. For teams in their early stages, the primary challenge is the cost of developing for these two platforms as weighed against a cross-platfrom solution that attempts at broad reuse of the same code. Let’s explore the available options and their relative merits.

Native Development: The Traditional Approach

When developing natively for mobile platforms, developers must navigate two distinct ecosystems: iOS and Android.

For iOS development, Apple provides Swift—a modern, safe, and efficient programming language—along with SwiftUI, their latest declarative framework for building fluid, responsive user interfaces. Swift's focus on safety and clarity, combined with SwiftUI's intuitive approach to UI development, enables developers to create sophisticated applications that feel natural on Apple devices. Some libraries, especially those that have been around for a while, may include Objective-C, with idiosyncrasies for iOS, or more broadly across Apple platforms, which date back to the NextStep OS that Jobs brought back to Apple as the foundation for what was originally Mac OS X (now known as macOS) and proliferated across all the other flavors (eg. tvOS, watchOS, iPadOS, and visionOS). It’s taken a few years for SwiftUI to gain prominence, but the approach stabilized in the past two years, but it is still not fully backwards compatible, which requires developers to reach back into the older UIKit framework, which relies on an imperative approach, to cover some corner cases of Apple platform development.

Android’s foundation was built on Java, with its own unique challenges, such as performance of the JVM and how it handles garbage collection. For modern Android development, developers primarily use Kotlin, an expressive language with strong Java compatibility, and Jetpack Compose, Google's modern toolkit for building native UIs. Android there is still the occasional need to fall back to the legacy XML-based UI approach, such as supporting rich notifications on older versions

Jetpack Compose and SwiftUI both use a declarative approach, helping developers build complex interfaces with cleaner code while maintaining the performance and flexibility of native applications. This declarative paradigm is valuable as it does bring the platforms closer together by letting the developer “declare” what they want the UI to look like and leave it up to the system to render it appropriately.

While both native approaches offer the current best-in-class experience for each of the platforms, neither are very well easily translatable to the other. For example, Android's greater flexibility within the operating systems is not easily duplicated in iOS, but this comes at the cost of great complexity. Similarly Android, especially with Google’s layer that sits on top of the open-source platform (AOSP), has some stronger opinions on how certain things should be built, while Apple’s approach is to provide some guidance while also forcing the developer to do more of the heavy lifting at times.

While native development offers the best performance and full access to platform features, providing unparalleled speed and responsiveness along with seamless integration of device capabilities, it comes with some potential trade-offs: the necessity of maintaining entirely separate codebases for iOS and Android, increased development resources to support multiple platform-specific teams, and the added complexity of managing parallel development cycles and testing processes across both platforms. Depending on the complexity of the app, this approach may require a broader developer background to leverage the full extent of each platform. On the flip side, it usually allows the teams to stay as close to the platform without an extra layer of indirection that needs to be understood and can potentially lag behind the native platform.

This leads to an interesting set of product questions. Most notably, how do you create a single product experience that leverages both platforms to their maximum extent, even when those extents are mutually incompatible with each other?

Web-Based Solutions: The Browser Approach

For teams with web expertise or simpler app requirements, web-based solutions offer viable alternatives:

Progressive Web Apps (PWAs)

These modern web applications leverage standard web technologies to deliver app-like experiences directly through the browser, offering broad platform compatibility and simplified deployment. PWAs provide seamless accessibility across devices without requiring installation from app stores, while automatic updates ensure users always have the latest version.

Although they excel in reach and maintainability, PWAs face certain limitations regarding deep platform integration and may not match the full capabilities of native applications when it comes to hardware access and system features. Additionally, in the face of changing regulations, Apple temporarily signaled that PWAs would not be allowed on iOS home screens. While this decision was reversed within a couple of months, there still remains the possibility that PWAs will lose abilities or otherwise be constrained over the long term.

Hybrid Applications

By leveraging WebView-based solutions such as Cordova or Capacitor, development teams can create applications that seamlessly blend web technologies with native platform capabilities. These hybrid frameworks enable developers to write applications using familiar web technologies like HTML, CSS, and JavaScript while still accessing device features through a comprehensive plugin ecosystem. This approach allows for efficient development cycles by maintaining a single codebase that can target multiple platforms, while still providing access to platform-specific functionality when needed. The WebView container acts as a bridge between web content and native features, offering a balanced compromise between development efficiency and platform integration. This approach is optimal if the same components are needed to not only be available on both iOS and Android, but in a traditional web setting across desktop and mobile browsers as well.

Cross-Platform Frameworks: Universal Solutions

For teams seeking a single codebase approach, but with a desire to leverage system functionality and a more native-like appearance, several frameworks offer compelling options:

React Native

Originally developed by Facebook (now Meta) in 2015 to address their own cross-platform development needs, React Native has evolved into a popular open-source framework. The vast ecosystem of libraries and tools, combined with the familiarity of React development patterns, makes it particularly attractive for web developers transitioning to mobile development. While leveraging JavaScript/TypeScript (JS/TS) and providing native UI components, its journey hasn't been without challenges. Notable companies like Airbnb and Udacity have documented their experiences of adopting and later moving away from React Native, citing performance issues and maintenance overhead. However, the framework continues to evolve under open-source governance, with tools like Expo significantly simplifying development by providing a managed environment, pre-built components, and deployment solutions.

Flutter

Flutter is Google's modern UI toolkit utilizing the Dart programming language to achieve exceptional performance and streamlined developer productivity. Its innovative custom rendering engine, known as Skia, ensures pixel-perfect visual consistency across different platforms (not just mobile, but including desktop applications on macOS, Windows, and Linux) by directly handling all rendering operations. This approach allows Flutter to maintain precise control over every pixel on the screen, resulting in highly consistent and beautiful user interfaces that look and behave identically whether running on iOS or Android. Flutter's hot reload feature enables rapid iteration during development, while its widget-based architecture promotes code reusability and maintainable application structure. Flutter has gained significant popularity in developing economies (such as India), offering a cost-effective solution for cross-platform app development. However, recent developments have raised concerns within the Flutter community. In early 2024, Google implemented layoffs affecting key teams, including those working on Flutter and Dart. These reductions have led to uncertainty about the framework’s future and Google’s commitment to its ongoing development. Also, while Dart appears to be very similar to JS/TS, it is a different language with a limited set of experienced developers. As Gergely Orosz points out, the popularity of Flutter appears to have overtaken React Native, but it’s popularity depends on different geographies. Also, it shows that teams using Flutter have been less likely to generate revenue. It is hard to suggest why this is the case.

Other

MAUI

Microsoft App UI (MAUI) is a solution enabling .NET developers to use C# while maintaining near-native performance through platform-specific optimizations. For teams already steeped in C# and the .NET platform, this choice can make sense. Especially if it allows for reusing business logic originating elsewhere outside the mobile app, it follows a similar idea as utilizing something like JS/TS in Node for the backend for React Native. It is built on Xamarin, which predates MAUI as Microsoft’s cross-platform mobile solution.

Qt

Qt is a comprehensive C++ framework that allows developers to create cross-platform mobile applications using a single codebase. It provides a rich set of libraries and tools for building native-looking UIs, along with extensive hardware access capabilities. While Qt offers strong performance and a mature ecosystem, it's less commonly used for mobile development compared to other frameworks, being more popular in Linux desktop and embedded systems development. In many ways, Qt moved in the opposite direction as Flutter and React Native; whereas the latter two started off focused on mobile, then web and desktop apps, Qt started primarily as a desktop solution that migrated into mobile.

Even more?

Additionally, other more obscure and less broadly used solutions also exist. It seems like every language that a developer might prefer eventually ends up with a framework for building mobile apps. For example, Rust developers might want to try Tauri, while Python developers could lean on Kivy.

So Why Bother with Native?

Read more

Read more

Why Agile Often Devolves into Waterfall

For the last quarter century, Agile development has been hailed as a revolutionary methodology for making software development more flexible, collaborative, and responsive to change. Traditionally, teams relied on the Waterfall approach—mirroring other engineering disciplines where blueprints and permits precede construction. This desired predictability appeals to stakeholders. However, given the complexity of software systems, such predictability rarely materializes in practice. Unlike other engineering disciplines where upfront planning enables consistent production of physical items (from bridges to automobiles to electronics), software behaves more like an organic system, benefiting from continuous monitoring and adaptation—much more like gardening than might be expected in a field labeled as engineering.

The need to address Waterfall's shortcomings led to the Agile Manifesto. Published in 2001, the manifesto outlined core values and principles that emphasized adaptability, collaboration, and swift value delivery. Yet despite these ideals, many organizations find their Agile practices gradually transforming into rigid, bureaucratic processes reminiscent of the very Waterfall methodology they sought to escape. This raises a crucial question: why does Agile so often devolve into Waterfall? To answer this, we must first understand how Waterfall works and compares to Agile. From there, we can explore how to maintain an Agile footing and realize its full potential.

What is the Waterfall Methodology?

The Waterfall methodology is a traditional approach to software development that follows a linear, sequential process. It consists of distinct phases—requirements gathering, design, implementation, testing, deployment, and maintenance—with each phase requiring completion before the next can begin. This approach demands extensive upfront planning and detailed documentation, leaving minimal flexibility for changes once development starts. While Waterfall offers apparent clarity and predictability, it struggles to handle evolving requirements or unexpected challenges, making it poorly suited for dynamic environments.

The methodology's inflexibility comes from its "plan-driven" nature, measuring success through adherence to predefined timelines and deliverables. This structure typically results in extended development cycles and delayed feedback since working software isn't delivered until the project's end. These characteristics position Waterfall as the opposite of Agile, which emphasizes flexibility and iterative progress—traits that better align with creating software that meets customer needs.

Early in my career, I encountered the Waterfall approach in my first job at L-3 Communications. Not only was the Agile Manifesto barely only a year old and not widely disseminated, but the work we were doing was highly controlled as a Department of Defense contractor working on projects that involved secure cryptography and needed to be launched into space and run untouched for decades. This required careful requirements analysis and planning, but even then it was clear that Waterfall processes could do more harm than good as the software was being developed.

How does Agile Differ?

Agile emerged as a response to Waterfall projects that consistently missed deadlines and failed to satisfy customers. Waterfall's linear progression—from requirements gathering through system design to implementation and release—couldn't provide the flexibility needed as software became increasingly prevalent. This lead to a more obvious dichotomy between what the customer thought they wanted as opposed to what they truly wanted or needed in a finished product. As hardware became commoditized and software distribution grew more dynamic through the Internet, the ability to adapt plans for moving targets became crucial. The shift from mainframe computing to portable physical media (e.g., floppy disks and CDs) and later to cloud services and app stores enabled teams to deliver software incrementally through continuous updates. This is when the “softness” of software to truly shine as development teams found the ability to mold the solution to the customers’ needs.

The Agile Manifesto emphasized rapid delivery of working functionality and frequent iteration to achieve more predictable and desirable outcomes. While this flexibility is appealing in theory, implementing it effectively requires both diligence and intention—it demands a comfort with ambiguity and uncertainty that many find challenging. Teams must be able to see enough of the future to understand their general trajectory without getting caught up in the specifics of the final outcome.

My first experience with an Agile approach didn’t happen until a decade after the Manifesto was released. Working for a small team on a direct-to-consumer product, it was a revelation to be able to chunk up the work into consumable parts and deliver intermediary functionality. Even with a greenfield project, we focused on regular test and internal delivery cycles to ensure the effort was worth the investment and the original overall goal of the project was met with regular feedback loops to ensure we stayed on course.

The Spirit vs. The Letter of Agile

At the heart of this issue lies a fundamental misunderstanding of the Agile Manifesto. The manifesto emphasizes individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan. These values are intentionally broad, giving teams flexibility to adapt them to their unique context. However, many organizations implement Agile practices superficially—following the "letter of the law"—without embracing the underlying philosophy. This leads to an overemphasis on rigid ceremonies, metrics, and processes that ironically mirror Waterfall development's inflexibility. This rigidity ultimately undermines Agile's intended goals. While the resulting linear process gives management and stakeholders a way to monitor progress, organizations often overlook the crucial tradeoff between flexibility and observability—and worse, some mistakenly believe they can maximize both. By trying to apply the facade of something that sounds “cool” like Agile while still holding onto the underlying tenets of Waterfall, teams end up with the worst parts and limitations of both, rather than the opposite as intended.

The Rise of "Agile Theater"

The desire to appear agile while maintaining predictable long-term plans often leads to "Agile Theater"—where teams merely go through the motions of agile practices without embracing their principles. Consider these examples:

• Strict Ceremonies: Daily stand-ups, sprint planning, and retrospectives devolve into mandatory rituals with little purpose. Instead of fostering collaboration and transparency, these ceremonies become checkbox exercises. Stand-ups turn into rote status reports rather than meaningful team interactions. Retrospectives, meant to generate constructive feedback, either descend into complaint sessions or focus on superficial "vanity metrics" that serve only to document their own existence.

• Story Pointing as a Performance Metric: The concept of “story points” originated as a concept of “ideal days” from the concept of “Extreme Programming”, and didn’t even come from the more common Scrum form of Agile. As the intent was never to get to the granularity of less than a day, it lead to confusion and tension with management. This lead to teams misusing story points to measure productivity instead of estimating effort and complexity. As a result, teams would inevitably end up gaming the system and pressuring developers to inflate estimates. When story points get converted to time estimates, they lack the precision needed for anything beyond rough approximations.

The Waterfall Mindset in Disguise

These practices often lead to the re-emergence of a waterfall mindset in an agile disguise. For example:

• Fixed Scopes and Timelines: Organizations often insist on fixed scopes and deadlines, treating sprints as miniature waterfalls where changes are discouraged until the next planning cycle. Stakeholders attempt to divide predetermined roadmaps into two-week sprint increments—essentially just splitting up available time. This contradicts the core principle that teams should be able to adapt. While maintaining focus within a sprint helps avoid context switching, truly agile teams must be able to adjust at sprint boundaries, if not more frequently. This flexibility may disrupt longer-term roadmaps, but it's essential. Particularly in early-stage projects and companies, excessive advanced planning—a hallmark of Waterfall—limits the ability to respond to business needs.

  These fixed timelines are especially prevalent in event-driven deliverables, such as working for a sports league such as Major League Baseball. When timelines can’t slip, as games will be played whether or not the MLB App is ready, it is important to focus on figuring out how to adjust scope. Additionally, those timelines are generally known well in advance, often over a year in the future, so while it is inherently Waterfall-esque to try to hit those future deadlines from far out, sticking to an Agile approach where adjustments are made often to ensure calibration with those timelines end up resulting in the best outcome for both the development team and the users.

• Command-and-Control Management: Instead of empowering teams to self-organize, some managers use agile artifacts like burndown charts or velocity metrics to enforce top-down control. Teams can easily game metrics such as story points completed, pull requests merged, or lines of code contributed to meet arbitrary goals—satisfying management without delivering real value.

• Water-Agile-Fall: While engineering teams experienced with Agile development may follow agile processes successfully, other product development functions like design and QA often fall into waterfall patterns. Design and requirements aren't delivered until fully complete, while testing begins only after all development is finished, creating burdensome testing cycles before production release.

  I have seen this problem across companies both small (Diamond Kinetics) and large (Major League Baseball), where those other functions are treated as services that happen out of band with the software development process. This makes coordinated delivery difficult, as the communication across those boundaries are rarely in sync and therefore it becomes more linear like Waterfall than the iterative that Agile prescribes. If engineers aren’t able to review early designs and help shape and scope the effort, then they feel the need to implement the functionality to completion before handing it off to testing. This results in mini-waterfalls that break with the spirit of the manifesto and result in falling back to a Waterfall process under the hood.

  

How to Avoid the Trap

In order to successfully implement Agile processes, the team must be committed to the concepts and strive to “keep the faith” in the intent of the Manifesto, focusing on people over processes and early and often delivery. To prevent Agile from devolving into Waterfall, teams and organizations must actively resist the temptation to prioritize form over function. Here are some practical steps:

1. Revisit the Agile Manifesto Regularly: Make the manifesto a living document within your organization. Discuss its values and principles in team meetings to ensure everyone understands the "why" behind Agile practices. Use this understanding to tailor ceremonies and processes to your team's needs. If a stand-up feels unnecessary, skip it. If retrospectives aren't yielding actionable insights, try a different format. This approach enables following the spirit of the law without getting tied up in the letter of the law.

2. Focus on Outcomes, Not Outputs: Shift the emphasis from metrics like velocity and story points to the value delivered to customers. Encourage teams to measure success based on impact, not activity. A regular release schedule can help maintain focus on outcomes. Trust your teams to make decisions and adapt processes as needed. Avoid micromanagement and foster a culture of autonomy that prioritizes results.

   Especially when working at a smaller and earlier stage startup, such as Diamond Kinetics, it was important to focus on what could be delivered with confidence. A “release train” where releases were possible on any given week provided the necessary guardrails to plan work against while accounting for the necessary steps of testing and submission for app review. Leveraging this approach effectively tripled the number of successful releases from one year to the next.

   Additionally, this involves including all product development functions, from product to design to testing, in the outcome. Doing so provides better accuracy around true task estimation. When two or more people are involved across multiple disciplines, it is easier to understand why it is impossible to estimate durations at finer granularities of days if not weeks. While the programming effort for a small feature may take only a few hours, including the effort to define, design, implement, and test that functionality, and the necessary handoffs between disciplines results in any effort worth doing taking the better part of a working week in all but the most trivial examples.

3. Avoid Sprinting the Marathon: Most Agile teams settle on two-week sprints, yet meaningful software projects span far longer. They're more like marathons than sprints—and no one sprints a marathon. Cramming each two-week sprint with maximum work leads to team burnout. Build in flexibility by allowing sprints that tackle technical debt or enable exploration and learning. Teams need these occasional deviations from fixed schedules to remain adaptive and responsive to business needs.

4. Plan Roadmaps against Sprints: Rather than splitting a long-term roadmap into sprints, adopt a Now/Next/Future roadmap template with a sliding window:

   • Now - What is being worked on in the current sprint and possibly the next one

   • Next - What's up for the next month or two

   • Future - What's on the horizon for the rest of the quarter

   This approach, with constant review during sprint planning, provides a bottom-up view of what the team can produce. While accurate prediction may be challenging initially, refining this approach will serve the team well through its growth. By focusing on what the team can produce and how that evolves over time, you'll achieve the best possible outcome.

5. Ensure Feedback Loops Exist: Multiple feedback loops are essential for maintaining agility and effective iteration. Internally, regular and productive retrospectives enable teams to provide candid feedback to leadership and stakeholders about their work environment and needs. Externally, user experience studies and A/B testing generate insights that help teams better serve their customers. Proper system observability and monitoring round out these feedback loops by ensuring teams can maintain and improve product health.

Is Agile Always the Answer?

While Agile is widely accepted as the best practice for modern software development, we shouldn't automatically assume it's always the right choice. As with our earlier discussion about the "letter of the law," teams should carefully consider whether an Agile approach truly fits their project. Some circumstances—like developing low-level firmware or creating software with strict security requirements (such as military or space applications)—may actually benefit from a Waterfall approach. These scenarios often demand extensive upfront planning to ensure all requirements are met, and their deliverables are less likely to change over time.

Nevertheless, incorporating some degree of agility can still prove valuable. Given today's complex software systems, it's best to adapt your process to suit both your team and project. Rather than viewing it as a strict choice between Agile and Waterfall, consider it a spectrum that skilled teams and leaders can navigate to their advantage.

Conclusion

Agile is a mindset, not a checklist. Specifically, that mindset is about flexibility in the name of delivery. When organizations lose sight of this fundamental truth, they risk falling into the same traps of rigidity and inefficiency that plague Waterfall development. By staying true to the Agile Manifesto's principles and prioritizing outcomes over processes, teams can avoid the pitfalls of "Agile Theater" and unlock agile development's full potential. This approach not only leads to better software but also creates an environment where creativity, collaboration, and adaptability thrive. While teams can adopt Waterfall practices when truly necessary, this should be done cautiously and selectively. Ultimately, leadership must determine the most effective approach based on their experience and context.

Subscribe now

History

“You are responsible for the assembly your code creates.” - Mike Acton

So about 20 years ago, I was at Insomniac Games and I was attending a software lecture by Mike Acton, who said the above quote during the meeting. I was very early in my career and had just gotten done with my second (and final) game with the company.

And, to be honest, the above quote intimidated me. I was just getting good enough to make sure that what I did at the C++ level worked (and trying to get it done on time). The thought have having to then go and check everything that my code produced at the assembly level felt like another world.

I don’t think I was alone in this sentiment, as the general discussion continued onto how we were doing work. Someone made a crack about Fortran in the meeting, to which everyone laughed, with the exception of Mike. Completely deadpan, Mike responded with “What’s funny about Fortran…it creates incredible assembly.”

Needless to say, this left a mark.

Practice with Assembly (or More to the Point…Not)

Admittedly, I really did admire all the folks that were focused at the lowest level (my business partner Rob Wyatt included). And a big part of me really wanted to be that focused there, figuring out how to make it run faster, always.

I would love to tell you that from that moment on, I was diligent about checking my assembly after I wrote my C++ code.

I didn’t.

Mostly I still had my head up my own ass about the style of the C++ code, did it look good, did it seem maintainable (for whatever the whims of a given place dictated that to mean). And to be fair, it was the main critiques of code reviews at the time (and largely ever since).

To my credit, I will say that my code did get the job done, whatever it needed to be.

And this was despite the fact that I never looked at the assembly!!!*

(*Except that one time where I needed to understand why the strict aliasing optimization was failing on some older code and had to explain it. Thanks to Mike Acton for that tip as well).

But, how?!? How is such a thing possible?

Well, because the code produced the result that I wanted it to. The behavior matched my needs, I could write tests against it (when I wrote tests), and it worked (mostly) without bugs.

So I just moved onto the next problem, confident that I have solved the problem.

AI in Development…Is This a Joke?

With that out of the way, let’s fast forward to today’s world. There seems to be a big debate around whether or not AI models are going to change everything for coding, mainly centering around two camps of “these are all just toy models and the code it produces isn’t worth a damn” to “oh shit, this is changing everything.”

To be honest, this time last year, I was definitely in the first camp…albeit without having really dived in to understand what was happening. I saw the videos online and largely brushed them off with the same sort of arrogance that I used to judge other people’s code.

“Well, stylistically it shouldn’t be like that…”

“Wait, it isn’t even separating them into different files? Hah, I could do a much better job of making this more modular.”

“Sure, it is fast to develop…but will it be maintainable by humans?”

Over the decades, my instincts have been trained to look at code, having the equivalent of a compiler and runtime running in my head to look at the code and see where it could go wrong. To be sure, this is what every good software has done (…or just smashes the compile command enough times and fixes stuff until the job is done).

As code reviews came in, I was given instruction on how to restructure it, make it better, make it this or that…and I did it…believing that this was the end-all-be-all. Hell, it was what I got rewarded for as so many places.

But I missed a big point about all software development.

It’s the Customer, Stupid

As much as I can claim to have “mastered” the art of code development (and there many that would claim that I had not achieved that by any stretch), there was something fundamental missing.

I would love to believe that the code that I’ve written is the ultimate combination of language mastery, insightful use of algorithmic tricks, and, frankly, art. To be honest, most programmers want to believe this. We are a master class of doomsaying in our analysis of code (especially other people’s) and supreme arrogance in the righteousness of our own code and products. We love to show off for the other programmers to show how badass we are.

But there is one fact that we typically miss: people don’t pay you for your code.

Ever.

Yep, I can hear you all screaming now, “but we develop this way so we can easily maintain it” and “there is no way a computer can be as insightful as I am” and “this guy just sucks.” And while I’ll cop to the last point readily, none of these (and more) actually matter.

What matters is whether or not the functionality that you produce actually matters to other people.

Does it solve a problem for them?
Does it create a great experience for them?
Does it save them time?
Does it create a great product?
Does it make them happy?

Will they pay for it?

It is the complement of Mike’s quote above:
“You are responsible for the functionality that your code produces.”

It’s Always Been Product Engineering

So earlier I admitted that a year ago I had been firmly in the camp of “AI…what a funny little toy.” Today I’ve drunk the Kool Aid and am now in the second camp (the one that says: “oh shit, this is changing everything”).

And the reason for this has a lot to do with why I don’t check my assembly.

I started dabbling with AI development, helped by a useful video that expanded my understanding for how I could actually use the tools. It was like having to unlearn everything I had learnt…I just could not get my head around everything what the code was doing…and I felt like I had to…because that was what good engineers do, right?

Then it clicked…maybe it didn’t matter what the code looked like. Maybe I could focus more on whether I was getting the functionality I wanted and less about the code.

(As Rob points out: “After working as a contractor on tons of code bases I don’t control, I always say that code is just text in a file, compiler doesn’t care about your fancy formatting and spell checked comments.”

In other words…it does not affect the generated assembly or the functionality one bit.)

With that in mind, I started tearing into the development system, attempting thing that I would not have previously…some thing that were not possible previously:

• Developing a UI from an image

  • I’m no expert in CSS and HTML (nor will I be)…but I could just drop an image and let the robot create it for me

  • To be sure…it did not get it right the first time…but I could art direct the CSS and HTML (and later PHP) by just telling it to the robot

• Commanding the creation of a Backend Service

  • I’m not an expert in writing Python…but by giving just enough info about the behaviors I wanted, the robot could create a very useful service for me.

  • Again, where it needed augmentations, improvements, or fixes…I just told it what to do

• Creating a silly game from clip art and sounds

  • I asked the robot to create a simple side scroller…adding bits and pieces here and there

  • Generating music for the game using Udio

  • And dammit…it worked…

Fundamentally, I started caring more about what I was to achieve, rather than how it was being done.

And honestly, that is what our customers care about as well. Not about how it looks in code…but what it does…and how fast I can tweak it, make it better.

To be sure…it is fast. And now I don’t have to worry about whether I have learnt all the languages I need to make something work.

Hard Stuff is Still Hard…but Maybe a Little Easier

Now, before you believe that it is all fun and games…I’ll be honest and agree that “yeah, it is not all there yet.” There are times when the robot gets messed up, hooked into loops, can’t figure its own code out.

So sometimes it is worth sweating it out a bit, falling back on those decades of engineering experience…and sometimes I just say “fuck it” and start all over again…after all, it might have only taken 20 minutes…and I know what I want now…starting from scratch isn’t so bad.

(And let’s face, it programmers…how many times have you walked into a code base and said “I don’t understand what the hell these folks were saying…I have to rewrite all of this to make sense.” Is this really any different?)

To be sure, product-market fit has always been hard…it has been hard to bring something up to speed, get to a set of customers, get them to try it…see if it actually would do anything for anyone. And the amount of effort to get something up and running has been herculean.

Figuring out what customers want is a challenge…because a lot of times they don’t even know.

But imagine a world where you are focusing on the outcomes vs. the construction. Imagine that world is far easier and far faster to get something up and running that let’s you test your hypotheses for the “next great product.”

Imagine a world where we’ve shortened the distance between the product person and designer and the actual product. Because that is the world we are heading to increasingly fast. Where your idea can come to life faster than ever before.

And that’s a great thing. Because that means that someone doesn’t need to know all of software engineering (frontend, backend, databases, etc.) to do something amazing.

It is not perfect today and I would be remiss to make you believe that it all going to be easy. It isn’t. But much in the same way that compilers and software changed the needs of the underlying assembly and architecture, you can bet your ass that everything will change with this new type of development. We’ve already architected new types of processors…how much longer do you really think it will be before we’ve rearchitected new languages and frameworks to better meet the needs of developing through AI.

And that just keeps making everyone into a product engineer.

To The Diehard Coders

To be sure, I don’t want to go back to the old way of working…I admittedly want to be focused on the products I create and not how I have to slog to get there.

Now, for those of you that still want to be coding in the same way that you have, I respect you. Honestly and truly I respect you because I understand where you are coming from. Because I’ve been there, I grew up there.

I get the pride. I get the amazing work that you’ve done. And I get the desire to keep being amazing in that way.

But I have a bit of advice for you…which takes us back to the very beginning: I hope you are checking the assembly that your code produces.

Because if you aren’t…then you are accepting that “it just works” below whatever level you feel most comfortable working.

And doesn’t that describe everyone working with AI to create products today?

(Ironically…this was written without the help of AI…unless you count autocorrect….)

Addendum - Talking About Optimizations

Rob rightly pointed out the following: “does this mean we no longer care about traditional performance metrics as long as it's fast enough? small enough? What about optimizing for power? Are we going to start getting generally bad code because the robot made something that technically works?”

So, this is a great point about whether we are going to push important performance metrics by the wayside…and actually, I’ll argue that the exact opposite will occur.

Questions around any performance metrics are fundamentally about behavioral constraints. That is, when I am saying “I want this faster, I want it smaller, I want it to use less power”, I am not making any statements about how I do this…I am back to asking what I want to happen.

I will fully cop to my early assertion about holding onto code as a form of art…preciously crafting my class structure in some way like I was Leonardo da Vinci…but really not giving a damn about performance as long as it “felt” good (and admittedly was pretty extensible for what I was using it for).

Because I held onto the code as a form of art, I wasn’t doing any kind of measurement around how well it would perform, let alone whether it was small or how much power it used.

But what if you didn’t care about the “look” of the code, but instead what it did, like I’ve been saying all along. Then you just add in your constraints for speed, size, power…in each of these cases, they are all *measurable constraints…*the kind of things that an optimizers can go nuts with.

In a world where AlphaGo invented new moves for Go and JIT HotSpot compilation exists, it is not a far fetched world to imagine a combination of an LLM and a compiler to loop through and optimize the hell out of whatever functionality with we give it on a particular architecture.

To ground this in a real world example, let’s use my earlier example of the Rust ray tracer. Now, developing a ray tracer today in a world of RTX and MoonRay is a bit silly but, it does have some nice characteristics:

1. It is simple

2. It lends itself to testing out lots of different optimizations

3. It generates a fun result (i.e. an image)

4. It makes me feel like Kevin Flynn (if you don’t get the reference, I pity you)

After I got it working, I made it a bit interactive, but it was slow as molasses (I did exactly what I ask, which was to create your run of the mill ray tracer, as simple as can be). Curious if I could make it better, I asked it to make it faster.

The robot jumped at taking advantage of multi-threading to parallelize the ray casting, which was a big bump in performance (~2 fps to ~20 fps). Then I asked it if it could make any further improvements, where it added in acceleration structures (don’t quite remember the bump then).

(To be fair, when I tried to get it to optimize for ARM using NEON instructions, the output image started to fall apart…so it isn’t quite all there yet.)

Nevertheless, despite being an admittedly simple example, it does go back to the original point…because I cared more about the behavior than the mechanics, I was able to get the robot to keep pushing the performance.

Who knows, if my belief in LLM + compiler is right, then maybe the language of the future is actually…assembly.

Next Up - PJ’s Money Where His Mouth Is: Shipping an Application That People Pay For (Can he do it?!?)

Big shout out to Hans Ku and Maggie Wiedner for handing me some Kool Aid on AI development.

Subscribe now

WWDC is coming up and Rob and PJ have some opinions!

AI is in the air for everyone this season and WWDC is likely going to be no exception....but with so much AI processing happening in the cloud and Apple having been so focused on data privacy, security, AND user experience...how IS Apple going to thread this needle?

Will they push more stuff to the cloud to bring the widest support to folks?
WIll they deprecate more hardware to make the user experience the best they can do?
Should they even bother with AI at this time in any meaningful way ahead of having some strong use cases?

Join Rob and PJ as they make some prediction ahead of WWDC...and see how well they did during the event itself!

Microsoft recently introduced its CoPilot + PCs as a fully integrated lineup...based on ARM processors (Qualcomm's Snapdragon specifically), unified memory, and requirements around the onboard TPUs.

Hmm...sound very Apple-like, no?

With an apparent revitalized push back into the consumer space, what does this mean for the future of the hardware? Specifically, the Wintel era meant infinite flexibility, configurability, and backwards compatibility, stemming all the way back in time.

Will the voices of prosumer builders be silenced on Windows? Will developers rejoice as Windows powered machines with MacBook form factors come to bear? Could this actually be better for everyone to have more stable hardware platforms?

Come join us as we dive into these topics to prognosticate towards a potentially better future for computing...

Hype is all around around the tech stories that are out there....

Apple has a terrible ad for a thinner iPad!
Google is leaving San Fran and moving jobs overseas!
Announcements around AI that is going to revolutionize everything!!!
Tech companies are acting like big corporations!!!

Maybe it is just us...but does anyone get the impression that tech news is actually a bit boring right now? 

Perhaps we are just exhausted from the constant barrage of "bigger", "smaller", "better", "faster", "newer", "revolutionary, "evolutionary", "different"...

But it does feel like a moment when tech is really trying to keep its whizz bang growth cycle going...and not start to admit that these companies, while still wildly profitably, are transforming into the value, blue chip stocks (oh, no, we're IBM!) who are trying to eek out more profit from ads on smart TVs rather than actually innovate.

Join us as we rant a little bit on the current state of affairs!

CPUs have been around for forever as the heart of computers....then came GPUs to enable cool real time graphics...and later bitcoin mining...then some clever folks figured out how train AIs on them...which leads to a new explosion of processing units tuned specifically for AI development (see Google's TPUs, Apple's Neural Engines, Meta's MTIAs).

Why do we have them all? And what the heck do they all mean? And why should we care? 

Come explore a bit of the magic and differences amongst these different types of processing units that is powering your phones, games, and the explosion in AI development!

In the arms race of computational performance gains and big market splashes...security issues can pop their ugly head, something going all the way down to the hardware level...and when they happen, they can be brutal.

In this episode of Tricky Bits, Rob and PJ discuss a recent article and research on hacking into the M-series chips, using clever attacks to extract data that is, ideally, supposed to be kept secret. 

What are the nitty gritty details of how this attack is successful?

How does company culture and processes affect approaches to hardware development? 

And did we back ourselves into a corner by becoming less efficient programmers?

Link to article: 
https://arstechnica.com/security/2024/03/hackers-can-extract-secret-encryption-keys-from-apples-mac-chips/

Link to research:
https://gofetch.fail/

Ever wondered why a virtual rollercoaster can leave your stomach in knots, despite never leaving your living room? Strap in and brace yourself as we unpack the enigma of VR-induced nausea with the expert insights of Nemanja Milosavljevic from Neurosync Labs. Our latest episode of Tricky Bits zooms in on the battle between our eyes and our inner ear when diving into the digital deep end, bringing to light the vestibular system's pivotal role in the physical disorientation we face in virtual landscapes.

Hold onto your headsets — this session isn't just about identifying problems; we're all about the solutions. Learn about how the groundbreaking C-Infinity device can sync your senses, offering an immersive experience with a solid grip on reality. By integrating force measurements with natural motion, this ergonomic innovation could be the key to unlocking a future of comfortable, nausea-free virtual exploration for a number of VR experiences.

Join us as we envision an electrifying era of VR, where the technology we discuss today might just redefine our gaming experiences tomorrow.

NeuroSync Labs:  https://neurosyncvr.com/

Hardware is where the digital meets up with physical reality.

In the modern world, software can be developed and deployed worldwide in timescales measured in minutes....but despite advances in development, consumer hardware at scale still presents many difficult challenges.

Although there are many modern avenues to prototyping hardware solutions, making the leap beyond RPi, Arduino, ESP level projects into mass market devices encounters challenges including funding, development, manufacturing, shipping, supply chains, and revenue models.

In this episode, Rob and PJ explore the world of hardware development, understand its pitfalls, and look at where it might be going.

So much of Tricky Bits is an ongoing conversation between Rob and PJ about various tech topics. This particular episode revisits some old topics around game development, its history and costs, and what factors come into play that shape the market today.

As a heads up...this one is more conversational than some of the past episodes as we  build up to a larger discussion for a future conversation around big and small media in various contexts (games, movies, television).

 Finally...we announce our Discord server, where you can comment on individual episodes. You can find the link to the thread for this episode here: https://discord.com/channels/1146929485694902282/1217606185159491695

Come let us know what you think!

Could Apple be quietly gearing up to redefine the AI landscape? With all eyes on Siri's evolution and Apple's famed secrecy, we delve into the tech giant's delicate dance between user privacy and the cutting edge of AI innovations. Join Rob and PJ  as we unravel Apple's enigmatic strategy, considering the possible hardware constraints and their steadfast commitment to a reliable assistant amidst the whirlwind of AI advancements.

Apple's Siri has been a household name, but it's not without its critics. In this episode, we dissect the challenges it faces, balancing user privacy with the need for more personal and proactive functionalities. We speculate on the potential for Apple to introduce more generative AI capabilities and how this might revolutionize user experience while sticking to their privacy-first mantra. Moreover, we explore the juxtaposition of Apple's hardware prowess with their approach to personalized AI, pondering the efficiency and practicality of running inference models on mobile devices.

Wrapping up our deep dive, we tackle the broad implications of AI in our daily tech interactions. From ethical concerns over data usage to the burgeoning role of edge computing, we question whether AI will be the next big revolution or a passing trend.

(And yes...this description WAS in fact generated from the Episode transcript and AI...wild times, right folks?

You know what what is really going to get your noodle later on? Did the AI write this footer? )

2024 and Tech Layoffs still continue...more than a headline; they're a wake-up call to evaluate the true resilience of our skills in an ever-changing industry. In this episode, we take on a tough subject to better understand the realities of job security, debunking the myth that exceptional programmers are safeguarded from the chopping block. 

We also challenge the notion of generalist vs. specialist and instead focus on the all important question of "what is the value delivered" and "what skills do you need to deliver that value."

Finally...we remind everyone... your self worth is not something that is defined by any company...you have value far and above your job.

Maps...we use them almost every day of our lives.....but...what makes maps hard? From the paper maps we used to pull out on car trips to the digital and dynamic maps that we have our smart phones, the problems around making and maintaining maps have been with us for a while. Fortunately...we've got an expert to help us out here.

Cory Whittaker, a long time self professed "map nerd", takes Rob and PJ on a journey on maps and how they are used in the modern day.

Come and join us for our first interview episode!

Rumblings of a major XBox announcement in a post Activision/Blizzard acquisition world slated for mid-Feb 2024 started a number of rumors around what might happen...would all XBox games go cross platform?  What would this mean for the XBox hardware?!?

Then on Feb 15th....Phil Spencer, Sarah Bond, and Matt Booty fulfilled the special announcement...with a whopping 4 games (all unnamed) slated to go cross platform...followed by a lot of corp speak. And, ahead of that, there was an internal meeting where they seemed to be doubling down on the XBox hardware.

So it seemed like the wild speculations...were untrue....or were they?

In this episode, Rob and PJ pick apart the XBox strategy...analyze what is there and speculate what is still yet to come...

We chatted about the Apple Vision Pro previously and now the reviews are in....how are they stacking up for the Vision Pro and how did our predictions work out for themselves, so far? And what predictions to we have from here?

Also...Meta announced dividends at its last earnings call...and now Goldman Sachs analysts are (maybe) portending that this will lead Alphabet and Amazon to do the same. But do we actually think that this is as imminent as the headlines suggest? 

As an aside....PJ published a blog post about this very topic on 2.7.2024...check it out here: https://trickybitsblog.github.io/2024/02/07/meta-earnings-report-dividend.html

Performance Reviews! Everyone's most beloved topic and favorite activity...but how does that impact company culture...and what kind of incentives do they give rise to? 

Rob and PJ dive deep into these questions and explore how much past behaviors that gave rise to success....actually become more important that the pursuit of success itself?