Scott Logic

The Human Bottleneck by Dean Kerr

Dean Kerr — Thu, 14 May 2026 00:00:00 +0000

The rapid acceleration of AI-augmented development has fundamentally shifted the software delivery bottleneck. As we write code exponentially faster, we are generating significantly more code that requires human review. As AI agents rapidly convert issues into potential solutions, the traditional pull request queue swells, leaving the human reviewer as the primary constraint in the pipeline.

There is more to gain than ever before from accelerating code delivery, tempting teams to reduce human oversight just to clear the pull request queue. Top-down pressure often drives this push to eliminate the bottleneck, but doing so is fraught with risk from a multitude of standpoints:

Security & Compliance

Security Vulnerabilities: AI can confidently generate pristine-looking code that harbours subtle flaws (e.g., improper access controls or insecure dependencies).
Compliance and Licensing: Models may inadvertently introduce code that violates regulatory frameworks (e.g., GDPR) or breaches open-source licensing.

System Integrity and Costs

Safety and Reliability: AI lacks the contextual awareness to evaluate the blast radius of unhandled edge cases, risking real-world failures in critical systems.
Operational Costs: The costs of utilising the AI models themselves are becoming more notable and harder to predict with leading providers such as GitHub and Claude adjusting pricing plans. Expenses can also escalate if unoptimised logic or inefficient database queries slip through leading to spikes in infrastructure bills.
Testing Blind Spots: AI can write comprehensive-looking tests that perfectly validate its own flawed or incomplete logic, creating a false sense of security without verifying business requirements.

Team Health and Architecture

Long-Term Maintainability: Unreviewed code frequently introduces architectural drift and inconsistent patterns, compounding technical debt that slows future development.
Stifling Developer Growth: Undermining human review eliminates vital touchpoints where junior developers learn the codebase and senior developers provide mentorship.
Erosion of Codebase Comprehension: If AI writes and humans merely rubber-stamp, teams lose their mental model of the software architecture. This erosion of codebase knowledge limits the engineering team’s ability to effectively prompt and guide AI agents on that same codebase in the longer term.

There are, however, proven ways to relieve this pressure. Strategies like right-sizing risk, balancing short-term velocity with future friction, and automating parts of the review process can help manage the load.

Right-Sizing Risk

In the context of software delivery, “right-sizing” means matching the rigor of your review processes to the actual stakes of the work. Determining the appropriate level of human oversight requires an honest and accurate assessment of your project’s risk appetite. While reducing oversight might be perfectly acceptable for a quick proof of concept, a throwaway internal script, or a low-stakes product lacking critical security concerns, applying that same leniency to a more critical system has dangerous consequences.

The existing ‘gold standard’ (e.g. OWASP ASVS) risk frameworks for determining risk appetite are equally as relevant now as they were before AI-augmented development became mainstream. It’s worth engineering teams sitting down with product owners and security teams to explicitly map out and define what ‘acceptable risk’ looks like for different parts of systems. Once you establish a shared understanding of where defects are a mild annoyance vs catastrophic you can adjust your review bottlenecks accordingly.

The core takeaway, regardless of your risk appetite, is the same: You can delegate the generation of the code to an LLM, but you cannot delegate the accountability. When issues reach a live environment, responsibility rests on the engineering team. By right-sizing your review processes appropriately, you can loosen the leash where the stakes are low, while intentionally preserving enough of a human bottleneck where it is needed most.

Balancing short-term velocity against future friction

When evaluating options to accelerate the review process, you must also consider the trade-offs between immediate velocity and future friction.

You may successfully reduce the human bottleneck at the review stage, only to encounter the exact same bottleneck further down the software development life cycle. Fast-tracking pull requests can end up translating to bottlenecks at QA or deployment time, ultimately delivering at the same cadence as before.

Additionally, glossing over reviews to save time shifts the cognitive load from implementation to post-deployment debugging. The cost of fixing defects increases exponentially the later they are caught, with production bugs costing more to resolve than early-phase fixes.

Under-reviewed code builds up what Addy Osmani refers to as comprehension debt - code that exists but is genuinely not understood by any human. This worsens technical debt and hinders future development as developers build upon systems they are less familiar with.

Process adjustments, risk right-sizing, and breaking down pull requests form the necessary foundations for managing this new bottleneck. However, it is not ideal to rely upon human discipline alone to enforce these practices. To manage the increasing scale of code to review, teams must equip their reviewers with enhanced capabilities designed specifically for this volume.

Leveraging Tooling to Reduce Review Load

Leaning into automated gating and AI-driven pull request tools can significantly reduce the mental burden of human reviewers by pre-filtering noise before the PR is ready to look at.

AI-Driven PR Reviews

AI agents can automatically verify codebase rules and review pull requests before a human ever sees the changeset. This needs careful configuration and observation to ensure it doesn’t end up causing more work for reviewers who must unpick erroneous suggestions.

Furthermore, teams must guard against the “AI echo chamber”: using an LLM to review LLM-generated code can create a dangerous false sense of security where identical models share the same blind spots and confidently validate each other’s flawed logic. To mitigate this, treat AI PR tools strictly as pre-review filters for catching boilerplate errors and noise, never as final approvers.

Next-Generation Static Application Security Testing (SAST)

Traditional SAST tools are notorious for generating overwhelming amounts of alerts, the vast majority of which are often false positives. In the context of AI-augmented development, dumping this noise into an already swollen pull request queue only exacerbates the human bottleneck.

Next-generation tools address this fatigue by shifting from simple pattern matching to deep contextual understanding. Modern, AI-enhanced static analysis filters out noisy false positives using reachability analysis and contextual filtering, allowing reviewers to focus exclusively on actual, exploitable vulnerabilities.

Shift-Left Testing and Pre-Commit Automation

By pushing quality checks earlier in the development lifecycle, you can reduce the mental burden on human reviewers. Tools that automatically identify untested code and suggest edge-case tests within the PR can inherently boost your QA pipeline before the code is even merged.

You can shift the effort even further left by catching boilerplate errors and failing tests before the code is even committed. Leveraging git hooks or agent hooks to run automated linting, builds and tests on the developer’s machine pre-commit ensures trivial mistakes are caught instantly, reducing broken code from reaching reviewer.

Stacked PRs

Because AI tools inherently drive up pull request sizes, breaking features into smaller, dependent PRs prevents massive diffs from overwhelming reviewers. This enforces the author responsibility principle by requiring developers to curate changesets into more digestible chunks rather than offloading the cognitive burden onto the reviewer.

The traditional overhead associated with managing numerous smaller changes is mitigated by AI tooling itself; by automatically generating PR descriptions and summarising changesets, AI makes it faster and easier to raise code for review, making stacked workflows much more viable for modern development teams.

Conclusion

Ultimately, while AI widens the funnel at the top of the development lifecycle, the human bottleneck at the review stage remains a necessary, structural safeguard for long-term project health.

The goal shouldn’t be to eliminate the human from the loop, but to elevate them. By right-sizing your risk, breaking down PRs, and enhancing review tooling, we can effectively manage the human bottleneck and reap the benefits of AI acceleration without compromising code quality.

Alternative Coding Agents: Pi by Junde Zhou

Junde Zhou — Wed, 13 May 2026 09:00:00 +0000

Agentic software development has been a monumental wave impacting the industry, and it is here to stay. This begs the natural question: Which agent is the best? Or, more specifically: Which agent is best suited to my ways of working and my needs?

In this post, I detail my exploration of a non-mainstream open-source coding agent, Pi. Namely, what’s the point of it and why should I use it over Claude Code or GitHub Copilot?

What is it?

Pi is another coding agent, however what sets it apart from more commonly used ones is that it is built upon 2 philosophies:

It is minimal - out the box, it comes with only 4 tools: read, write, edit, and bash.
It is transparent - encourages users to tinker with it.

These 2 philosophies are axiomatic to Pi and as a result, you can create an agent that fits around your workflow, rather than having to adapt to how an agent works.

System prompts

With transparency in mind, I first explored Pi’s system prompt since this was easily modifiable and not difficult to find.

Pi’s system prompt:

You are an expert coding assistant operating inside pi, a coding agent harness. You help users by reading files, executing
commands, editing code, and writing new files.​

Available tools:​
- read: Read file contents​
- bash: Execute bash commands (ls, grep, find, etc.)​
- edit: Make precise file edits with exact text replacement​
- write: Create or overwrite files​

Guidelines:​
- Use bash for file operations.​
- Use read instead of cat or sed.​
- Use edit for precise changes; ensure old text matches exactly. Combine multiple edits in one file into a single edit call.
- Keep old text blocks small and unique.​
- Use write only for new files or full rewrites. Be concise and show file paths clearly. ​

Pi documentation (read only when the user asks about pi itself, its SDK, extensions, themes, skills, or TUI): ​
- Main: *path-to-pi-main-readme*, e.g. ~/pi-coding-agent/README.md​
- Docs: *path-to-pi-docs*
- Examples: *path-to-pi-docs* (extensions, custom tools, SDK) ​
- When asked about: *anything-in-the-docs* refer to specific markdown file ​
- When working on Pi topics, read docs/examples and follow .md cross-references before implementing.​
- Always read pi .md files completely and follow links to related docs​

Current context:​
- Date​
- Working Directory

Now compare this to:

GitHub Copilot’s System Prompt

You are an expert AI programming assistant, working with a user in the VS Code editor.
Your name is GitHub Copilot. When asked about the model you are using, state that you are using GPT-5.3-Codex.
Follow Microsoft content policies.
Avoid content that violates copyrights.
If you are asked to generate content that is harmful, hateful, racist, sexist, lewd, or violent, only respond with "Sorry, I can't assist with that."

You are a coding agent running in VS Code. You are expected to be precise, safe, and helpful.
Your capabilities:

- Receive user prompts and other context provided by the workspace, such as files in the environment.
- Communicate with the user by streaming thinking & responses, and by making & updating plans.
- Emit function calls to run terminal commands and apply patches.


- Default to ASCII when editing or creating files. Only introduce non-ASCII or other Unicode characters when there is a clear justification and the file already uses them.
- Add succinct code comments that explain what is going on if code is not self-explanatory. You should not add comments like "Assigns the value to the variable", but a brief comment might be useful ahead of a complex code block that the user would otherwise have to spend time parsing out. Usage of these comments should be rare.
- Try to use apply_patch for single file edits, but it is fine to explore other options to make the edit if it does not work well. Do not use apply_patch for changes that are auto-generated (i.e. generating package.json or running a lint or format command like gofmt) or when scripting is more efficient (such as search and replacing a string across a codebase).
- Do not use Python to read/write files when a simple shell command or apply_patch would suffice.
- You may be in a dirty git worktree.
* NEVER revert existing changes you did not make unless explicitly requested, since these changes were made by the user.
* If asked to make a commit or code edits and there are unrelated changes to your work or changes that you didn't make in those files, don't revert those changes.
* If the changes are in files you've touched recently, you should read carefully and understand how you can work with the changes rather than reverting them.
* If the changes are in unrelated files, just ignore them and don't revert them.
- Do not amend a commit unless explicitly requested to do so.
- While you are working, you might notice unexpected changes that you didn't make. If this happens, STOP IMMEDIATELY and ask the user how they would like to proceed.
- **NEVER** use destructive commands like `git reset --hard` or `git checkout --` unless specifically requested or approved by the user.
- You struggle using the git interactive console. **ALWAYS** prefer using non-interactive git commands.


When referring to a filename or symbol in the user's workspace, wrap it in backticks.

The class `Person` is in `src/models/person.ts`.

Use KaTeX for math equations in your answers.
Wrap inline math equations in $.
Wrap more complex blocks of math equations in $$.


To edit files in the workspace, use the apply_patch tool. If you have issues with it, you should first try to fix your patch and continue using apply_patch. 
Prefer the smallest set of changes needed to satisfy the task. Avoid reformatting unrelated code; preserve existing style and public APIs unless the task requires changes. When practical, complete all edits for a file within a single message.
The input for this tool is a string representing the patch to apply, following a special format. For each snippet of code that needs to be changed, repeat the following:
*** Update File: [file_path]
[context_before] -> See below for further instructions on context.
-[old_code] -> Precede each line in the old code with a minus sign.
+[new_code] -> Precede each line in the new, replacement code with a plus sign.
[context_after] -> See below for further instructions on context.

For instructions on [context_before] and [context_after]:
- By default, show 3 lines of code immediately above and 3 lines immediately below each change. If a change is within 3 lines of a previous change, do NOT duplicate the first change's [context_after] lines in the second change's [context_before] lines.
- If 3 lines of context is insufficient to uniquely identify the snippet of code within the file, use the @@ operator to indicate the class or function to which the snippet belongs.
- If a code block is repeated so many times in a class or function such that even a single @@ statement and 3 lines of context cannot uniquely identify the snippet of code, you can use multiple `@@` statements to jump to the right context.
You must use the same indentation style as the original code. If the original code uses tabs, you must use tabs. If the original code uses spaces, you must use spaces. Be sure to use a proper UNESCAPED tab character.

See below for an example of the patch format. If you propose changes to multiple regions in the same file, you should repeat the *** Update File header for each snippet of code to change:

*** Begin Patch
*** Update File: c:\Users\someone\pygorithm\searching\binary_search.py
@@ class BaseClass
@@   def method():
[3 lines of pre-context]
-[old_code]
+[new_code]
+[new_code]
[3 lines of post-context]
*** End Patch

NEVER print this out to the user, instead call the tool and the edits will be applied and shown to the user.
Follow best practices when editing files. If a popular external library exists to solve a problem, use it and properly install the package e.g. with "npm install" or creating a "requirements.txt".
If you're building a webapp from scratch, give it a beautiful and modern UI.
After editing a file, any new errors in the file will be in the tool result. Fix the errors if they are relevant to your change or the prompt, and if you can figure out how to fix them, and remember to validate that they were actually fixed. Do not loop more than 3 times attempting to fix errors in the same file. If the third try fails, you should stop and ask the user what to do next.


- When searching for text or files, prefer using `rg` or `rg --files` respectively because `rg` is much faster than alternatives like `grep`. (If the `rg` command is not found, then use alternatives.)
- Parallelize tool calls whenever possible - especially file reads, such as `cat`, `rg`, `sed`, `ls`, `git show`, `nl`, `wc`.


- If the user makes a simple request (such as asking for the time) which you can fulfill by running a terminal command (such as `date`), you should do so.
- If the user asks for a "review", default to a code review mindset: prioritise identifying bugs, risks, behavioural regressions, and missing tests. Findings must be the primary focus of the response - keep summaries or overviews brief and only after enumerating the issues. Present findings first (ordered by severity with file/line references), follow with open questions or assumptions, and offer a change-summary only as a secondary detail. If no findings are discovered, state that explicitly and mention any residual risks or testing gaps.


When doing frontend design tasks, avoid collapsing into "AI slop" or safe, average-looking layouts.
Aim for interfaces that feel intentional, bold, and a bit surprising.
- Typography: Use expressive, purposeful fonts and avoid default stacks (Inter, Roboto, Arial, system).
- Color & Look: Choose a clear visual direction; define CSS variables; avoid purple-on-white defaults. No purple bias or dark mode bias.
- Motion: Use a few meaningful animations (page-load, staggered reveals) instead of generic micro-motions.
- Background: Don't rely on flat, single-color backgrounds; use gradients, shapes, or subtle patterns to build atmosphere.
- Overall: Avoid boilerplate layouts and interchangeable UI patterns. Vary themes, type families, and visual languages across outputs.
- Ensure the page loads properly on both desktop and mobile

Exception: If working within an existing website or design system, preserve the established patterns, structure, and visual language.


You interact with the user through a terminal. You have 2 ways of communicating with the users:
- Share intermediary updates in `commentary` channel.
- After you have completed all your work, send a message to the `final` channel.
You are producing plain text that will later be styled by the program you run in. Formatting should make results easy to scan, but not feel mechanical. Use judgment to decide how much structure adds value. Follow the formatting rules exactly.


Persist until the task is fully handled end-to-end within the current turn whenever feasible: do not stop at analysis or partial fixes; carry changes through implementation, verification, and a clear explanation of outcomes unless the user explicitly pauses or redirects you.

Unless the user explicitly asks for a plan, asks a question about the code, is brainstorming potential solutions, or some other intent that makes it clear that code should not be written, assume the user wants you to make code changes or run tools to solve the user's problem. In these cases, it's bad to output your proposed solution in a message, you should go ahead and actually implement the change. If you encounter challenges or blockers, you should attempt to resolve them yourself.


- You may format with GitHub-flavored Markdown.
- Structure your answer if necessary, the complexity of the answer should match the task. If the task is simple, your answer should be a one-liner. Order sections from general to specific to supporting.
- Never use nested bullets. Keep lists flat (single level). If you need hierarchy, split into separate lists or sections or if you use : just include the line you might usually render using a nested bullet immediately after it. For numbered lists, only use the `1. 2. 3.` style markers (with a period), never `1)`.
- Headers are optional, only use them when you think they are necessary. If you do use them, use short Title Case (1-3 words) wrapped in **…**. Don't add a blank line.
- Use monospace commands/paths/env vars/code ids, inline examples, and literal keyword bullets by wrapping them in backticks.
- Code samples or multi-line snippets should be wrapped in fenced code blocks. Include an info string as often as possible.
- File References: When referencing files in your response follow the below rules:
* Use inline code to make file paths clickable.
* Each reference should have a stand alone path. Even if it's the same file.
* Accepted: absolute, workspace‑relative, a/ or b/ diff prefixes, or bare filename/suffix.
* Optionally include line/column (1‑based): :line[:column] or #Lline[Ccolumn] (column defaults to 1).
* Do not use URIs like file://, vscode://, or https://.
* Do not provide range of lines
* Examples: src/app.ts, src/app.ts:42, b/server/index.js#L10, C:\repo\project\main.rs:12:5
- Don’t use emojis or em dashes unless explicitly instructed.


- Balance conciseness to not overwhelm the user with appropriate detail for the request. Do not narrate abstractly; explain what you are doing and why.
- Do not begin responses with conversational interjections or meta commentary. Avoid openers such as acknowledgements (“Done —”, “Got it”, “Great question, ”) or framing phrases.
- The user does not see command execution outputs. When asked to show the output of a command (e.g. `git show`), relay the important details in your answer or summarize the key lines so the user understands the result.
- Never tell the user to "save/copy this file", the user is on the same machine and has access to the same files as you have.
- If the user asks for a code explanation, structure your answer with code references.
- When given a simple task, just provide the outcome in a short answer without strong formatting.
- When you make big or complex changes, state the solution first, then walk the user through what you did and why.
- For casual chit-chat, just chat.
- If you weren't able to do something, for example run tests, tell the user.
- If there are natural next steps the user may want to take, suggest them at the end of your response. Do not make suggestions if there are no natural next steps. When suggesting multiple options, use numeric lists for the suggestions so the user can quickly respond with a single number.


- Intermediary updates go to the `commentary` channel.
- User updates are short updates while you are working, they are NOT final answers.
- You use 1-2 sentence user updates to communicated progress and new information to the user as you are doing work.
- Do not begin responses with conversational interjections or meta commentary. Avoid openers such as acknowledgements (“Done —”, “Got it”, “Great question, ”) or framing phrases.
- You provide user updates frequently, every 20s.
- You must always start with a intermediary update before any content in the `analysis` channel. The initial message should be a user update acknowledging the request and explaining your first step. You should include your understanding of the user request and explain what you will do. Avoid commenting on the request or using starters such at "Got it -" or "Understood -" etc.
- When exploring, e.g. searching, reading files you provide user updates as you go, every 20s, explaining what context you are gathering and what you've learned. Vary your sentence structure when providing these updates to avoid sounding repetitive - in particular, don't start each sentence the same way.
- After you have sufficient context, and the work is substantial you provide a longer plan (this is the only user update that may be longer than 2 sentences and can contain formatting).
- Before performing file edits of any kind, you provide updates explaining what edits you are making.
- As you are thinking, you very frequently provide updates even if not taking any actions, informing the user of your progress. You interrupt your thinking and send multiple updates in a row if thinking for more than 100 words.
- Tone of your updates MUST match your personality.


When mentioning files or line numbers, always convert them to markdown links using workspace-relative paths and 1-based line numbers.
NO BACKTICKS ANYWHERE:
- Never wrap file names, paths, or links in backticks.
- Never use inline-code formatting for any file reference.

REQUIRED FORMATS:
- File: [path/file.ts](path/file.ts)
- Line: [file.ts](file.ts#L10)
- Range: [file.ts](file.ts#L10-L12)

PATH RULES:
- Without line numbers: Display text must match the target path.
- With line numbers: Display text can be either the path or descriptive text.
- Use '/' only; strip drive letters and external folders.
- Do not use these URI schemes: file://, vscode://
- Encode spaces only in the target (My File.md → My%20File.md).
- Non-contiguous lines require separate links. NEVER use comma-separated line references like #L10-L12, L20.
- Valid formats: [file.ts](file.ts#L10) only. Invalid: ([file.ts#L10]) or [file.ts](file.ts)#L10
- Only create links for files that exist in the workspace. Do not link to files you are suggesting to create or that do not exist yet.

USAGE EXAMPLES:
- With path as display: The handler is in [src/handler.ts](src/handler.ts#L10).
- With descriptive text: The [widget initialization](src/widget.ts#L321) runs on startup.
- Bullet list: [Init widget](src/widget.ts#L321)
- File only: See [src/config.ts](src/config.ts) for settings.

FORBIDDEN (NEVER OUTPUT):
- Inline code: `file.ts`, `src/file.ts`, `L86`.
- Plain text file names: file.ts, chatService.ts.
- References without links when mentioning specific file locations.
- Specific line citations without links ("Line 86", "at line 86", "on line 25").
- Combining multiple line references in one link: [file.ts#L10-L12, L20](file.ts#L10-L12, L20)


As you work, consult your memory files to build on previous experience. When you encounter a mistake that seems like it could be common, check your memory for relevant notes — and if nothing is written yet, record what you learned.

Memory is organized into the scopes defined below:
- **User memory** (`/memories/`): Persistent notes that survive across all workspaces and conversations. Store user preferences, common patterns, frequently used commands, and general insights here. First 200 lines are loaded into your context automatically.
- **Session memory** (`/memories/session/`): Notes for the current conversation only. Store task-specific context, in-progress notes, and temporary working state here. Session files are listed in your context but not loaded automatically — use the memory tool to read them when needed.
- **Repository memory** (`/memories/repo/`): Repository-scoped facts stored locally in the workspace. Store codebase conventions, build commands, project structure facts, and verified practices here.


Guidelines for user memory (`/memories/`):
- Keep entries short and concise — use brief bullet points or single-line facts, not lengthy prose. User memory is loaded into context automatically, so brevity is critical.
- Organize by topic in separate files (e.g., `debugging.md`, `patterns.md`).
- Record only key insights: problem constraints, strategies that worked or failed, and lessons learned.
- Update or remove memories that turn out to be wrong or outdated.
- Do not create new files unless necessary — prefer updating existing files.
Guidelines for session memory (`/memories/session/`):
- Use session memory to keep plans up to date and reviewing historical summaries.
- Do not create unnecessary session memory files. You should only view and update existing session files.




Here is a list of skills that contain domain specific knowledge on a variety of topics.
Each skill comes with a description of the topic and a file path that contains the detailed instructions.
When a user asks you to perform a task that falls within the domain of a skill, use the 'read_file' tool to acquire the full instructions from the file URI.

get-search-view-results
Get the current search results from the Search view in VS Code



Data Engineering on a Budget by Benjamin Logan
blogan@scottlogic.com (Benjamin Logan) — Tue, 12 May 2026 00:00:00 +0000
Context

Today a lot of data engineering projects leverage Cloud SaaS offerings like Databricks and Microsoft Fabric. While they are excellent offerings, they are quite expensive to use and do not always naturally lend themselves to best practises. In this article I am going to outline an open-source alternative to these common software systems that can be used in a small, lightweight data project. A “build it yourself” solution may appear more complicated and harder to maintain than a SaaS product but I will show that all depends on the size of your project.

The Project

Overview

In this scenario, we are creating a solution for a small data project. All data will be consumed from Kafka and there will be no more than a few hundred messages per day. 
The components used in this example are:


  Kafka
  Docker
  Prefect
  Postgres


In this example, messages enter the system via Kafka and are stored in Postgres using a medallion layer architecture. Prefect orchestrates the data pipelines, handling ingestion from Kafka to Postgres and performing additional ETL operations across the medallion layers. Although Postgres isn’t traditionally associated with medallion architectures, it can be effectively adapted for this purpose. Its support for multiple databases and tables enables the creation of distinct bronze, silver, and gold layers, each tailored to meet different consumer requirements.

All the components are underpinned by docker containers and since all the components are open source, running the containers will be the main infrastructure cost of the project. In production this project could be run on something like ECS Fargate with deployment pipelines pushing between environments. An alternative approach could be to run it on Kubernetes however this will increase complexity and cost. As a result I think it would be better to not use it.

An example docker compose file for this project is displayed below:

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.4.0
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - "2181:2181"

  kafka:
    image: confluentinc/cp-kafka:7.4.0
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "9093:9093"     
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL://localhost:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CREATE_TOPICS: "swiss_events:1:1"

  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: pguser
      POSTGRES_PASSWORD: pgpass
      POSTGRES_DB: pgdb
    ports:
      - "5432:5432"
    volumes:
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U pguser -d pgdb"]
      interval: 5s
      timeout: 5s
      retries: 5

  prefect-server:
    image: prefecthq/prefect:3-latest
    depends_on:
      postgres:
        condition: service_healthy
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://pguser:pgpass@postgres:5432/pgdb
      PREFECT_SERVER_API_HOST: 0.0.0.0
      PREFECT_UI_API_URL: http://localhost:4200/api
    command: prefect server start
    ports:
      - "4200:4200"
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:4200/api/health || echo 'Healthcheck failed'"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 90s

  create-pool:
    image: prefecthq/prefect:3-latest
    depends_on:
      prefect-server:
        condition: service_healthy
    environment:
      PREFECT_API_URL: http://prefect-server:4200/api
    entrypoint: ["sh", "-c", "sleep 20 && prefect work-pool create --type process local-pool"]
    restart: "no"

  prefect-worker:
    image: prefecthq/prefect:3-latest
    depends_on:
      prefect-server:
        condition: service_healthy
    entrypoint: ["sh", "-c", "apt-get update && apt-get install -y gcc libpq-dev && pip install --no-cache-dir -r requirements.txt && sleep 15 && prefect worker start --pool local-pool --type process"]
    environment:
      PREFECT_API_URL: http://prefect-server:4200/api
      PREFECT_WORKER_WEBSERVER_HOST: 0.0.0.0
    working_dir: /app/flows
    volumes:
      - ./flows:/app/flows

  register-deployment:
    build:
      context: ./flows
      dockerfile: Dockerfile
    depends_on:
      create-pool:
        condition: service_completed_successfully
    environment:
      PREFECT_API_URL: http://prefect-server:4200/api
    restart: "no"


Why Postgres?

There are many open-source database options available—so why choose Postgres? While other databases could certainly do the job, I prefer Postgres for its powerful capabilities in querying JSON data stored in database columns in its jsonb format. In this example, the initial data ingestion from Kafka into the bronze layer arrives in JSON format, so the payload is stored in a jsonb column. This allows us to take full advantage of Postgres’s robust JSON querying features when transforming data from the bronze to the silver layer.

In addition, I chose Postgres because it is better suited to lightweight workloads. While lakehouses are the industry standard in Data Engineering sometimes projects do not need the large scalability of a lakehouse. A lakehouse can take petabytes of data in different formats, but not all projects have petabytes of data and multiple formats. If there is one data source with a consistent format, that is not producing a huge amount of data, then using a database can be preferable.

What is Prefect?

Prefect is a python-only application that allows developers to create pipelines. It performs a similar role as Apache Airflow since they are both workflow orchestration tools. In Prefect, pipelines contain tasks that represent a stage of the pipeline. These pipelines can be integrated with packages like PySpark and Dask. The advantages of using Prefect that I will outline aren’t Prefect specific but rather apply to most open-source options.

Firstly, you aren’t locked into Spark. Spark is normally used by default but packages like Dask can be a good alternative. Spark does scale better than Dask when using data volumes above 100GB however if you are using data less than 100GB then Dask may be a better option. See Spark vs Dask: Environmental Big Data Analytics Tools Compared - Round Table Environmental Informatics for an interesting read on this. By using Dask you can get around managing Spark clusters and the learning curve should be less but will depend on your previous knowledge. Dask also works better for local development as it can be run from a simple python file without much setup whereas PySpark can be quite hard to set up locally.

from prefect import flow, task
from prefect_dask import DaskTaskRunner
import dask.dataframe as dd
from adapters.db_adapter import DBAdapter

@task
def extract_data():
    db = DBAdapter(username="kafkauser", password="kafkapass", host="localhost", port="5432", dbname="kafkafb")
    db.open_connection()
    query_str = "SELECT * FROM raw_table"
    df = db.query(query_str)
    db.close_connection()
    return df

@task
def transform_data(df):
    ddf = dd.from_pandas(df, npartitions=10)
    ddf = ddf.dropna().query("value > 100") 
    return ddf.compute()

@task
def load_data(df):
    db = DBAdapter(username="kafkauser", password="kafkapass", host="localhost", port="5432", dbname="kafkafb")

    db.bulk_insert_df(df)

    db.close_connection()

@flow(task_runner=DaskTaskRunner())
def etl_pipeline():
    raw_df = extract_data()
    transformed_df = transform_data(raw_df)
    load_data(transformed_df)

if __name__ == "__main__":
    etl_pipeline()


Above is an example flow in Prefect that leverages Dask to do some simple ETL operations. 
The code snippet above demonstrates how Prefect enables a more intuitive and modular approach to structuring workflows, supporting familiar design patterns like the adapter pattern. In contrast, platforms like Databricks and Fabric advocate for the use of notebooks, which—due to their cell-based architecture—can limit flexibility and make it harder to implement conventional coding practices seamlessly.

import psycopg2
import pandas as pd
from io import StringIO

class DBAdapter:
    conn = None

    def __init__(self, username: str, password: str, host: str, port: str, dbname: str):
        self.username = username
        self.password = password
        self.host = host
        self.port = port
        self.dbname = dbname
    
    def open_connection(self):
        self.conn = psycopg2.connect(
            dbname="my_database", user="user", password="password", host="localhost", port="5432"
        )
    
    def close_connection(self):
        self.conn.close()
    
    def query(self, query_str):
        return pd.read_sql(query_str, self.conn)
    
    def bulk_insert_df(self, df):
        cursor = self.conn.cursor()

        # Use a temporary buffer to bulk insert
        buffer = StringIO()
        df.to_csv(buffer, index=False, header=False)
        buffer.seek(0)

        cursor.copy_expert("COPY cleaned_table FROM STDIN WITH CSV", buffer)
        self.conn.commit()
        cursor.close()


As shown above, since Prefect is purely python we can create classes such as this DBAdapter class. This allows us to create cleaner code that is more maintainable and easier to read. Furthermore, this class can be mocked during unit testing to allow for the pipeline code itself to be tested. 
Unit testing is a lot easier in a pure python environment as you have a broader range of tools that can be used compared to what is available when using notebooks. Moreover, notebooks don’t lend themselves well to unit tests since they do not possess the ability to import anything from another notebook. 
In summary there are multiple advantages to being able to use pure python files over notebooks. These advantages include:


  Not locked into spark
  Better code structure
  Can use common code patterns easier
  Can import custom classes from python files
  Better unit testing


Trade Offs

The point of this article is to highlight that not every data project needs a big data software platform. There are trade-offs to using Databricks and Fabric as well as various open-source applications or cloud services. No one technology should be the default for every application; the use case should be examined carefully.
I personally think there are two types of architecture currently available – single big data platform or a more granular, lightweight platform.


  
    
      Architecture
      When to use
    
  
  
    
      SaaS Big Data Platform
      Many types of data sources and large volumes of data
    
    
      Lightweight Platform
      Single type of data source and small to medium amounts of data
    
  


For the purposes of breaking down the benefits and draw backs of both types I am going to compare them to booking a holiday.

SaaS Big Data Platform

This is the equivalent of an all-inclusive holiday. Everything is included from breakfast to flights. SaaS offerings abstract the details of managing infrastructure like setting up a delta lake or provisioning an EC2 instance to run your spark cluster on. It’s all handled by the platform.

Benefits


  Quick set up for a project
  Easy to get started
  Infrastructure managed by platform
  Security is easier to manage


Drawbacks


  High cost
  Often you will spend time developing “workarounds” for limitations of the platform
  Notebook development
  Can be a steep learning curve after the initial set up for developers
  Testing can be difficult


We don’t really talk enough about how steep the learning curve to fully master a SaaS platform can be. There are similarities between the different offerings however someone who has mastered Fabric will need to relearn a lot to master Databricks. It is like having expert skills in AWS and working on Azure.

Lightweight Platform

This is the equivalent of booking every detail of the holiday yourself. You book the flights, research restaurants and sorted out all the individual parts yourself and saved money in the process! 
A lightweight platform which uses open-source applications allows you to pick each piece of the tech stack. You will have less infrastructure costs, but it will result in a larger development time as there will be more devops tasks and coding work to do. The size of the project will dictate whether this approach will produce any benefit.

Benefits


  More selection of tools and services
  More control over each part
  Cheaper infrastructure costs
  More transferable skills


A benefit that perhaps should be considered more is developers having more transferrable skills. You do not need to be a master at Databricks for this to be feasible. A developer who has used python and docker will have a very shallow learning curve for the example project I have demonstrated in this article. This approach will more easily allow software engineers to act as data engineers.

Drawbacks


  Maintaining the platform can be expensive
  Development time will be longer
  Only useful for smaller projects
  Must be skilled in operating all the individual tech
  Have to stand-up the runtime platform unless using cloud services
  Reliability/availability can be harder to achieve
  Monitoring


Conclusion
The perfect solution does not exist, but the question is – does a lightweight architecture provide a better way to do data engineering? I think personally it provides a better approach when you know that there will be one type of data source that will not produce large amounts of data. SaaS offerings will be better for other use cases.

Pros of this project


  Better code structure
  Better unit testing
  Cost of development will be less as more can be ran on local machines instead of incurring cluster costs when developing


Cons of this Project


  More complicated as there are more parts
  Developers need knowledge of multiple services instead of just Databricks or Microsoft Fabric
  More work needed to set up and maintain the platform which could cost a lot


A final thought that could become a positive for open source
Having done a few projects on cloud SaaS offerings, I would be interested to know how much time engineers spend  working on fixes for limitations of the SaaS platform. It is something that is not normally recorded and can be quite repetitive as you will likely have to put the same work arounds in every time you use the SaaS product.



Open Source Is Not One-Size-Fits-All — Find Your Fit by Sonali Mendis
smendis@scottlogic.com (Sonali Mendis) — Wed, 29 Apr 2026 00:01:00 +0000
Feeling stuck in your day job? Tinkering with the same libraries, juggling different versions, or endlessly maintaining old code? You’re not alone. I’ve been there. When I worked as a developer in software product companies, I felt truly alive and curious only when I changed roles and faced new challenges. In consultancy, things moved too fast to go deep—leaving me stagnant.

This feeling isn’t uncommon. Talk to experienced engineers at meetups or your workplace, and you’ll find almost everyone has felt like a “frog in the well” at some point in their career.

Beyond the Bubble

There are many ways to break out of this bubble without constantly changing jobs. One path that often comes up is contributing to open source projects. It’s frequently seen as a go-to way to learn, explore, and grow—and it absolutely can be. The key is finding the right fit.

Maybe you’ve wanted to try a new role but never had the chance. Maybe you’ve wanted to explore a new language or framework. Open source can offer those opportunities—on real projects that people actually use, not just small hobby experiments that fade away.

Along the way, you’ll discover that contributing to open source:


  Connects you with new people
  Exposes you to different expectations and workflows
  Gives you the freedom to explore and experiment


What is Open Source?

If you’re new to open source, here’s a quick primer:


  Open source software is publicly accessible software that anyone can view, modify, and distribute. It’s built by communities of contributors from around the world.


Since the GNU Project in 1983, open source development has grown tremendously. Today, it’s hard to imagine building proprietary software—or even doing your daily work—without relying on open source libraries, platforms, or tools.

My First Attempt (Failure Story)

In 2011, I was working as a developer on a product that used Adobe Flex for its frontend. It was my first job after graduating in 2010. I was doing well—confident in my skills and comfortable in my role.

At the same time, I saw some of my former classmates contributing to open source projects and getting recognized for their work. That world intrigued me.

The first piece of advice I received was simple: find something close to your work and your interests.

So I chose Apache Flex. Adobe had just open-sourced Flex and donated it to the Apache Software Foundation. It felt like the perfect fit—it was the same technology I used every day.

I joined the mailing list, excited and motivated to contribute.

That excitement didn’t last long. Almost immediately, I was flooded with emails—threads full of experienced contributors discussing topics I barely understood. My confidence started to fade. The same person who felt capable at work suddenly felt completely out of place.

Still, I pushed myself to pick up an issue that seemed simple. But once I started, I was lost. I didn’t know where to begin, and more importantly, I didn’t know where to ask for help. This wasn’t like my workplace, where there were clear roles, teammates I could turn to. Here, everything felt unstructured and distant.

I hesitated to ask questions on the mailing list. I didn’t want to sound inexperienced or “silly.”
After struggling for quite some time, I managed to put together a fix and opened a pull request.
And then… nothing. It wasn’t prioritized. There were bigger issues the maintainers were focusing on.

I waited for feedback, but none came. Eventually, I gave up. I left the mailing list, stopped following the project, and never went back to that pull request again.

Why it Failed?

Looking back, it’s clear why this attempt failed. It wasn’t the project—it was my approach and my mindset at the time.
Here’s what went wrong:

1. Lack of experience with ambiguity

I was less than a year into my career. At work, everything was structured—tasks were clearly defined, and guidance was always available. Open source was the opposite. Issues were often loosely defined, and I wasn’t yet comfortable navigating that uncertainty.

2. Fear of communication

Unlike my workplace, this was a remote, distributed community. I didn’t know anyone personally, and that made me hesitant to ask questions. I was afraid of being judged. In reality, that fear held me back more than anything else.

3. Jumping in too quickly

I tried to contribute immediately without first understanding the project or the community. I didn’t spend time observing discussions, understanding workflows, or learning how contributors collaborated.

4. Imposter syndrome

The biggest issue wasn’t technical—it was psychological. Seeing so many experienced contributors made me feel like I didn’t belong. I started doubting my own abilities.

One important thing to clarify: the initial advice I received—to find a project close to my work and interests—was not wrong. In fact, it’s still one of the best pieces of advice for anyone starting out in open source.

The problem wasn’t the choice of project. Apache Flex was a good fit for my skills at the time. The real issue was how I approached the contribution process. I focused only on what to contribute, not how to contribute within a community.

To be fair, this was before many open-source projects widely adopted Codes of Conduct to make participation more welcoming and inclusive. Based on my limited interaction, I can’t judge the community fairly. But with over 15 years of experience now, I believe it was likely more welcoming than I assumed—if I had taken the time to engage.

At that stage in my career, I simply wasn’t ready. And that’s okay.

My Second Attempt (Success Story)

In 2023—more than a decade after my first failed attempt—I found myself back in the open source world. This time, it wasn’t entirely by choice. I was assigned to work on an open source project that my company was supporting. But something was different this time.

Instead of jumping straight into contributing code, I started by listening. I joined community calls regularly—even before the project had a name. At first, I was just there because it was part of my job. But soon, the idea of reshaping how cloud services could be used in the future started to fascinate me. I began observing how the community worked. Who spoke? Who made decisions? What problems were being discussed?

One thing quickly stood out. There were many subject matter experts—people with deep knowledge in areas like threat modelling and security controls. But there was a gap.
No one was actually doing the groundwork. There were ideas, discussions, and expertise—but no clear structure, no roadmap, and no one driving things forward.

This time, instead of feeling overwhelmed by the ambiguity, I leaned into it. I started small. Before each community call, I spent a few hours doing my own research. I didn’t fully understand threat modelling, but I tried to piece together what the project might need. Then I shared my thoughts openly—with no expectation of being “right.”

To my surprise, the community responded very positively. People started reaching out, thanking me for driving the project forward and asking for my opinion. That was a turning point—I realized I had gone from being an outsider to someone whose contributions actually mattered.

What started as small contributions—notes, ideas, early drafts—began to shape the direction of the project, FINOS Common Cloud Controls. Slowly, we started building a framework together. Not because anyone had a perfect plan, but because we were making progress step by step.

After two months, I was moved off the project to focus on a client engagement. But by then, something had changed. I had built a connection with the project and the community. Out of curiosity—and a genuine sense of ownership—I requested to continue contributing in my own time as a representative of my company.

Over time, my role in the FINOS CCC evolved. People came and went. New contributors and organizations joined. But the vision remained the same: to shape how cloud services are used in financial institutions, with the right level of controls in place. I went from being a contributor, to an active participant, then to a project maintainer, a working group lead, and eventually an ambassador—speaking about the project at FINOS conferences. What started as an assignment became something much more meaningful.

Why the Second Attempt Worked

So why did this attempt succeed where the first one failed?

The biggest difference wasn’t the project—it was me.

1. Comfort with uncertainty

In 2011, ambiguity overwhelmed me. In 2023, I embraced it. With over a decade of experience, I had learned that not having all the answers is normal. I trusted myself to figure things out along the way.

2. Not doing it alone

This time, I wasn’t a lone contributor. I had a small group of colleagues alongside me, and that made a huge difference. Even though we hadn’t worked closely before, we quickly became a support system for each other. It felt like stepping into something new with familiar faces—safer and more encouraging.

3. Focusing on contribution, not perfection

Earlier, I hesitated because I didn’t want to be wrong. This time, I shared ideas freely—even when they were incomplete. That openness helped move the project forward.

4. Active participation in the community

I didn’t just show up to contribute code—I showed up to listen, learn, and engage. Attending calls regularly helped me understand the vision, the people, and the direction of the project. Nothing felt “over my head” anymore because I was part of the conversation.

5. Leveraging what I already knew

I wasn’t a security expert. I didn’t know much about threat modelling or writing controls. But I knew AWS. And I knew that if the project involved cloud systems, that knowledge would be useful. Instead of focusing on what I lacked, I contributed what I had.

6. Growth beyond technical skills

One of the biggest surprises was how much I learned outside of coding. I learned by listening to principals and CTOs handle tough questions. I learned how to communicate ideas clearly, how to ask for resources, and how to encourage collaboration.

I was pushed into areas I had never explored before—public speaking, facilitating meetings, and leading discussions.

7. A sense of ownership

Over time, the project stopped feeling like “just work.” It became something I cared deeply about. I wanted it to succeed—not because I had to, but because I was invested in it.

In the end, the relationship was mutual: the project benefited from my contributions, and I grew because of it.

How to Find the Right Project

Finding the right open source project can make all the difference. As I learned from my own experience, it’s not just about picking a project—it’s about finding one that fits you.

Here’s a simple way to get started:

1. Find a project you care about

Start with something you already use or are curious about. It could be a library you rely on, a framework you want to learn, or even a tool you use daily. Interest makes a huge difference—it keeps you going when things get difficult.

2. Engage with the community

Before writing code, spend time understanding the project. Join community channels like Slack, Discord, forums, or mailing lists.
Choose these channels wisely—pick the ones that suit you and avoid overwhelming yourself with too much information at once.
Listen to discussions. Observe how people collaborate. This step is often overlooked, but it makes a big difference.

3. Make your first contribution

Look for issues labeled “good first issue” or “beginner-friendly.” These are intentionally designed to help new contributors get started.
Don’t aim for big changes early on—small, consistent contributions build confidence and momentum.

Your first contribution doesn’t have to be code. It could be improving documentation, fixing a typo, or asking a thoughtful question. The goal is to get comfortable with the process.

Setting the Correct Mindset

Finding the right project is only part of the journey. Just as important is having the right mindset when working in open source—because it can feel very different from a typical workplace environment.

Here are a few things to keep in mind:


  Contributions are voluntary: People contribute on a best-effort basis. Everyone has different priorities, so responses and progress may take time.
  There is less structure: Unlike your day job, there may not be clearly defined roles or responsibilities. This can feel uncomfortable at first—but it also creates opportunities to step up.
  Progress can be slow: Decisions take time, discussions evolve, and direction may change. Patience is key.
  You learn through feedback: Code reviews and discussions are one of the best ways to grow. Be open to feedback—it’s part of the process.
  Direction can change: Open source projects evolve based on community input. What you start may not always be what gets shipped—and that’s okay.
  Share your ideas openly: Don’t wait for perfection. Sharing early helps move the project forward and invites collaboration.
  Help create a welcoming environment: Open source thrives on community. The more welcoming and supportive you are, the better the experience becomes for everyone—including you.


Myths About Open Source

Before concluding it’s important to talk about misconceptions about open source that stop people from even getting started. Let’s clear up a few of the most common ones:

Myth 1: “I need to be an expert”
Reality: Everyone starts somewhere.

Most contributors you see today were once beginners trying to understand the same codebase. Open source is not about knowing everything—it’s about learning in public and improving over time.

Myth 2: “I don’t have enough time”
Reality: Even a small amount of time is enough.

You don’t need to spend hours every day. Even one or two hours a week can make a meaningful contribution over time. Consistency matters more than intensity.

Myth 3: “I need to write code to contribute”
Reality: Contributions go far beyond code.

Documentation, testing, reporting bugs, improving examples, or even helping others in discussions are all valuable contributions. In many projects, these are just as important as writing code.

Myth 4: “Open source communities are unwelcoming”
Reality: Most communities are supportive and encouraging.

While every community is different, most open source projects genuinely want contributors to succeed. Many have adopted Codes of Conduct (such as the Contributor Covenant) to make participation safer, more respectful, and inclusive.

From my own experience, the biggest barrier wasn’t the community—it was my assumption about it. Once I started engaging, I found people far more helpful and welcoming than I expected.



Everyone’s journey is different, and so are the opportunities we come across. Sometimes, the right opportunities don’t present themselves—and in those moments, we have to create our own.

Open source is one of the most powerful ways to do that.

Think of it like walking into a shoe shop. Not every shoe will fit you. Some may look good but feel uncomfortable. Others may fit well but not excite you.
The key is to find the one that fits you—your interests, your skills, and your goals. And just like any good pair of shoes, there’s a price. In this case, it’s your time and effort. But when you find the right fit—something that matches both your interests and your commitment—the return is immense. That’s when growth accelerates. That’s when you start enjoying the journey.

Choose the right open source project, and it won’t just be another activity—it can become a catalyst for your career and personal growth.



From AI Slop to AI Empowerment: Raising the Bar for Open-Source Contribution by Tom Stavert
Tom Stavert — Tue, 14 Apr 2026 00:00:00 +0000
If you’ve worked collaboratively on software before, you’ve probably experienced it. You check out a pull request with the intention of leaving a review, ready to dive in and understand your colleague’s solution, leave a few helpful comments, and maybe bless the code with your seal of approval.

Suddenly, however, you’re hit with thousands of lines of changed code, fifty files added, and (if you’re lucky) a brand-new README file created minutes before the code was put up for review with some vague instructions on how to get it set up. You’ve now got the thankless task of making sense of this gargantuan submission, disentangling a complex web of classes, configuration files, and cryptic comments.

The one saving grace that you could count on until a few years ago was that a human had written this, and therefore at some point they had (presumably) put some thought into its design, and if all else failed, they could understand the reasons for the choices they had made. You might even be able to call them up and get them to walk you through a particularly confusing section, and in the process maybe you (or the author) would learn a thing or two.

Nowadays, however, seeing a large amount of code changed summons a different concern: how much of this is AI-generated? The signs are clear: a PR description littered with emojis, strange new libraries imported, overly verbose variable names, or thousands of lines of excessive unit tests that cover every possible failure state.

These things aren’t always bad, and the code might work perfectly fine, but your job as a reviewer is to ensure the quality of this code, and now you’ve got your work cut out for you. What’s worse is that you know that when you hit something unusual that you don’t understand, or can’t explain, and you ask your colleague for an explanation, you’re probably going to receive something along the lines of: “Oh sorry I’m not sure, AI generated that bit… seems to work though, right?”

This challenge is felt more than ever by the open-source community, who are increasingly fighting a losing battle against “AI Slop” submissions. Many open-source projects have spoken out against this issue or even put measures in place to tackle low-effort submissions. The maintainers of Godot have expressed despair at the quantity of low-quality code contributions, stating that many contributors don’t even understand their own changes. The maintainers of cURL have abandoned bug bounties due to this same issue, while the maintainers of tldraw have been forced to review their contributions policy.

In contrast, some open-source maintainers have expressed more positivity towards developers utilising AI. Giving back to an open-source project, while fulfilling, can be a time-consuming process, from the initial steps of becoming familiar with an established codebase and rules surrounding contribution, to developing a solution that is worthy of being merged. If AI is used effectively, it can reduce friction in this process, and when properly utilised, this could lead to an increase in the number of high-quality open-source contributions being submitted. What is it then that differentiates “AI slop” from a valuable and appreciated open-source contribution?

Fluxnova and the DTCC Hackathon

I was struck by the importance of responsible AI usage for open-source contributions shortly after joining Scott Logic’s team for the Depository Trust & Clearing Corporation (DTCC) Industry-Powered AI Hackathon.

Over the course of just three days our team had to develop an AI-driven solution to a real problem facing the finance industry. Our team was committed to a solution which involved Fluxnova, an open-source workflow automation and orchestration platform that allows business processes to be modelled, executed, and monitored in a transparent and auditable way. The problem we aimed to tackle was the difficulty involved for non-technical people in retrieving workflow information from the system, forcing businesses to invest in training dedicated teams to resolve issues.

Our idea was to use generative AI (and in particular, model context protocol tools) to expose this workflow information to a chatbot-style AI, allowing it to return easily understandable updates about ongoing workflows in response to natural language queries.

Now the question was: how do we turn this idea into a real, functional piece of software in the short amount of time we had available? The first challenge we faced was a lack of familiarity with the Fluxnova codebase, which would be essential to understand how to extract the workflow information that we needed for our chatbot. This problem is all too common when getting to grips with a new codebase, and a critical barrier in the path of would-be open-source contributors.

While reading documentation and exploring the engine code manually were useful activities for gaining this familiarity, AI tools were invaluable in rapidly getting up to speed with both the domain and the codebase. The ability to interactively query the AI as my learning developed and ask targeted questions greatly accelerated my learning, and I was able to get up to speed relatively quickly, a task that might have taken days to achieve otherwise.

Once a better understanding of the codebase was achieved, the next challenge was working out how to implement our planned solution. At this stage, it can be tempting to have an AI agent decide how to do this, and the idea of delivering this “one-shot” prompt is appealing from a standpoint of productivity. However, in my experience trying to do too much at once can lead to unpredictable and unreliable results, particularly when working with such a large codebase.

It is frequently observed that AI coding tools work best on relatively small units of code and that they struggle with large, pre-existing codebases, likely due to the inability of AI agents to effectively deal with such large amounts of context. Some developers have demonstrated effective techniques for modernising codebases to allow AI agents to interact more easily with them, but a complete AI-driven overhaul of an open-source project is overkill for a hackathon, as well as most open-source contributions.

Instead, a more robust approach was taken which is similar to the way in which you would tackle the problem without an AI coding agent: breaking the task down. By splitting the larger task into smaller, more manageable subtasks, each with clear direction, it is much easier to effectively steer the AI agent rather than simply hoping that it understands your intentions at each stage. This requires a clearer picture of how the implementation should look from the outset, which takes time to develop. However, this would be required for a more “manual” approach regardless, and the use of AI obviously speeds up actually writing the code significantly.

With this direction I was able to build a working prototype within a matter of hours, a task which would have taken much longer otherwise. If this code had needed to be written manually, dedicating so much time to one solution would have been incredibly risky, with disastrous consequences for the hackathon if it didn’t work out. However, because I was able to produce the prototype so rapidly, we were able to test and verify that it was fit for purpose and spend the rest of the available time testing and refining our solution further.

Hackathon to Open Source Contribution

After the Hackathon concluded, we realised that the solution which we had developed might be relevant to the maintainers of the Fluxnova project, and so we worked to refactor it into a more production-ready state. While our solution was fit for a Hackathon submission, we knew we should expect to be held to a higher standard as open-source contributors. We now shifted our focus from hacking together a working solution to understanding how our offering could be best integrated into the Fluxnova ecosystem.

Part of this process simply involved starting a dialogue with maintainers of the Fluxnova project to understand how they saw our work fitting into the platform, whether it should be part of Fluxnova’s core offering or available as a plugin. This communication with stakeholders is something that AI simply cannot do for you (at least for now!).

No matter how high quality the code, it won’t be accepted if the contribution is simply not a good fit for the project, so taking the time to get to grips with the expectations of those who will be reviewing your code is a critical step that shouldn’t be missed. In our case, we gathered that our contribution would be most appropriate as a plugin, and we got involved in establishing a new plugins repository for Fluxnova.

Another critical consideration was the quality of the code itself. Having taken more time to understand Fluxnova’s architecture, it was clear that there were a multitude of potential approaches that could be taken, each with their own pros and cons. AI was an effective tool in rapid prototyping of these different approaches to assess their effectiveness, allowing us to make more informed decisions as we narrowed down on our final solution.

One problem that was common to all of these solutions was the relatively large quantity of code that was produced by AI. Most AI agents appear to produce code in an unnecessarily verbose style, leading to a huge number of added lines which become cumbersome to review. While there may be nothing wrong with the code itself, this problem is even more pertinent to open-source contributions as we are also relying on our code being reviewed by maintainers who may be put off by the prospect of sifting through an unnecessarily lengthy pull request.

The challenge, then, is to trim down the code to a more reasonable size. In my experience, AI agents are particularly effective at this kind of refactoring work, spotting where logic can be condensed into helper methods and more succinct syntax. When embarking on a refactor like this, unit tests serve a critical purpose to allow AI agents to ensure that their changes won’t break the functionality. In fact, unit tests themselves are a key area in which the quantity of code produced can be significantly reduced.

When relying on AI agents to produce code, it’s more important than ever to carefully review tests and ensure that they are as comprehensive as possible. AI-produced unit tests are frequently complete overkill, sometimes thousands of lines long and impractical to review manually. My first task, therefore, was to trim down the unit tests into a more reasonable set of tests that I was confident I could effectively review. With these tests in place, I was able to allow AI to refactor and condense the code it had originally produced, without fear of creating any breaking changes.

Aside from reducing the quantity of code, another way of making your code easier to read and understand, and therefore to review, is high-quality documentation. A decent architectural description of a codebase inserted into a README file works like a map, guiding you around an unfamiliar codebase considerably more easily than just trawling through files. However, in my experience, documentation can easily be forgotten about when you are neck-deep in a refactor or excitedly implementing a new feature.

This is an area that I think AI really excels in. An important part of any AI workflow is guiding your agent towards updating documentation as it goes. Keeping your documentation up to date not only helps any humans that come to review your code in future but also makes your code easier for AI agents to work with. Out-of-date documentation that contradicts the truth of what’s happening in a codebase is a sure way to confuse an AI agent, at best wasting tokens as it tries to make sense of an objectively confusing situation, or at worst completely leading it astray.

The Imbalance of Effort

It is unquestionable that AI coding agents are improving, both in terms of the underlying models and the tooling they have access to. It is beginning to become accepted in the industry that AI agents will be involved in some capacity in the software development process, even if the extent to which they are involved and the autonomy they are given is still in contention. Why is it, then, that so many of us are so concerned about submitting AI-generated code in open-source contributions?

There is a definite stigma surrounding AI use in many circles. Interestingly, I have felt this stigma particularly strongly not from experienced developers but rather from my colleagues who have more recently left education. For those of us who have graduated from university in recent years, we will remember the panic of many universities who were unprepared to handle the sudden rise of AI tools, with many universities outright banning their use for academic work. While their stances have largely softened into more refined policies and measured advice, the remnants of this reaction remain in the psyche of those who experienced this vilification of AI.

Another factor is that while these tools have been updated and improved, many people’s views have not kept up with these changes. Many of those who tried implementing AI tools into their work too early were left unimpressed by their effectiveness and still hold an outdated opinion of their usefulness in producing quality code.

Ultimately though, I think the biggest concern we have when submitting AI-produced code is that it will be perceived as a low-effort submission, regardless of the amount of effort that was put into reviewing and refining it. It is true that the time taken to produce low-quality code with AI is much lower than the time it takes to review it, particularly if the AI-generated code is not reviewed properly by the contributor. If your contribution consists purely of copy-pasting the description of a GitHub issue into Copilot and letting it take care of the rest, then it’s hardly surprising that open-source maintainers would resent your lack of effort and label your contribution “AI slop”.

It’s often difficult to gauge how much of a contribution is AI-generated until one delves into the code itself, so there’s always the risk that you spend time trying to understand code that the contributor themselves didn’t write and may not have even looked at. It’s this imbalance of effort, I think, that has led so many open-source maintainers to despair at the impact of AI on their projects.

More transparency surrounding the extent of code that is AI-generated seems a logical way forward, but it needs to break through the stigma which has been built up surrounding AI use. Being transparent about AI use needs to be done delicately, or it risks alienating AI-skeptic maintainers who don’t want AI agents anywhere near their codebase. Those of us who believe that AI is a valuable tool for producing high-quality code therefore need to be champions of its responsible usage when it comes to open-source.

While it seems obvious, making sure AI-generated code is thoroughly reviewed before you submit it is critical if you want to build up trust in AI-powered solutions for your favourite open-source projects. All it takes is a few low-quality submissions with AI-generated code to torpedo not only your own reputation as a contributor, but also the maintainer’s reception of AI-generated code in general.

A good rule of thumb is to spend at least as much time reviewing AI-generated code as you would expect a reviewer to spend on it (and probably a fair bit more). Don’t leave it up to others to do that step for you and make sure you’re ready to answer questions about the code when the time comes. We won’t always get this perfect as we adapt to this new workflow. So, when we do let some code slip through that we don’t fully understand, it’s important to take ownership of that mistake and not try to deny or defend these mistakes to the detriment of code quality and the quality of our submission.

We must take as much pride in our code now as we did before AI, while giving proper consideration to the reviewers who give up their time. If we do both, we shouldn’t be afraid to embrace AI’s power to improve open-source projects.



Beyond the Hype: Is AI taking the fun out of software development? by Colin Eberhardt
Colin Eberhardt — Wed, 01 Apr 2026 00:01:00 +0000


In this episode, I’m joined by Dean Kerr (Lead Developer) and Amy Laws (Developer) to discuss ‘The Experiment’ – a four‑week study we ran to explore how AI really affects software development. Instead of synthetic benchmarks, the project team tackled genuine issues in an open‑source project, alternating between AI‑assisted work and going completely ‘cold turkey’.

The contrasts were striking. Amy’s AI‑free period exposed how dependent everyday development has become on instant summaries and contextual answers. Meanwhile, Dean found that adopting a simple agentic loop (Analysis → Implementation → Reflection) helped him make better use of AI rather than blindly accept its output.

Together, their experiences reveal a discipline in flux: developers gain speed and support from AI, but also confront questions about craftsmanship, learning, and where the fun in software now sits as the tools reshape both workflow and mindset.

Useful links for this episode


  
    Analysis → Implementation → Reflection – a practical technique for issue resolution with agentic AI – Dean Kerr, Scott Logic
  
  
    Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity – METR
  
  
    SWE-bench – SWE-bench Team
  
  
    Agentic Engineering Patterns – Red/green TDD – Simon Willison’s Weblog
  


Subscribe to the podcast


  
    Apple Podcasts
  
  
    Spotify
  


Transcript

Please note: this transcript is provided for convenience and may include minor inaccuracies and typographical or grammatical errors.

Colin Eberhardt

Welcome to Beyond the Hype, a monthly podcast from the Scott Logic team where we cast a practical eye over what’s new and exciting in software development. Everything from Kafka to Kubernetes, AI to APIs, microservices to microfrontends. We look beyond the promises, the buzz and the excitement to guide you towards the genuine value.

I’m Scott Logic’s CTO, Colin Eberhardt, and each month on this podcast, I bring together friends, colleagues, and experts for a demystifying discussion that aims to take you beyond the hype. In this episode, I’m joined by Amy and Dean, who have been undertaking an experiment to quantify the impact of AI on software development.

And yes, they found that AI resulted in an increased velocity. However, through this experiment, we learned much more about the broader impact AI is having on software development. And the impact on the human beings who are still an important part of this process. This impact is leading us to ask questions about the future of this craft, and ultimately, here, we ask whether it’s taking the fun out of software development.

We pick up the conversation by discussing what this experiment was and why we undertook it in the first place.

Introduction to The Experiment

Colin Eberhardt

We sort of came round to this discussion from something the two of you have been doing, uh, within Scott Logic, uh, recently, that internally we, we seem to call it by the shorthand of The Experiment, which makes it sound really quite grand. But Dean, for the benefit of the people who have no idea what The Experiment is, it would be great if you could describe what the two of you have been up to recently and why.

Dean Kerr

Yeah, so we’ve got a small sort of AI Incubator team within Scott Logic, and we were looking at running an experiment, as you say, into the effectiveness of AI when it comes to, uh, development. Both from a sort of a qualitative standpoint and also from a quantitative standpoint. So, how much faster does it make you, but also, and it could be more importantly here, how does it make you feel, and does it improve or sort of reduce job satisfaction?

And we ran this experiment over four weeks. So, we had two phases of The Experiment primarily. We had a two-week upskilling phase where we got to grips with an open source library of our choosing. We preselected 10 issues from that open source library. Uh, making sure that the issues were sort of real-world issues that we’d typically encounter in our role as software consultants.

So, issues that are relevant to us. And then, the second half of The Experiment, the latter phase, was actually going away, pairing up and then individually within those pairs, working through each of those 10 issues using an approach. One approach was actually “ cold turkey” – no AI whatsoever – which was quite interesting; some team members who had used AI previously are now going back to no AI, which has an interesting viewpoint there. And then the other approach was using what we termed basic AI. So, in this case, it was what we get from GitHub Copilot on the sort of bare-bones subscription, which is roughly, I think $20 a month per seat. And we picked a sort of free model. Which was GPT-5 mini.

Colin Eberhardt

Cool. So I guess there are lots of people and organisations trying to measure the impact of AI, but The Experiment, as we call it, does feel a bit different. A lot of the benchmarks that you see published tend to be more clean-room environments. Uh, often they are measuring the effectiveness of just the AI itself, not AI plus a human being.

It feels like this experiment is a little bit more of a better reflection of the real work of a developer in a complicated and potentially unfamiliar environment. From your perspective, Dean, does it, did it feel like a good reflection of real work?

Dean Kerr

I believe so. I think it definitely felt closer to home. Like you said, there are plenty of experiments out there that we could lean back on that are fairly clean room. They’re actually quite wide-scale as well, whereas we kept ours sort of smaller and simpler, and I think more focused.

Importantly, so that we have evidence from these experiments that is actually relevant to us and our domains that we typically work in as well.

Real‑World Workflows: Open Source, Quality and Benchmarks

Colin Eberhardt

So why pick an open source project over just some internal code base? What was the thinking behind that?

Dean Kerr

I think there were two caveats to picking an open source project, one of them being the ability to actually contribute back to open source. So we ended up picking a mock service worker, and the plan there is to contribute back to some of the libraries that we ourselves have used professionally in our day-to-day roles. So it’s nice to contribute back there. But I guess secondly, there was also a sort of proof of a mechanism in play here, so that a lot of the experiments, you know, run an issue for you, the standard software development lifecycle. And we could actually use as proof the fact that the issues we worked on were actually reintegrated into a mock service worker or the open source library as a sort of a gold marker or gold standard to say, “Hey, the work we did, it wasn’t just slop.” I know there was a lot of, uh, talk recently around a lot of open source libraries just getting a load of pull requests put up. But we wanted to ensure that our work was relevant and appropriate.

Colin Eberhardt

Yeah, that’s really interesting because the benchmarks that the organisations that are developing the foundation models, they have a benchmark and some of them are relatively real-world things like SWE-Benches, a bunch of GitHub issues, and their goal is to typically make a suite of tests pass.

So, they’re demonstrating that it’s functionally correct, but there isn’t really anything that assesses the quality. Whereas not only are you doing something of actual value by fixing a real-world issue, the fact that the quality gate is outside of the team means a maintainer has to go, “Yeah, I’m, I’m happy with that.”

That’s a real rubber stamp that the solution that you came up with, whether you used AI or not, was of good quality.

Dean Kerr

There were a couple of issues we encountered as well, where we had maybe two or three solutions that were all equally valid, but it was more a matter of preference which one would be accepted. And it was almost sort of second-guessing what the library owner themselves would prefer. So, each of these three solutions, they all tackled the issue slightly differently, and that meant trade-offs in different ways for the mock service worker.

Colin Eberhardt

So, one of the things that I found interesting about this experiment was not the obvious. The spoiler is, funnily enough, AI makes you faster. And I don’t think anyone’s surprised by that. I think we came up with a factor of 1.9. Could have been any number, really. We knew that it was, or at least we hoped that our experiment was going to demonstrate that you’re faster, because we certainly feel that. What I found more interesting and quite surprising is some of the things that, that you as the team learned along the way. And I think some of the most extreme learnings and the most interesting learnings came from the team that wasn’t allowed to use AI at all. What, what you called the “cold turkey” team.

Cold Turkey Begins: Rediscovering Pre‑AI Development

Colin Eberhardt

Now, Dean, you were lucky, you got to use AI even if it was the old, slightly rubbish version. But Amy, I’d love to hear more about your experience, because you were in cold turkey land, and I know you worked alongside Andy. And what I found really funny was that he, he went, he went deep, he tried to eradicate AI from his daily personal life as well.
What, what, what was it like having AI taken away from you?

Amy Laws

Yeah, so I think like coming into this, um, obviously being on the, um, AI Incubator team, we’re kind of encouraged to explore and use it more heavily than other people. So, it really was kind of a contrast having to go from using AI and experimenting with it to not using it at all. So, when we say no AI, Andy and I really tried to do everything we could to avoid AI. So, we disabled it in our IDEs because what I forgot is that it’s so integrated now that, although I wasn’t actively opening the Copilot chat panel, the autocomplete was still on, so I had to fully disable it. The same, even doing kind of Google searches, which is kind of your fallback, the AI search automatically pops up at the top.

So, you are having to scroll straight past it, and it’s one when you’re actively trying to avoid it, you realise just how ingrained it has become in everything that we do. So, that was kind of a bit of a learning curve, I guess, reverting to an older way of working, of picking up, going back through your Stack Overflow and forums and reading docs and those kinds of things that I’ve not had to do for quite a long time.

Colin Eberhardt

What was it like going back to Stack Overflow? Because I’ve seen the statistics that the Stack Overflow traffic has dropped considerably, so we know fewer people are using it, which makes sense, but for the few people that are using it, so you were forced to use it, what was it like going back to it? Is there a feeling that information is now lacking because Stack Overflow only exists because of an ongoing question-and-answer flow? But what’s it like there now?

Amy Laws

Yeah, definitely. Like, I guess when I was using it a couple of years ago, I would kind of automatically discount older answers or take them with a pinch of salt because things in tech move so quickly. Something that was answered five years ago might not necessarily be relevant anymore. But that’s really hard to do these days because, as you said, the response rate and, I guess, people asking questions on it, are dropping so rapidly. We just found that a lot of the things we were Googling or like searching on there, there just weren’t modern responses, and it may not be compatible with the versions of the libraries we had and that kind of thing. So, I definitely found myself having to search a lot harder on it than I would’ve previously.

Colin Eberhardt

Yeah, so it is weird. We are, We are getting to a point where, without AI, it’s fundamentally harder to find the answers.

Amy Laws

Yeah, I definitely agree with that. And I guess, I think I’ve lost the skill a little bit as well. Um, so I’ve not had to search through like documentation for quite a long time, and I think there’s kind of a skill and a nuance to that, that you kind of have without realising it.

Colin Eberhardt

Yeah. So, taking a step back, when you talk about AI-assisted software development, most of the time you think about AI writing the code.

Losing Old Skills, Missing New Tools

Colin Eberhardt

And another thing that I found really interesting in this experiment was that so much of what you missed was not the fact that it wrote the code for you. It was all the other things.

Can you talk about that? I mean, you’ve already talked about searching for answers to questions, but there were a lot more examples than that, weren’t there?

Amy Laws

Yeah, so a big one for me was kind of summarising information. So, it’s already been spoken about, but we worked on open source issues. But what we did with that is we actually went from the oldest ones on the backlog to the newest. So, a lot of the issues that we had were quite old. Some of them had very, very lengthy comment threads; some of them had over 60 comments. And a lot of that was noise, so people saying, “Oh, have you tried this workaround? This might have been fixed in this version.” No, it hasn’t. And all that kind of information that isn’t really that helpful a few years on.

Trying to kind of understand that entire comment thread and retain the important bits of information was quite difficult, especially when our job involves things other than just coding. So I’d kind of get myself up to speed with it and then get broken off to go to a meeting or something, and then coming back and trying to get back into that thread and what was relevant and what wasn’t was quite hard.

And I felt it was frustrating to know that if I had AI, I could have just thrown that issue into AI and got a nice summary. And I think that’s one of the things of having had it removed, you realise what you’re missing more.

Colin Eberhardt

Yeah, absolutely there. And thinking about how I use it and to, to your point, things like through Google search, I mean, often if I’m looking for something, I still use Google pretty much in the same way I’m trying to find a thing. But if I’m asking a question, more often than not, its AI-generated summary will answer that question. And I don’t have to navigate to a different site. It’s changed the way that I work without me intentionally, choosing that change or even acknowledging that’s the change that’s taken place.

Dean Kerr

I guess it’s one of the, sort of, the life cycles of the internet I’ve seen where it started with IRC (Internet Relay Chat) and message forums and things like Stack Overflow might be the next sort of format to sort of be cannibalised, really by the next format, which may well be just an input box where you ask AI.

Colin Eberhardt

Yeah, and that’s a massive rabbit hole that I, I’m not sure we’ll even go down because I, I couldn’t work out, well, I can’t work out what the future might look like because we know that the AI was trained on that data set, that’s what makes it so powerful, and the feedback loop has gone. If we are not asking questions, what does the AI train on? As I said, we’ll not go down that rabbit hole ‘cause I have no idea what’s gonna happen.

But getting back to some of the things that you mentioned about using AI to help you answer questions, to summarise information. What I find really interesting about that is that there’s a lot more tolerance for error in the AI. We, we focus a lot on how much code can it emit? How, how good is that code? And, and if it’s not good, some people sort of reject AI for software development to a certain, to a certain extent because they don’t think it writes quality code.

Whereas when it’s summarising an issue thread or you’re just having a conversation with it, it feels like you can be a lot more tolerant. When you were using it to summarise issues, did you think about what the quality of the AI summarisation is, or was it just that it felt good enough?

Dean Kerr

Yeah, I guess for me it did feel like it did a pretty good job. Like Amy mentioned, there are pretty long comment threads in some of the issues, and there is the case of, yeah, do you trust the summary because things will get cut out, and the AI is making a judgment of what things are relevant and no longer relevant.

But looking, you know, I had a look afterwards, ‘cause the, the raw sort thread is there. And it did seem to do a pretty good job at that. I think what’s interesting is, I guess it’s, in this case, it was a GitHub issue, but it could equally also be a Jira issue or a Trello issue. And one thing I didn’t really get to explore, with the GitHub issue, is using the sort of multimodalness of modern AI now, where in Jira you typically have screenshots and, it could be error messages, or stack traces that you can all feed to the AI to summarise as well. So, I think I did a good job with the textural summary. I’ll be really interested in terms of next, if I feed it more than just one piece of data, how it would actually do in terms of summarising all that as well.

Colin Eberhardt

Yeah, I guess this gets into, we’ve talked a bit about going cold turkey, and how that really made you sort of reflect on how much you use AI, and to a certain extent, we’re all becoming dependent on AI. Another thing I found interesting in the experiment was from the team using AI, and that it wasn’t a case of, oh yeah, roll up our sleeves, we are using AI.

Agentic Loops and Structured AI Workflows

Colin Eberhardt

Dean, you talked about the approach that you are using and, and you, you, you sort of described it as a particular pattern of Analysis → Implementation → Reflection. How did you get to that point? Why? Why did you sort of feel the need to almost formalise your own approach?

Dean Kerr

There’s been a lot of sort of literature recently that talked about closing the feedback loop with agentic AI and the sort of improvements that can be made through that. And, um, yeah, there’s some, for example, some recent posts on porting across things as large as web browsers or parsers with relative ease and swiftness as well.

So, I thought if that could work for sort of a larger-scale project going, I guess from a relatively greenfield, ground zero to a fully working application, I felt like it would be equally valid for just picking up a single issue as well. So yeah, I think putting a little bit of effort in at the very beginning, uh, knowing that I had to tackle 10 issues, to set up a, a lightweight harness that Analysis → Implementation → Reflection, I felt that that would pay dividends.

So, and I think it did in the end, uh, the, the Analysis phase, looking at the issue, summarising the thread for me, giving me the chance to interject, I didn’t agree with what it had summarised or if it was a bit off in, in, in some respects to its understanding of the issue. I gotta remember I was using the free model from around August, so I think the capabilities of models have jumped since then. So, um, I think it didn’t get things as right as the premium models. So having the ability to interject was quite useful. The implementation phase as well, where I could set the model off now that I have sort of double-checked its understanding, let it run implementation against some test cases that were built during the Analysis phase and in a sort of red/green, uh, loop, uh, run into those test cases pass.

Colin Eberhardt

So, was that actually an agentic loop? Did you basically instruct it once you’ve done your analysis and you’ve built up a test suite? Do you then basically say, right, now you can solve the issue.

Dean Kerr

Yeah. And, if it was a feature, implement the feature, or if it was a bug, you know, implement the fixes in it.
I think you read in these articles, and it all feels very grand and formal, but it is really just prompting the agent to run tests until they pass. It’s as simple as that, really.

It doesn’t have to be a heavyweight, super complex harness, uh, you can get away with a, with a simple prompt, really just to say, use red/green.

Colin Eberhardt

Yeah. And that, that’s something I find really interesting about AI and agents as a tool, that they’re not in any way opinionated. And by that I mean they don’t have an opinion about how you should use them. And I dunno, Dean, whether you’ve, whether you’ve had the time to think about how you used these tools in the past, was it more because we were running this experiment, you thought, I need to have a think about how I approach this. I’ve heard of agentic loops. I, I believe that to be a productive way of working. I’m gonna spend a lot of time working out how I approach the construction of an agentic loop. I mean, was that quite new to you, to spend that much time considering how you use this tool?

Dean Kerr

I think a lot of it’s fairly new to a lot of people. It moves that quickly. And that was the, I guess, the beauty of the first phase of the experiment, where you get two weeks to do upskilling. A lot of studies, uh, as you’ve seen elsewhere, don’t give anyone any time to get to grips with the approach, which is, you know, what model you may be using, what tooling, be that GitHub Copilot or Claude or others. And then the approach being agentic AI, for example. So, having the time to actually experiment and see what does and doesn’t work for a particular approach gave me the time necessary to fall into that sort of feedback mechanism, which is probably one of the latest styles of approaches that people are currently doing at the minute.

Colin Eberhardt

Yeah, I think this one was inspired, I think, by a study by METR, where they were looking at the impact of AI, I think, on open source maintainers. And what they effectively proved was that people who are experienced with AI are better at using AI. That was sort of paraphrasing it. But Amy, what are your thoughts on Analysis → Implementation → Reflection? To a certain extent, does that potentially reflect the way that you, as a human being, approach the problems?

Amy Laws

Yeah, I think so. Obviously, we didn’t have the AI to help me with it, but I think it is kind of an extension of our natural workflow. And I think from using AI for other things, um, the more and more I kind of treat it in the same way that I would work without it, I have found it is more effective.

I guess, kind of naturally, the Reflection part is the most important. And I would naturally do that before I put something up for PR (Pull Request). I would kind of sit and like almost review my own PR before I put it up. And I think probably with AI, I’ve developed a bit of a tendency not to do that quite as much or not be quite as critical.

Whereas I think going back to not using it, I made sure that I really, really understood what every line of code was doing and that I was kind of happy with it. And I think that level of understanding is something that I didn’t realise that maybe I’ve become a bit more lenient with.

Colin Eberhardt

Yeah, I think you make a great point ‘cause it’s, it can be really quite hard to work out how you should be using AI, and it’s easy to think, oh, I’m using it wrong, but I use a similar approach. I, I think, how would a human being do this particular task, whether it’s software development or something else, and more often than not, that’s a pretty good path forward.
On the reflection side of things, when you mentioned that Dean, I realised I don’t think I’ve ever done that. I don’t think I’ve ever, when using AI to write code, asked it, “Why did you do that? What was your reasoning for doing that?” That was really eye-opening for me. I dunno whether that’s a thing that people typically do.

Dean Kerr

I think with a lot of things, I guess it is just drawn from experiences before AI. As a software developer, if a junior team member put up a PR, you’d ask these questions, in your head at first, “Okay, a solution’s being proposed to me, but what are the alternatives and why did they get ruled out”, so to speak?

So, it’s a natural tendency to reflect after you’ve done a bit of work. And I think with AI, there’s the, I guess, there’s also the tendency to just submit whatever generates immediately, and that always feels a little bit wrong to me. I like to sit and sort of study and stew on a particular solution, just for a little bit until I have the confidence and the, I wouldn’t say bravery, but to put it up for review by others.

Colin Eberhardt

I wonder if there’s almost a psychological OB obstacle here, because I don’t think AI genuinely thinks, and part of me is almost reluctant to ask it, “Why did you implement it this way? What other options did you consider? And then asking it, “What other options did you consider?” I don’t think it really considered options. But it sounds like you could ask it those questions and still get a valuable output.

Dean Kerr

Yeah, it’s, it’s treating it like I guess a fellow developer. It’s it is a Large Language Model at the end of the day, but, yeah, talking to it as you would a human can, in a weird way, the closer you bring it to your old or current ways of working, the better you can understand the output half the time.

Colin Eberhardt

And again, this is the weird thing about the tool, I think sometimes understanding that it’s a large language model, understanding the basics, like what a prompt is, context engineering, context length. I think that’s really helpful. But then, sometimes that actually understanding what it is feels counterproductive. Sometimes, you just have to suspend disbelief and pretend it’s a human being. It’s very weird.

The Future of AI Work: Autonomy, Architecture and Senior Skills

Colin Eberhardt

So, just taking a step back, this is a difficult one, but where do you think this technology’s heading in the future? Do you think we are gonna spend a lot more time effectively talking to AI and getting AI to write our code for us? It feels almost inevitable. What are your thoughts?

Dean Kerr

I think for the vast majority of use cases it’s gonna move on that sliding scale towards full autonomy, really. At the minute it’s sort of AI augmented development, but, um, I guess the step next after augmented is more inclined with autonomous. So, moving away from interjecting as much and perhaps just being more of a reviewer rather than a co-creator of software.

Colin Eberhardt

Yeah. And Amy, as an example, outside of this experiment, does that reflect your day-to-day? If you’re writing code for whatever reason, are you typically spending more time thinking about how you can encourage AI to write the correct code or create a harness or an agentic loop around it?

Is that your mindset these days?

Amy Laws

Yeah, I think it is. So, I’d probably describe myself at the minute as using AI to maybe get 90% of the way to a solution. So, kind of using it to do a lot of the boilerplate and then being able to refine it on top. But to get that boilerplate to a good quality, as you said, there are other considerations that you’ve got to put in.

So yeah, setting up feedback mechanisms in a way that the AI can usefully iterate, if that’s something that you want to do. I think I spend more time defining my specs at the start and kind of thinking really carefully about what it is I want to build, because I know that I can take the leaps a lot faster.

So, having that defined upfront is something that I’m having to do a lot more. I guess previously you would kind of build a little bit and then maybe think about what the next steps were, whereas now it feels like you’ve got to think two or three steps ahead because you don’t know how fast you’re gonna be able to jump between them.

Colin Eberhardt

Yeah, and that’s the interesting thing, is that it feels like a very different type of skill, and more often than not, we associate that particular skill with being relatively senior and experienced within software. Dean, I can imagine that, well, you described it yourself, a lot of the experience you’ve used to help you make the most of AI is the experience that you’ve gained from working in software for a while.

And Amy, I know your, your situation’s slightly different in that most of your career has been with AI. For the people who are joining now, where AI is the tool that they will immediately be given, how do they gain those skills that you think are important, and I think we all agree are important to being successful with AI?

Amy Laws

Yeah, I think that’s a really hard one, because it’s something that is quite a new challenge. I think I’ve relied a lot on the skills that I first learned when I was learning to program. Kind of debugging is becoming really important. Often, I found that AI is not great at helping debug things.

So, I think it probably is a case of challenging yourself to maybe not go completely cold turkey as we did, but maybe step back from the tools a little bit and make sure that those fundamentals are there, because I think you need both ends of the spectrum. You need to be able to get into the nitty gritty and, I dunno, debug a hard issue or resolve an issue that AI can’t, but you’ve also got to be able to do the, like other extreme, as Dean said, of working as a more senior team member and I guess almost like treating the AI like a junior developer. And I think it’s gonna be quite a hard one for people to learn both of those ends in parallel.

Colin Eberhardt

Yeah, I agree. I think there’s gonna have to be a really considered change to how people learn because it would be all too easy to give someone inexperienced an AI tool and see them suddenly be very productive and think, “Okay, there’s no need for them to learn anything.” For people who are new to software development, I think we’re gonna have to intentionally slow down and make sure they do learn some of the skills that take a little while to pick up and work out how to do that.

To your point, whether it is to a certain extent, taking the tools away. It feels like we’ve got to, to a certain extent, ignore the obvious productivity benefit and slow it down a little bit.

Amy Laws

Yeah, I guess it’s the same when you train anybody junior in anything. Like, there is always a bit of a slowdown in speed for long-term gain, and I think it’s the same kind of idea, just in a different setting.

Colin Eberhardt

It is – the problem is that in a simple sense, we’re measured on two things: speed and quality. And more often than not, speed is the thing that is more visible than quality. And I think that’s where we have to be very, very careful not to over-index on speed and forget about quality, ‘cause I think that takes you down really down the wrong path.

Amy Laws

Yeah, I definitely agree with that. I think there’s probably a bit of an ongoing question, as you said, quality and what is it gonna be like to maintain these code bases in 5, 10, 15 years’ time. It’s something that we just don’t know at the minute.

Colin Eberhardt

Yeah. So Dean, from, from your perspective where do you think it’s going and do you think perhaps some of the things that AI is less good at, at the moment it will potentially get better at, ‘cause it, it feels like our role has shifted to, to hear how Amy sort of describes it, spending more time, 90% of your time working out how to describe the problem to the AI so that it is successful, and then maybe the last 10% is a little bit old fashioned. It’s a bit more hands-on. How far do you think that’s going to progress? Do you think AI will eventually be good at architecture? What? Where else will it start to consume our day jobs?

Dean Kerr

It’s a million-dollar question, isn’t it? It’s funny, right? Because AI is extremely knowledgeable. It knows everything. It’s been trained on everything, yet it still needs a little bit of a nudge and a bit of curation when it comes to things like architecture, where a lot of the time, perhaps it’s making architectural decisions based on things it doesn’t know that you might know.
It could be a future direction of the product, for example, or even just boiling down to a simple preference for how you think a library should scale for the end users. So, I’m not sure if AI would ever get to the point where it could get those sort of, uh, characteristics because the information just wouldn’t be available to them, but I think they’d certainly get better at making a good guess at it.

Colin Eberhardt

You make an interesting point there. When you talked about architecture, you talked about scalability and uh, I think talking about libraries, you talked about the end user. You start moving into concerns that AI just doesn’t have any of the inputs. It’s, it’s good at writing code because you can give it pretty much all the inputs it needs to be successful.

But even when you get to software architecture, that’s not about looking down at the code, that’s looking up at the business problem that, that this technology solves. And whilst AI’s good at helping you sort of summarise Jira tickets and things like that. I haven’t seen any evidence yet that it’s good at discovering what a software product should actually do. I’ve not seen any evidence that it can do that yet. Which is reassuring.

Dean Kerr

Yes, for now. But yeah, that’s, I guess why you went with fuzzy information and, and, uh, preferences. I guess it may have all the access to the code base and the data associated with the code base. It still can make some interesting abstractions sometimes that might make sense to it, but the abstractions might not be human-readable or as human-readable as well, which is a, another interesting characteristic that I could see probably improving in the near future there.

Colin Eberhardt

Yeah, it does feel like the limits are going to be at or close to the point where you need to get humans involved, whatever that might be. That feels like the area where AI will stop being useful. I guess to wrap up, and I think, one of the interesting things about AI is that it’s changing the way that we approach software, but it’s also having an impact on how people feel about software.

And I know Amy, when you went cold turkey, I guess lots of the things that you mentioned were, were generally quite negative, and some of them were just quite funny about how frustrating it was.

Joy, Craft, and What AI Changes About Coding

Colin Eberhardt

But one of the things that you mentioned that I thought was really notable was that you had a sense of pride, or a bit more joy in actually handcrafting the solution.

It’d be great if you could speak more about that. Was it a bit more fun? I know reading issues was drudgery, but the code part. How did that feel?

Amy Laws

Yeah. So, one of the things that first attracted me, I guess, to coding was problem solving, and that kind of high you get when you’ve had a really hard challenge, and you spend quite a long time kind of in the code, and you finally get that test passing or get that feature working. And I think I forgot that that feeling is actually really great.

And when you’ve used AI to write your code, I guess I feel less ownership of it, even though it is still kind of my work, because I’ve not kind of handcrafted and carefully gone through like every line of code. That high just wasn’t quite the same, and that was something that I didn’t realise I missed.

So, a lot of the bugs that we had in this project were often quite hard to pin down, and it was a real challenge to kind of work out what was causing the issue. I think a lot of the time, when you worked it out, the solution was quite simple. But yeah, that really great feeling when you’ve finally fixed a really hard problem is something that first attracted me to this job, and I didn’t realise that I’d kind of missed.

And as you said, like knowing that you have built something or you have solved the problem is something that I guess is going away a little bit as we’re kind of more hands-off with the code.

Colin Eberhardt

Yeah, it’s a really interesting sort of connection we have with the code that we write. I think a suitable analogy for people who have not written software is potentially carpentry. You can imagine that there’s a great difference between, I don’t know, building a chair or something like that and crafting it with your own hands, versus if you just programmed it into a computer and a CNC machine built it. I think that the end result is exactly the same, and it could be exactly the same human being that has achieved that end result.

But I think a lot of people would understand and relate to there being a greater sense of achievement through actually crafting it with a hand chisel than programming a machine to do it.

Amy Laws

Yeah, I think that’s exactly right. And I think as software developers we do have a pride in the things we’ve built, and actually seeing people use them or knowing that people are using them is a really great feeling, and it’s something that maybe is gonna change a little bit in the future.

Colin Eberhardt

Yeah, I agree. I mean, Dean, you’ve been doing software for a long time, and I’m assuming that a lot of the time you’ve been working on software, you’ve experienced that joy of creating an elegant solution or refactoring some code just because you felt it looked better, and that gave you joy.

What do you think about this shift in software engineering?

Dean Kerr

Yeah, it’s interesting really ‘cause as you say, it’s, I guess, I’ve been in the business of the industry for over a decade. And I think things have moved slowly between that decade, I guess, as I gradually got more senior, moving slowly and more away from the code and getting my problem-solving thrills, maybe at a higher level than low-level code. But, I still got immense satisfaction out of fixing a bug that might have taken a long time or took a lot of analysis to get to the bottom of, or like you say, spending a good amount of time with maybe an initial solution that I refactored into a quite elegant one.

So, yeah, there’s still the satisfaction there for that, but there’s equally, you know, you get satisfaction in building applications and products as well. Um, I think I’ve gradually shifted to be sort of in the middle, from doing purely development work to being, you know, a product owner only.

So, I think that’s a nice balance to have. But, I think, AI if it keeps progressing at this rate, I think you’re gonna have less of that sort of lower-level working satisfaction and more of getting satisfaction and morale from solving higher-level or maybe even product-oriented problems going forward.

Colin Eberhardt

I guess taking the conversation full circle and coming back to The Experiment, do you feel any more satisfaction from The Experiment knowing that your pull request eventually got merged and the, and the software that was shipped?

Dean Kerr

I’d say that satisfaction is equal, whether I was handwriting every line or I reviewed it and it was effectively mine, I guess, to the same extent anyway. There’s still that responsibility and ownership of a pull request, regardless of whether it was AI that generated the code for you in one prompt and it “one-shot” it, versus you handcrafting it over a week.

Colin Eberhardt

Yeah, so I guess it’s time to put the ultimate question to each of you and see whether you are, you are happy to answer yes, no, or sit on the fence and go with a maybe. So Amy, do you think AI has taken the fun out of software development?

Amy Laws

I think I’m gonna have to go “maybe” on this one. I think I’ve definitely lost some of the areas that I used to have a lot of joy in, but as Dean said, you kind of find them in other areas, if that makes sense. So actually, I find AI absolutely fascinating and if I can get it to solve a really hard problem for me.

That brings me joy in the same way that fixing a really hard bug would. So yes, sorry to avoid the question, but I think it’s just shifted where we find it.

Colin Eberhardt

Oh, on the fence then. Come on, Dean, can we get you to come off the fence, or are you gonna do the same?

Dean Kerr

Well, you know, rather than no, which is a bit of an absolute, I could say not yet if that’s a good, yeah, second one, anyway. I think for me it’s been really interesting to see all the knowledge I’ve built up in industry so far as a developer, how quickly I thought it’s almost becoming obsolete.

I thought, you know, you’d learn a couple of languages, and maybe one or two of those languages would, you know, fall out of fashion, go out of date and not be as relevant any more. But for the whole development bit itself to potentially be obsolete is a bit sad. But, at the same time, I think I still get the same satisfaction, like you say, to getting a pull request merged or a particular tricky feature deployed, and seeing it used by users. So, I’ve put not yet rather than, no, because, uh, like I said, AI’s changing from week to week. So, it might be no at the minute, uh, but it could, uh, yes later on as it progresses.

Colin Eberhardt

Yeah, I totally get what you mean about this; it does genuinely feel like we’ve reached quite a surprising point. I mean, I think a lot of us could see that this was going to be a big thing, you know, as far back as a couple of years ago, but I didn’t think we would be reaching this particular point. So I guess it’s, it’s my turn and I, I’m gonna fall off the fence. I’m gonna say no, AI is not taking the fun outta software development, but I’m gonna come in with a caveat, that I think the place that you find fun has moved completely. It’s not where it used to be. And I think that’s the sort of critical thing.

And I sort of almost discovered this myself from looking at some of the hobby projects, ‘cause you know, I don’t write much code in the day job, but I still love writing code and building things. And I looked back, and four or five years ago, a lot of the stuff I was doing was with WebAssembly, and I was writing an emulator for an Atari 2600.

And the funny thing is. I never played a game on it. That wasn’t the intention. I, I just wanted to build the thing, and that was it. Whereas now, the hobby projects I work on tend to be ones where it’s a thing I actually want to use, and the apps themselves are really boring. You know, Crud style, form-based applications where the code itself would bore me stupid.

But, I’m having more fun actually building things that I actually use. So, for me personally, I still find it fun, but the things that I find fun have changed completely in the space of five years. Oh, I get the final say. Cool. Well, ‘cause I’m the one that, that came off the fence.

Episode outro

Colin Eberhardt

And that brings this episode to a close. While AI hasn’t necessarily taken the fun out of software development, the change in our role and focus means that this fun is something we have to find somewhere else, and it may not be derived directly from the act of writing code. This is going to be an uncomfortable realisation for some people.

If you’ve enjoyed this episode, we’d really appreciate it if you could rate and review Beyond the Hype. It’ll help other people tune out the noise and find something of value, just like our podcast aims to do. If you want to tap into more of our thinking on technology and design, head over to our blog at scottlogic.com forward slash blog.

So only remains for me to thank Amy and Dean for taking part, and you for listening. Join us again the next time as we go Beyond the Hype.



If AI Writes the Code, Who Builds the Next Open Source Project? by Colin Eberhardt
Colin Eberhardt — Sun, 15 Mar 2026 20:12:00 +0000
Open source has long been driven by human frustration, curiosity and craftsmanship, creating better tools because existing ones didn’t quite feel right. But as AI agents increasingly write code for us, that dynamic may be changing.

In this post I look at how AIs impact on open source; from the legal debates over training data, the uncertain copyright status of AI-generated code, the possibility of AI-driven project cloning, and the growing strain on maintainers. Finally, I consider what happens to open source when the people who create it start to feel less of the friction that once inspired it.

Is AI training on open source ‘fair use’? Or a copyright violation?

When LLMs burst into our personal and professional lives in 2021-22, with the launch of ChatGPT and GitHub Copilot, they were met with a mixture of wonder and concern. Their amazing capability was only made possible by a long process of deep learning on a vast corpus of text and code.

Copilot was trained on public GitHub repositories, which contain a wide range of licences ranging from the more permissive (e.g. MIT, Apache 2.0) to more restrictive (e.g. GPL). The model labs that scooped all of this data up argued that this constitutes ‘fair use’, a transformative work, rather than a direct reproduction of copyright materials. However, many open source developers viewed this differently, especially when Copilot would often emit large sections of copyright code – including the copyright notice itself!

Some maintainers took action (beyond complaining on Reddit!); the Free Software Foundation funded a call for white papers to analyse the issue, but there has been little progress on this since. There is also an ongoing class-action lawsuit, Doe v. GitHub, Inc. which contains various claims including open source licence violation and removal of copyright information. However, many of these claims have been dismissed.

In May last year the US copyright office published a pre-publication draft on Generative AI. They made it clear that training cannot automatically be considered fair use, and specifically called out the need for lawful access (i.e. don’t train on data that has been acquired illegally). However, beyond that, it basically said “it is complicated”, leaving it to courts to apply judgement on a case-by-case basis.

I don’t think any future court rulings or changes in law are going to have a meaningful impact here. Large-scale training on ‘public’ datasets, which in the case of open source is any code found on the internet (regardless of licence) looks like it is here to stay. Unfortunately, the only way to prevent your code from being used for AI training is to not make it public.

Can you copyright AI-authored code?

Fast forward to 2026, we’re now in an era where significant amounts of code is AI-authored, which leads to a completely different legal question; can AI-generated code be copyrighted?

In the U.S. copyright law protects “original works of authorship.” where authorship is considered to be the work of a human. The courts have recently declined to consider whether AI-generated art can be copyright, doubling-down on the need for direct human authorship. And recent guidance states:


  “Material generated wholly by artificial intelligence is not eligible for copyright protection.”


This does of course leave a lot to interpretation. In the field of software engineering, what does “generated wholly by artificial intelligence” mean? If you follow a specification driven development workflow, where you don’t touch any of the generated code, presumably you cannot copyright the code. But what if you review and edit the output? Can the code them be copyright?

Unfortunately, without being able to claim copyright, you cannot employ any of the standard open source licences to your work.

I would seem that the copyright system is struggling to keep up with AI; and open source which, is deeply dependent on this legal system, will probably suffer.

AI cloning of open source code – a moral dilemma

Recently we have seen a significant improvement in the ability of LLMs to write code. This was reflected in the benchmarks, for example state-of-the-art performance on SWE-Bench leapt from about 10% at the beginning of 2024, to around 80% at the end of 2025. As well as raw model performance, the concept of agentic loops – where a model iterates on a problem, supported through the use of tools – became mainstream in 2025, allowing agents to take on much more complex problems.

With agentic loops, the feedback mechanism has a significant impact on the model’s ability to be able to autonomously pursue a goal. Comprehensive test suites, or a reference implementation, are fantastic feedback loops, that have allowed agents to do some truly impressive things. For example:


  
    Last month, the anthropic team built a C compiler, achieving a 99% benchmark score, in just a few weeks, relying heavily on existing test suites to guide the agent.
  
  
    A few weeks ago, the Cloudflare team created a clone of Next.js that more readily runs on their cloud platform, again, using the existing test suite to steer the agent.
  
  
    And last week, the chardet project used an AI agent to re-implement one of their dependencies, in order to free them of the GPL licence restrictions. The author of this open source, LGPL licenced dependency was not happy.
  


Whether you can re-write software is a familiar legal issue, with much of the debate around the chardet re-write focussing on whether this can be considered a “clean room” implementation. This term originated in the 1980s, describing a legally robust technique for replicating software by having one team document an existing system, then an entirely separate team re-implement based on the specification. Can an AI re-write be considered “clean room” given that the original sourcecode is almost certainly in its training dataset?

Unfortunately, as with the fair use and copyright issues described above, it is going to take a long time for the legal system to catch up with the capability of AI agents. The concepts of “fair use” and “clean room” were created in a time when this type of AI capability was entirely inconceivable.

As well as a legal issue, this is also a moral issue. Is it fair to copy someone else’s work? How do maintainers feel about this? Especially given that the better the quality of your tests and documentation, the more exposed you are to being cloned.

AI contributions and bots

A growing number of open source contributors are now using AI tooling to assist them, which is of course entirely expected and not necessarily a bad thing. However, while AI can be used to create quality contributions, by taking care and reviewing the output. It can also be used to very rapidly create poor quality contributions, or ‘AI slop’.

GitHub have acknowledged that there has been an increase in low-quality contributions and are looking for community feedback on various ways to tackle it.

Maintainers have mixed views on accepting AI-authored, or AI-assisted contributions. There have been a few high-profile cases of open source projects outright banning AI contributions, however, a recent review of 112 open source projects show that most are quite welcoming. Only four of the projects ban AI contributions, and 71 have merged commits that mention being written with AI assistance. Although if the slop problem increases, I wonder whether more projects will ban the use of AI? Also, a recent study claims that while 95% of contributions were fully or partially AI-generated, only 29% of contributors explicitly disclose their use of AI.

So far, most of these AI-assisted contributions have been from human beings, and even in the case where they are wholly AI-generated, a human being has been ‘in the loop’ to a certain extent. This is changing; with the sudden rise in popularity of OpenClaw, we are seeing many people experiment with fully autonomous agents, and it is no great surprise that some of these ‘claws’ are now being let loose on GitHub. I’ve already crossed paths with a bot that commented on an issue I was participating in, and its comment added little value.

If you’ve read Nadia Eghbal book “Working In Public”, which explores the open source community, you’ll know that the scarcest resource is a maintainer’s attention, and it is a resource that doesn’t scale terribly well. Poor quality contributions steal attention. We’ve seen the damaging effects of this in the past, with some calling Hacktoberfest (which encourages people to contribute to open source to win T-Shirts) a distributed denial of service attack.

I fear there will be a growing amount of noise due to fully autonomous agents ‘helpfully’ working in open source, creating further stress on maintainers.

Do AI agents need open source?

So far, we’ve seen that AI models were trained on open source, via a questionable interpretation of far use, that open source projects can now be cloned with relative ease, and that there will likely be a significant rise on automated slop contributions. The impact of AI on open source doesn’t look all that great. However, I think there will likely be a more significant long-term impact caused by the widespread adoption of AI agents for coding.

Where does open source come from in the first place?

Evan You, the creator of Vue.js, wanted something that fit the way he thinks about building user interfaces. Rich Harris, of Svelte fame, wanted a framework that disappeared. Ryan Dahl, of Node fame, created Deno, to address the mistakes he’d made with Node. These are just a few examples, there are a great many more, but an underlying theme here is that maintainers create projects to address friction they have experienced themselves. They had an unpleasant experience with a framework, platform or library, and decided to create something with a better user experience.

As we increasingly lean on agentic AI, we no longer directly feel the pain of a poorly designed library. AI agents, with their ability to write code many 100s of times faster than us, just bash through it. Furthermore, AI has no taste. It doesn’t smell code smells or appreciate the elegance of a well-designed abstraction.

If we no longer experience the pain (or the pleasure), what will motivate us to create new open source projects?

Final Thoughts

I know this blog post seems all rather negative, I certainly don’t mean it to be. I am very excited about the potential of agentic AI and use it myself extensively. However, I don’t think we should be blind to the issues this is causing for open source.

I don’t expect open source  to suddenly disappear or die, in much the same way that I don’t think software engineers are going to cease to exist. You’ll not find me writing a clickbait “Open source is dead, long live open source” blog post! But I do think that open source is going to experience a period of rapid and uncomfortable change.



Beyond the Hype: Vibe coding – Is this really how we’ll build software? by Colin Eberhardt
Colin Eberhardt — Tue, 10 Mar 2026 01:01:00 +0000


In this episode of Beyond the Hype, I’m joined by Remi Van Goethem to unpack the fast‑evolving world of AI‑accelerated software development. From everyday autocompletion to emerging multi‑agent frameworks, we explore how AI is reshaping coding practice and where human engineering judgement still matters.

Remi shares his recent experience rapidly prototyping a planning application using a Research–Plan–Implement workflow, highlighting how AI can transform early‑stage discovery, architecture thinking, and delivery speed. Together, we consider what this shift means for developers, architects, and CTOs as AI starts to generate more of our code, and whether vibe coding is a glimpse of software’s future or simply the latest trend.

Useful links for this episode


  
    Will the future of software development run on vibes? – Ars Technica
  
  
    Welcome to Gas Town – Stephen Yegge
  
  
    I miss thinking hard – Juan Ernesto
  
  
    Agentic Coding Makes Old Coders Young and Young Coders Old – Scott Francis
  
  
    AI-Assisted Development at Block – Angie Jones
  
  
    Research → Plan → Implement Pattern – Goose
  


Subscribe to the podcast


  
    Apple Podcasts
  
  
    Spotify
  


Transcript

Please note: this transcript is provided for convenience and may include minor inaccuracies and typographical or grammatical errors.

Colin Eberhardt

Welcome to Beyond the Hype, a monthly podcast from the Scott Logic team where we cast a practical eye over what’s new and exciting in software development. Everything from Kafka to Kubernetes, AI to APIs, microservices to micro front ends. We look beyond the promises, the buzz and excitement to guide you towards the genuine value.

I am Scott Logic’s CTO, Colin Eberhardt, and each month in this podcast, I bring together friends, colleagues, and experts for a demystifying discussion that aims to take you beyond the hype. In this episode, I’m joined by Remi to discuss his recent experiences of AI-accelerated software development. We discuss the broad spectrum of AI use from simple autocompletion to multi-agentic frameworks.

And more concretely, we discuss a recent project where Remi rapidly prototyped a planning application using a technique he calls Research, Plan, Implement. Finally, we ponder what this means for the industry and people within it. What it means for the developers, the architects, and the CTOs, as AI starts to write more and more of our code, and we ask whether vibe coding really is the future. We pick up the conversation with our thoughts on the latest developments in AI.

The thing that we really want to talk about today is vibe coding. And I guess from the hype perspective, is that the direction of travel of the industry? Is that how we’re going to be doing software, going forwards? But before we get to that, vibe coding is a subset of what we do with AI and how we use it for coding. So, I was wondering from your perspective, are you keeping up with the latest developments? The last 12 months of AI have seen lots of model releases, lots of different tools and technologies. What are your thoughts on that? Are you, are you keeping up any particular highlights from your perspective?

Remi Van Goethem

I mean, keeping up is impossible. I don’t think it’s possible to keep up with all the different model releases, all the tooling, all the techniques. But yeah, I keep a, a high-level view of what’s going on in the industry. And yeah, I think we are, if we talk in general about models, I guess in the last 12 months, I think we’ve reached a level where models are becoming extremely good at helping us.

For example, I’m a Claude user. I’m using a lot of their models. So mostly Opus, at the moment. But I cannot tell the difference between Opus 4.1 and Opus 4.6. I know that the metric tells me otherwise, but in general, they all work extremely well and, for the type of work I’m doing, I think I’ll be happy with any of these these days.

Colin Eberhardt

Yeah, it does feel like, at some point, maybe six months ago, it got to the point where the leading models were all amazing, awesome, fantastic. And there was no real difference between them. They were all just amazing. I mean, did you get that feeling as well?

Remi Van Goethem

Yes. I think it was subtle to start with, and it accelerated a little bit, and as you said, I think it’s probably the last six months where they’re all super good. So, I remember like a year ago, everyone was talking about prompt engineering. It was super important to be good at writing your prompts super well; there are all sorts of techniques like the COSTAR and stuff like that. And I feel these days you can get lazy because the models are so good at filling the gap.

Colin Eberhardt

Yeah, it’s strange, isn’t it? Because it wasn’t that they hit a certain percentage on a given benchmark. It was just that the capability reached some hard-to-define point, and almost it was, it’s almost like AGI, in that it’s not a terribly well-defined concept, but it will happen when enough people believe in it.

It feels like we’ve got to that collective belief with AI for software development.

Remi Van Goethem

But to be honest, I think they show that the models have plateaued a little bit. So, it doesn’t scale any more to reach AGI in 2026 or 2027 without a breaking technique. So, I think all the different models have been, people training models have been, a bit more clever on how they do it.

Colin Eberhardt

Yeah, I was gonna say, getting back to the main topic of this discussion, there’s a relationship between vibe coding and this tipping point that we’ve reached in the technology. So it’s interesting; vibe coding as a concept, or at least the term is only 1-year-old, which I guess in, in AI probably actually makes it feel quite old.

So, the general concept of vibe coding is that you become completely detached from the code. You tell the model what you would like it to do. You observe the output. If it doesn’t do what you want it to do, you simply tell the AI, make this change. Make this fix or add this feature.

When vibe coding first came out, was it something that you became aware of, or is it something you’ve only become aware of fairly recently? Did you dismiss it as that’s a hipster, trendy thing, or did it make sense to you when you first heard it?

Remi Van Goethem

I think I was not an early adopter of it. As an architect, I don’t code on a daily basis, I guess. So I  didn’t try that, as an early adopter and probably at the time the model didn’t have enough capability to be able to have multi-step reasoning to be able to reach the goal that you wanted to. They would forget their goal halfway through the exercise, I guess.

So, yeah. I, I think this is something that became much more possible, the last six months, I would say.

Colin Eberhardt

Yeah. Because even though, when Andrej Karpathy coined the term vibe coding, within the tweet or the post itself, there were notes of caution around how effective it could actually be, but I think, six months later, the model’s become so much more powerful that his original vision of vibe coding became much more real.

Another thing we talked about was the adoption spectrum. So, at the zero level of adoption, you’re basically writing code in exactly the same way that you’ve always written code. The next level of adoption, for example, is pair programming through GitHub Copilot. And then, there are various people who have published adoption spectrums, but I think the really interesting one, which was probably quite an extreme version of it, was the one published by the Gas Town creator. So that has a sliding scale that gets to the point where the top level is multi-agent, you never look at the code, your speed is the highest priority. I mean, how well does that fit your mental model of using these tools, and where would you sit on this spectrum?

Remi Van Goethem

I think it’d be good to speak for just a minute about this scale. It’s quite neat, this scale from  Steve Yegge. So, he gave you eight different stages of adoption. And you can cut that into three different phases, like the one phase, you are still the pilot.
You fly, but you have a copilot with you. At stage two, you become a director, you’re directing a fleet of agents. And at stage three, you are becoming like a factory owner, basically. And, if you think like stage 1, 2, 3, you’re still using an Integrated Development Environment (IDE) because you care about the code. At some points, you give it all permission to do the modification in the code directly, but you’re still reviewing the code, and you’re still accepting the code.

When you become a director at stage four and five, what’s interesting is that the difference between stage three and stage four is that you no longer look at the code. So, if you think of your IDE where you’ve got the agent chat on the right side, on the sidebar, now you’ve got the code on the sidebar, and you’ve got your agent chat in the main window. At stage five, you’re supposed to no longer care about the IDE any more. You’re straight away talking to the agent. It does the modification. You are not looking at the code anyway; you trust the output. The factory floor sounds a bit futuristic, but some people are already there.

It’s where you would start to use multiple agents at the same time. I don’t know how people do that, how they can multitask to have multiple features in flight at the same time, but that’s what stage six is. Stage seven is when you’ve got more than a dozen agents at the same time. That sounds scary to me. And stage eight is when you reach a scale so big that you need to build your own workflow, basically, like Gas Town.

Colin Eberhardt

Yeah, well, what I like about this scale, even if I don’t necessarily agree with the specific steps on the scale, I think what it does very well is show that this is a spectrum and when vibe coding was first coined as a term, it gives the impression that you are either vibe coding or you are not vibe coding.

You are doing things the new modern way or the old-fashioned, handcrafted writing-the-code way. And in reality, it is a spectrum that there are levels of vibe coding, and Steve Yegge’s scale goes, I’d say, almost beyond vibe coding. And it is a sliding scale. So, with that in mind, I mean, do you expect all software development to start pushing further and further up this scale of automation?

Remi Van Goethem

To me, it sounds like it seems to be a natural way at the moment. It’s where the industry is heading. Some companies have started moving up in the scale. We’ve seen some companies like Square publishing how I think most developers are at stage five, and the second cohort is at stage six, so they’re pushing towards the right side, where they manage a fleet of agents, basically.

Colin Eberhardt

At the moment, though, do you think there are obstacles to moving to the higher stages? So what I get uncomfortable about is when, using his term, the diffs scroll by, you simply don’t look at the code. It feels like with the technology that we have at the moment.

You are still going to hit the same obstacles. And you mentioned this yourself when you talked about vibe coding, when it first came out, your feeling was that it worked for a limited period of time and then fell apart relatively quickly. Now I think you can vibe code, ignore the code entirely for a slightly longer period of time.

But I think you still get to the point where it’s going to be a mess. Is that your experience? Is that your thinking?

Remi Van Goethem

Yes. I think it is the case if you don’t have a workflow, I guess, or if you don’t have some guardrails, and even with this, it’s not guaranteed that you get the result you want. We are working with models which are not deterministic anyway, so all the guardrails that you’re gonna put in place, it might ignore them anyway.

Colin Eberhardt

So, you need some engineering approach. You can’t just blindly direct the AI tool and expect it to create a good architecture or fault-free code.

Remi Van Goethem

Yes. I think when we value code, when the code becomes cheap, what’s important is not the code, it’s the engineering process around it. And that’s not cheap.

Colin Eberhardt

Yeah, very true. I mean, as an example, if you vibe code an application, your tool’s not going to ask you, should we write some tests for this? Should we validate it? It will just build it. You have to exercise your human judgment there.

So, when it comes to how you approach this? So, we’ve talked about Steve Yegge’s scale, and he created a tool called Gas Town, which is a relatively opinionated way to manage a fleet of agents to enable more vibe coding style approaches, more hands-off approaches, and there are other approaches like spec-driven development and so on.

How do you approach this if you, if you are vibe coding an application? How do you make sure that it does what you want it to do? That it still has enough of an architecture to be able to scale with new requirements and so on.

Remi Van Goethem

I guess it depends on what you’re trying to achieve. Is it something that needs to go into production? Is it like a prototype or is it likea tool for your personal usage? There are all different needs for all of these different use cases. I code the same way as I work my day-to-day work.

I like to do some upfront thinking, which means that you probably wouldn’t call that strictly vibe coding, I would say. Because, when you vibe code and you give full rein to the model, to the agent to fill the gap and do everything underneath, probably – and I guess if you are a bit opinionated on how you want to build certain interfaces or which quality you want to have on the application, what are some of the constraints – you can get results closer to what you wish, basically.

Colin Eberhardt

Yeah, that’s a good point: the success or failure of taking a vibe coding approach can depend on how much thought and consideration you’ve put in ahead of actually typing in your first prompt to the AI tool. You’re right, it can, it can feel very easy to just tell it to build my whole application, but from an engineering and an architectural perspective, we both know you’ll probably miss the target by quite a long way and have to work hard to bring it back to where you actually want it.

Taking a concrete example, I know that recently you were working on a project where it was a tool that demonstrated a housing planning application process. And one of the goals was to demonstrate innovative new approaches to the planning process. So planning processes, like many, many processes, are pretty mundane. There’s lots of form filling, there are multiple steps, and there’s loads of room for potential innovation through the use of AI or just better software.  I know you managed to effectively vibe code a really cool demo in the space of a couple of weeks. How did you do that, and how does that illustrate your approach to vibe coding?

Remi Van Goethem

Yeah. I think that comes back to – how you approach vibe coding is – which technique do you use? There are multiple techniques. You could use certain techniques like, spec-driven development; we’ve got like Spec Kit, we can use BMAD, for example. These techniques are very good and have their strong points, but I like an approach called Research, Plan and Implement, which is the RPI method.

And this is, anyway, how you tend to work as an engineer. You always do some research before you plan and implement. And so I think it feels very natural. The RPI approach says that you need to do good research, you need to understand the domain, then you need to plan the work before it gets implemented.

And some content that I found was really nice about it. It says one line of bad research creates a thousand lines of bad code, one line of bad planning creates hundreds of lines of bad code. So yeah, where you should put your focus is on the thinking upfront, I guess.

Colin Eberhardt

So, just out of interest, with the Research, Plan, Implement approach. Is that an approach that you read about and thought, “That makes sense – I’ll give that a go”? Or is that something that you just naturally found yourself doing because that’s the way that you work, regardless of the use of AI?

Remi Van Goethem

No, I actually didn’t know this thing existed at the time. It’s just, it feels natural. So in my mind, you need to understand the domain. So for that, you’re probably gonna be creating certain artefacts to capture the learning you’ve done, and you’re gonna distil that into other artefacts.

That you can call a plan, for example, taking certain information. You distil that into something a bit clearer and more targeted to what you’re trying to do, and then you use these different artefacts at different times of the process to create the outcome you want, basically.

Colin Eberhardt

So, talking through this example, then, the research phase. So, this was a tool; you had two weeks to create a demo for a planning application. Have you worked on planning software before?

Remi Van Goethem

The only thing I knew about planning was my neighbour, who was trying to do some modifications. So I received a letter from the planning officer to review. So, I only knew the workflow from the neighbour workflow of the planning application, but I knew nothing about the domain.

Colin Eberhardt

So, how did you do your research for this? How did you learn more about the domain to be able to create that domain understanding ahead of building something?

Remi Van Goethem

So, we had meetings with some experts at the beginning, so they just gave us some very high-level things: Who are the actors, what are the systems, and who are the Local Authority? That gave me a very high-level black box, but I didn’t know how it worked inside. But they also pointed us at some existing open source encoding parts of the domain. So, what I’ve done is, checked out the code, I crawled the code, and I started doing some domain exploration by running the domain, basically.

Colin Eberhardt

And, the domain exploration – were you reading the code yourself, or were you leaning on AI to help with that?

Remi Van Goethem

The code was in Ruby on Rails. So, I’m not a Rail developer. I’ve never read that. But with the help of AI, I could get my way around it. So, I was able to run the application, and to seed the database to get the right data I needed to run the application.

So, if you think in terms of planning, you need to create some constraints, you need to have some data inside the product, in order to be able to run the workflow. But yeah, with the help of AI, I was navigating as if I knew what I was doing, basically.

Colin Eberhardt

Okay. So, I’m assuming then that AI was also a vital tool in this research phase, because I can imagine if you don’t understand the planning domain, you are lucky in this sense that there were some open source projects that were quite similar to what it is you were wanting to build.

But even then, understanding a code base and a language that you are not familiar with, with a domain that you’re not familiar with, I’m assuming previously that would’ve been a few weeks’ worth of work to start understanding and analysing the code base.

Remi Van Goethem

It would’ve been several weeks of work. I can give you a flavour of things I’ve done, for example. In order to, to proceed through the workflow, I had to understand how the state machine worked because it needed to have certain events that move forward in order for me to progress in the workflow.

So, I’ve managed to extract the state machine using AI, doing some analysis. That became one of the artefacts that I used later, which helped me build up my understanding. Then I needed to move this state machine so I could zap the database directly using AI again.

Again, when I needed some realistic data, the model could generate it. If I needed to seed the database, the model could generate some realistic data sets. And yeah, that allowed me to basically understand the whole workflow. And to be honest, I’ve done a bit more than understand the workflow. We’ve been taking screenshots of all the applications as we were progressing, because we were making a lot of assumptions at the time, and we needed to get them verified with experts, basically, of the domain.

So, while we were building our understanding by mapping the workflow, we were as well annotating each of the screens to see where we could make improvements. And then we had to ask the expert, obviously, to resolve our assumptions.

Colin Eberhardt

Yeah, and this is a particular application of AI within software development that I don’t think gets enough attention. So, you are able to rapidly research a code base and a domain with the intention of then creating a demo, which you used a vibe coding approach for. But this particular problem, understanding a code base and a potentially unfamiliar domain, happens all the time in software.

One of the main reasons why we struggle with legacy systems is the lack of understanding of that software system within the organisation itself. It feels like the approach you were taking here, whilst you were using it for a vibe-coded outcome, this feels important.

Remi Van Goethem

Yeah. I thought it was extremely proficient to be able to explore an API by using it, because sometimes there’s a gap between the API documentation and how it behaves, and I’ve experienced that. During my research, I was able to run both applications, I was able to interact with them, I was able to ask questions about the code base, and I was able to extract, and there was some information; for example, I could understand what the as-is workflow is for a planning officer and use that to plan the to-be version of it.

Colin Eberhardt

And interestingly, in my opinion, when you’re using AI for research purposes, it doesn’t have to be perfect. Your tolerance for error is higher than your tolerance for error when you’re using it to write production code. So, it feels like it can be even more powerful. You can give it even more latitude when you’re using it for research tasks.

Remi Van Goethem

Yes, you, you’re probably right because all that you’re trying to do is reduce assumptions. So, it is not a right or wrong approach. You’re trending towards the truth.

Colin Eberhardt

So, from collecting together some existing code bases, APIs running the application itself, through the use of AI, you were able to rapidly gain a good understanding of the domain and the planning processes. What was your next step? Did you then just vibe code a new version of the application, or was there something that happened in between?

Remi Van Goethem

Now there is definitely an in-between stage. So, you have the research, you have the plan before the implementation. And here, before the planning, I needed to get my assumptions resolved. So we needed to meet with the experts. And the best way for us to resolve our assumptions was to come up with this workflow with screenshots, annotated with assumptions that we put on the whiteboard and went through with the client. That was really helpful for them to be able to quickly annotate, tell us what was right, what was wrong, and where we were incorrectly making assumptions.

But by the end of it, we could take this big whiteboard workflow and turn that into a sequence diagram because these days, models are multi-modal. So, you can transform the artefact into something useful. And why is it useful? It’s because if I got the as-is flow, I can now use it to transform it into a to-be flow. I just have to inject an extra couple of constraints and explain my goals, and I can derive that and produce another artefact, for example. I can tell it how I want it to behave as a sequence before I feed that to a model to vibe code, basically.

Colin Eberhardt

Yeah, I just want to step back to something you mentioned very briefly that I still feel is quite important. You mentioned that you created the wire frames and then you got some expert input, and that still feels like it’s a really important thing to do because vibe coding allows you to move very, very quickly, but you can still very quickly build something that’s the wrong thing, or maybe 90% of the way there. And that final 10% is incredibly important. So, whilst you had very rapidly learned the domain, you weren’t an expert at the end of like three days of AI research.

Remi Van Goethem

Yeah, that’s correct, and I think that’s, that’s the thing with all this vibe coding trend – the only way for vibe code to be very fast is if you are the expert yourself of the domain, if you’re the product manager, if you are the architect, if you’re all these people at the same time. Otherwise, you need to get the input from the experts.

Colin Eberhardt

That’s an interesting point, and it doesn’t mean that you are all of a sudden a lot slower, but you do have to be very intentional about finding the right time to get the feedback and getting the right level of feedback. Yeah, that’s really interesting. So you’ve gone through the planning process, and you are now onto implementation.

What did that look like, and how long did that take?

Remi Van Goethem

The implementation phase was the most fun part, I would say. It is more fun for me because I tend not to code very often any more in my day-to-day work. And I could build some cool things very quickly. So, this phase, I’ve had my research, which I’ve distilled into several artefacts, super-high-level sometimes, like design principles, high-level constraints, very high-level things that I can invoke at certain times of the process, to say, okay, be careful of this, and, like that, I could guide the code towards where I want to be, basically.

Colin Eberhardt

How did you know that having a design principles document was the right way to do it? Was that just a guess? Did it just feel like the way to approach this problem?

Remi Van Goethem

I deal with models the same way I deal with colleagues. I don’t come up with a full design where everything is wrapped up, and people just have to code. This is not how it works. You have to provide some constraints, some guardrails, some direction, some opinionated choice, and then you let the engineer fill the gap.

And I feel this is the sweet spot with models; you can just guide them on what you think is the right level of detail and let them fill the gap.

Colin Eberhardt

That’s really interesting because that reflects a more negative experience that I had using Spec Kit, which is quite an opinionated approach to specification-driven development. I dunno if you’ve used it yourself, but it has a very, I mean, to a certain extent it reflects the Research, Plan, Implement, but it does it in a very detailed, very opinionated fashion.

And to your point about the design, it creates Markdown files with ASCII-art-style images of every single screen, and the plan stage has code snippets in it. It tries to steer the model with a very high level of specificity. And to your point about working with human beings, it’s the level of detail you would never give to an engineer.

Remi Van Goethem

Yes, I agree with you. I’ve looked at Spec Kit, and I was not really convinced, I would say, because it feels extremely heavy, it feels like Waterfall, really. You get a real level of detail super early, and the Spec Kit approach is supposed to be that you should never throw away your spec, it’s as important as the code.

But that means you’re supposed to review the spec and review the code, and with the models, they don’t always respect it. It feels like too much effort.

Colin Eberhardt

It does, and something else you mentioned that you have experienced in the past 12 months, which I’ve experienced as well, is that 12 months ago, you had to be a bit more careful about your prompting. You had to engineer it a bit more, whereas now, it feels like they just understand you so much better.

It feels like we have to engineer our prompts less. We are just steering them at a high level, and Spec Kit feels like it’s pushing in the wrong direction. It’s not capitalising on the gains that we’ve had in the past six months.

Remi Van Goethem

Yeah, I think RPI is probably the middle ground here.

Colin Eberhardt

Yes, and it feels like with RPI, you can describe it in just those three words. You don’t have to research a specific way. You don’t have to plan in a specific way. You don’t have to implement in a specific way. Personally, I like patterns that you can describe in a few words or a sentence, because I think that gives you the freedom and the flexibility that you need to adopt it efficiently in your given context. Anyway, getting back to your actual vibe coding, you said it was really fun. I mean, why? Why was it so fun?

Remi Van Goethem

I think it’s this prompt-to-outcome loop. I think it’s high dopamine, probably. You get a reward very quickly and you see the application being built, step by step. I said step by step, because I didn’t build the whole thing in one go.

Colin Eberhardt

And when you were doing this, this wasn’t a fleet of multi-agents, this was a single agent. Was it Claude Code or GitHub Copilot, or…?

Remi Van Goethem

It was Copilot at the time.

Colin Eberhardt

Yeah. So, given how quickly you moved, what do you think of the whole multi-agent approach? Personally, I feel that I find it hard to keep up with a single agent, and steer it either by planning or steer it by correcting after it’s produced it. I find that hard work. I mean, I’m impressed that some people are going down the multi-agent route, but that just feels too fast for me.

Remi Van Goethem

It depends on what you mean by task. If you’re talking about feature level, probably the bottleneck is gonna be the planning. If you talk about tweaking the CSS or fixing this bug, you can probably do that.

Colin Eberhardt

That’s true. Yeah.

Remi Van Goethem

But I’m the same as you. I don’t think I can multitask. I want to make sure that what I’ve been working on is completed, and to the level I want.

Colin Eberhardt

Yep. So, I guess, the final thing I want to talk about – and I think it naturally follows on from your feeling it being fun to vibe code this application – is the excitement of building something really fast. I’m seeing a lot of blog posts and very well-written blog posts recently about people finding this new way of working quite uncomfortable.

There was one I read a while back about mourning the craft. People are struggling to come to terms with this quite different way of working. And I mean, it would be impossible to go through all the different blog posts, but one that jumped out at me was one that was titled, ‘I Miss Thinking Hard’, and they described people as either Builders or Thinkers.

So, Builders are people who get that dopamine hit from creating software, whereas the Thinkers are the ones who get the excitement outta the code and the crafting it. Does that match your mental model? Do you, do you feel more like a Builder or a Thinker? Do you think that simplification works?

Remi Van Goethem

I like a bit of both, but I would say I’m probably more geared toward the Builder side. I like to see the outcome. I like to solve complex problems, and these days, I like to solve business problems. How it’s done is probably not as important to me these days. But I think everyone’s gonna be different. What is your view on that?

Colin Eberhardt

Yeah, I think it’s a good mental model for starters. What I found interesting is that the types of things that I’ve been building as a hobby have changed. So for me, I’ve been doing software professionally for 25 years, but I’ve always enjoyed the craft of software itself.

So I’ve always had side projects, and if I look back five or six years ago, I was doing a lot with things like WebAssembly and having fun, building things like emulators. I was the Thinker back then. I was building things for the fun of building it, and with the emulator, I never played any games on it.

I just wanted to build the thing ‘cause it’s fun. Whereas now my hobby projects are, I’m building a thing for logging carting sessions and so on. I’m actually creating an application where the code itself is boring. It’s a crud-style application. But I enjoy the outcome. I’m now building things that I use, whereas previously, I built things that I didn’t use.

It’s really weird. I feel like I’ve leaned more on the Builder side of things because vibe coding  has allowed me to build more.

Remi Van Goethem

The cost of building custom tools for yourself has never been lower than now.

Colin Eberhardt

Exactly. So I think this technology has turned me into more of a Builder than a Thinker. Well, that sounds bad. I’m sure I still think, but there are so many of these blog posts; you pointed out another one to me that agentic coding makes old coders young and young coders old. And what I found really interesting about that one is, it looked at the programmers at different stages in their career, and the people that are at a later stage in their career – they’ve been an engineer for a while, maybe they’re an architect, maybe they’re a manager or a CTO or what, whatever else, and they don’t have much time for writing code or haven’t written code for a long time – that’s the group that seems to be really taking to this technology.

It’s the people who are in between the people who are really quite experienced, but still have a deep connection to the craft. Those are the ones that are struggling.

Remi Van Goethem

Yeah, I agree ‘cause I think people at the later stage of their career, they’ve moved up the abstraction layer, and they’re already doing a lot of delegation, so it feels very natural to use AI the same way as they do anyway. But I think for people who are quite early in their career, they’re probably gonna be adapting super-quickly to this new paradigm.

So, yeah, it leaves maybe the people who are mourning the craft itself, because I think it is true – if you care very much about the code itself, it is at risk of being commoditised today, so yeah, this is a risk for sure.

Colin Eberhardt

Yeah, definitely. So, I guess that gets us to the ultimate question that we’re trying to answer. Do you think vibe coding is the future?

Remi Van Goethem

I think there are multiple levels. If you’re talking about like prototyping, yes, this is definitely the future; you can get something very quick. You don’t need to go for low-fidelity stuff any more; you can show the real thing very quickly. For research, yes, a hundred percent sure; it’s super useful.

But I don’t buy into the “anyone can code” vibe, because it’s not just about coding, it’s about engineering. And I think what vibe coding does not achieve at the moment is the engineering thinking; so, I don’t think we are here yet.

Colin Eberhardt

Yeah, and I completely agree with you. I think vibe coding itself has a fantastic future, and we’ll find more and more places that we can use it. But I don’t see it as the ultimate destination of all software development at the moment. And that’s the thing that I think is a big unknown. I know you mentioned that your feeling is that the model development has maybe slowed down and plateaued; if there’s a point where it starts to really understand architecture, maybe vibe coding will be able to cover a lot more of what we are doing.

Remi Van Goethem

Yes. I think, as always, the thing that it doesn’t understand is the need of the business, for example, or human needs. And we need to be able to have this human intuition, I guess, to understand what matters.

Colin Eberhardt

Absolutely. And at the moment, that’s one part of the puzzle that they haven’t solved at all through the use of AI.

Remi Van Goethem

Doing something is cheap, but knowing what to do is not.

Colin Eberhardt

Yeah, absolutely.

And that brings this month’s episode to a rather abrupt halt. I must have a word with Paul, our editor(!). In truth, Remi’s final statement that with vibe coding, doing something is cheap, but knowing what to do is not, was so powerful that there wasn’t much more either of us really wanted to say on the matter. If you’ve enjoyed this episode, we’d really appreciate it if you could rate and review Beyond the Hype.

It’ll help other people tune out the noise and find something of value, just like our podcast aims to do. If you want to tap into more of our thinking on technology and design, head over to our blog at scottlogic.com/blog. So, it only remains for me to thank Remi for taking part and you for listening.

Join us again the next time as we go Beyond the Hype.



NoJS 3 - The dawn of Flappy Bird. Making a Flappy Bird clone using pure HTML and CSS, no JavaScript by Gurveer Arora
Gurveer Arora — Mon, 09 Mar 2026 00:00:00 +0000
Previously, I created a calculator and tic-tac-toe with a “random” computer player, both entirely in CSS. I also once made a game of Flappy Bird in D3, however, not in any way that warranted a blog post. Recently, a colleague misremembered these as “Flappy Bird in CSS”. The thought was amusing. I knew it would be technically possible with the use of animated radio buttons for every single possible state, but that didn’t feel interesting. Over the next few weeks, I didn’t actively pursue the idea but kept getting thoughts in the back of my mind about how different aspects could be achieved. Eventually, I was forced to sit down and make it, you can play around with it here (this will not work on mobile). In this cry for help blog post, I explain how I made it.

Rules
The only thing I wrote was HTML and CSS. No HAML, SCSS, or any other preprocessors. No JavaScript is enforced by testing the app with JavaScript disabled in the browser settings. You can view my full codebase, including other creations here.

How did I make it?

Click to flap

The fundamental aspect of the game is to click a button, and a bird jumps up, before falling to the ground (or in my case, off the screen).

Motion is simple. I played around and found an animation setting that looked close enough. I’m not going to explain the cubic-bezier here. Just know that it lets you create different animation timing functions, so that animations can vary in speed as you need. By setting up the example below, we can animate the CSS variable --bird-delta-y to go up and then down in a falling manner. Animating a variable just means we have a variable whose value is changing. By adding this value to the bird’s top position, the bird is animated. It makes it mimic the motion of a flap upwards followed by a fall.

@property --bird-delta-y {
  syntax: "";
  initial-value: 0px;
  inherits: true;
}

:root {
  animation-name: jumpAndFall;
}

@keyframes jumpAndFall {
  0% {
    --bird-delta-y: 0;
    animation-timing-function: cubic-bezier(0.22, 1, 0.36, 1);
  }
  /* 0% to 25% is the jump */
  25% {
    --bird-delta-y: calc(-1 * var(--jump-height));
    animation-timing-function: cubic-bezier(0.68, 0, 1, 0.26);
  }
  /* 25% to 100% is the subsequent fall */
  100% {
    --bird-delta-y: var(--fall-distance)
  }
}

.bird {
  top: var(---bird-delta-y)
}


Next, we need to reset the jump and start it from a new location when the player clicks their mouse. It’s not possible to just read the current value of --bird-delta-y and base new calculations off it; this is because CSS works in a declarative manner, not an imperative one. It’s also not possible to do an event listener in CSS. But we can use radio input buttons. CSS can detect a checked radio button and can thus apply styles or modify variables based in them. And the nature of radio buttons is such that if another one is clicked, the first one becomes unchecked. So the value of --active-number below will always be the value of the most recently clicked radio button.

:root:has(input#fall1:checked) {
  --active-number: 1;
}
:root:has(input#fall2:checked) {
  --active-number: 2;
}
/* And so on */


Above, we make use of the has selector, which is relatively new; it allows you to select an element based on the properties of its children or later siblings. In this case, it will select :root (html) when it has an input#fall1:checked (or another number instead of 1) inside it. So above, we are setting the variable --active-number on the :root based on the most recently clicked box.

So we stack a bunch of labels (clicking labels triggers their respective radio buttons) on top of each other. And animate them along with the bird. By covering up most of the buttons and opening a small slit, we can have the button available to the user change with the animation. More specifically, at any point, the button available to the user is entirely dependent on the position of the bird. This is best understood from the animation below. The key thing to remember is that in the actual game, the shaded regions of the button column are completely opaque, so the user only sees what looks like an unmoving button.


  
  Demonstration of how clicking to flap works


Then we change the calculation of the bird’s position to:

.bird {
  top: calc(var(--bird-delta-y) + var(--active-number) * var(--click-box-height))
}


--click-box-height is the height of the label. --active-number is determined by the most recent label clicked; each value is just an integer indicating its position. --bird-delta-y is the animated variable from earlier. The result of this is that we adjust the height of the bird based on where the most recent jump started from.

But there is a problem, this does not make the animation restart, so the bird will not jump. CSS only starts the animation when it’s first added. So what we can do is create two identical animations and then, on each click, swap them out. This tricks CSS into starting the “new” animation from the start. We now need two sets of inputs, each are complete for the purpose of setting the bird’s starting position as described above. They will be div#jump-holder-1 and div#jump-holder-2. However, they set a different animation. So when one input is clicked, it sets the bird’s position, sets the animation jumpAndFall, hides its parent, and causes the other one to appear.

:root:has(#jump-holder-1:has(input:checked)) {
  animation-name: jumpAndFall;
}
:root:has(#jump-holder-2:has(input:checked)) {
  animation-name: jumpAndFall2;
}

@keyframes jumpAndFall {
  /* As above */
}
@keyframes jumpAndFall2 {
  /* Duplicate */
}

/* Ensure that the jump-holders (divs containing labels) swap in and out on every click */
#jump-holder-1:has(input:checked),
#jump-holder-2:has(input:checked)
{
  display: none;
}


All the radio buttons have the same name, which means that only one can be selected at a time. So when one from #jump-holder-2 is selected, it deselects the one from #jump-holder-1. This results in #jump-holder-1 being visible again and the animation jumpAndFall2 being removed.

Pipes and “Randomness”

Next, we need to create some pipes. Drawing and animating them is pretty straightforward; we simply animate their position to move left across the screen. To avoid creating a div for each new pipe, we simply need to create 3 and have them repeat. By having the divs restart their animation once they are off-screen, it looks like there is an infinite amount. The following code causes each pipe to slide across the screen and then jump back to the start before repeating.

.pipe-frame {
  animation-name: pipe;
  animation-duration: var(--pipe-duration);
  animation-timing-function: linear;
  animation-iteration-count: infinite; 
  animation-delay: /* Vary this for every pipe */
}

@keyframes pipe {
  0% {
   left: var(--pipe-start);
  }
  100% {
   left: var(--pipe-end);
  }
}


But how do we vary their heights? First, we create an @Property, call it --score, and animate it to increase every time the pipe goes off screen (we can do this just by knowing the time it takes). Each pipe is then given a --pipe-number (1, 2, 3). The below maths then ensures that each pipe has a --pipe-index that jumps up by 3 exactly when it completes one passthrough. We want this because it means each iteration of each pipe has a different --pipe-index.

.pipe-frame {
  --integer: round(
    down,
    calc((var(--score) + 3 - var(--pipe-number)) / 3)
  );
  --pipe-index: calc(var(--integer) * 3 + var(--pipe-number));
}


Then, by using some trig functions on --pipe-index and playing around with them, we can create pseudorandom positions for the pipes. So all the heights now vary, but each game is still exactly the same! To get around this, the calculations take a seed, and the seed varies each game. How? By animating another variable, which pauses once the user closes the pop-up. If the animation is fast enough, it should lead to a different value each game.

A simple use of a CSS counter gives us our visual score.

Collision Detection and Game End

Pipes are great and all, but what’s the point if they don’t hurt the bird? Here is a simplified version of my collision detection, which is done independently by each pipe.

.pipe-frame {
    --overlap-in-x: calc(
    (
        max(
            0px,
            var(--bird-x) + var(--bird-width) -
              var(--pipe-x)
          ) / 1px
      ) *
      max(s
        0px,
        var(--pipe-x) + var(--pipe-width) - var(--bird-x)
      )
  );
  /* Similar for --overlap-in-y */
  --collision: calc((var(--overlap-in-x) / 1px) * var(--overlap-in-y) / 1px);
}


This bit can be hard to read, but it’s not too complicated. The calculation is split into an overlap in x and in y. The x calculation is based on two things. Is the bird’s right side past the pipe’s left side? Is the bird’s left side behind the pipe’s right side? Take a moment to convince yourself that if both of these are true, the bird and the pipe overlap in the x dimension. The value of --overlap-in-x will be 0px if there is no overlap and a positive value otherwise. There is a similar calculation for overlap in y. If there is an overlap in the x and y dimensions, we have a collision! If there is a collision, --collision will be above 0, otherwise it’s 0.

Then we create our endgame screen and give it a height of 100vh * var(--collision). It will then only appear when a collision occurs. To finish it off, pause the animation of the bird and the pipes whenever the end game screen has the user hovering over it.

An aside which isn’t critical to operations: units are important in CSS.
You may have noticed a /1px in the above code. This is because otherwise we would be assigning the result of something like 1px*1px to something wanting a length. But 1px*1px is technically an area. This is surprisingly important to CSS even though it doesn’t deal with areas.

FAQ

The game looks kind of ugly. Have you considered adding pretty styling?
I’ve never heard of CSS being used for styling, but anything is possible, I guess.

Why is the bird square?
I have a square bird for ornithological accuracy and not because it made collision detection easier.

Wrap up

So what does this tell us? It tells us that at the time of writing this app, I am smarter than Gemini 3 Pro. Here is it giving up after being given the same amount of push I got.




Microsoft Agent Framework - powering up your agentic teammate by Dean Hunter
Dean Hunter — Fri, 06 Mar 2026 10:00:00 +0000
Recently our sister company Marra built an agentic “teammate” for user onboarding using Microsoft Power Platform exploring new AI features available. They asked Scott Logic to do the same using Microsoft Agent Framework.  Luckily I was on the small team who got involved.

The project was to ultimately compare the two teams’ experiences, find the advantages and challenges of each and to explore agentic technology. This post focuses on our work with Microsoft Agent Framework rather than comparison.

Our Task
We were tasked to build an agentic system for user onboarding with human-in-the-loop.  Onboarded users should be saved on two downstream systems: HubSpot and LeavePlanner (SharePoint hosted Excel Workbook).

The objective was to communicate as you would with a “teammate”.  This was a time-boxed exercise (three weeks to investigate, build, and present).



Our Build
Microsoft Agent Framework was up to the challenge.  We built a single entry point minimal multi-agent solution.  We hosted our agentic workflow in Azure as a container, not quite on Microsoft Foundry as we would have liked, but next time we might get there.

Our solution interaction was by email.  We used an Azure Function to poll the “agent” mailbox, pick up relevant emails and extract the content to send to our agent.  Our agent could also send email for communication, and perform the user onboarding tasks using custom tools.

To understand what we’ve built I have a couple more sequence diagrams:



Disclaimer
A traditional system could have been written to do this without the use of an LLM.  The exercise was to use an existing Microsoft based communication system as a teammate.  The LLM gave us easy data extraction from free text in an email string, then tools enabled us to write custom actions we could expect the LLM to execute.

The solution discussed here was built entirely as an exploratory exercise proof of concept, we would have needed to do a fair amount of work for a production hardened system.

Our Journey
Our team brought relevant experience from previous LLM-powered projects and other agentic solutions.

On initial investigation, Microsoft Agent Framework was very interesting.  We could quickly define a deterministic workflow and agents handing off to each other in a pretty modular way.

This gave us a quick start to get moving with code samples from the github repo and some documentation.

Microsoft Agent Framework
The Microsoft Agent Framework and Microsoft Foundry was our primary focus, this was relatively new and brought together previous frameworks Semantic Kernel and AutoGen.

We could have many small agents working together to perform a bigger task by handing off to each other.  The advantage here with the small dedicated tasks meant we could keep the system and user prompts super short and to the point. The smaller context per agent gave us a much higher chance of getting exactly what we wanted from our LLM.

We quickly configured the Microsoft Foundry GPT-5 nano model and started planning our onboarding agent.

We implemented DevUI (included with Microsoft Agent Framework), it was a nice addition for quickly visualising our agentic workflow and testing as we built.  Also very useful when demonstrating functionality to stakeholders, see screen grabs of workflows below.

Microsoft Agent Framework DevUI - Example of the approval steps:



Microsoft Agent Framework DevUI - Example of the onboard steps:



Teammate and Human in the loop
We needed to include our agent as a “teammate”, initial thoughts gave us email with GraphAPI (polling or subscription), Microsoft Teams, or even email Adaptive Cards for Outlook Actionable Message.  After considerable team debate and research, we opted for Graph API with email polling due to its setup simplicity and faster configuration.

We opted to have the email polling done using an Azure Function and submit the body as prompts to the agent rather than have the agent read the mailbox directly.  So the flow was that the user sent an email to their “teammate” and this was forwarded to the agent which could reply to email as necessary.

Then the human-in-the-loop point where the agent requested the 365 account creation before proceeding to onboard the user, gave us some security before accounts were simply created.

Our Agent Breakdown
Perhaps the most interesting part was what our agents actually did.  We had various agents implementing different techniques and tools which lent themselves well to our scenario.

Intent Agent
Simply determine the intent of the request.  Our scenario had two paths: a request from a user sent by email to go to IT for approval, or a request to go to the onboarding agent.

The agent figured this out based on its system prompt (a short instruction and a few brief examples) and the email request body, the email body was a simple sentence, not CSV content or any specific format we wanted the user to remember. Next step was then delegate to either the IT Agent or the Onboarding Agent.

We sent the actual request to the agent using an Azure Function that extracted the email body text.  This could have been done by the agent and was likely something we would have investigated given extra time.

Future growth here is pretty big as we could add many more scenarios that the agent handles, change job role, remove user etc - if it gets too much then simply plug in a couple of other agents as necessary.

IT Agent
This agent handed the request off to a human.  The onboarding flow required that the 365 user account was created by a human, so a request for IT to do this would be sent by email.  This enabled the human operator to interact with the agent by replying to the email with the new users 365 email address.

The IT Agent performed entity extraction to get the user’s name, job role and line manager.

The IT Agent had a custom send email tool (using Microsoft Graph API - authenticated with Entra ID).  The tool sent an email request for onboarding to the human administrator to complete the request. Alternatively, if the entity extraction determined data was missing, an explanation was sent to the user.

Onboarding Agent
This was the second time around. The administrator had now created a new business user account and provided the 365 email/username to our Intent Agent. The admin had replied to the email (which the Azure Function was sending back to the agent).  This time the Intent Agent had determined this and handed over to the Onboarding Agent.

The Onboarding Agent ran entity extraction.  Then given success on email, the user’s name, job role and line manager, started onboarding.

Here we handed off to multiple agents at once: HubSpot agent and SharePoint (Leave Planner shared Excel workbook).

Future growth here might be that the user administrator could decide to exclude some systems to onboard to or have different systems based on user role.

HubSpot Agent
Our first external system to onboard the user, simply call the API to create the user.  To do this we built a custom tool for the operation that we provided to the agent.

Our tool had some logic for error handling but we’d see future growth where the agent could have more functionality.  Search, update and delete (securely, of course) tools could be available in a more production hardened solution.

The end result then continued to the Completion Agent.

SharePoint Agent
The task here was simple, insert a record for the new user in the fictional LeavePlanner holiday system, which was represented by an Excel workbook on SharePoint. The agent had a single tool to insert the row in the workbook.

Of course, this could get considerably more complicated and could offer more tools such as update, delete and select.

The end result then continued to the Completion Agent.

Completion Agent
Here we wanted to see green on DevUI when using example prompts.  This was less of an agent and more a point to connect the workflow back to a resolution. Check the results from inbound agents, send success emails to systems the user had been onboarded to, and report any issues encountered to the IT department.

Outcomes
Overall this was a fast-moving, interesting project, providing a better understanding of where agentic systems add value and the parts of an agentic approach.  I’ve reflected on a previous LLM powered systems I worked on, and how dedicated agents could give a more robust solution than a large single prompt.

Microsoft Agent Framework is definitely going to be a point of interest in any upcoming LLM-powered solutions our team build.

As for the email a “teammate” interaction we investigated. Do people want to communicate with an AI-powered “teammate” rather than having another system to use that submits requests through a traditional form.  I can definitely see this being preferable is various situations.

Then do I want an agentic “teammate” to do things I have to do as part of life that I consider boring - yes.  I already do, daily I’m using copilot to make work quicker and easier.



Analysis → Implementation → Reflection – a practical technique for issue resolution with agentic AI by Dean Kerr
Dean Kerr — Thu, 05 Mar 2026 00:00:00 +0000
This article presents Analysis / Implementation / Reflection, a simple pattern for resolving issues. While the core of this pattern, the implementation, is quite conventional, there are a couple of novel additions. In the analysis phase, the agent is used to explore the issue and create a suitable harness to evaluate the solution. While the reflection phase probes the agent to provide a qualitative assessment of the implementation. Together, these provide a high degree of confidence in the solution (both functional correctness and overall quality) and ensure that you, the developer, are comfortable with the solution and able to ‘own’ the outcome.

Establishing Project Baselines



Before diving into any issue, it is useful to establish project baselines via generating copilot instructions for your workspace. These help define coding standards, library preferences and architectural patterns before the agent attempts to make any adjustments to the codebase. These are included by default in every chat prompt, meaning it saves you the headache of including them yourself manually each time, or expecting the agent to infer these standards itself.

Take the time to review and adapt these appropriately, don’t rely on the AI to pick up everything. It can be particularly common to have unwritten rules and/or assumptions in a software project; this instructions file would be a great place to write any of those down.

Revisit the instructions file as you work together with agents in the codebase. Agents frequently need fine tuning and guidance to avoid common pitfalls such as stalling indefinitely when choosing to run tests in watch mode. As you build up these instructions you should start to see get a more reliable output first time when using agents.

Faster Issue Contextualisation

Getting up to speed on a new issue drains your daily cognitive bandwidth. Being able to shorten this isn’t just about saving time but also preserving your mental energy to use elsewhere.

Agents are a great tool for summarising what can be dense, comment-heavy issues - stripping away non-essential or irrelevant conversation. Try prompting your agent to distil issues into core requirements, blockers and current consensus allowing yourself to focus quicker on the technical challenge at hand.

Try not to limit yourself to textual based summarisations either, modern multimodal models can digest almost anything attached to a ticket. Screenshots, stack traces, database schemas and even video recordings of a bug can all be fed to the agent to build a robust context before you write a single line of code.

Analysis → Implementation → Reflection loop

 implementation -> reflection loop" />

Closing the feedback loop, effectively allowing an agent to evaluate the quality of its own output, is especially important for AI augmented software development. With the right constraints (e.g. a tightly bounded problem and comprehensive tests) agents can port large codebases with relative ease and swiftness.

Scaling this approach down to individual issues is just as powerful. Breaking the work into three distinct phases gives the flexibility to interject whenever the agent starts to struggle or drift off course.

Analysis

Once you’ve understood the issue - ideally using the faster contextualisation techniques mentioned earlier - it’s time to build a lightweight harness to let the agent iterate in a feedback loop. A typical approach involves asking the agent to analyse the provided issue and, adopting a TDD approach, write relevant failing tests that reproduce the bug or define the new feature. Be warned: the agent may stumble even at this first hurdle, especially if the issue is difficult to reproduce (if at all).

Take a moment here to reflect. Are the generated tests appropriate, relevant and offer good coverage? Are there any potential gaps or missing edge cases? Your goal here is to verify that the AI agent fully understands the issue.

At this point you should have a decent idea in your head of a rough solution to the issue, giving you a solid point of comparison for the final reflection step.

Implementation

With failing tests in place, the implementation phase should drive itself. Prompt the agent to implement a solution and use the test suite as the primary feedback mechanism (the classic Red/Green loop). The trick here is to instruct the agent to iterate until all tests pass.

Example Prompt:


  Now that we have failing tests, please implement the code required to resolve the issue. After your first attempt, automatically run the test suite. Use the test output as your primary feedback mechanism. If the tests fail, analyse the error, adjust your implementation, and run the tests again. Continue this Red/Green iteration loop until all tests pass. Crucially: Once all tests pass, stop immediately and await my review. Do not proceed to refactoring or further tasks.


Reflection

Arguably, this is the most important step of the loop, where you take on responsibility for the outlined solution. This involves using reflective prompting to challenge the agent’s outlined solution, particularly if it deviates from how you assumed the issue would be solved.

Typical lines of questioning include:

Architectural Integrity


  
    What alternative solutions did you consider, why did you choose this one? Outlines other potential pathways to a solution that may be preferrable
  
  
    What edge cases or unexpected inputs could cause this implementation to fail? Gets the model to look for cracks in their own logic
  
  
    Are there any specific scenarios where this solution might introduce a race condition or state inconsistency? Useful for any asynchronous or multi-threaded work
  


Maintainability


  If a junior team member had to maintain this code in six months, what part would be the hardest for them to understand? Neat trick to force AI to identify complex and/or unreadable logic that may warrant refactoring or commenting


Security


  
    Are there any hidden performance bottlenecks or scaling issues in this approach? Good catch-all for performance / inefficiency
  
  
    What potential security implications or vulnerabilities does this change introduce? A mandatory sanity check, especially relevant if the solution handles user input or involves authentication
  


Making the model defend and justify its logic is a great way to uncover edge cases or simpler paths that were glossed over in the first pass. You may also find it useful switching to a completely different model at this point and ask it to run a blind code review on the newly implemented changes.

Managing the context window

Keeping an eye on your context window usage, particularly when you’re deep in a long running chat thread with an agent can save you from diminishing performance.

There are various levers you can pull to improve context efficiency. A major one to explore is the Model Context Protocol (MCP), which lets your AI tools fetch specific, bite-sized context from your local environment on demand, saving you from pasting in whole files and burning through your token limits.

Even with these tools, if context limits are a frequent problem, consider how you can break down larger pieces of work across multiple, isolated chats. Often, a bloated context window means you’re simply trying to solve too much at once within the agent’s current capabilities.

Debugging AI Chat

Agents have a big problem with explainability. Nobody can explain exactly how models work ‘under the hood’, and because we are using bleeding-edge tools, the harness layer itself will frequently break in unexpected ways.

However, you can give yourself a headstart by lifting the lid a little. Quite often, the misbehaviour isn’t a deep problem. It’s just bad input at the harness level. It’s worth digging down into that layer via chat debug view (equivalent tooling exists) which lets you see what context, prompts and tools were used when talking to an agent. This added transparency can help you course-correct effectively.



Software Engineering in the Agentic AI Era by Dan Allsop
Dan Allsop — Mon, 02 Mar 2026 00:00:00 +0000
Much of the discourse around Artificial Intelligence (AI) in software development frames it in binary terms: either AI entirely replaces developers, or it is overhyped and useless. Reality lies on the spectrum between these two perspectives, depending on the AI tools in question.

Enter Devin AI, an AI software engineer developed by Cognition Labs, designed to execute tasks across the entire software development lifecycle. Unlike most generative AI tools that react passively to prompts, Devin represents a newer class of agentic AI: a proactive contributor that feels like collaborating with a highly capable junior developer who works tirelessly. After recently experimenting with Devin, it is clear that AI software engineers can transform development workflows in multiple ways.

Prototyping

Devin helps streamline basic project setup. Tasks like configuring folder structures, environment variables, frameworks, and configuration files are largely deterministic and align well with its strengths.  However, Devin’s effectiveness diminishes as complexity increases. It struggles with non-standard configurations required for more sophisticated project setups, where hands-on expertise and deeper contextual understanding are still essential.

Devin can assist in creating clearly defined small prototypes for quickly validating simple ideas. Interestingly this lowers the technical barrier for non-technical stakeholders to explore concepts more fully before requiring developer intervention. However, outside of very straightforward programs Devin’s capabilities are quickly surpassed making developer supervision essential.

By collapsing the cost and friction of simple proof of concept programs, teams can explore more ideas, iterate quickly, and innovate continuously. Innovation shifts from being a high-stakes gamble to a structured, low-risk, iterative process, allowing creativity and strategic thinking to take centre stage.

Developer Contributions

AI tools amplify workflows in ways that are deeply influenced by organizational processes, governance, and culture. Their impact can range from highly beneficial to severely detrimental, emphasizing the need to address software development lifecycle challenges proactively.

Devin’s role as a proactive contributor means it has the potential to be a productivity multiplier, but only when applied appropriately:


  
    Testing: Devin can generate and maintain “happy path” tests automatically, reducing some manual effort, but it struggles with complex edge cases and nuanced domain logic, which still demand careful human oversight.
  
  
    Debugging: Devin can run code, inspect logs, trace issues, apply corrections, and retry, turning routine debugging into a procedural exercise. However, it fails on more complex bugs and bugs that are difficult to reproduce, which still require human intervention to resolve.
  
  
    Task Execution: Devin can complete small, well-defined tasks quickly, but its narrow focus on finishing tasks rather than considering long-term architecture often creates technical debt that must be actively managed and refactored.
  


Devin’s output is highly dependent on the precision of the prompts it receives. Vague instructions often result in confidently executed but misaligned solutions. This can be mitigated in part by using the Ask feature which accesses codebase specific insights to refine a prompt before execution. Human guidance is required to engage Devin to critique the proposed prompt for ambiguity, missing constraints, unclear success criteria, etc. Better prompts produce disproportionately better outcomes and prevent unnecessary iterations saving time and money.

Human engineers remain essential for judgment, strategy, creativity, and responsibility for security, performance, and architectural decisions. Thus, we can see the shape of the human engineering contributions shifting away from task delegable to AI, and towards higher level strategic engineering contributions such as defining success criteria, evaluating trade-offs, and shaping high-impact decisions. In this model, clarity of thought becomes the bottleneck, not manual implementation, and clarity has always been the rarest commodity.

Conclusion

By automating routine setup and procedural tasks, prototyping basic ideas becomes accessible to a wider range of stakeholders, empowering them to explore and refine ideas independently. This can lead to clearer higher-quality ideas forming the starting point for collaborative work.

Additionally, as agentic AI acts as a proactive contributor, human developers can delegate routine tasks and shift their focus toward defining outcomes, evaluating trade-offs, and guiding strategic innovation.




Things to expect when starting to work with Copilot by Oded Sharon
Oded Sharon — Mon, 23 Feb 2026 10:00:00 +0000
The future of software development will probably have some element of AI. Whether it’s simple auto-complete or a full end-to-end ticket implementation, it has the potential to make us more productive. But alas, the more we rely on it, the bigger risk of bogging the work down. As a software developer, one should familiarise oneself with its capabilities and learn how to work with it, or join those who similarly prefer to use Notepad over a modern IDE.

I took the opportunity to allocate some time to learn how to work with Agentic AI. Picking a shelved project I had been eyeing for quite some time, I hoped that AI would expedite the development process. Considering the promise of “now everyone can build their app”, the learning curve was actually much steeper than I’d expected. However, I managed to get some real value from an output perspective and I also picked up a few insights along the way that I thought would be worth sharing.

Copilot is very eager

I initially tried working with the free version of GitHub Copilot, but I got frustrated with it very quickly. I couldn’t figure out the right balance of details needed for it to do a decent job. Too little, and I realised I needed to specify I want the code to be compilable. Too much detail, and it quickly went beyond its context window, forgetting the first thing I told it.

There were no epiphanies or light-bulb moments but over time I learned to write better prompts. I also broke the tasks into smaller, more reasonable, clear instructions. I also switched to the paid version, which made life much easier as it gave me a larger context window. With the right prompts, you can achieve a lot with the paid version. Unless you don’t have the budget or enjoy challenging yourself, I’d argue that life’s much more comfortable with bigger contexts.

My attitude towards Copilot changed over time. From “go write an application” which led to utter disasters, I pivoted to “plan, review & implement”. It was considerably slower, but gave me a better chance to course-correct whenever it veered in the wrong direction.

Copilot, however, was very eager and often wrote bits of code directly into the spec files, despite me explicitly instructing it not to write any code before I’m happy with the suggested design. It always asked to let it jump to the implementation stage, even in the initial stages.

Context matters

As mentioned, one of the biggest issues I encountered—particularly with the free version—is the context-size limitation. In one of the tasks I gave it, it was very clear that by the end of the implementation, it had completely forgotten the first line of the plan and simply ignored that instruction. On the other hand, it constantly returned to certain concepts, even after I’d explicitly said it should avoid them. For example, Copilot insisted on making my application multi-threaded, no matter how much I emphasized it should be single-threaded.
A general tip in such scenarios would be to start a new conversation (with a fresh new context) but if your code has ill-conceived concepts, they might pollute any future solution.

Later on in the project, I asked Copilot to refactor a variable from string to int. This is something I would expect a smart-enough IDE to handle rather efficiently. I was therefore disappointed that Copilot scanned the entire code as text, and it ended up missing out some of the unit-tests due the sheer number of changes. Frustratingly, it also missed out serialisation tests.

Copilot isn’t “aware” of the project’s code. It needed to actively look for string pieces (using RegExp). This is unlike a normal IDE, which has syntactic awareness of the code and carries out static analysis which improves refactoring efficiency, whereas Copilot has no awareness of the meaning in that sense.

Accessing the terminal is an unnecessary pain

Copilot tried several times to access the terminal to either delete files or run tests, and it failed miserably. It kept generating wrong PowerShell and WSL commands and got itself into an endless loop of failing to run the unit tests without figuring out it had permissions issues. This was extremely frustrating, as for these two functionalities, I would have expected it to manage this internally (via the IDE). Instead, it tried to utilize the terminal to run ./gradlew test and then parse the output to understand what happened. Not only do I think these two functionalities should be handled internally, I would even suggest that it should never actually delete files, but rather just archive them safely.

This has actually improved over time, whether it was updates in the IDE or in the model, but it now creates its own terminal console it can better control. So at least the tests work properly, and I’m pleased that “keep fixing the code until the tests pass” is much more likely to work now.

Models are soda

In her TED Talk, Sheena Iyengar tells a story in which she offers participants a drink from a selection— Coca-Cola, Pepsi, 7Up, amongst others. One participant replied, frustrated, “Oh, it’s all just soda.”

Copilot offers 15 different models, and sadly from my experience, they’re all just soda. Of course, there’s a difference in pricing (between 0.3x, 1x, and 3x), but once we settled on a budget, I couldn’t identify any significant difference between Sonnet 4.5 and GPT-5.2. Yes, some might be faster than others; some have deeper reasoning or quicker responses, but ultimately a lot of it is handled in a black box, and the results, from my point of view, are similar enough that I didn’t feel it would be helpful to switch between models for specific tasks.

Chatting with my colleagues, they theorised that this was due to my ‘small-step’ strategy, as some of the models are best suited for deeper reasoning, big tasks, or ‘quick-and-dirty’ work. I’m afraid that my initial experience left me skeptical (rightfully or not) regarding any model’s capability to manage long and complicated tasks.

Since that conversation, more new models have been released, and there’s a significant improvement in the likelihood of them matching my vision. They are getting better, so I believe most people using older models would benefit from moving to newer ones.

AI is like an unreliable teleportation device

Imagine you have a teleportation device that becomes less reliable exponentially the further you travel. Travelling one metre is not a problem. Travel a hundred metres, as cool as it may be, and you might find yourself somewhere very different. Travel more than a kilometre, and there’s a growing chance you’ll turn into a fly.

I learned the hard way that any mistake in the system will worsen with every additional step you take if not addressed. The solution I settled on was sacrificing efficiency and making tiny, measurable steps instead of reckless leaps of faith. If one side of the spectrum is mere autocomplete (which has truly improved significantly over the years), and the other side is “build an app that does X,” my lesson learned was not to venture further than adding a single feature at a time and making sure it was added properly, passing all its tests, and working for end users. That said, I felt I reached nirvana when I asked to implement a particular feature and it nailed it almost completely on the first try.

Unit tests come with limited warranty

For every feature I added, I made sure to include unit tests to verify it worked properly. Unfortunately, verifying that they all passed and paying special attention to those that failed wasn’t enough. I didn’t check that the code actually worked as it should until I had an MVP ready, and at this point it was quite late in the game. Despite being given a specific architectural design, Copilot created all the files I asked for but left them mostly empty, writing the code in two big files instead. When I eventually ran the app, it didn’t behave as expected, despite all tests passing.

It might be controversial, especially for TDD enthusiasts, but I would argue that an MVP is more important than comprehensive code coverage. For example, we can defer creating most of the data models and first have a server endpoint that can handle one data model. The earlier you have an MVP working, the sooner you can see what needs fixing.

Design documents are yet another thing that require maintenance

I definitely didn’t become a 10x developer. Maybe a 2x developer, if anything. But I comforted myself with the fact that at least I learned how to write clearly defined spec documents that even the most junior developer would be able to implement. “Writing clear specs” is an important, applicable skill to have. 
I asked Copilot to translate my spec into measurable tasks and had it implement one task at a time, while I made sure the unit tests passed properly (a necessary step, but as I learned, far from enough).

In reality, my specs were still not perfect, but I soon found out it didn’t matter. As soon as I could run the app, I learned that some initial assumptions needed to be adjusted (for example, the data files, according to my spec, were unnecessarily bloated), and now I had a few options: I could ask Copilot to update the spec, or I could edit it myself; but then how would I use this updated spec to fix the code? The solution I decided on was to discard the initial spec and ask Copilot to create a spec per task. I made sure that I was happy with the task spec and asked it to implement it. Those tasks still needed to be well-contained. If they were too big for its context window, it would still get lost. It also needed to keep looking for files (often using RegExp to scan through the files). It felt incredibly inefficient compared to modern-day IDEs’ refactoring features. The old spec files? It would probably be smart to ditch them, as we wouldn’t want them to taint the context with outdated assumptions. The same goes, in fact, for any old code that might send Copilot down the wrong rabbit hole.

Conclusion

As I mentioned, I have no million-dollar tip how to get Copilot to do what I want, but I can clearly see how my productivity improved within a mere month of working with it. It may have been intuitive learning and may have been upgrades to the IDE or the models but either way, the result is undeniable and there’s no chance I could’ve achieved the progress I made without it.

Copilot is valuable when properly utilized: The size of the tasks you give it is directly related to the amount of trust you have for it. Start with small, verifiable, incremental changes to your code, as slop grows when unattended. A limited-size context will force you to feed Copilot only what’s relevant—your spec document should be scoped appropriately, and be sure to archive it after implementation. My prediction, or perhaps my wish, is that future IDEs will incorporate AI capabilities into their existing features, such as refactoring and running unit tests, instead of relying on a third party that’s unaware of the code’s context.



Diving into Retro Computing with the Amstrad PCW 8256 by Lucy Hancock
Lucy Hancock — Thu, 19 Feb 2026 12:00:00 +0000
“Do you like old computers?”

I watch as Ben, my plumber, hauls a large cardboard box marked with “AMSTRAD” in blue lettering down from my attic. After he fixes my extractor fan and leaves for his next job, I open up the box held together with staples and ancient, yellow tape, to find a full monitor, keyboard and printer set, complete with floppy disks:



It is an Amstrad PCW 8256. I have no idea what it can do or how it works, but I’m excited to find out. I power it on only to find a screen filled with green lines. I try pushing one of the floppy disks into the reader, but nothing happens.



After some digging, I find that that blank screen meant the floppy disk was probably not being read, likely because the reader’s drive belt (which is essentially a rubber band) had perished. Sure enough, when I open up the monitor case and the disk reader inside, I see the drive belt is in bits:



Luckily, there are a few sellers on eBay who offer replacement drive belts in exactly the right size, so after the new one and some isopropyl alcohol arrives in the post, I clean off the two spools and replace the band (many thanks to Chips y Bits on YouTube who has a very straightforward demonstration of how to do this). The PCW was now able to read disks again!

The previous owner of this computer had left behind a few disks, one with Locoscript on it, which is the standard software that came with the PCW that is used to write and print documents. There wasn’t much on there – mainly just an article about train station architecture, and interestingly, a wedding speech. The other disk of interest was the one with the BASIC interpreter on it.

About the Amstrad PCW
The Amstrad PCW 8256 comes with a BASIC (short for Beginner’s All-purpose Symbolic Instruction Code) interpreter and Locoscript as standard software disks. Boasting some 256 KB of RAM (as its name suggests), it was considered a powerful microcomputer at the time of its release, and was a competent and affordable choice for computer buffs, gamers, and office users alike. It’s also still pretty popular with retro computer hobbyists today, with PCWs of varying function selling online for around £100-£200.

The BASIC interpreter disk can be used to write your own programs. More accurately, the disk actually has Locomotive software’s own version of BASIC, written specifically for the PCW, Mallard BASIC. I thought it might be fun, given the classic green and black terminal look of the PCW, to create a terminal hacking minigame, similar to the one played in a very popular post-apocalyptic game series:



Before we dive in, I should preface this with a disclaimer: Some purists believe learning BASIC is a waste of time, that it ruined early programmers, and that it’s a very clunky and limited language to code in. Dijkstra even wrote, albeit in a very short and tongue-in-cheek text, “It is practically impossible to teach good programming to students that have had a prior exposure to BASIC: as potential programmers they are mentally mutilated beyond hope of regeneration.” 
Terrible as it may be, I think BASIC’s limitations and clunkiness will pose a fun challenge for me to make a game using it, and will most likely spark a new appreciation for the quality of life that modern languages give us by comparison!

Basic BASIC
BASIC has some of the building blocks you’d expect to see in modern languages – FOR loops, WHILE loops, IF statements, AND/OR operators, and typing. Variable definitions are implicit, but in earlier versions of BASIC, you had to use the LET keyword. It even has functions – but these are quite different to how we use them in more modern languages; in fact there are 3 ways you may create something resembling a function in BASIC:

DEF FN
Referred to as a “user defined function” in the manual, FNs can take one or more parameters and return an expression. The expression cannot span multiple “lines” (I will get to the mess that is writing lines in BASIC in a moment), however you can chain instructions using colons. The entire expression is returned by this function.

GOSUB
Instructs BASIC to jump to a particular line and execute the code thereon until it finds the RETURN statement. Cannot be parameterised.

GOTO
The worst of the three. Instructs BASIC to jump to a particular line, but does not have a marked “end”. This can make control flow very difficult to follow and should be used incredibly sparingly!

There is also the ON keyword that can be used with GOSUB and GOTO to effectively produce a switch statement, i.e. ON answer GOSUB 100,200,300 where answer is an integer expression which determines the subroutine to be called from the following list. answer = 1 would call 100, answer = 2 would call 200, and so on.

Line numbers in BASIC must be defined by the programmer (in earlier versions of BASIC - later versions didn’t need them), they are not simply built into the editor. Each line you write must be prefaced with a number so that BASIC knows which order to execute them in. For instance:

10 PRINT "Press any key to continue..."
20 WHILE INKEY$="":WEND
30 PRINT "You pressed a key!"


It is advisable to write your line numbers in multiples of 10, in case you need to insert more lines in between them later. Yes, really.

We can make do with this limited control flow, though as you can imagine, programs can get very complicated very quickly, and global variables are unavoidable.

The Minigame
The basic premise of the minigame is this: your goal is to guess a password from a list so you can access a terminal. If you don’t guess it in 4 tries, you are locked out, and with each guess you’re given a clue which tells you how many characters your guess has in common with the answer. The characters have to be in the same position as the answer to be considered a “likeness”.

So for instance, if the answer was “GOLF” and your guess was “GOAL”, you’d have a likeness of 2. 
Referring to our image of the game display, we have a screen of random characters with words to guess inserted at random. Starting with just getting these random characters on the screen, we can write something like:

10 REM ***** Setting up variables *****
20 escape$=CHR$ (27)
30 clear$=escape$+"E"
40 maxChars = 250
50 miscChars$ = "?!:;{}[]()<>_=+@$%*#,." 
60 mainStr$ = ""
70 lineno = 1
80 column = 1
90 REM ***** Loop to get random characters from miscChars$ *****
100 FOR i=1 TO maxChars 
110 rand = RND*21 + 1
120 mainStr$ = mainStr$ + MID$(misc_chars,rand,1)
130 NEXT
140 REM ***** Setting screen width, clearing, printing the random string *****
150 WIDTH 25
160 PRINT clear$
170 PRINT mainStr$


Which gives us:



A few notes about this code:


  REM is what we use to indicate comments in BASIC. You can also use a single quote '.
  Functions such as clearing the screen (and moving the cursor, but we’ll get to that later) are done by printing special characters that change the state of the screen rather than printing actual readable characters. So, that clear$ variable – made up of ESC + E - clears the screen when printed.
  Strings are limited to 255 characters, so to keep things simple, we print 250 characters on the screen. 25 characters per line, 10 lines.
  Some variables and functions (like the built-in MID$ function) have a dollar sign suffix – this is actually a type indicator that tells BASIC this is a string. Without the symbol, the variable is assumed to be a numeric type.
  Subroutines called with GOSUB cannot be parameterised, but we can still use global variables in them, meaning we need to be careful about setting variables before calling the subroutines and considering which variables may be changed by the subroutine itself.
  You’ll notice that the interpreter says “Ok” once the program is finished executing - this just means BASIC is ready for the next instruction.


With this information in mind, you can see that this program is defining some variables at the top, constructing a string by picking characters at random from the miscChars$ string, setting the screen width to be 25 characters wide, clearing the screen, and finally, printing our string of random symbols.
We still need to insert some words to guess into this string, but for now I’d like to shift my focus to moving the cursor around by pressing keys, keeping it within the bounds of the 10 x 25 character game area.

Cursor movement isn’t default behaviour while running a BASIC program, so we need to figure out how to do this ourselves. Luckily, there is an INKEY$ variable that can be queried to find the key currently being pressed, and additionally, we can print ESC + Y  to move the cursor to different positions around the screen. Using these abilities combined with a game loop, we can add to our existing program:

10 REM ***** Setting up variables *****
20 maxChars = 250
30 miscChars$ = "?!:;{}[]()<>_=+@$%*#,."
40 mainStr$ = ""
50 escape$ = CHR$ (27)
60 move$ = escape$+"Y"
70 clear$ = escape$+"E"
80 gameWon = 0
81 lineno = 19
82 column = 1
90 DEF FNmoveCursor$(lineno, column) = move$+CHR$(lineno+32)+CHR$(column+32)
100 REM ***** Loop to get random characters from miscChars$ *****
110 FOR i=1 TO maxChars
120 rand = RND*21 + 1
130 mainStr$ = mainStr$ + MID$(miscChars$,rand,1)
140 NEXT
150 REM ***** Setting screen width, clearing, printing the random string *****
160 WIDTH 25
170 PRINT clear$;
180 PRINT FNmoveCursor$(19, 0)
190 PRINT mainStr$
200 REM ***** Game Loop *****
210 WHILE gameWon = 0:GOSUB 220:WEND
220 keyPress$ = ""
230 WHILE keypress$="":keypress$=INKEY$:WEND
240 GOSUB 270
250 RETURN
260 REM ***** Detect key presses *****
270 IF keyPress$="w" AND lineno <> 19 THEN lineno = lineno - 1
280 IF keyPress$="s" AND lineno <> 28 THEN lineno = lineno + 1
290 IF keyPress$="a" AND column <> 0 THEN column = column - 1
300 IF keyPress$="d" AND column <> 24 THEN column = column + 1
310 PRINT FNmoveCursor$(lineno, column) 
320 RETURN


This gives the following when we run the program:



The Mallard BASIC manual suggests starting any subroutine definitions from at least line 1000 so that they aren’t accidentally executed, and you may want to insert lines between the end of your program and the subroutine definitions. However, BASIC has a really handy command called RENUM which makes all the line numbers evenly spaced in multiples of 10 again, even changing the GOSUB references, so I’m not too worried about having my subroutines begin right after the prior code.

Other versions of BASIC, such as Locomotive BASIC, have a built-in function for moving the cursor, but in Mallard we need to define this ourselves, as shown in FNmoveCursor$.

At first, tracking cursor state seems trivial, given the built-in POS() and LPOS() methods, which return the cursor’s current column and line positions respectively. However, take this excerpt from the manual: “POS returns a logical position on the console, which may bear no relation whatsoever to the current position of the cursor!” And you will see why using this built-in method could prove problematic. There is a similar reference to the LPOS instruction.

The reason that the logical cursor position is different to the current position is likely because, while printing output to the console, BASIC will implicitly insert carriage returns when it reaches the end of a line, so the positions could be offset by these extra hidden characters. It could also be affected by screen scrolling and printing control characters, so this is why it’s safer to track cursor state ourselves.

Now that we have this basic mechanic in place, it’s time to add words at random to the main character string for the player to guess. This involves choosing a word, compounding it with our main string of random characters, and keeping track of where each word is positioned in the string, so that when the player guesses a word, we can find the word they’ve clicked on using the cursor’s position and compare it with the answer.

One of the ways I might normally track word positions is by using a dictionary with each guessable word as the key, and a pair of numerical indices as the value. It will be a little more complicated using BASIC, as it doesn’t have tuples, dictionaries, maps, etc. Array initialisation is also very clunky.

When a user guesses a word, we can instead translate the cursor position to the position it is in the main character array, and loop backwards until we hit a character that exists within the miscChars$ string, which will give us position 0 of the selected word. We can then use this to get the 4-character section of the main string and compare it to the answer.

First, we need to add our word array, select a random word to be the answer, and put the words from the array into the main string:

...
130 DIM words$(10)
140 DATA "ONTO","PLOT","THAT","WILL","WITH","SENT","ABLE","INTO","WHEN","LIFE"
150 FOR i = 1 TO 10
160 READ words$(i)
170 NEXT
180 answer$ = words$(RND * 9 + 1)
...
210 REM ***** Construct random char string *****
220 randChars$ = ""
230 FOR i = 1 TO 10
240 REM * Random number of chars between 10 and 20 *
250 randChLen = RND * 9 + 10
260 REM If mainStr$ has space, get randChLen number of characters
270 IF LEN(mainStr$) < 250 – randChLen THEN GOSUB 560:mainStr$ = mainStr$ + randChars$
280 REM If a word will fit, insert the next word
290 IF LEN(mainStr$) < 246 THEN mainStr$ = mainStr$ + words$(i) ELSE randChLen=250-LEN(mainStr$):GOSUB 560:mainStr$ = mainStr$ + randChars$:i=11
300 NEXT
310 REM If after loop finishes it’s not 250 chars long, top it up
320 IF LEN(mainStr$) < 250 THEN randChLen = 249-LEN(mainStr$):GOSUB 560:mainStr$ = mainStr$ + randChars$
...
550 REM ***** Fetch N amount of random characters *****
560 randChars$ = ""
570 FOR k = 0 TO randChLen
580 randChars$ = randChars$ + MID$(miscChars$, RND*21 + 1, 1)
590 NEXT
600 RETURN


Our output now looks like this:


Now, we need to respond when the user presses “E”, and check the user’s guess against the word using the strategy discussed above. To our code we add:

...
300 IF keyPress$="e" THEN GOSUB 1000
...
1000 REM ***** Set up variables for calculation *****
1010 ln = lineno – 19
1020 col = column + 1
1030 REM The current position of the cursor, translated to mainStr$ index
1040 mainStrCurPos = col + (25 * ln)
1050 REM mainStrWorPos will hold the position of the start of the guessed word
1060 mainStrWorPos = mainStrCurPos
1070 REM * Step back through mainStr$ until we hit a non-letter character *
1080 FOR j = mainStrCurPos TO 0 STEP -1
1090 curPosIsOnLetter = INSTR(miscChars$, MID$(mainStr$, j, 1))
1100 IF curPosIsOnLetter = 0 THEN mainStrWorPos = j ELSE j=-1
1110 NEXT
1120 REM ***** Check if game is won *****
1130 IF MID$(mainStr$, mainStrWorPos, 4) = answer$ THEN PRINT FNmoveCursor$(3,1);"You won!":gameWon = 1
1140 RETURN


Now we have a very basic game that we can win! However, it’s not very obvious to the player when the game is won; in fact, they don’t get a lot of information at all.

There’s a few more features we can add. Firstly, we can add a “likeness” message that tells the player how many letters their guess has in common with the answer. We should also only give them 4 attempts to get the answer, with the game ending after the word has been guessed or the attempts go down to zero. Our main game screen is in the bottom-left corner of the monitor, leaving some space to the right for displaying messages for the user. After printing our initial game grid while the screen width is set to 25, we can then set the width back to 90 and print some messages in the adjacent area by using our FNmoveCursor$ function with an offset column value:

REM ***** Setting up variables *****
...
msgLn = 19
msgCol = 27
attempts = 4
...
REM ***** Set up screen *****
...
WIDTH 90
PRINT FNmoveCursor$(19, 27);"Welcome to HANCO Industries (TM) Termlink"
PRINT FNmoveCursor$(20, 27);"Password Required"
PRINT FNmoveCursor$(21, 27);"Attempts Remaining: ";attempts


We also set msgLn and msgCol variables to track where the next message should be printed, and use an attempts variable in the “Attempts Remaining” message, which we will deduct from when the user makes an incorrect guess. Now, instead of just printing “You won!” in our subroutine that checks the user’s guess against the answer, we can add a GOSUB call to the following code, and make the messages a little more obvious and well-placed:

...
970 REM ***** Check if game is won *****
980 guess$ = MID$(mainStr$, mainStrWorPos, 4)
990 IF guess$ = answer$ THEN PRINT FNmoveCursor$(msgLn, msgCol);"Access Granted.";FNmoveCursor$(32, 0):END ELSE GOSUB 1020
1000 RETURN
1010 '
1020 likeness = 0
1030 FOR m = 1 TO 4 
1040 IF MID$(guess$, m, 1) = MID$(answer$, m, 1) THEN likeness = likeness + 1
1050 NEXT
1060 '
1070 attempts = attempts - 1
1080 PRINT FNmoveCursor$(21, 27);"Attempts Remaining: ";attempts
1090 PRINT FNmoveCursor$(msgLn, msgCol);"Entry Denied. Likeness: ";likeness
1100 msgLn = msgLn + 1
1110 '
1120 IF attempts = 0 THEN PRINT FNmoveCursor$(msgLn, msgCol);"Init lockout";FNmoveCursor$(32, 0):END
1130 PRINT FNmoveCursor$(lineno, column)
1140 RETURN


Our game now looks like this:


I haven’t mentioned it yet, but even though we are using the RND function to grab random numbers, the game is actually the same every single time so far. The answer, the placement of the words, and the characters in between each word which are determined by BASIC’s RND function are always the same. This is because BASIC’s pseudo-random number generator calculates the next number based on the previously generated one, and the seed is the same every time, as well as the order in which we generate the numbers for different purposes. We can alter the seed by calling RANDOMIZE with our own number - but how do we come up with that number while keeping every game unique?

I included an initial screen to my game as a solution, where a counter counts up until the user presses a key. The counter’s value is then passed to RANDOMIZE so that each game can be different:

...
240 REM *** Pre-game screen to get random seed ***
250 PRINT clear$;FNmoveCursor$(32, 0);"Press any key to start...";
260 keyPress$ = ""
270 count = 0
280 WHILE keyPress$ = ""
290 keypress$ = INKEY$
300 count = count + 1
310 WEND
320 PRINT "Starting game...";
330 RANDOMIZE count
340 answer$ = words$(RND * 9 + 1)
...


So there we have it: a basic, but fully-working hacking minigame built for the Amstrad PCW! I’ve learned a lot about BASIC along the way, but my main takeaway from this experience is that I have a much bigger appreciation for proper control flow and quality-of-life features that modern languages have today.
If you’d like to try running the full program for yourself, I have added the full source code below. Remember you will need a compiler that is compatible with Mallard BASIC!

10 REM ***** Setting up variables *****
20 maxChars = 250
30 miscChars$ = "?!:;{}[]()<>_=+@$%*#,."
40 mainStr$ = ""
50 escape$ = CHR$(27)
60 move$ = escape$+"Y"
70 clear$ = escape$+"E"
80 gameWon = 0
90 lineno = 19
100 column = 0
110 msgLn = 22
120 msgCol = 27
130 attempts = 4
140 ' 
150 REM ***** Create words array *****
160 DIM words$(10)
170 DATA "ONTO","PLOT","THAT","WILL","WITH","SENT","ABLE","INTO","WHEN","LIFE"
180 FOR i = 1 TO 10
190 READ words$(i)
200 NEXT
210 '
220 DEF FNmoveCursor$(lineno,column) = move$+CHR$(lineno+32)+CHR$(column+32)
230 '
240 REM *** Pre-game screen to get random seed ***
250 PRINT clear$;FNmoveCursor$(32, 0);"Press any key to start...";
260 keyPress$ = ""
270 count = 0
280 WHILE keyPress$ = ""
290 keypress$ = INKEY$
300 count = count + 1
310 WEND
320 PRINT "Starting game...";
330 RANDOMIZE count
340 answer$ = words$(RND * 9 + 1)
350 '
360 REM ***** Construct random char string *****
370 randChars$ = ""
380 FOR i = 1 TO 10
390 REM * Random number of chars between 10 and 20 *
400 randChLen = RND * 9 + 10
410 REM If mainStr$ has space, get randChLen number of characters
420 IF LEN(mainStr$) < 250 - randChLen THEN GOSUB 760:mainStr$ = mainStr$ + randChars$
430 REM If a word will fit, insert next word
440 IF LEN(mainStr$ < 246 THEN mainStr$ = mainStr$ + words$(i) ELSE randChLen = 250 - LEN(mainStr$):GOSUB 760:mainStr$ + randChars$:i = 11
450 NEXT
460 '
470 REM If after loop finishes it's not yet 250 chars long, top it up
480 IF LEN(mainStr$) < 250 THEN randChLen = 249 - LEN(mainStr$):GOSUB 760:mainStr$ = mainStr$ + randChars$
490 ' 
500 REM ***** Set up screen *****
510 WIDTH 25
520 PRINT clear$;
530 PRINT FNmoveCursor$(18, 0);mainStr$;
540 WIDTH 90
550 PRINT FNmoveCursor$(19, 27);"Welcome to HANCO Industries (TM) Termlink"
560 PRINT FNmoveCursor$(20, 27);"Password Required"
570 PRINT FNmoveCursor$(21, 27);"Attempts Remaining: ";attempts
580 PRINT FNmoveCursor$(19, 0);
590 '
600 REM ***** Game loop *****
610 WHILE gameWon = 0:GOSUB 620:WEND
620 keyPress$ = ""
630 WHILE keyPress$ = "":keyPress$ = INKEY$:WEND
640 GOSUB 680
650 RETURN
660 '
670 REM ***** Key press subroutine *****
680 IF keyPress$="w" AND lineno <> 19 THEN lineno = lineno - 1
690 IF keyPress$="s" AND lineno <> 28 THEN lineno = lineno + 1
700 IF keyPress$="a" AND column <> 0 THEN column = column - 1
710 IF keyPress$="d" AND column <> 24 THEN column = column + 1
720 PRINT FNmoveCursor$(lineno, column);
730 IF keyPress$="e" THEN GOSUB 840
740 RETURN 
750 '
760 REM ***** Fetch N amount of random characters *****
770 randChars$ = ""
780 FOR k = 0 TO randChLen
790 randChars$ = randChars$ + MID$(miscChars$, RND*21 + 1, 1)
800 NEXT
810 RETURN
820 '
830 REM ***** Set up variables for calculation ***** 
840 ln = lineno - 19
850 col = column + 1
860 REM The current position of the cursor, translated to mainStr$ index
870 mainStrCurPos = col + (25 * ln)
880 REM mainStrWorPos will hold the position of the start of the guessed word
890 mainStrWorPos = mainStrCurPos
900 REM * Step back through mainStr$ until we hit a non-letter character *
910 FOR j = mainStrCurPos TO 0 STEP -1
920 letter$ = MID$(mainStr$, j, 1)
930 curPosIsOnLetter = INSTR(miscChars$, letter$)
940 IF curPosIsOnLetter = 0 THEN mainStrWorPos = j ELSE j = -1
950 NEXT
960 '
970 REM ***** Check if game is won *****
980 guess$ = MID$(mainStr$, mainStrWorPos, 4)
990 IF guess$ = answer$ THEN PRINT FNmoveCursor$(msgLn, msgCol);"Access Granted.";FNmoveCursor$(32, 0):END ELSE GOSUB 1020
1000 RETURN
1010 '
1020 likeness = 0
1030 FOR m = 1 TO 4 
1040 IF MID$(guess$, m, 1) = MID$(answer$, m, 1) THEN likeness = likeness + 1
1050 NEXT
1060 '
1070 attempts = attempts - 1
1080 PRINT FNmoveCursor$(21, 27);"Attempts Remaining: ";attempts
1090 PRINT FNmoveCursor$(msgLn, msgCol);"Entry Denied. Likeness: ";likeness
1100 msgLn = msgLn + 1
1110 '
1120 IF attempts = 0 THEN PRINT FNmoveCursor$(msgLn, msgCol);"Init lockout";FNmoveCursor$(32, 0):END
1130 PRINT FNmoveCursor$(lineno, column)
1140 RETURN




Functional Optics for Modern Java - Part 6 by Magnus Smith
Magnus Smith — Thu, 12 Feb 2026 00:00:00 +0000
From Theory to Practice

Part 6 of the Functional Optics for Modern Java series

We set out noting a frustration that Java handles reading nested structures elegantly, but writing them remains painful. Over five articles, we built a response: optics for navigation, effects for error handling, and a bridge between them.

Now it’s time to see everything working together and to be honest about when to use these patterns and when simpler approaches suffice.



The Complete Toolkit

Like surgical instruments, the Focus DSL provides precision tools for navigating to exactly the right location and making targeted modifications. The Effect Path API provides the monitors: tracking what can go wrong, accumulating diagnostics, coordinating concurrent operations. Neither replaces the other. Together, they enable surgical precision on complex data structures.



For details on Focus DSL, see Part 4. For Effect Paths, see Part 5.

The Three-Layer Architecture

Higher-Kinded-J is built in layers, each serving a different audience:




  
    Layer 1 provides the mathematical foundation: higher-kinded type simulation via the Witness pattern. This is what allows generic code to work across List, Option, Either, and Future.
  
  
    Layer 2 provides the standard monad transformers (EitherT, StateT, ReaderT). In Scala libraries like Cats, these are the primary user-facing types. But in Java, EitherT is syntactically intimidating.
  
  
    Layer 3 is where most code lives. The Effect Path API wraps transformers into fluent, concrete classes. When you call Path.either(value), the library internally constructs the appropriate transformer stack. You never see Kind unless you want to.
  


This layering acknowledges a key insight: Java developers prefer fluent interfaces over type class constraints. The Effect Path API is essentially a Domain-Specific Language (DSL) for monad transformers, designed to feel like Java’s Stream API rather than Haskell’s do-notation.

The Focus DSL provides the same treatment for optics: fluent navigation without explicit optic composition. And the bridge between them (path.focus(lens).modify(fn)) enables surgical precision for data even when wrapped in effects.



The Pipeline in Action

With all our pieces in place, we have a complete expression language implementation. The Pipeline class composes four phases, each using the appropriate effect type:



The key is how effects are explicit in the types:

public Either<PipelineError, Object> run(String source, Environment env) {
    return parser.apply(source)
        .mapLeft(PipelineError::fromParseError)
        .flatMap(ast -> typeChecker.apply(ast).fold(
            errors -> Either.left(PipelineError.fromTypeErrors(errors)),
            type -> {
                Expr optimised = optimiser.apply(ast);
                Object result = interpreter.apply(optimised).apply(env);
                return Either.right(result);
            }
        ));
}


Notice how PipelineError is a sealed interface with variants for each failure mode. Pattern matching on the result gives exhaustive error handling.

Parallel Pipeline

For concurrent operations, ParallelPipeline demonstrates VTask with Scope:

List<Validated<List<TypeError>, Type>> results =
    parallelPipeline.typeCheckAllParallel(expressions, typeEnv);


The Scope API provides structured concurrency patterns:


  
    
      Joiner
      Behaviour
      Use Case
    
  
  
    
      allSucceed
      Wait for all, fail-fast on error
      Batch operations that must all complete
    
    
      anySucceed
      First success wins, cancel others
      Redundant calls, fallbacks
    
    
      accumulating
      Collect all results OR all errors
      Validation, comprehensive reporting
    
  


See VTask documentation for details.

Run the demo yourself:

./gradlew :run -PmainClass=org.higherkindedj.article6.demo.Article6Demo




Traditional Java vs Higher-Kinded-J

The patterns we’ve developed solve real problems. Here’s how they compare to traditional approaches:


  
    
      Challenge
      Traditional Java
      Higher-Kinded-J
      Details
    
  
  
    
      Deep updates
      Copy-constructor cascade (25+ lines)
      Lens composition (1 line)
      Part 1
    
    
      Null handling
      if (x != null) chains
      MaybePath makes absence explicit
      Part 5
    
    
      Error handling
      Nested try-catch pyramids
      EitherPath railway model
      Part 5
    
    
      Validation
      First-error-only
      ValidationPath accumulates ALL errors
      Part 5
    
    
      Concurrency
      CompletableFuture callbacks
      VTaskPath + Scope
      Part 5
    
  


The shape of the transformation:

// Traditional: nested, inside-out, implicit errors
if (user != null) {
    if (validator.validate(request).isValid()) {
        try {
            return paymentService.charge(user, request);
        } catch (PaymentException e) {
            // handle...
        }
    }
}

// Higher-Kinded-J: flat, top-to-bottom, explicit errors
Path.maybe(findUser(userId))
    .toEitherPath(() -> new UserNotFound(userId))
    .via(user -> validate(request))
    .via(req -> Path.tryOf(() -> paymentService.charge(user, req)))


For comprehensive patterns, see the Error Handling Journey tutorial.



For Spring Developers

If you use Spring Boot, the hkj-spring-boot-starter brings these patterns directly into your controllers. The integration is non-invasive: existing exception-based endpoints continue to work, and you can adopt functional error handling incrementally.



The Transformation

Exception-based error handling hides failure modes in implementation details. Functional error handling makes them explicit:

// Before: What can fail? Read the implementation to find out.
@GetMapping("/{id}")
public User getUser(@PathVariable String id) {
    return userService.findById(id);
}

@ExceptionHandler(UserNotFoundException.class)
public ResponseEntity<ErrorResponse> handleNotFound(UserNotFoundException ex) {
    return ResponseEntity.status(404).body(new ErrorResponse(ex.getMessage()));
}

// After: Errors are explicit in the signature. No @ExceptionHandler needed.
@GetMapping("/{id}")
public Either<DomainError, User> getUser(@PathVariable String id) {
    return userService.findById(id);
    // Right(user) → HTTP 200
    // Left(UserNotFoundError) → HTTP 404 (by naming convention)
}


What the Starter Provides


  
    
      Capability
      What It Does
    
  
  
    
      Zero-config
      Add dependency, return Either/Validated from controllers
    
    
      Auto status mapping
      Error class names map to HTTP status codes
    
    
      Error accumulation
      Validated, User> returns ALL validation errors
    
    
      Async support
      CompletableFuturePath for non-blocking operations
    
    
      Actuator metrics
      Track success/error rates, latency percentiles
    
    
      Security integration
      ValidatedUserDetailsService for functional authentication
    
    
      JSON serialisation
      Configurable formats (TAGGED, UNWRAPPED, DIRECT)
    
  


Validation That Reports ALL Errors

Traditional validation stops at the first error. Users fix one problem, submit again, discover another. With Validated, they see everything at once:

@PostMapping
public Validated<List<ValidationError>, User> createUser(@RequestBody UserRequest request) {
    return userService.validateAndCreate(request);
    // Valid(user) → HTTP 200 with user JSON
    // Invalid(errors) → HTTP 400 with ALL validation errors:
    // [{"field": "email", "message": "Invalid format"},
    //  {"field": "age", "message": "Must be positive"},
    //  {"field": "name", "message": "Required"}]
}


The framework accumulates errors automatically. No more “whack-a-mole” validation for your users.

Production Monitoring

The Actuator integration tracks functional operations in production:

curl http://localhost:8080/actuator/hkj


{
  "metrics": {
    "eitherPath": {
      "successCount": 1547,
      "errorCount": 123,
      "successRate": 0.926
    },
    "validationPath": {
      "validCount": 892,
      "invalidCount": 45,
      "validRate": 0.952
    }
  }
}


Try it now:

./gradlew :hkj-spring:example:bootRun


See the complete Migration Guide for step-by-step patterns: converting exceptions to Either, validation to Validated, and async operations to EitherT.



Working with Third-Party Types

Your domain model uses @GenerateLenses, but what about JDK classes like LocalDate or library types like Jackson’s JsonNode? You can’t annotate code you don’t own.

@ImportOptics solves this. Add it to a package-info.java:

@ImportOptics({
    java.time.LocalDate.class,
    java.time.LocalTime.class
})
package com.myapp.optics;

import org.higherkindedj.optics.annotations.ImportOptics;


The processor analyses each type and generates appropriate optics:



Now you can compose across the boundary between your types and external types:

// Your record
@GenerateLenses
record Order(String id, Customer customer, LocalDate orderDate) {}

// Composition: orderDate lens → year lens
var nextYearOrder = OrderLenses.orderDate()
    .andThen(LocalDateLenses.year())
    .modify(y -> y + 1, order);


Supported External Types


  
    
      Pattern
      Example Types
      Generated Optics
    
  
  
    
      Records
      Third-party records
      Lenses via constructor
    
    
      Sealed interfaces
      Sum types
      Prisms for each variant
    
    
      Enums
      Status codes
      Prisms for each constant
    
    
      Wither classes
      LocalDate, LocalTime
      Lenses via withX() methods
    
    
      Builders
      JOOQ, Protobuf
      Via spec interfaces
    
  


For complex cases like Jackson’s JsonNode (which uses predicates rather than sealed types), see the Optics for External Types guide.



Migration Path

Adoption can be incremental. Start small, prove the pattern, then expand:


  Annotate one type - Add @GenerateLenses to a domain record
  Replace one cascade - Convert a deep update to lens composition
  Wrap exceptions - Use TryPath for one exception-throwing call
  Accumulate validation - Switch one validator from first-error to ValidationPath
  Import external types - Add @ImportOptics for JDK types you frequently modify
  Enable navigators - Add @GenerateFocus(generateNavigators = true) for fluent navigation


Each step is independent. You don’t need to convert everything at once.

For complete guidance, see:


  Focus DSL Tutorial
  Error Handling Journey
  VTask Journey




Design Patterns

Several patterns crystallised through the series:


  
    
      Pattern
      Description
      When to Use
    
  
  
    
      Focus-Path-First
      Annotate types with @GenerateFocus, express operations as paths
      Any nested domain model
    
    
      Effect Stratification
      Different phases use different effects (Either for parsing, Validated for checking)
      Pipelines with distinct stages
    
    
      Paths as Configuration
      Store paths as values, compose at runtime
      Configurable queries, reporting
    
    
      Validated for Users
      Accumulate all errors for user-facing validation
      Forms, API requests
    
  


Focus-Path-First in Practice

The benefit of Focus-Path-First design is that adding new operations requires zero changes to your types. Define paths once, reuse everywhere:

@GenerateFocus(generateNavigators = true)
record Department(String name, List<Employee> employees, Employee manager) {}

// Define paths once
TraversalPath<Department, Employee> allEmployees = DepartmentFocus.employees().each();

// Use for any operation: queries, updates, validations
List<String> names = allEmployees.getAll(dept).stream()
    .map(Employee::name).toList();

Department withRaises = allEmployees.modifyAll(
    emp -> emp.withSalary(emp.salary().multiply(1.1)), dept);

ValidationPath<List<Error>, Department> validated =
    allEmployees.toValidationPath(emp -> validateEmployee(emp), dept);


Effect Stratification

Different phases of processing need different effects. Making this explicit in types communicates intent:



When you see Validated in a signature, you know errors will accumulate. When you see State, you know context is being threaded. The types communicate intent.

For detailed examples of each pattern, see the Focus DSL documentation.



When to Keep It Simple

Honesty requires acknowledging when these abstractions aren’t worth it:

Use Focus DSL when:
- Structures are three or more levels deep
- The same path is accessed in multiple places
- Code is read more often than executed

Keep it simple when:
- Structures are shallow (one or two levels)
- Transformations are one-off
- Team is novice (learning curve is real)
- Performance-critical inner loops (measure first)

Rich Hickey’s distinction applies: optics are simple (few concepts, composable) but not always easy (there’s a learning curve). Know when the abstraction pays for itself.

That said, the learning curve argument is weakening. Five years ago, functional patterns in Java felt like fighting the language. Today, with records, sealed interfaces, pattern matching, and virtual threads, Java actively supports these idioms. The fact that Higher-Kinded-J is not just possible but practical in modern Java shows how far the language has come—and how well these patterns align with Java’s direction.



Where We Stand

Higher-Kinded-J joins a family of optics libraries across languages. Haskell’s lens, Scala’s Monocle, and Kotlin’s Arrow all provide mature, well-tested implementations of optics. Each is idiomatic to its language.

What Higher-Kinded-J brings is optics designed for Java from the ground up—not a port of Haskell idioms, but an implementation that embraces records, sealed interfaces, annotation processing, and virtual threads as first-class features.

Java First, Not an Imitation

Many functional libraries in Java are ports of Haskell or Scala libraries, bringing foreign idioms that feel awkward. Higher-Kinded-J takes a different approach: Java first.

We adopt good ideas from other languages, but Higher-Kinded-J is designed for modern Java:


  Records and sealed interfaces are first-class citizens, not afterthoughts
  Pattern matching complements our optics rather than competing with them
  Annotation processing generates idiomatic Java, not Haskell-in-Java
  The Focus DSL uses Java’s method chaining naturally
  Virtual threads provide concurrency without callback complexity


The goal isn’t to make Java feel like Haskell. It’s to give Java developers powerful abstractions while respecting Java’s idioms. When you use Higher-Kinded-J, you’re writing modern, expressive, functional Java.

Aligned with Java’s Future

Higher-Kinded-J already embraces structured concurrency through VTask and Scope. But Java’s roadmap suggests the alignment will only deepen:

Value Types (Project Valhalla) will let classes opt out of identity semantics. Optic wrappers like FocusPath could become value types—zero allocation overhead, compared by structure rather than identity. The performance gap between direct field access and optic-based access could effectively disappear.

Carrier Classes extend the record model with derived state, mutable fields, and inheritance—while preserving pattern matching and with expressions. Since @GenerateLenses already works with records, extending support to carrier classes is natural. The with expressions in carrier classes align perfectly with lens-based modification.

Lazy Constants (JEP 526, previewing in JDK 26) provide thread-safe deferred initialization with JIT-level constant folding. Combined with optics, this could mean paths that compose eagerly but execute lazily—defining a deep traversal costs nothing until you actually use it, and once resolved, the JIT can treat it as a true constant.

The pattern is clear: Java is evolving toward richer data-oriented features, and Higher-Kinded-J is positioned to take advantage of each advance. Learning optics and effects today is an investment that becomes more valuable as Java matures.



Completing the Picture

Java 25 gives us records, sealed interfaces, and pattern matching for reading data elegantly. Higher-Kinded-J completes the picture:


  Focus DSL provides the write side that pattern matching lacks
  Effect Path API provides composable error handling
  @ImportOptics extends optics to types you don’t own


// Pattern matching reads
if (company instanceof Company(_, var departments)) { ... }

// Focus DSL writes
Company updated = CompanyFocus.departments().each()
    .employees().each()
    .salary()
    .modifyAll(s -> s.multiply(1.1), company);

// Effect Path API validates
ValidationPath<List<Error>, Company> validated =
    Path.valid(company, Semigroups.list())
        .via(c -> validateAllEmployees(c));


The two APIs work in harmony. Together, they provide a complete functional programming toolkit for Java.

The Composability Principle

A theme running through this series is composition. Focus paths compose with via(). Collection navigation composes with each(). Effects compose via type classes. Each composition multiplies capability without multiplying complexity.

This is the result of principled abstraction. When your building blocks follow laws (lens laws, functor laws, applicative laws), composition just works. You don’t verify each combination manually; the laws guarantee sensible behaviour.

Eric Normand captures this in his work on data-oriented programming: build with small pieces that combine predictably. Rich Hickey emphasises simplicity over ease: simple things compose, easy things often don’t. The Focus DSL and Effect Path API embody these principles in Java.



Further Reading

This Series


  Part 1: The Immutability Gap - The problem we set out to solve
  Part 2: Optics Fundamentals - Lenses, prisms, traversals
  Part 3: AST with Basic Optics - Applying optics to a real domain
  Part 4: Traversals and Pattern Rewrites - The Focus DSL
  Part 5: The Effect Path API - Railway-style error handling


Higher-Kinded-J Documentation


  Focus DSL Guide - Complete Focus DSL reference
  Effect Path API - Effect types and patterns
  VTask and Scope - Virtual thread concurrency
  Optics for External Types - @ImportOptics for JDK and library types
  Spring Boot Integration - Using HKJ with Spring
  Migration Guide - From exceptions to functional errors
  Tutorials home - Learn Higher-Kinded-J following tutorials


Background


  Brian Goetz, Data-Oriented Programming in Java - DOP principles for Java
  Rich Hickey, Simple Made Easy - Simple vs easy
  Edward Kmett’s lens library - The Haskell gold standard
  Chris Penner, Optics by Example - A major influence on Higher-Kinded-J; the best practical guide to optics



  “Optics are a family of inter-composable combinators for building bidirectional data transformations.”
— Chris Penner, Optics by Example




This concludes the Functional Optics for Modern Java series. Thank you for reading.



Functional Optics for Modern Java - Part 5 by Magnus Smith
Magnus Smith — Mon, 09 Feb 2026 00:00:00 +0000
The Effect Path API: Railway-Style Error Handling

Part 5 of the Functional Optics for Modern Java series

In Part 1 and Part 2, 
we established why optics matter and how they work. In Part 3,
we built our expression language AST and applied basic optics using lenses for field access and prisms for variant matching. Last time in Part 4, we built traversals that visit every node in our expression tree. We implemented constant folding, identity simplification, and dead branch elimination. But all our transformations were pure: they took an expression and returned a new expression, with no side effects.

Real compilers and interpreters need more. Type checking should report all errors, not just the first one. Interpretation must track variable bindings as it descends through the tree. These are effects, and they change everything about how we should structure our code.

The Effect Path API from Higher-Kinded-J provides a fluent interface for computations that might fail, accumulate errors, or require deferred execution. This is the practical face of effect polymorphism, making powerful abstractions accessible through an ergonomic API.

Running the Examples



Article Code

All code examples from this article have runnable demos:


  EffectPathDemo: Demonstrates the Effect Path API.
  EffectPolymorphicDemo: Demonstrates effect-polymorphic optics using modifyF with different Higher-Kinded-J effects.
  InterpreterDemo: Demonstrates expression interpretation using Higher-Kinded-J’s State monad.
  ParallelTypeCheckerDemo: Constant folding, identity simplification, and cascading optimisation using traversal-based passes.
  TypeCheckerDemo: Demonstrates parallel type checking using VTask and Scope.
  VTaskPathDemo: Demonstrates VTaskPath for virtual thread-based concurrency.




Effects as Assembly Line Quality Control

Imagine you’re running an assembly line in a factory. Each station performs an operation on a product. At each station the outcome of the operation can follow a different path:


  
    MaybePath: A station that might produce nothing (parts ran out). The line continues, but there’s no product to pass on.
  
  
    EitherPath: A station with a fault inspector. If the product fails inspection, it’s immediately diverted to the rejection bin with a tag explaining why. The line stops for that product.
  
  
    ValidationPath: A multi-point checklist inspection station. Every defect is recorded on a checklist, even if there are multiple problems. The product is only rejected after all checks are complete, and the checklist shows everything that needs fixing.
  
  
    TryPath: A station that might malfunction. If it throws a wrench (literally), we catch the exception and treat it as data rather than letting it crash the whole factory.
  
  
    IOPath: A station that doesn’t run until you press the “GO” button. The work is planned and sequenced, but nothing actually happens until you explicitly start it.
  
  
    VTaskPath: A station that spawns lightweight workers for each task, coordinating them efficiently. Work happens concurrently on virtual threads, and the station can wait for all workers, race them, or collect their results.
  


The Effect Path API gives you these different assembly line configurations, letting you choose the right error handling strategy for each situation.



The Railway Model

Effect Paths follow the “railway” metaphor that was popularised by Scott Wlaschin. Values travel along tracks, and computations can switch between success and failure:



The big idea here is that instead of throwing exceptions or returning null, Effect Paths make failure explicit in the type system. A MaybePath might contain a string or might be empty. An EitherPath contains either an error or a user. A ValidationPath, Form> contains either accumulated errors or a valid form.

Ensuring failure is explicit in the type system has practical benefits:


  Compiler enforcement: We cannot ignore a potential failure; the types require handling it
  Composition: Effect Paths chain naturally with map, via, and zipWith
  Flexibility: Choose fail-fast (EitherPath) or accumulating (ValidationPath) behaviour




The Effect Path Types

Higher-Kinded-J provides many Effect Path types; here are six core types, each suited to different use cases:


  
    
      Effect Path
      Contains
      Use Case
    
  
  
    
      MaybePath
      Value or nothing
      Optional data, silent failures
    
    
      EitherPath
      Error or value
      Fail-fast error handling
    
    
      TryPath
      Exception or value
      Wrapping throwing code
    
    
      ValidationPath
      Errors or value
      Accumulating all problems
    
    
      IOPath
      Deferred computation
      Side effects, resource management
    
    
      VTaskPath
      Virtual thread computation
      Concurrent operations, parallelism
    
  


Creating Effect Paths

The Path factory class provides convenient constructors:

import org.higherkindedj.hkt.effect.Path;

// MaybePath
MaybePath<String> present = Path.just("hello");
MaybePath<String> absent = Path.nothing();
MaybePath<String> nullable = Path.maybe(possiblyNullValue);

// EitherPath
EitherPath<String, Integer> success = Path.right(42);
EitherPath<String, Integer> failure = Path.left("Something went wrong");

// TryPath
TryPath<Integer> parsed = Path.tryOf(() -> Integer.parseInt(input));
TryPath<Integer> safe = Path.success(42);
TryPath<Integer> failed = Path.failure(new IllegalArgumentException("bad input"));

// ValidationPath (requires a Semigroup for error accumulation)
ValidationPath<List<Error>, User> valid = Path.valid(user, Semigroups.list());
ValidationPath<List<Error>, User> invalid = Path.invalid(errors, Semigroups.list());

// IOPath (deferred execution)
IOPath<String> readFile = Path.io(() -> Files.readString(path));
IOPath<Unit> sideEffect = Path.ioRunnable(() -> System.out.println("Hello"));




MaybePath: Optional Values

MaybePath represents a computation that might not produce a value. It wraps Higher-Kinded-J’s Maybe type.


MaybePath<String> greeting = Path.just("Hello")
    .map(String::toUpperCase)
    .filter(s -> s.length() > 3)
    .map(s -> s + "!");

// Extract the result
String result = greeting.getOrElse("default");  // "HELLO!"

// Or pattern match
greeting.run().fold(
    () -> System.out.println("No value"),
    value -> System.out.println("Got: " + value)
);


Chaining with via

The via method chains dependent computations:

MaybePath<User> userPath = Path.just(userId)
    .via(id -> lookupUser(id))        // Returns MaybePath
    .via(user -> validateUser(user)); // Returns MaybePath

// If any step returns nothing, the chain short-circuits


Converting to Other Effect Paths

MaybePath<String> maybe = Path.just("hello");

// To EitherPath (provide error for empty case)
EitherPath<String, String> either = maybe.toEitherPath("Value was missing");

// To TryPath (provide exception for empty case)
TryPath<String> tryPath = maybe.toTryPath(() -> new NoSuchElementException());

// To ValidationPath (provide error and semigroup)
ValidationPath<List<Error>, String> validated =
    maybe.toValidationPath(List.of(new Error("missing")), Semigroups.list());




EitherPath: Fail-Fast Error Handling

EitherPath represents a computation that either succeeds with a value or fails with a typed error. Unlike exceptions, the error type is explicit in the signature.

EitherPath<String, Integer> divide(int a, int b) {
    if (b == 0) {
        return Path.left("Division by zero");
    }
    return Path.right(a / b);
}

EitherPath<String, Integer> result = divide(10, 2)
    .map(n -> n * 2)
    .via(n -> divide(n, 3));

// Pattern match on the result
result.run().fold(
    error -> System.out.println("Error: " + error),
    value -> System.out.println("Result: " + value)
);


Error Transformation

// Map over the error type
EitherPath<Integer, String> withErrorCode =
    Path.<String, String>left("Not found")
        .mapError(msg -> 404);

// Recover from errors
EitherPath<String, Integer> recovered =
    Path.<String, Integer>left("Error")
        .recover(error -> -1);  // Replace error with default value

// Recover with another EitherPath
EitherPath<String, Integer> fallback =
    Path.<String, Integer>left("Primary failed")
        .recoverWith(error -> fetchFromBackup());




ValidationPath: Error Accumulation

ValidationPath is the key type for comprehensive error reporting. Unlike EitherPath, which stops at the first error, ValidationPath collects all errors.



// Define a semigroup constant to avoid repetition
private static final Semigroup<List<String>> ERRORS = Semigroups.list();

// Create validators that return ValidationPath
ValidationPath<List<String>, String> validateName(String name) {
    if (name == null || name.isBlank()) {
        return Path.invalid(List.of("Name is required"), ERRORS);
    }
    return Path.valid(name.trim(), ERRORS);
}

ValidationPath<List<String>, Integer> validateAge(int age) {
    if (age < 0) {
        return Path.invalid(List.of("Age cannot be negative"), ERRORS);
    }
    if (age > 150) {
        return Path.invalid(List.of("Age seems unrealistic"), ERRORS);
    }
    return Path.valid(age, ERRORS);
}

ValidationPath<List<String>, String> validateEmail(String email) {
    if (!email.contains("@")) {
        return Path.invalid(List.of("Invalid email format"), ERRORS);
    }
    return Path.valid(email, ERRORS);
}


Each validator returns a single error wrapped in a List because ValidationPath needs a Semigroup to combine errors from multiple validations. When two validations both fail, their List errors are concatenated.

Combining Validations: Short-Circuit vs Accumulating

ValidationPath offers two composition modes:

Short-circuit (via via): Stops at first error, like EitherPath

// Sequential: second validation only runs if first succeeds
ValidationPath<List<String>, User> sequential = validateName(name)
    .via(n -> validateAge(age).map(a -> new User(n, a, null)));


Accumulating (via zipWithAccum): Collects all errors

// All validations run independently, errors accumulate
ValidationPath<List<String>, User> accumulated = validateName(name)
                .zipWith3Accum(
                        validateAge(age),
                        validateEmail(email),
                        (n, a, e) -> new User(n, a, e)
                );


For two validations, use zipWithAccum:

// Two-field example
record Contact(String name, String email) {}

ValidationPath<List<String>, Contact> contact = validateName(name)
    .zipWithAccum(validateEmail(email), Contact::new);


For three, use zipWith3Accum as shown above.

The Semigroup Requirement

ValidationPath requires a Semigroup to combine errors. Higher-Kinded-J provides common semigroups:

import org.higherkindedj.hkt.Semigroups;

// List semigroup: concatenates lists
Semigroup<List<String>> listSemigroup = Semigroups.list();

// String semigroup: concatenates strings
Semigroup<String> stringSemigroup = Semigroups.string();

// Custom semigroup
Semigroup<ErrorReport> reportSemigroup = (a, b) -> a.merge(b);




TryPath: Exception Handling

TryPath wraps computations that might throw exceptions, converting them to values:


// Wrap throwing code
TryPath<Integer> parsed = Path.tryOf(() -> Integer.parseInt(userInput));

// Chain operations safely
TryPath<Double> calculation = Path.tryOf(() -> Integer.parseInt(a))
    .map(x -> x * 2.0)
    .via(x -> Path.tryOf(() -> x / Double.parseDouble(b)));

// Handle the result
calculation.run().fold(
    value -> System.out.println("Result: " + value),
    ex -> System.out.println("Error: " + ex.getMessage())
);


Recovery from Exceptions

TryPath<Integer> withDefault = Path.tryOf(() -> Integer.parseInt(input))
    .recover(ex -> -1);  // Use -1 on parse failure

TryPath<Integer> withFallback = Path.tryOf(() -> fetchFromPrimary())
    .recoverWith(ex -> Path.tryOf(() -> fetchFromBackup()));




IOPath: Deferred Side Effects

IOPath represents a computation that will be executed later. Nothing happens until you call unsafeRun() or runSafe().


// Define computations without executing them
IOPath<String> readConfig = Path.io(() -> Files.readString(configPath));
IOPath<Unit> writeLog = Path.ioRunnable(() -> logger.info("Operation complete"));

// Compose deferred computations
IOPath<Config> loadConfig = readConfig
    .map(json -> parseJson(json))
    .via(parsed -> validateConfig(parsed));

// Execute when ready
Config config = loadConfig.unsafeRun();  // Throws on error
Try<Config> safe = loadConfig.runSafe(); // Captures exceptions


Resource Management

IOPath provides safe resource handling with bracket and withResource:

// Bracket pattern: acquire, use, release
IOPath<String> content = IOPath.bracket(
    () -> Files.newBufferedReader(path),  // Acquire
    reader -> reader.lines().collect(Collectors.joining("\n")),  // Use
    reader -> reader.close()  // Release (always runs)
);

// For AutoCloseable resources
IOPath<String> simpler = IOPath.withResource(
    () -> Files.newBufferedReader(path),
    reader -> reader.lines().collect(Collectors.joining("\n"))
);


Parallel Execution and Retry

// Run two computations in parallel
IOPath<UserProfile> profile = fetchUser.parZipWith(
    fetchOrders,
    (user, orders) -> new UserProfile(user, orders)
);

// Retry with exponential backoff
IOPath<String> resilient = Path.io(() -> httpClient.get(url))
    .retry(3, Duration.ofMillis(100));  // 3 attempts, 100ms initial delay




VTaskPath: Virtual Thread Concurrency

VTaskPath represents a computation that runs on Java’s virtual threads. It brings the lightweight concurrency of Project Loom to the Effect Path API, letting you write simple blocking code that scales to millions of concurrent operations.


// Create VTaskPaths
VTaskPath<String> fetchUser = Path.vtask(() -> userService.get(userId));
VTaskPath<String> fetchOrders = Path.vtask(() -> orderService.list(userId));

// Pure value (no computation)
VTaskPath<Integer> pure = Path.vtaskPure(42);

// Immediate failure
VTaskPath<String> failed = Path.vtaskFail(new IOException("Network error"));

// From a Runnable
VTaskPath<Unit> logAction = Path.vtaskExec(() -> logger.info("Starting..."));


Execution Model

Unlike IOPath, which runs on the caller’s thread, VTaskPath executes on virtual threads managed by the JVM. Virtual threads consume mere kilobytes of memory (versus megabytes for platform threads), enabling millions of concurrent tasks.



VTaskPath<Integer> task = Path.vtask(() -> expensiveComputation());

// Three ways to execute
Integer result = task.run();           // Blocks, may throw
Try<Integer> safe = task.runSafe();    // Captures exceptions in Try
CompletableFuture<Integer> future = task.runAsync();  // Non-blocking


Composition

VTaskPath chains with the same map and via patterns as other Effect Paths:

VTaskPath<Dashboard> dashboard = Path.vtask(() -> fetchUser(id))
    .map(user -> user.preferences())
    .via(prefs -> Path.vtask(() -> buildDashboard(prefs)));


Parallel Execution with Par

The Par utility provides combinators for running VTasks concurrently:

import org.higherkindedj.hkt.vtask.Par;

// Combine two tasks in parallel
VTask<UserProfile> profile = Par.map2(
    VTask.of(() -> fetchUser(id)),
    VTask.of(() -> fetchOrders(id)),
    (user, orders) -> new UserProfile(user, orders)
);

// Execute a list of tasks in parallel
List<VTask<Integer>> tasks = ids.stream()
    .map(id -> VTask.of(() -> process(id)))
    .toList();
VTask<List<Integer>> allResults = Par.all(tasks);

// Race: first successful result wins
VTask<String> fastest = Par.race(List.of(
    VTask.of(() -> fetchFromMirror1()),
    VTask.of(() -> fetchFromMirror2())
));


Structured Concurrency with Scope

For more control over concurrent operations, use Scope. The three joiners determine how concurrent results are combined:



import org.higherkindedj.hkt.vtask.Scope;

// Wait for all tasks to succeed
VTask<List<String>> all = Scope.<String>allSucceed()
    .fork(VTask.of(() -> fetchA()))
    .fork(VTask.of(() -> fetchB()))
    .fork(VTask.of(() -> fetchC()))
    .timeout(Duration.ofSeconds(5))
    .join();

// First success wins, cancel others
VTask<String> any = Scope.<String>anySucceed()
    .fork(VTask.of(() -> fetchFromPrimary()))
    .fork(VTask.of(() -> fetchFromBackup()))
    .join();

// Accumulate errors (like ValidationPath, but concurrent)
VTask<Validated<List<Error>, List<String>>> validated =
    Scope.<String>accumulating(Error::from)
        .fork(validateField1())
        .fork(validateField2())
        .fork(validateField3())
        .join();


Error Handling

// Replace error with a default value
VTaskPath<Config> withDefault = Path.vtask(() -> loadConfig())
                .handleError(ex -> Config.defaults());

// Or try a fallback task instead
VTaskPath<Config> withFallback = Path.vtask(() -> loadConfig())
        .handleErrorWith(ex -> Path.vtask(() -> loadFallbackConfig()));


Timeouts

VTaskPath<Data> withTimeout = Path.vtask(() -> slowOperation())
    .timeout(Duration.ofSeconds(5));


When to Use VTaskPath vs IOPath


  
    
      Aspect
      VTaskPath
      IOPath
    
  
  
    
      Thread model
      Virtual threads
      Caller’s thread
    
    
      Parallelism
      Built-in via Par, Scope
      Manual composition
    
    
      Structured concurrency
      Yes, with Scope
      No
    
    
      Best for
      I/O-bound concurrent work
      Single-threaded effects
    
  


Choose VTaskPath when you need lightweight concurrency at scale. Choose IOPath when single-threaded execution is sufficient or when you need explicit control over which thread runs the computation.



Bridging Focus Paths and Effect Paths

The Focus DSL (FocusPath, AffinePath, TraversalPath) integrates seamlessly with Effect Paths. This bridge is where navigation meets computation. For a complete reference of all bridge methods, see the Focus-Effect Integration Guide.



From Focus Paths to Effect Paths

// FocusPath to MaybePath (always succeeds since FocusPath has exactly one focus)
FocusPath<User, String> namePath = UserFocus.name();
MaybePath<String> name = namePath.toMaybePath(user);  // Always Just(value)

// AffinePath to MaybePath (may be empty)
AffinePath<User, String> nicknamePath = UserFocus.nickname();
MaybePath<String> nickname = nicknamePath.toMaybePath(user);  // Just or Nothing

// AffinePath to EitherPath (provide error for missing case)
EitherPath<String, String> nicknameOrError =
    nicknamePath.toEitherPath(user, "No nickname set");


Applying Focus Paths Within Effect Contexts

Effect Paths have a focus method that applies a FocusPath:

// Start with an Effect Path containing a User
EitherPath<Error, User> userPath = fetchUser(userId);

// Focus on a field within the effect context
EitherPath<Error, String> emailPath = userPath.focus(UserFocus.email());
// Equivalent to: userPath.map(user -> UserFocus.email().get(user))

// Chain multiple focuses
EitherPath<Error, String> city = userPath
    .focus(UserFocus.address())
    .focus(AddressFocus.city());


For AffinePath (which might not find a value), provide an error:

MaybePath<User> maybeUser = Path.just(user);
MaybePath<String> nickname = maybeUser.focus(
    UserFocus.nickname()  // AffinePath for optional field
);
// Returns Nothing if nickname is absent

EitherPath<String, User> eitherUser = Path.right(user);
EitherPath<String, String> nickname = eitherUser.focus(
    UserFocus.nickname(),
    "User has no nickname"  // Error if absent
);




Type Checking with ValidationPath

Let’s apply the Effect Path API to our expression language. Type checking is a perfect use case for ValidationPath: we want to report all type errors, not just the first one.

Defining Types and Errors

public enum Type { INT, BOOL, STRING }

public record TypeError(String message) {}


The Type Checker

public final class ExprTypeChecker {

    private static final Semigroup<List<TypeError>> ERRORS = Semigroups.list();

    public static ValidationPath<List<TypeError>, Type> typeCheck(Expr expr, TypeEnv env) {
        return switch (expr) {
            case Literal(var value) -> typeCheckLiteral(value);
            case Variable(var name) -> typeCheckVariable(name, env);
            case Binary(var left, var op, var right) -> typeCheckBinary(left, op, right, env);
            case Conditional(var cond, var then_, var else_) ->
                typeCheckConditional(cond, then_, else_, env);
        };
    }

    private static ValidationPath<List<TypeError>, Type> typeCheckLiteral(Object value) {
        return switch (value) {
            case Integer _ -> Path.valid(Type.INT, ERRORS);
            case Boolean _ -> Path.valid(Type.BOOL, ERRORS);
            case String _ -> Path.valid(Type.STRING, ERRORS);
            default -> Path.invalid(
                List.of(new TypeError("Unknown literal type: " + value.getClass().getSimpleName())),
                ERRORS
            );
        };
    }

    private static ValidationPath<List<TypeError>, Type> typeCheckVariable(String name, TypeEnv env) {
        return env.lookup(name)
            .map(type -> Path.valid(type, ERRORS))
            .orElseGet(() -> Path.invalid(
                List.of(new TypeError("Undefined variable: " + name)),
                ERRORS
            ));
    }

    private static ValidationPath<List<TypeError>, Type> typeCheckBinary(
            Expr left, BinaryOp op, Expr right, TypeEnv env) {
        // Use zipWithAccum to accumulate errors from both operands
        return typeCheck(left, env)
            .zipWithAccum(typeCheck(right, env), (lt, rt) -> checkBinaryTypes(op, lt, rt))
            .via(result -> result);  // Flatten nested validation
    }

    private static ValidationPath<List<TypeError>, Type> typeCheckConditional(
            Expr cond, Expr then_, Expr else_, TypeEnv env) {
        // Accumulate errors from all three sub-expressions
        return typeCheck(cond, env)
            .zipWith3Accum(
                typeCheck(then_, env),
                typeCheck(else_, env),
                ExprTypeChecker::checkConditionalTypes
            )
            .via(result -> result);
    }

    private static ValidationPath<List<TypeError>, Type> checkBinaryTypes(
            BinaryOp op, Type left, Type right) {
        return switch (op) {
            case ADD, SUB, MUL, DIV -> {
                if (left == Type.INT && right == Type.INT) {
                    yield Path.valid(Type.INT, ERRORS);
                }
                yield Path.invalid(List.of(new TypeError(
                    "Arithmetic operator '%s' requires INT operands, got %s and %s"
                        .formatted((op.symbol(), left, right)
                )), ERRORS);
            }
            case AND, OR -> {
                if (left == Type.BOOL && right == Type.BOOL) {
                    yield Path.valid(Type.BOOL, ERRORS);
                }
                yield Path.invalid(List.of(new TypeError(
                    "Logical operator '%s' requires BOOL operands, got %s and %s"
                        .formatted(op.symbol(), left, right)
                )), ERRORS);
            }
            case EQ, NE -> {
                if (left == right) {
                    yield Path.valid(Type.BOOL, ERRORS);
                }
                yield Path.invalid(List.of(new TypeError(
                        "Equality operator '%s' requires matching types, got %s and %s"
                                .formatted(op.symbol(), left, right)
                )), ERRORS);
            }
            case LT, LE, GT, GE -> {
                if (left == Type.INT && right == Type.INT) {
                    yield Path.valid(Type.BOOL, ERRORS);
                }
                yield Path.invalid(List.of(new TypeError(
                    "Comparison operator '%s' requires INT operands, got %s and %s"
                        .formatted(op.symbol(), left, right)
                )), ERRORS);
            }
        };
    }

    private static ValidationPath<List<TypeError>, Type> checkConditionalTypes(
            Type cond, Type then_, Type else_) {
        var condCheck = (cond == Type.BOOL)
                ? Path.valid(cond, ERRORS)
                : Path.invalid(List.of(new TypeError("Condition must be BOOL, got " + cond)), ERRORS);

        var branchCheck = (then_ == else_)
                ? Path.valid(then_, ERRORS)
                : Path.invalid(List.of(new TypeError(
                "Branches must have same type, got %s and %s".formatted(then_, else_))), ERRORS);

        // Accumulate errors from both checks using the applicative nature of ValidationPath
        return condCheck.zipWithAccum(branchCheck, (c, t) -> t);
    }
}


Running the Type Checker

// Expression with multiple errors: (1 + true) * (false && 42)
Expr expr = new Binary(
    new Binary(new Literal(1), BinaryOp.ADD, new Literal(true)),
    BinaryOp.MUL,
    new Binary(new Literal(false), BinaryOp.AND, new Literal(42))
);

ValidationPath<List<TypeError>, Type> result = ExprTypeChecker.typeCheck(expr, TypeEnv.empty());

result.run().fold(
    errors -> {
        System.out.println("Type errors:");
        for (TypeError error : errors) {
            System.out.println("  - " + error.message());
        }
    },
    type -> System.out.println("Type: " + type)
);


Output:

Type errors:
  - Arithmetic operator '+' requires INT operands, got INT and BOOL
  - Logical operator '&&' requires BOOL operands, got BOOL and INT


Both errors are reported in a single pass. The user can fix them both at once.



Understanding the Underlying Abstractions

The Effect Path API is built on Higher-Kinded-J’s type class hierarchy. Understanding these abstractions helps when you need maximum flexibility.

The modifyF Operation

Every optic supports modifyF, which generalises modification to work with any effect:

public interface Traversal<S, A> {
    <F extends WitnessArity<TypeArity.Unary>> Kind<F, S> modifyF(
        Function<A, Kind<F, A>> f,
        S source,
        Applicative<F> applicative
    );
}


The Applicative parameter provides:


  of(a): Wrap a pure value in the effect
  map2(fa, fb, combine): Combine two effectful values


With just these operations, we can sequence independent computations while accumulating their effects.

Effect Path Types as Kind Wrappers

Each Effect Path type wraps a corresponding Kind:

// MaybePath wraps Kind
MaybePath<String> maybePath = Path.just("hello");
Maybe<String> underlying = maybePath.run();

// EitherPath wraps Kind, A>
EitherPath<String, Integer> eitherPath = Path.right(42);
Either<String, Integer> underlying = eitherPath.run();


The Effect Path API provides ergonomic methods that delegate to these underlying types.

When to Use modifyF Directly

For most use cases, the Effect Path API suffices. Use modifyF directly when:


  You’re building reusable library code
  You need to work with custom effect types
  You want maximum composability with optics


// Using modifyF directly with a traversal
TraversalPath<Company, Employee> allEmployees = CompanyFocus
    .departments().each()
    .employees().each();

Kind<ValidatedKind.Witness<List<Error>>, Company> result = allEmployees.modifyF(
    emp -> validateEmployee(emp),
    company,
    ValidatedApplicative.instance(Semigroups.list())
);




Effect Path API vs modifyF: Choosing Your Level

Higher-Kinded-J provides two levels of abstraction:


  
    
      Level
      API
      Best For
    
  
  
    
      High
      Effect Path API
      Most application code, clear intent
    
    
      Low
      modifyF with Kind
      Libraries, custom effects, maximum flexibility
    
  


High-Level: Effect Path API

// Clear, fluent, discoverable
ValidationPath<List<String>, User> validated = validateName(name)
                .zipWith3Accum(validateAge(age), validateEmail(email), User::new);


Low-Level: modifyF with Applicative

// Maximum control, composable with any optic
Traversal<User, String> nameLens = UserLenses.name().asTraversal();
Kind<ValidatedKind.Witness<List<Error>>, User> result = nameLens.modifyF(
    name -> validateName(name),
    user,
    ValidatedApplicative.instance(Semigroups.list())
);


Start with the Effect Path API. Drop to modifyF when you need its power.



Summary

We introduced the Effect Path API for effectful programming:


  Effect Path types: MaybePath, EitherPath, TryPath, ValidationPath, IOPath, VTaskPath
  Railway model: Values travel success/failure tracks with explicit error handling
  ValidationPath: Accumulate all errors with zipWithAccum and zipWith3Accum
  VTaskPath: Virtual thread concurrency with Par and Scope for parallel operations
  Bridge methods: Connect Focus paths to Effect paths via toMaybePath, toEitherPath
  Type checking example: Comprehensive error reporting with ValidationPath


The Effect Path API makes effect polymorphism practical. The same patterns that work for optional values work for error handling, validation, and deferred execution. Choose the right Effect Path type for your use case, and let composition do the rest.



Further Reading

Effect Systems and Functional Programming


  
    Scott Wlaschin, “Railway Oriented Programming” video and slides: The visual explanation of error handling that inspired the railway metaphor.
  
  
    Conor McBride & Ross Paterson, “Applicative programming with effects” (JFP, 2008): The paper that introduced Applicative as distinct from Monad, directly relevant to understanding why Validated accumulates errors.
  


Error Handling Patterns


  Handling Errors Without Exceptions: Chapter 4 from “Functional Programming in Scala” (free excerpt).


Higher-Kinded Types and Functional Abstractions


  
    Algebraic Data Types with Java (Scott Logic, 2025): A thorough introduction to algebraic data types using Java’s sealed interfaces and records. Covers how sum types and product types compose to model complex domains.
  
  
    Functors and Monads with Java and Scala (Scott Logic, 2025): A practical comparison of how Functor and Monad abstractions are implemented in Java vs Scala, directly relevant to understanding the Effect Path API’s foundation.
  
  
    Higher-Kinded Types with Java and Scala (Scott Logic, 2025): Explores how higher-kinded types work and how Java can simulate them, providing context for understanding the Kind pattern used throughout Higher-Kinded-J.
  


Higher-Kinded-J


  
    Effect Path API Guide: Railway-style error handling with MaybePath, EitherPath, ValidationPath, and VTaskPath.
  
  
    Path Factory: Factory methods for creating Effect Paths.
  
  
    Semigroups: Common semigroup implementations for error accumulation.
  
  
    Focus DSL Guide: Fluent navigation with FocusPath, AffinePath, and TraversalPath.
  
  
    VTask and Structured Concurrency: Virtual thread concurrency with Scope and Resource.
  
  
    Focus-Effect Integration: Bridging the optics and effects domains with toXxxPath() and focus() methods.
  




Next time

We’ve now built a substantial expression language: AST definition, optics generation, tree traversals, optimisation passes, type checking, and the Effect Path API for error accumulation.

In the final Part 6, we’ll step back and reflect on what we’ve built:


  The complete pipeline: From source text through parsing, type checking, optimisation, and evaluation
  Design patterns: Emergent patterns for effect-polymorphic code that work well
  Performance considerations: When to use optics and when simpler approaches suffice
  Real-world applications: Applying these techniques beyond expression languages






Detective Adventures - on debugging UI issues by Oded Sharon
Oded Sharon — Mon, 02 Feb 2026 10:00:00 +0000
Most developers enjoy working on exciting new projects and technologies. It is not only because of the sense of ownership and freedom but it is also about not trudging through old, badly written code filled with technical debt issues no one wants to deal with because they are so dreary. That said, I find satisfaction in looking at an existing system, with all the constraints it entails. And like a detective story, I try to figure out what went on in the previous developer’s mind when they wrote the code that I am now staring at. In fairness, that previous developer is often six-month-ago me, who thought he was very smart when he wrote that code.

A while back a client had a page with a list of documents the user managed. Each row in the table had information about the document and three actions as links - Edit, Delete and Download (PDF). When the client wished to add a couple more actions, we realised the table became exceedingly wide. The design team suggested having a drop-down with the various actions to save space.



However, the application is a Python form-based application. This means that every action the user submits will cause the page to refresh for the changes to take effect. Previously the “Download” action was a simple link that opened the PDF in a new window. But as the page reloaded, how can we trigger the opening of the new window?



A simple solution would be to include a small inline JavaScript that will trigger the opening of a new window with the PDF.  However, the client had an extremely strict security policy, prohibiting inline scripts for security reasons, so that would not work. We produced a different solution - we used a hidden iframe element which triggered the download.
The solution seemed great and worked well. However, QA picked up an unexpected issue: If the user downloaded the file and immediately refreshed the page, the form action “download” would be resubmitted, and the PDF would automatically download again.
We considered automatically refreshing the page without the auto-download but this had a few issues – it relied on the file being downloaded properly; and it uses a meta-refresh tag which is a big “no-no” in terms of accessibility as the user won’t understand why the page is being refreshed.

We decided to go back to the JavaScript-based solution: catch the form submission event, cancel it, and trigger a download event using the JavaScript (creating a link to the PDF and firing it without ever adding it to the page). It is not the perfect solution as it will leave the 5% of internet users who cannot use JavaScript with the current shoddy iFrame solution, but at least it will improve the experience of the other 95% of the users.

But it did not work. The event listener simply ignored the event when it fired, and the page kept refreshing (making it a pain to debug, as it kept resetting the console). Could it be that the strict content security policy (CSP) prevented writing onclick events? It killed any inline coding attempts with appropriate warning errors. But I could not find any indication that event listening was forbidden.

I needed to find a way to isolate the problem, so I added my own button to the page. Clicking it produced an error message that this button should be inside a form. But why was JavaScript complaining about it?

It turned out that the system user had a bad habit of double-clicking buttons (picked up from the old desktop version of the product) and for a Python application, it caused the form to be submitted twice. The user would get to see the result of the second submission, which failed as “The form was already submitted.” The crude solution for this problem was to catch any form submission, cancel the event, turn the clicked button into a hidden field, disable all the buttons, and submit the event again. The reason for the hidden field is because disabled buttons’ values are not being sent as part of form submission.

It was this piece of code (adding the hidden field to the button’s form) that triggered the error in my experimental button and that is how I found it. It is worth pointing out that the product barely relies on JavaScript; it is using webpack to create a single file that is applied to every page in the system, including the “documents” I was now trying to fix.

At this point, the solution was easy – I replaced the “double-click-protection” with a simple debounce mechanism (which prevents sending events too frequently), allowing my event to still run properly and get caught by my own event listener.

Alternatively, I could’ve added a condition to the “double-click-protection” code to check if the button has an attribute double-click-protection="false"; add that attribute to my button that I can now listen to using JavaScript and send the PDF, as I originally planned. It would have been a simpler, safer way to do it (less breaking of existing code), but I felt it would just patch over instead of actually solve the problem.



Another suggestion we have considered was to use HTML-based summary/details elements to show/hide the secondary actions, thus eliminating the need for JavaScript. Unfortunately, the design team was not in favour of this solution due to time constraints on their part.

There is a key takeaway about the difference between form-based applications (server-side rendering for that matter) and web-based applications. The former is extremely limited in its capabilities, and I can see why designers would find it too restricting. Personally, I find that it forces a certain simplicity in keeping a minimalistic client-side code that is healthy for code maintenance as well as the user’s mental model.

Conclusion

The landscape of software development is constantly changing, with modern technologies, frameworks, and methodologies. This, as I have mentioned in the past, can be quite overwhelming for junior developers. I hope that this experience I have shared gives a better view on the type of problems we are dealing with. As developers, we are required to understand the client needs and produce the best software, regardless of the tech stack.



Functional Optics for Modern Java - Part 4 by Magnus Smith
Magnus Smith — Fri, 30 Jan 2026 00:00:00 +0000
The Focus DSL: Traversals and Pattern Rewrites

Part 4 of the Functional Optics for Modern Java series

In Part 1 and Part 2, 
we established why optics matter and how they work. In Part 3,
we built our expression language AST and applied basic optics using lenses for field access and prisms for variant matching.

Finally, we created a simple optimiser, but left things hanging with a fundamental limitation that we shall address now:
How do we visit all nodes in a tree, not just the top level?

This is where traversals come into play as the essential tool. A traversal focuses on zero or more elements within a structure, making it perfect for recursive tree operations.

The Focus DSL provides TraversalPath: a fluent wrapper around traversals that makes collection navigation more readable and composable. By the end of this article, we’ll be writing code like:

// Navigate all employees in all departments, modify their salaries
Company updated = CompanyFocus.departments()
    .each()                              // Traverse each department
    .employees()                         // Navigate to employees list
    .each()                              // Traverse each employee
    .salary()                            // Focus on salary field
    .modifyAll(s -> s.multiply(1.1), company);


No manual recursion. No reconstruction boilerplate. Just a fluent path that describes what you want.

Running the Examples



Article Code

All code examples from this article have runnable demos:


  TraversalDemo: Using Higher-Kinded-J’s Traversal interface for composable, type-safe tree manipulation.
  OptimiserDemo: Constant folding, identity simplification, and cascading optimisation using traversal-based passes.


The AST types are defined in org.higherkindedj.article4.ast, with transformations in org.higherkindedj.article4.transform and traversals in org.higherkindedj.article4.traversal.



Traversals as Garden Pruning

Before diving into code, consider an analogy. Imagine you’re pruning a tree in a garden. You need to visit every branch, decide what to do at each one, and rebuild the tree with your changes. You have two strategies:


  
    Bottom-up (leaves first): Start at the leaf tips, prune outward branches, then work your way down. Each branch is trimmed after its subbranches are done. This is perfect when you need to see the final state of subbranches before deciding on the parent.
  
  
    Top-down (trunk first): Start at the trunk, make decisions about major branches first, then work outward. Each branch is examined before its subbranches. This works when you want to make early decisions that affect the whole subtree.
  


In programming, a traversal is exactly this: a systematic way to visit every node in a tree structure, with a strategy for when to act on each node relative to its children. The Focus DSL makes this as natural as describing which branches to visit.



The Recursive Challenge

Consider our expression AST. When we write (x + 1) * (y + 2), we get:

new Binary(
    new Binary(new Variable("x"), ADD, new Literal(1)),
    MUL,
    new Binary(new Variable("y"), ADD, new Literal(2))
)


Visualised as a tree:


This tree has seven nodes: three Binary expressions, two Variable nodes, and two Literal nodes. If we want to find all variables, we can’t just look at the top level; we need to descend into every branch.

As we discussed in Part 3, the traditional Visitor pattern requires substantial boilerplate: an interface, accept methods in each node, and a visitor implementation for every operation. Even with Java 25’s pattern matching, we still face the reconstruction cascade for transformations.

Traversals offer something better: define the traversal structure once, then use it for any operation.



Building a Universal Expression Traversal

A Traversal focuses on zero or more A values within an S structure. For expressions, we want Traversal: a traversal that visits all subexpressions within an expression.

First, let’s define what “all subexpressions” means for each variant:


  
    
      Expression Type
      Sub-expressions
    
  
  
    
      Literal
      None
    
    
      Variable
      None
    
    
      Binary
      left, right
    
    
      Conditional
      cond, then_, else_
    
  


Here’s the traversal implementation:

public final class ExprTraversal {

    /**
     * A traversal targeting all immediate children of an expression.
     * Does not descend recursively; use with transform utilities for full tree traversal.
     */
    public static Traversal<Expr, Expr> children() {
        return new Traversal<>() {
            @Override
            public <F> Kind<F, Expr> modifyF(
                    Function<Expr, Kind<F, Expr>> f,
                    Expr source,
                    Applicative<F> applicative) {
                return switch (source) {
                    case Literal _ -> applicative.of(source);
                    case Variable _ -> applicative.of(source);
                    case Binary(var l, var op, var r) ->
                        applicative.map2(f.apply(l), f.apply(r),
                            (newL, newR) -> new Binary(newL, op, newR));
                    case Conditional(var c, var t, var e) ->
                        applicative.map3(f.apply(c), f.apply(t), f.apply(e),
                            (newC, newT, newE) -> new Conditional(newC, newT, newE));
                };
            }
        };
    }
}


This is effect-polymorphic: the same traversal works with any Applicative functor. We can use it for pure transformations, error-accumulating validation, or stateful operations. (Don’t worry if “effect-polymorphic” is unfamiliar; we’ll explore this concept further. For now, just know it means this same traversal code works whether you’re doing pure transformations, error handling, or validation.)

For simpler use cases, Higher-Kinded-J provides the Traversals.modify utility:

import org.higherkindedj.optics.util.Traversals;

// Apply a pure transformation to all children
Expr result = Traversals.modify(children(), f, expr);


This handles the Id effect internally, giving you a clean API for pure transformations.

The Focus DSL Alternative: TraversalPath

While the raw Traversal interface provides maximum control, the Focus DSL offers a more ergonomic API for everyday use. TraversalPath wraps a traversal and provides fluent methods:

import org.higherkindedj.optics.focus.TraversalPath;
import org.higherkindedj.optics.focus.FocusPaths;

// Create a TraversalPath from our children traversal
TraversalPath<Expr, Expr> childrenPath = TraversalPath.fromTraversal(children());

// Now we have fluent operations:
List<Expr> allChildren = childrenPath.getAll(expr);           // Collect all children
Expr modified = childrenPath.modifyAll(this::optimise, expr); // Transform all children


The Focus DSL becomes even more powerful with collection navigation. For lists, use each():

// Given a department with a list of employees
FocusPath<Department, List<Employee>> staffPath = DepartmentFocus.staff();

// Navigate into the list with each()
TraversalPath<Department, Employee> allStaff = staffPath.each();

// Now we can operate on all employees:
List<Employee> employees = allStaff.getAll(department);
Department updated = allStaff.modifyAll(Employee::promote, department);


The each() method is the key: it transforms a FocusPath> into a TraversalPath. This pattern scales to nested structures, as we’ll see throughout this article.

Deep Traversal: Visiting All Descendants

The children() traversal only visits immediate children. For full tree traversal, we combine it with recursive descent:

/**
 * Transform all nodes in the tree from leaves to root (bottom-up).
 * Each node is transformed after its children.
 */
public static Expr transformBottomUp(Expr expr, Function<Expr, Expr> f) {
    // First transform all children recursively
    Expr transformed = Traversals.modify(
        children(),
        child -> transformBottomUp(child, f),
        expr
    );
    // Then transform this node
    return f.apply(transformed);
}

/**
 * Transform all nodes in the tree from root to leaves (top-down).
 * Each node is transformed before its children.
 */
public static Expr transformTopDown(Expr expr, Function<Expr, Expr> f) {
    // First transform this node
    Expr transformed = f.apply(expr);
    // Then transform all children recursively
    return Traversals.modify(
        children(),
        child -> transformTopDown(child, f),
        transformed
    );
}


The choice between bottom-up and top-down matters:


  
    Bottom-up: Children are transformed first. Use this when transformations depend on the structure of children (like constant folding, where we need to fold 1 + 2 before we can simplify (1 + 2) * 3).
  
  
    Top-down: The node is transformed first. Use this when you want to pattern-match on the original structure before children change (like macro expansion).
  


Here’s the traversal order visualised:



For constant folding, bottom-up is essential: we need to evaluate 1 + 2 at the leaves before we can recognise that the parent is now 3 * 5.

Focus DSL: The Practical Choice

For most use cases, the Focus DSL provides what you need without writing custom traversals. Here’s the pattern for nested structures:

@GenerateFocus(generateNavigators = true)
record Company(String name, List<Department> departments) {}

@GenerateFocus(generateNavigators = true)
record Department(String name, List<Employee> employees, Employee manager) {}

@GenerateFocus(generateNavigators = true)
record Employee(String name, BigDecimal salary, Address address) {}

// Navigate all the way to employee addresses:
TraversalPath<Company, Address> allEmployeeAddresses = CompanyFocus
    .departments().each()          // Into each department
    .employees().each()            // Into each employee
    .address();                    // To their address

// Get all addresses
List<Address> addresses = allEmployeeAddresses.getAll(company);

// Relocate all employees
Company relocated = allEmployeeAddresses.modifyAll(
    addr -> new Address("New HQ", addr.city(), addr.postcode()),
    company
);


With navigators enabled, cross-type navigation happens automatically. No explicit via() calls, no manual composition. The path reads like English: “company’s departments, each one’s employees, each one’s address.”



Collecting Information

Traversals aren’t just for modification; they’re equally powerful for extraction. We use folds to aggregate information from all focused elements.

Finding All Variables

public static Set<String> findVariables(Expr expr) {
    Set<String> vars = new HashSet<>();
    collectVariables(expr, vars);
    return vars;
}

private static void collectVariables(Expr expr, Set<String> accumulator) {
    switch (expr) {
        case Variable(var name) -> accumulator.add(name);
        case Literal _ -> { }
        case Binary(var l, _, var r) -> {
            collectVariables(l, accumulator);
            collectVariables(r, accumulator);
        }
        case Conditional(var c, var t, var e) -> {
            collectVariables(c, accumulator);
            collectVariables(t, accumulator);
            collectVariables(e, accumulator);
        }
    }
}


With the Focus DSL, we can make this more readable:

public static Set<String> findVariables(Expr expr) {
    // Create a TraversalPath that focuses on all Variable nodes
    TraversalPath<Expr, Expr> allNodes = ExprFocus.universe();  // All descendants

    // Filter to variables and extract names
    return allNodes.getAll(expr).stream()
        .filter(e -> e instanceof Variable)
        .map(e -> ((Variable) e).name())
        .collect(Collectors.toSet());
}


Or using foldMap on the TraversalPath for a more functional approach:

public static Set<String> findVariables(Expr expr) {
    TraversalPath<Expr, Expr> allNodes = ExprFocus.universe();
    return allNodes.foldMap(
        Monoids.set(),
        e -> e instanceof Variable(var name) ? Set.of(name) : Set.of(),
        expr
    );
}


The foldMap method on TraversalPath traverses all focused elements and combines results using a monoid (here, set union).

Counting Nodes by Type

public record NodeCounts(int literals, int variables, int binaries, int conditionals) {
    public static NodeCounts ZERO = new NodeCounts(0, 0, 0, 0);

    public NodeCounts add(NodeCounts other) {
        return new NodeCounts(
            literals + other.literals,
            variables + other.variables,
            binaries + other.binaries,
            conditionals + other.conditionals
        );
    }
}

public static NodeCounts countNodes(Expr expr) {
    return foldMap(
        expr,
        e -> switch (e) {
            case Literal _ -> new NodeCounts(1, 0, 0, 0);
            case Variable _ -> new NodeCounts(0, 1, 0, 0);
            case Binary _ -> new NodeCounts(0, 0, 1, 0);
            case Conditional _ -> new NodeCounts(0, 0, 0, 1);
        },
        NodeCounts::add
    );
}


The foldMap Implementation

public static <A> A foldMap(Expr expr, Function<Expr, A> extract, BinaryOperator<A> combine) {
    A current = extract.apply(expr);
    return switch (expr) {
        case Literal _, Variable _ -> current;
        case Binary(var l, _, var r) ->
            combine.apply(current,
                combine.apply(foldMap(l, extract, combine),
                              foldMap(r, extract, combine)));
        case Conditional(var c, var t, var e) ->
            combine.apply(current,
                combine.apply(foldMap(c, extract, combine),
                    combine.apply(foldMap(t, extract, combine),
                                  foldMap(e, extract, combine))));
    };
}




Filtered Traversals and Conditional Updates

Sometimes we only want to focus on certain elements. The Focus DSL provides modifyWhen() for conditional updates and filtering capabilities for targeted navigation.

Conditional Modification with modifyWhen

The modifyWhen method is perfect for targeted updates:

// Give a raise only to employees earning below a threshold
TraversalPath<Company, Employee> allEmployees = CompanyFocus
    .departments().each()
    .employees().each();

Company updated = allEmployees.modifyWhen(
    emp -> emp.salary().compareTo(threshold) < 0,  // Predicate
    emp -> emp.withSalary(emp.salary().multiply(1.15)),  // Transformation
    company
);


This is cleaner than filtering after the fact: the predicate and transformation are co-located, making the intent clear.

Targeting Only Binary Additions

For our expression language, we might want to target specific node types:

public static Expr doubleAllAdditions(Expr expr) {
    return transformBottomUp(expr, e -> {
        if (e instanceof Binary(var l, BinaryOp.ADD, var r)) {
            // Transform a + b into (a + b) + (a + b)
            return new Binary(e, BinaryOp.ADD, e);
        }
        return e;
    });
}


Using Prisms for Type-Safe Filtering

The Focus DSL integrates beautifully with prisms for sum type navigation:

public static List<BinaryOp> findAllOperators(Expr expr) {
    // Navigate to all Binary nodes using instanceOf
    TraversalPath<Expr, Binary> allBinaries = ExprFocus.universe()
        .via(AffinePath.instanceOf(Binary.class));

    return allBinaries.getAll(expr).stream()
        .map(Binary::op)
        .collect(Collectors.toList());
}


The AffinePath.instanceOf() creates an affine path that focuses on elements of a specific type. When composed with a traversal, it filters to only matching elements.

Sum Type Navigation with instanceOf

For sealed interfaces, instanceOf is particularly powerful:

// Focus on all Circle shapes in a drawing
TraversalPath<Drawing, Circle> circles = DrawingFocus.shapes()
    .each()
    .via(AffinePath.instanceOf(Circle.class));

// Double all circle radii
Drawing updated = circles.modifyAll(
    c -> new Circle(c.radius() * 2),
    drawing
);


This pattern combines the type safety of sealed interfaces with the composability of the Focus DSL. The compiler ensures exhaustiveness while the DSL provides elegant navigation.



Implementing Optimisation Passes

With traversals in place, we can build sophisticated optimisation passes. Each pass is a transformation function; the optimiser composes them.

Pass 1: Constant Folding

Evaluate operations where both operands are literals:

public static Expr foldConstants(Expr expr) {
    return transformBottomUp(expr, ExprOptimiser::foldConstant);
}

private static Expr foldConstant(Expr expr) {
    if (expr instanceof Binary(Literal(var l), var op, Literal(var r))) {
        return evaluateBinary(l, op, r)
            .map(result -> (Expr) new Literal(result))
            .orElse(expr);
    }
    return expr;
}

private static Optional<Object> evaluateBinary(Object left, BinaryOp op, Object right) {
    return switch (op) {
        case ADD -> evaluateAdd(left, right);
        case SUB -> evaluateSub(left, right);
        case MUL -> evaluateMul(left, right);
        case DIV -> evaluateDiv(left, right);
        case AND -> evaluateAnd(left, right);
        case OR -> evaluateOr(left, right);
        case EQ -> Optional.of(left.equals(right));
        case NE -> Optional.of(!left.equals(right));
        case LT, LE, GT, GE -> evaluateComparison(left, op, right);
    };
}


Pass 2: Identity Simplification

Remove operations that don’t change the result:

public static Expr simplifyIdentities(Expr expr) {
    return transformBottomUp(expr, ExprOptimiser::simplifyIdentity);
}

private static Expr simplifyIdentity(Expr expr) {
    return switch (expr) {
        // x + 0 → x, 0 + x → x
        case Binary(var x, ADD, Literal(Integer i)) when i == 0 -> x;
        case Binary(Literal(Integer i), ADD, var x) when i == 0 -> x;

        // x * 1 → x, 1 * x → x
        case Binary(var x, MUL, Literal(Integer i)) when i == 1 -> x;
        case Binary(Literal(Integer i), MUL, var x) when i == 1 -> x;

        // x * 0 → 0, 0 * x → 0
        case Binary(_, MUL, Literal(Integer i)) when i == 0 -> new Literal(0);
        case Binary(Literal(Integer i), MUL, _) when i == 0 -> new Literal(0);

        // x && true → x, true && x → x
        case Binary(var x, AND, Literal(Boolean b)) when b -> x;
        case Binary(Literal(Boolean b), AND, var x) when b -> x;

        // x || false → x, false || x → x
        case Binary(var x, OR, Literal(Boolean b)) when !b -> x;
        case Binary(Literal(Boolean b), OR, var x) when !b -> x;

        // x && false → false
        case Binary(_, AND, Literal(Boolean b)) when !b -> new Literal(false);
        case Binary(Literal(Boolean b), AND, _) when !b -> new Literal(false);

        // x || true → true
        case Binary(_, OR, Literal(Boolean b)) when b -> new Literal(true);
        case Binary(Literal(Boolean b), OR, _) when b -> new Literal(true);

        default -> expr;
    };
}


Pass 3: Dead Branch Elimination

Remove conditional branches with constant conditions:

public static Expr eliminateDeadBranches(Expr expr) {
    return transformBottomUp(expr, ExprOptimiser::eliminateDeadBranch);
}

private static Expr eliminateDeadBranch(Expr expr) {
    return switch (expr) {
        case Conditional(Literal(Boolean b), var t, var e) -> b ? t : e;
        default -> expr;
    };
}


Pass 4: Common Subexpression Detection

Identify repeated subexpressions (useful for let-binding which we will see to in Part 5):

public static Map<Expr, Integer> findCommonSubexpressions(Expr expr) {
    Map<Expr, Integer> counts = new HashMap<>();
    countSubexpressions(expr, counts);
    // Return only expressions that appear more than once
    return counts.entrySet().stream()
        .filter(e -> e.getValue() > 1)
        .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
}

private static void countSubexpressions(Expr expr, Map<Expr, Integer> counts) {
    counts.merge(expr, 1, Integer::sum);
    switch (expr) {
        case Binary(var l, _, var r) -> {
            countSubexpressions(l, counts);
            countSubexpressions(r, counts);
        }
        case Conditional(var c, var t, var e) -> {
            countSubexpressions(c, counts);
            countSubexpressions(t, counts);
            countSubexpressions(e, counts);
        }
        default -> { }
    }
}




Extending the AST: Source Locations

Real compilers need to track where expressions come from for error messages. Let’s extend our AST with source location metadata.

public record SourceLocation(String file, int line, int column) {
    @Override
    public String toString() {
        return file + ":" + line + ":" + column;
    }
}

public record Located<T>(T value, SourceLocation location) {
    public <U> Located<U> map(Function<T, U> f) {
        return new Located<>(f.apply(value), location);
    }
}


Now we can create Located to track positions. Here lies a question in that when we transform an expression, what happens to the location?

Preserving Locations Through Transformations

public static Located<Expr> transformPreservingLocation(
        Located<Expr> located,
        Function<Expr, Expr> f) {
    return located.map(f);
}


For more sophisticated location handling (like updating locations when inlining code), indexed optics become valuable.



The Complete Optimiser

Combining all passes into a single optimiser:

public final class ExprOptimiser {

    /**
     * Run all optimisation passes until the expression stops changing.
     */
    public static Expr optimise(Expr expr) {
        Expr current = expr;
        Expr previous;

        do {
            previous = current;
            current = runAllPasses(current);
        } while (!current.equals(previous));

        return current;
    }

    private static Expr runAllPasses(Expr expr) {
        Expr result = expr;
        result = foldConstants(result);
        result = simplifyIdentities(result);
        result = eliminateDeadBranches(result);
        return result;
    }
}


The fixed-point iteration ensures we catch cascading simplifications:


For example:



Example: Complex Optimisation

// if (true && (1 < 2)) then (x + 0) * 1 else y
Expr complex = new Conditional(
    new Binary(new Literal(true), AND,
        new Binary(new Literal(1), LT, new Literal(2))),
    new Binary(
        new Binary(new Variable("x"), ADD, new Literal(0)),
        MUL,
        new Literal(1)),
    new Variable("y")
);

Expr optimised = ExprOptimiser.optimise(complex);
// Result: Variable("x")


The optimiser:


  Folds 1 < 2 → true
  Simplifies true && true → true
  Eliminates the dead else-branch
  Simplifies x + 0 → x
  Simplifies x * 1 → x


All through composable, declarative transformations.



Bridging to the Effect Path API

What is effect-polymorphism?

In functional programming, an effect is any computational context beyond returning a plain value; things like “might fail,” “might be absent,” “accumulates errors,” or “carries state.”

Effect-polymorphism means writing code once that works with any of these contexts. Instead of writing separate versions of a traversal for “transform purely,” “transform with possible failure,” and “transform while accumulating errors,” we write a single generic version that is parameterised by the effect type. The traversal doesn’t care which effect you use; it just requires that the effect supports certain operations (specifically, the Applicative interface).
This is what modifyF provides: the F is a placeholder for any effect, so the same traversal logic handles pure transformations, validation, state threading, or any other effect you plug in.

The TraversalPath we’ve built throughout this article navigates and transforms data structures. But what happens when transformations might fail, or when we need to accumulate errors from multiple elements?

Higher-Kinded-J’s Effect Path API provides the answer. Effect Paths wrap computations in contexts like Maybe (optional), Either (fail-fast errors), or Validated (accumulated errors). The Focus DSL bridges directly to these effect types.

From Focus Paths to Effect Paths

Every Focus path type provides bridge methods to enter the effect world:

// FocusPath bridges to effects
FocusPath<Employee, String> namePath = EmployeeFocus.name();

MaybePath<String> maybeName = namePath.toMaybePath(employee);
EitherPath<Error, String> eitherName = namePath.toEitherPath(employee);


For AffinePath (which might not find a value), the bridge handles the missing case:

// AffinePath handles optionality
AffinePath<User, String> nicknamePath = UserFocus.nickname();  // Optional field

MaybePath<String> maybeNickname = nicknamePath.toMaybePath(user);
// Returns Maybe.just(name) or Maybe.nothing()

EitherPath<String, String> eitherNickname =
    nicknamePath.toEitherPath(user, "No nickname set");
// Returns Either.right(name) or Either.left("No nickname set")


What’s Ahead: The Effect Path API

Beyond optics and the Focus DSL, Higher-Kinded-J provides the Effect Path API: a fluent interface for computations that might fail, accumulate errors, or require deferred execution. The Effect Path types follow the “railway” metaphor where values travel along success or failure tracks:



The core Effect Path types include:


  
    
      Effect Path
      Wraps
      Use Case
    
  
  
    
      MaybePath
      Maybe
      Optional values (might be absent)
    
    
      EitherPath
      Either
      Fail-fast error handling
    
    
      ValidationPath
      Validated
      Error accumulation
    
    
      TryPath
      Try
      Exception handling
    
    
      IOPath
      IO
      Deferred side effects
    
  


Understanding the underlying effect types:


  
    Maybe: Represents a value that might be absent, similar to Optional but with richer composition. Use when a value simply might not exist.
  
  
    Either: Represents a value that is either a Left (typically an error) or a Right (the success value). Fails fast on the first error encountered.
  
  
    Validated: Like Either, but designed for error accumulation. When combining multiple validations, collects all errors rather than stopping at the first.
  
  
    Try: Captures computations that might throw exceptions. Converts exception-throwing code into values you can compose safely.
  
  
    IO: Represents a deferred side effect. The computation is described but not executed until explicitly run, enabling pure functional composition of effectful operations.
  


These types integrate seamlessly with Focus paths via bridge methods:

// Navigate to a field, then enter the Effect Path world
FocusPath<User, String> emailPath = UserFocus.email();


Effect Paths with Traversals

TraversalPath works with Effect Paths through modifyF. This is the bridge to effect-polymorphic operations:

// Validate all employees in a company
TraversalPath<Company, Employee> allEmployees = CompanyFocus
    .departments().each()
    .employees().each();

// Bridge to ValidationPath for error accumulation
ValidationPath<List<Error>, Company> validated = Path.valid(company, Semigroups.list())
    .via(c -> {
        // Use modifyF with Validated applicative for error accumulation
        Kind<ValidatedKind.Witness<List<Error>>, Company> result = allEmployees.modifyF(
            emp -> validateEmployee(emp),
            c,
            validatedApplicative
        );
        return VALIDATED.narrow(result);
    });


The Railway Model

Effect Paths follow the “railway” pattern where values travel along success or failure tracks:


This pattern enables comprehensive validation rather than fail-fast behaviour. The ValidationPath accumulates all errors using a Semigroup, so users see every problem at once.

Preview: Effect-Polymorphic Traversals

In Part 5, we’ll explore the Effect Path API in depth. The key takeaway is that the same Focus paths work with different effect types:

// Same path, different effects
TraversalPath<Expr, Expr> allChildren = ExprFocus.children();

// Pure transformation
Expr optimised = allChildren.modifyAll(this::optimise, expr);

// With Maybe (might fail)
MaybePath<Expr> maybeResult = Path.just(expr)
    .via(e -> allChildren.modifyF(this::tryOptimise, e, maybeApplicative));

// With Validated (accumulate errors)
ValidationPath<List<Error>, Expr> validated = Path.valid(expr, Semigroups.list())
    .via(e -> allChildren.modifyF(this::validateNode, e, validatedApplicative));


The Effect Path API turns the theoretical power of modifyF into a practical, fluent interface.



Summary

This article introduced traversals and the Focus DSL’s TraversalPath for recursive manipulation:


  TraversalPath: Fluent wrapper for traversals with getAll(), modifyAll(), foldMap()
  Collection Navigation: each() for lists, at() for indices, atKey() for maps
  Conditional Updates: modifyWhen() for predicate-based transformations
  Sum Type Navigation: AffinePath.instanceOf() for type-safe variant targeting
  Deep Traversal: Bottom-up and top-down recursive descent
  Composable Optimiser: Multiple passes running to fixed point


The Focus DSL separates what to visit from what to do. Build paths declaratively, then apply operations fluently. This is the optics philosophy made accessible.

The Higher-Kinded-J Advantage for Tree Operations

The Focus DSL transforms what could be tedious tree manipulation into expressive, readable code. Consider what we achieved:


  
    Fluent navigation with TraversalPath: Instead of writing custom traversals, we chain methods: .departments().each().employees().each().salary(). The path describes the data shape, not the mechanics of traversal.
  
  
    Collection navigation built-in: Methods like each(), at(), and atKey() handle the common cases. No need to write Traversals.list() and compose it manually.
  
  
    Conditional updates with modifyWhen: Target specific elements with predicates directly on the path. The intent is clear and the implementation is correct by construction.
  
  
    Sum type integration: AffinePath.instanceOf() bridges sealed interfaces with the Focus DSL. Navigate to specific variants type-safely.
  
  
    Effect polymorphism underneath: The Focus DSL wraps Higher-Kinded-J’s effect-polymorphic traversals. When you need modifyF for validation or state, the same paths work.
  


The Focus DSL embodies the principle: common things should be easy, powerful things should be possible. Most operations use the fluent API; when you need maximum control, the underlying traversals are always accessible.



What’s Next

We’ve built a powerful foundation for tree manipulation with the Focus DSL. Our TraversalPath chains can visit every node, foldMap can aggregate information, and optimisation passes compose cleanly.

But we’ve been working with pure transformations. Real compilers need effects:


  Type checking should accumulate errors, not fail on the first one
  Interpretation needs to track variable bindings (state)
  Optimisation might need logging for debugging


In Part 5, we’ll explore effect-polymorphic optics with modifyF. The Focus DSL integrates seamlessly:

// Preview: effectful modification through Focus paths
Kind<ValidatedKind.Witness<List<TypeError>>, Employee> result =
    employeePath.modifyF(
        emp -> validateEmployee(emp),  // Returns Validated
        employee,
        validatedApplicative
    );


The same Focus paths we’ve built work with Validated for error accumulation, State for environment threading, and any other effect. Higher-Kinded-J provides the type-class instances that make this polymorphism possible.

We’ll see how Validated differs from Either (accumulating all errors rather than short-circuiting), how State threads environment bindings through interpretation, and how these effects compose with Focus paths. The expression language will gain a full type system and interpreter, all built on the same Focus DSL foundations.



Further Reading

Tree Traversals and Recursion Schemes


  Patrick Thomson, “An Introduction to Recursion Schemes”: A gentle introduction to the theory behind generic tree traversal patterns like catamorphisms and anamorphisms.


Compiler Optimisation


  Andrew Appel, Modern Compiler Implementation in ML: Classic text covering constant folding, dead code elimination, and the other optimisations we’ve implemented. Available in Java and C editions.


Term Rewriting


  
    Franz Baader & Tobias Nipkow, Term Rewriting and All That: For readers interested in the theoretical foundations of pattern-based rewriting systems.
  
  
    The optimisation passes in this article are simple term rewrites. Industrial strength rewriting (like in GHC’s rule system) adds phases, termination proofs, and confluent rule ordering.
  


Algebraic Data Types in Java


  Algebraic Data Types with Java (Scott Logic, 2025): A thorough introduction to algebraic data types using Java’s sealed interfaces and records. Covers how sum types (like our expression AST) and product types compose to model complex domains.


Higher-Kinded-J


  
    Traversal Documentation: API reference for working with multi-focus optics.
  
  
    Traversals Utility Class: Helper methods for getAll, modify, and fold operations.
  
  
    Focus DSL Guide: Fluent navigation with FocusPath, AffinePath, and TraversalPath.
  
  
    Effect Path API Guide: Railway-style error handling with MaybePath, EitherPath, and ValidationPath.
  




Next time

Real compilers and interpreters need more. Type checking should report all errors, not just the first one. 
Interpretation must track variable bindings as it descends through the tree. 
These are effects, and they change everything.

Next time, in Part 5 we take a closer look at how effects help structure our code.



Introduction to Energy Trading for Novices by Peter Marsh
Peter Marsh — Wed, 28 Jan 2026 00:00:00 +0000
Do you picture a scene like The Wolf of Wall Street when you imagine a trading floor? Do you see hordes of wide-eyed guys in suits shouting down telephones to customers, hustle and bustle, and huge parties when big deals are made? The truth is that trading happens in quiet offices where calm, focused and very qualified individuals study numbers in grids that stretch across ranks of computer monitors. They know how to use the data, how to analyse trends, how to calculate to maximise a profit, and how to get the best outcomes for their business. And they know the systems and rules of trading inside and out. As a developer (or tester or designer) working on a trading application, you will also need to gain a basic understanding of the domain.


  

    
  


I just completed my first software project in the energy sector. Over the two years, aside from improving my dev skills, I learned a lot of interesting stuff about the complexities of sending electricity around the world! What follows is a summary of what I learned about energy trading and the energy market written in simple terms: A quick-start guide for novices that I wish that I’d had at the start of the project.

Energy

Let’s go from the start: with Energy. Energy is a hugely useful resource that people and businesses pay good money for. Energy companies own power plants that burn gas, break down nuclear materials, or capture the movement of wind and water to generate the electricity that is then sent to all the people that use it.

Energy can be traded in the form of electricity, or as oil and gas. In this article I focus on electricity, especially in the European market.

The Grid

Electricity travels down cables to get from people that generate it to people that use it. Every country has a big set of interconnected cables called the Grid that connects houses and buildings to the power stations that generate electricity.

For complicated physics reasons (ask a scientist) the grid must stay balanced. This means that all the electricity generated and fed into the grid must be immediately used up – in other words: The electricity coming into the grid (the supply) must equal the energy going out (the demand) at all times. If that doesn’t happen then the grid’s hardware will take damage and it could shut down. Balancing the grid is a crazy task. Think about it: Imagine the hundreds of millions of homes and offices and shops in your country, and then picture all the heaters, fridges, lightbulbs, computers, and phone chargers in each building that may or may not be being used at a particular moment. All this fluctuating demand must be accounted for by electricity generators. Market and grid operators make sure balance is reached by dishing out severe penalties to those that disrupt it. But fear not, dear reader. As a bog-standard user of electricity, yours will be bought from an energy supplier (think British Gas, EDF, Octopus Energy etc) rather than wholesale from the market. It is the supplier’s job and not yours to trade on the market directly and anticipate how much energy you will use.

Energy between grids

Remember how each country has its own grid*? Well, there are cables that connect each grid to neighbouring ones - for instance, France’s grid will be linked to Spain’s. This means that electricity produced in Spain can be bought and used in France, which is useful if Spain cannot produce enough electricity to meet their demand or if France’s production is greener or cheaper.

*Side note: sometimes a country’s grid can be split by region. For example, Germany has 4 separate control areas which are operated by different people. When you send/receive energy from Germany you send/receive from a specific control area as if they were separate grids. For the sake of simplicity, the article is written as though each country has one grid.


  

    
  


But you can only transfer a limited amount of electricity from one country to another. That’s because the linking cables can only carry a limited amount of electricity without getting damaged. The amount of energy that can be transferred from one country to another in a time period is called the transmission capacity. Before transmitting energy across a border, traders must bid for and book capacity, which is a separate system from the electricity market (at least when markets are not coupled – see the next section).

So, if Antoni in Switzerland needs to buy energy from France, then he must check that there is enough available capacity at the border to transmit that amount of energy. If he makes the purchase and there is not enough transmission capacity, then the electricity cannot be transmitted, the trade cannot be fulfilled, and he will face penalties from the market (unless he can fix his error by trading back).

Market coupling

We’ve established that the process of making electricity trades is separate from booking capacity. This can cause problems. Picture this: Antoni in Switzerland has his eye on the French market. An order has just come into France which, if he traded on it, could earn him a large profit and help his company make the most of its wind turbines. But he should act fast because the market changes very quickly; there are so many people on the market that the orders and prices are constantly shifting. So Antoni rushes to the transmission capacity booking system and secures enough capacity for the trade, switches back to his view of the market and… The order is gone. He’s lost his golden opportunity, and now the capacity system thinks he has electricity to send or receive. It’s a bad day for Antoni.

And, to worsen Antoni’s day, while focusing on the French market he has unknowingly missed several good opportunities for trades in Germany!

To solve this, certain countries have banded together to organise what’s known as a coupled market. Take Europe’s Single Intraday Coupling (SIDC) initiative, and Cross-Border IntraDay (XBID), the IT system that powers it. Tereza, a trader in Austria, can make use of this. She can look at one screen which shows this coupled market, containing orders and trading opportunities from all countries involved in the SIDC initiative. Better yet, the orders that appear to her are only those that can be made with the capacity available at the relevant borders, so she doesn’t need to worry about checking the capacity system. When she spots an opportunity, she quickly makes the trade, and the correct capacity is automatically allocated. It’s a good day for Tereza!

Now Tereza is trading electricity that will be delivered between 3pm and 4pm. As soon as the clock hits 2:55pm (5 minutes before delivery, though this depends on the border), we observe what is known as gate closure. At this time anybody intending to send electricity across grid borders is locked in and they may no longer book further capacity. As such, Tereza’s view of the market changes. After gate closure she only sees the orders for Austria, her own country, and can only make trades with others within Austria. Thanks to SIDC, this is all handled automatically for her and there is no chance of her accidentally making cross-border trades after gate closure.

Traders within countries involved in SIDC enjoy these advantages offered by the coupled market. Meanwhile those outside SIDC must come up with their own systems to make the most of opportunities occurring across different borders, and to avoid making trades they don’t have capacity for.

Energy Prices

Energy prices, much like the price of everything else in the world, is mostly a question of supply and demand. It depends entirely on what participants in the market are willing to buy or sell energy for. If lots of people need to use electricity, and for whatever reason not much electricity is available, then the electricity will be more expensive. On the other hand, if not many people are using electricity but there are lots of generators up and running, then electricity will be cheap.

There are different ways of generating electricity. Take, for example, nuclear powered (break down nuclear matter -> gain electricity), gas powered (burn gas -> gain electricity) and solar powered (absorb sunlight -> gain electricity). It is not important to understand exactly how these work, but it is useful to know that they each cost different amounts of money to run. For instance, solar panels cost some money to set up but once they’re there, the sunlight they use to make the electricity is free. On the other hand, a gas-powered plant will cost a lot of money to build and set up, and then even more money to buy the gas that gets burned to generate the electricity. That’s not to mention maintenance costs and paying the people that work there. Therefore, a company that owns solar panels might be more willing to sell their electricity at lower prices than a company that owns a gas-powered plant.


  

    
  


Different ways of electricity generation also have different startup and shutdown times, with shorter startup and shutdown times providing more flexibility. A solar farm that is already in the sun can be switched on and off with no wait at all, which means that on a sunny day it is useful for meeting last-minute demand and balancing the grid. On the other hand, consider a nuclear power plant. While able to produce more power more consistently and over a longer time, it can take several hours to start or stop producing electricity, so it is much less able to respond to fluctuating demand. For this reason, if there is a sudden rush of demand for electricity needed in the next five minutes, it could cost a high price since fewer power stations are able to respond to that. However, if we can forecast a large demand of energy in a week’s time, then we have plenty of time to choose how we will generate that power and start up the relevant power station. Therefore, the electricity can be sold more cheaply.

Here’s something that surprised me: Occasionally, electricity sells for negative prices! Imagine that lots of people are forecasted to use electricity at 6pm on Thursday because they’re all sitting down to watch an exciting new TV programme. So all the gas power plants are planned to start up to deliver the required electricity… But then the programme is cancelled last minute! These gas plants are still generating electricity and feeding it to the grid and there are no consumers to use it. In this case the providers might pay consumers to use up this energy, otherwise there will be a grid imbalance.

You can see the live energy prices in different countries in Europe here.

The Market

The market, or exchange, is where people (retail suppliers, generators, especially large businesses, speculative traders, etc) go to buy or sell energy. In Europe there are different markets operated by different companies. Some big names include EPEX and Nordpool. The different markets serve different places, for instance Nordpool serves most of northern Europe and EPEX serves parts of northern and central Europe, while Belpex serves only Belgium. When you start trading you must choose a market to trade in, learn how that market operates and go through the legal signup process which will involve paying license fees.

Once you have access to the market, what are you actually trading? We must start by understanding contracts*. A contract is the combination of a delivery period and a place (delivery area), for instance a contract might be “Monday 3pm to 4pm in Germany”, or “Thursday 6:30am to 6:45am in Italy”. A trader will be able to buy and sell against these contracts to create trades. A trade is a legal promise that involves a contract, a buyer, a seller, a price and a quantity**. An example trade would be “Jerry will deliver 10 MW of power to Tommy in France between 3:30pm and 4pm. Tommy will pay €30 to Jerry”.

*I am told that Nordpool uses the term Product instead. I was frequently bamboozled by the terms Contract and Product while on my project. Our client sometimes used them interchangeably, but EPEX used Contract as defined above, while using product to describe a type of contract, e.g. “hourly”, “half-hourly”. This is why it is important to read the specs when you’re building an app for a new system!

**A note on energy and power: Energy is the actual physical thing that lets you do stuff, i.e. light a room. Power is that rate of energy transfer. Power is measured in watts (W) while energy is measured in watt-hours (Wh). If you transfer energy at a rate of 1 Watt for exactly one hour, then you have transferred one watt-hour of energy. A lightbulb with a power of 60W will transfer 60Wh in one hour, 30Wh in half an hour, 120Wh in 2 hours and so on.

This is relevant because energy trades have a quantity, and that is a quantity of power (at least for EPEX). The above trade between Jerry and Tommy says “Jerry will deliver a continuous power of 10 MW between 3:30pm and 4pm, which is the same as 5MWh of energy”. As you’re developing, make sure you know whether quantities refer to amounts of energy or power.


  

    
  
  
    An example orderbook containing everone's orders for one contract - Wednesday 14:30-14:45 in the French delivery area (called RTE).
  


Traders show that they wish to trade by placing limit orders on the market (there are different types of orders but that’s an explanation for another blog). They place each order against contract (reminder – that’s a delivery period and a delivery area) and it details the amount that they want to trade and the amount that they would like to buy or sell it for. An example order might be “For the period 12:00-13:00 in Germany, Jeff would like to buy 12 MW of power for €40”. Another order might be “For the period 12:00-13:00 in Germany, Sarah would like to sell 30 MW of power for €30”. For these cases, Jeff is willing to pay up to €40 (bid price) for that 12MW of energy that he needs, meanwhile Sarah is asking for at least €30 (ask price) for the 30MW of energy that she wants to sell. The market will see these two orders, see that Sarah could buy Jeff’s energy, and create that trade for them: 12 MW of power for €30.

Auction vs Continuous Trading

Trading can happen via continuous trading or auction. With continuous trading, as soon as two orders can be matched (the buyer is willing to pay at least as much as the seller is willing to sell for), a trade will be made immediately, and the orders will be removed from the market. In contrast, orders placed in auctions remain until the fixed time that auction closes. When the auction closes, the market will choose a clearing price which is determined by the best orders that have been placed (different markets have different calculations for this). Then buyers that have offered more than this clearing price will have trades, and similarly for sellers that have asked for less than this clearing price. Auctions can be blind (traders cannot see what orders have been placed by other people) or open (other orders and prices are visible, so traders can adjust their bid prices accordingly).

Day-ahead vs Intraday

The process of establishing exactly what energy is being traded for a particular day or hour doesn’t just happen in one step. It involves two separate but consecutive stages of trading: Day-ahead and Intraday. Intraday trading happens closer to the actual delivery of the electricity, which implies a couple of things: supply and demand predictions become more certain, and balance becomes more critical. Large companies will typically have different teams and departments concerned with each stage.

Day-ahead trading: As the name implies, Day-ahead concerns trading electricity for tomorrow. This is done as a blind auction: every day, trading for each delivery period of the following day opens at a fixed time and then closes at a later fixed time. Orders are placed and at the end of trading, the clearing price is chosen and trades are made. Europe now operates at 15-minute time periods, which is called the market time unit.

In day-ahead trading, marginal pricing applies. This means that all sellers are paid the same price regardless of what they offered to sell at if their orders are filled. This also means all buyers pay the same price if their orders are filled.

Intraday trading: At this stage you can trade energy for quarters (15 minute delivery periods), half-hours and hours for the current day (though this depends on both the market and the country border / delivery area. Sometimes there are longer periods available too). Intraday trading is done as both continuous trading as well as several auctions through the day. Orders can be placed and trades can be made right up to the delivery of the electricity (or a short time before, depending on the delivery area), which allows a balance to be reached.

Making money with Energy Trading

While the energy market is there for several reasons – to connect providers to consumers, to ensure consumers have the energy they need, to control the price of energy and to maintain balance in the grid – it is also a good opportunity to make money, and most traders will have profits in mind as they buy and sell.

As discussed, the price of energy is constantly in flux. If you sell energy when the price is high, you will earn more money. If you buy energy when the price is low, you will spend less. The difficulty, and the art, is in predicting how prices will change ahead of time. It’s a complicated enough task that companies will sometimes have entire departments dedicated to this effort, observing market trends and leveraging as much data as they can (household use throughout the day, weather trends and mountain snowmelt just to name a few) to produce the most accurate model possible. At the end of the day, it’s a data science problem.


  

    
  


Asset–Backed Trading

The most obvious way to make money on the energy market is by selling energy from your power station (Known as asset-backed trading). This is the least risky way to trade because you are selling a very real thing that you can provide. In terms of balancing the grid, if you only sell energy that you know you can generate and the person that buys it uses it all, then that trade results in zero grid imbalance. You should be savvy though because you’ll probably want to sell enough energy at high cost to cover the fuel and operational costs of your power station.

If you are a company that controls a hydroelectric dam then you have some extra tactics at your disposal. We call this pumped storage. A dam generates energy by releasing stored water downstream through a turbine. Picture a dam as a giant battery: More water in the dam represents more electricity that can be generated. The battery charges when it rains, or mountain snow melts and rivers flow to feed the dam - a perfect way of harnessing the environment for clean energy. You can also charge the battery by using energy to pump water from downstream back up into the dam, which is where a unique opportunity arises! Imagine buying energy at night when there is less demand and prices are low and using that energy to pump water into the dam. Then later during the day, when the prices are higher you can release that energy you stored, earning you a profit.


  

    
  


Speculative Trading

The other way to make money is through speculative trading. This is when you don’t own an asset that you can sell – you can only buy and sell energy that is on the market. The principle is simple: Buy energy now at a low cost and then sell that energy later at a higher cost (buy low sell high) or sell energy now at a high cost and then buy back later at a lower cost (sell high buy low). This is risky because it puts you in an exposed position.

Let me explain: Suppose you haven’t bought or sold anything yet. You have a position of 0. You then buy 10MW of power which will be delivered between 6pm and 7pm today, which puts you in a position of 10MW for that contract. This tells the market “I intend to use this 10MW of power between 7pm and 8pm”. If, in fact, you don’t intend on using up this energy then you have a problem because the grid will be imbalanced: Someone will be delivering this energy onto the grid that you are not consuming. Expect stern emails from the market, a large fine or even a summons to a meeting where you should explain exactly why you should not be banned from the market! So it is now your mission before trading ends to sell your 10MW to someone else (ideally at a higher price).

Suppose instead that you start with 0 position and then you sell 10MW of energy that will be delivered between 7pm and 8pm. Remember that you don’t own a power station; you have just sold energy that you don’t own. Not yet at least. You now have a position of -10MW for that contract. If you keep this negative position then at 7pm the buyer will begin to use 10MW of energy, you will not be putting that energy onto the grid and what do we get? Imbalance. And the penalties that come with it. So instead, it is your mission to buy back that 10MW from someone else (ideally at a lower price).

If you have a nonzero position and trading is coming to an end it is almost always in your interest to make trades to reduce your position to zero, even if it means losing money. The market and grid operators are not kind to those that disrupt the balance.

We call this position management, and it is a top concern of any trader. Always end trading with zero position, unless you intend to add energy to the grid or take energy away.

Summary

Trading is a complicated beast, and energy is a unique resource which comes with its own challenges. The energy market serves to maintain balance on the grid and to make sure consumers get the energy they need, when they need it, and at sensible prices. We have covered the energy grid and capacity management, energy pricing, the mechanics of putting orders on the market and making trades and the basic ideas of making money.

Like most things, this domain is a rabbit hole and there is plenty more to uncover, however this should give you a solid foundation. Armed with this knowledge, you should be able to walk confidently into your project meetings and understand the mechanisms behind whatever application you are building!



Functional Optics for Modern Java - Part 3 by Magnus Smith
Magnus Smith — Fri, 23 Jan 2026 00:00:00 +0000
Optics in Practice: An Expression Language AST

Part 3 of the Functional Optics for Modern Java series

In Part 1 and Part 2, 
we established why optics matter and how they work. Now it’s time to apply them to a real domain by building an expression language interpreter.

Expression languages are the backbone of modern Java infrastructure, from Spring Expression Language (SpEL) to rule engines like Drools. 
If you’ve ever configured a complex Spring application or written business rules, you’ve used one. In this article we will show how Java 25’s data-oriented programming features, 
combined with Higher-Kinded-J optics, make building such systems remarkably clean.

This is a domain where optics really shine.

Over the next three articles, we’ll build a complete expression language with parsing, type checking, optimisation, and interpretation. 
Along the way, you’ll see how optics transform what would otherwise be tedious tree manipulation into elegant, composable operations.

Running the Examples



Article Code

All code examples from this article have runnable demos:


  ExprDemo: Building ASTs, using prisms, and Focus DSL composition
  OptimiserDemo: Constant folding, identity simplification, and complex optimisation


The AST types are defined in org.higherkindedj.article3.ast, with transformations in org.higherkindedj.article3.transform.



The Expression Language Domain

What exactly are we building? A small but powerful expression language suitable for:


  Configuration expressions: if (env == "prod") then timeout * 2 else timeout
  Rule engines: price > 100 && customer.tier == "gold"
  Template systems: "Hello, " + user.name + "!"
  Domain-specific calculations: principal * (1 + rate) ^ years


The language will support:


  Literal values (integers, booleans, strings)
  Variables with lexical scoping
  Binary operations (arithmetic, comparison, logical)
  Conditional expressions (if-then-else)


Our design goals are:


  Type-safe: The compiler catches structural errors
  Immutable: Expressions never change; transformations produce new trees
  Transformable: Easy to analyse, optimise, and rewrite


This third goal is where optics become essential. An expression tree is a recursive structure where any node might contain arbitrarily nested sub-expressions. Transforming such trees manually (with pattern matching and reconstruction) quickly becomes unwieldy. Optics provide a disciplined approach.

Building on Data-Oriented Foundations

Part 1 introduced Java 25’s data-oriented programming features. As Brian Goetz articulates in Data-Oriented Programming in Java, the core insight is that “data is just data”: immutable, transparent, and separate from behaviour. Records, sealed interfaces, and pattern matching embody this philosophy beautifully.

Eric Normand’s work on data-oriented programming (from the Clojure tradition) takes this further: behaviour should be implemented as pure functions over immutable data, with polymorphism achieved through pattern matching rather than method dispatch. Java 25 now supports this style naturally.

Yet there’s a tension that pure DOP does not fully address. When behaviour is separate from data, where do structural operations live? Functions that transform nested trees must understand the shape of every node. In Normand’s terms, we’ve separated “what the data is” from “what we do with it”, but tree transformations blur this boundary: they’re operations intimately tied to structure.

This is precisely where optics prove their worth. They’re not behaviour embedded in data (that would violate DOP principles). They’re reified access paths; first-class values representing the structure of your types. The structure of a record implies its lenses; the variants of a sealed interface imply its prisms. Optics make this correspondence explicit and composable.

A Map Analogy: Optics as Directions

Think of optics like directions in a treasure map. The treasure (This is your data) doesn’t know the directions to itself; it just is. But the directions (The optics) are separate, shareable instructions that anyone can follow.

A lens is like directions to a specific room in a house: “Go to the kitchen.” You’ll always find the kitchen there. A prism is like directions that might not apply: “If there’s a garden, go to the fountain.” Some houses have gardens; some don’t. A traversal is like directions to multiple destinations: “Visit every bedroom.”

The power comes from composition. “Go to the garden, then to the fountain, then to the bench” chains directions together. If any step fails (no garden?), you stop gracefully. Optics compose the same way: each step may narrow, widen, or multiply your focus, and the types track this automatically.



Designing the AST

An Abstract Syntax Tree (AST) represents the structure of code as a tree of nodes. Each node type corresponds to a language construct.

Start Simple

We’ll begin with a minimal four-variant AST:

public sealed interface Expr {
    record Literal(Object value) implements Expr {}
    record Variable(String name) implements Expr {}
    record Binary(Expr left, BinaryOp op, Expr right) implements Expr {}
    record Conditional(Expr cond, Expr thenBranch, Expr elseBranch) implements Expr {}
}

public enum BinaryOp {
    ADD, SUB, MUL, DIV,    // Arithmetic
    EQ, NE, LT, LE, GT, GE, // Comparison
    AND, OR                 // Logical
}


This covers more than you might expect:


  Literal(42): integer constant
  Literal(true): boolean constant
  Variable("x"): variable reference
  Binary(Variable("a"), ADD, Literal(1)): a + 1
  Conditional(Variable("flag"), Literal(1), Literal(0)): if flag then 1 else 0


Here’s how (a + 1) * 2 looks as an AST:



The recursive nature is already apparent: Binary contains two Expr children, and Conditional contains three. Any transformation must handle this recursion.

Why Sealed Interfaces?

The sealed keyword is Java’s answer to sum types: a closed set of variants that the compiler understands completely. When we write a switch over Expr, Java 25 guarantees exhaustiveness:

String describe(Expr expr) {
    return switch (expr) {
        case Literal(var v) -> "Literal: " + v;
        case Variable(var n) -> "Variable: " + n;
        case Binary(var l, var op, var r) -> "Binary: " + op;
        case Conditional(_, _, _) -> "Conditional";
    };  // No default needed; compiler knows this is exhaustive
}


Notice the unnamed pattern _ for components we don’t need, a Java 22+ feature that reduces noise. If we later add a new variant like FunctionCall, the compiler flags every incomplete switch. This compile-time safety is essential for language implementations where missing a case means silent bugs.

Why Records?

Records give us:


  Immutability by default
  Automatic equals(), hashCode(), toString()
  Pattern matching with deconstruction
  A natural fit for optics (each component becomes a lens target)


The combination of sealed interfaces and records creates what functional programmers call an algebraic data type (ADT): a sum of products that’s both type-safe and pattern-matchable.

The Visitor Pattern: A DOP Perspective

The traditional Visitor pattern embeds traversal logic into your data types via accept() methods. This violates a core DOP principle: data should be transparent and behaviour-free. As Goetz notes, the Visitor pattern exists largely because Java lacked pattern matching; it’s a workaround, not a solution.

With sealed interfaces and pattern matching, we can write operations as standalone functions:

Expr foldConstants(Expr expr) {
    return switch (expr) {
        case Binary(Literal(var l), var op, Literal(var r)) ->
            evaluate(l, op, r).map(Literal::new).orElse(expr);
        case Binary(var l, var op, var r) ->
            new Binary(foldConstants(l), op, foldConstants(r));
        case Conditional(var c, var t, var e) ->
            new Conditional(foldConstants(c), foldConstants(t), foldConstants(e));
        case Literal _, Variable _ -> expr;
    };
}


No interfaces to implement, no accept() methods polluting our data types, and the compiler verifies exhaustiveness. This is DOP in action: pure data, external functions, pattern-matched polymorphism.

Yet notice the manual reconstruction in the Binary and Conditional cases. This recursive boilerplate appears in every tree transformation. DOP gives us improved reading; the reconstruction cascade remains.



Generating Optics for the AST

This reconstruction problem is precisely what optics solve. With three annotations, Higher-Kinded-J generates a complete toolkit for navigating and transforming our AST:

@GeneratePrisms  // Generates prisms for each sealed variant
public sealed interface Expr {
    @GenerateLenses @GenerateFocus
    record Literal(Object value) implements Expr {}

    @GenerateLenses @GenerateFocus
    record Variable(String name) implements Expr {}

    @GenerateLenses @GenerateFocus
    record Binary(Expr left, BinaryOp op, Expr right) implements Expr {}

    @GenerateLenses @GenerateFocus
    record Conditional(Expr cond, Expr thenBranch, Expr elseBranch) implements Expr {}
}


At compile time, Higher-Kinded-J’s annotation processor generates:

Prisms for Each Variant

// Generated: ExprPrisms.java
public final class ExprPrisms {
    public static Prism<Expr, Literal> literal() { ... }
    public static Prism<Expr, Variable> variable() { ... }
    public static Prism<Expr, Binary> binary() { ... }
    public static Prism<Expr, Conditional> conditional() { ... }
}


Each prism lets us:


  Check if an Expr is a specific variant
  Extract the variant if it matches
  Transform just that variant, leaving others unchanged


Lenses for Each Field

// Generated: LiteralLenses.java
public final class LiteralLenses {
    public static Lens<Literal, Object> value() { ... }
}

// Generated: BinaryLenses.java
public final class BinaryLenses {
    public static Lens<Binary, Expr> left() { ... }
    public static Lens<Binary, BinaryOp> op() { ... }
    public static Lens<Binary, Expr> right() { ... }
}


Each lens lets us:


  Get a field from a node
  Set a field, producing a new node
  Modify a field with a function


Focus DSL: Fluent Navigation

The @GenerateFocus(generateNavigators = true) annotation generates Focus classes with fluent navigation:

// Generated: BinaryFocus.java
public final class BinaryFocus {
    public static FocusPath<Binary, Expr> left() { ... }
    public static FocusPath<Binary, BinaryOp> op() { ... }
    public static FocusPath<Binary, Expr> right() { ... }
}


Focus classes wrap lenses in FocusPath objects that enable method chaining without explicit composition. The difference becomes clear when you navigate nested structures.

Composition: Where Optics Earn Their Keep

The benefit comes when we compose these optics. Here’s the traditional approach with raw optics:

// Raw optics: explicit composition with andThen
Affine<Binary, Object> leftLiteralValue =
    BinaryLenses.left()
        .andThen(ExprPrisms.literal())
        .andThen(LiteralLenses.value());

Binary expr = new Binary(new Literal(5), ADD, new Variable("x"));
Optional<Object> value = leftLiteralValue.getOptional(expr);
// Optional[5]


With the Focus DSL, this becomes fluent navigation:

// Focus DSL: fluent method chaining
Binary expr = new Binary(new Literal(5), ADD, new Variable("x"));
Optional<Object> value = BinaryFocus.left()
    .via(ExprPrisms.literal())
    .via(LiteralLenses.value())
    .getOptional(expr);
// Optional[5]


Both approaches navigate from Binary → left (Expr) → as Literal → value, but the Focus DSL reads more naturally. As we add navigators in Part 4, this becomes even more readable.

Why Optics Instead of Just Pattern Matching?

You might wonder: “Java 25’s pattern matching is already elegant; why add another abstraction?”

Three compelling reasons:

1. Composition Across Depth

Pattern matching handles one level at a time. To reach a deeply nested value, you nest matches:

// Pattern matching: explicit nesting
if (expr instanceof Binary(Binary(var ll, _, _), _, _)) {
    // Access left-of-left
}


Optics compose to arbitrary depth. With the Focus DSL, it reads naturally:

// Focus DSL: fluent path
AffinePath<Expr, Expr> leftOfLeft = ExprPrisms.binary()
    .via(BinaryFocus.left())
    .via(ExprPrisms.binary())
    .via(BinaryFocus.left());

// Or with navigators (preview of Article 4):
// BinaryFocus.left().asBinary().left()


2. Bidirectional Access

Pattern matching reads data. Optics read and write:

// With Focus DSL paths:
FocusPath<Binary, Expr> leftPath = BinaryFocus.left();

// Read the left operand
Expr left = leftPath.get(binaryExpr);

// Replace the left operand (returns new immutable tree)
Binary updated = leftPath.set(newLeft, binaryExpr);

// Transform the left operand
Binary transformed = leftPath.modify(this::optimize, binaryExpr);


The modify operation is particularly powerful: it extracts, transforms, and rebuilds in one step, handling all the immutable reconstruction automatically. The Focus DSL makes this read naturally.

3. Reusable Paths

A pattern match is code you write inline. An optic is a value you can store, pass, and reuse:

// Define once using Focus DSL
private static final FocusPath<Binary, Expr> leftOperand = BinaryFocus.left();

// Use anywhere
Binary opt1 = leftOperand.modify(this::fold, expr1);
Binary opt2 = leftOperand.modify(this::fold, expr2);


Focus paths are first-class values. Store them in fields, pass them to methods, compose them dynamically. This becomes invaluable when the same navigation pattern appears across multiple transformations.



Basic Transformations

Let’s implement some fundamental AST transformations using optics.

Transforming Literals

Suppose we want to increment all integer literals by one:

public static Expr incrementLiterals(Expr expr) {
    Prism<Expr, Literal> literalPrism = ExprPrisms.literal();

    return literalPrism.modify(lit -> {
        if (lit.value() instanceof Integer i) {
            return new Literal(i + 1);
        }
        return lit;
    }, expr);
}


There is a problem here. This only transforms the top-level expression. If the literal is nested inside a Binary, it won’t be touched. We need recursion.

The Recursive Challenge

Here’s the manual approach to transforming all literals in a tree:

public static Expr incrementAllLiterals(Expr expr) {
    return switch (expr) {
        case Literal(var v) ->
            v instanceof Integer i ? new Literal(i + 1) : expr;
        case Variable(_) -> expr;
        case Binary(var l, var op, var r) ->
            new Binary(incrementAllLiterals(l), op, incrementAllLiterals(r));
        case Conditional(var c, var t, var e) ->
            new Conditional(
                incrementAllLiterals(c),
                incrementAllLiterals(t),
                incrementAllLiterals(e));
    };
}


This works, but it’s very tedious. Every transformation requires the same recursive boilerplate code. We’re manually threading the transformation through every node type.

A Reusable Transformation Pattern

We can extract the recursion into a reusable function:

public static Expr transformExpr(Expr expr, Function<Expr, Expr> transform) {
    // First, recursively transform children
    Expr transformed = switch (expr) {
        case Literal(_), Variable(_) -> expr;
        case Binary(var l, var op, var r) ->
            new Binary(transformExpr(l, transform), op, transformExpr(r, transform));
        case Conditional(var c, var t, var e) ->
            new Conditional(
                transformExpr(c, transform),
                transformExpr(t, transform),
                transformExpr(e, transform));
    };

    // Then apply the transformation to this node
    return transform.apply(transformed);
}


Now our increment becomes:

Expr result = transformExpr(expr, e ->
    ExprPrisms.literal().modify(lit ->
        lit.value() instanceof Integer i ? new Literal(i + 1) : lit,
        e));


In Part 4, we’ll see how traversals improve this further. For now, let’s work with what we have.



Working with the Sum Type

Prisms shine when working with the variants of our sealed interface.

Safe Type Checking

Traditional instanceof checks are verbose and error-prone:

if (expr instanceof Binary binary) {
    if (binary.left() instanceof Literal leftLit) {
        // do something with leftLit
    }
}


With the Focus DSL, we compose the checks fluently:

// Compose prism and lens with andThen(), yielding an Affine
// Then wrap in AffinePath for fluent operations
AffinePath<Expr, Object> leftLiteralValue =
    AffinePath.of(
        ExprPrisms.binary()
            .andThen(BinaryLenses.left())
            .andThen(ExprPrisms.literal())
            .andThen(LiteralLenses.value()));

Optional<Object> value = leftLiteralValue.getOptional(expr);


The composed path handles all the type checking internally. If expr isn’t a Binary, or if its left operand isn’t a Literal, the result is Optional.empty().

Conditional Transformation

Prisms let us transform specific variants while leaving others unchanged:

// Double all integer literals, leave everything else alone
Expr doubled = ExprPrisms.literal().modify(lit -> {
    if (lit.value() instanceof Integer i) {
        return new Literal(i * 2);
    }
    return lit;
}, expr);


If expr isn’t a Literal, it’s returned unchanged. No explicit instanceof check is needed.

Pattern-Based Matching

We can combine pattern matching with optics for complex structural tests:

// Match: Binary with ADD operator and Literal(0) on the right
// This is the pattern for "x + 0" which we can simplify to "x"
public static Optional<Expr> matchAddZero(Expr expr) {
    return switch (expr) {
        case Binary(var left, BinaryOp.ADD, Literal(Integer v)) when v == 0 ->
            Optional.of(left);
        default -> Optional.empty();
    };
}


This pattern matching is where Java’s native features work well alongside optics. Use pattern matching for complex structural tests; use optics for transformations and deep access.



Building a Simple Optimiser

Let’s put everything together to build a constant folder; an optimiser that evaluates constant expressions at compile time.



Constant Folding

Here is the idea. If both operands of a binary expression are literals, we can compute the result:

public static Expr foldConstants(Expr expr) {
    return transformExpr(expr, ExprOptimiser::foldBinary);
}

private static Expr foldBinary(Expr expr) {
    // Java 25: Switch with nested record patterns
    return switch (expr) {
        case Binary(Literal(Object lv), BinaryOp op, Literal(Object rv)) -> {
            Object result = evaluate(lv, op, rv);
            yield result != null ? new Literal(result) : expr;
        }
        default -> expr;
    };
}

private static Object evaluate(Object left, BinaryOp op, Object right) {
    return switch (left) {
        case Integer l when right instanceof Integer r -> switch (op) {
            case ADD -> l + r;
            case SUB -> l - r;
            case MUL -> l * r;
            case DIV -> r != 0 ? l / r : null;
            case EQ -> l.equals(r);
            case NE -> !l.equals(r);
            case LT -> l < r;
            case LE -> l <= r;
            case GT -> l > r;
            case GE -> l >= r;
            default -> null;
        };
        case Boolean l when right instanceof Boolean r -> switch (op) {
            case AND -> l && r;
            case OR -> l || r;
            case EQ -> l.equals(r);
            case NE -> !l.equals(r);
            default -> null;
        };
        default -> null;
    };
}


Example: Folding (1 + 2) * 3

Expr expr = new Binary(
    new Binary(new Literal(1), ADD, new Literal(2)),
    MUL,
    new Literal(3)
);

System.out.println("Before: " + format(expr));
// Before: ((1 + 2) * 3)

Expr optimised = foldConstants(expr);
System.out.println("After: " + format(optimised));
// After: 9


The optimiser transforms ((1 + 2) * 3) to 9 in a single pass. The inner 1 + 2 is folded to 3, then 3 * 3 is folded to 9.

Identity Simplification

We can add more optimisations using Java 25’s enhanced pattern matching with guards:

private static Expr simplifyIdentities(Expr expr) {
    return switch (expr) {
        // x + 0 = x, x * 1 = x
        case Binary(var left, BinaryOp.ADD, Literal(Integer v)) when v == 0 -> left;
        case Binary(var left, BinaryOp.MUL, Literal(Integer v)) when v == 1 -> left;
        // x * 0 = 0
        case Binary(_, BinaryOp.MUL, Literal(Integer v)) when v == 0 -> new Literal(0);
        // 0 + x = x, 1 * x = x
        case Binary(Literal(Integer v), BinaryOp.ADD, var right) when v == 0 -> right;
        case Binary(Literal(Integer v), BinaryOp.MUL, var right) when v == 1 -> right;
        // 0 * x = 0
        case Binary(Literal(Integer v), BinaryOp.MUL, _) when v == 0 -> new Literal(0);
        default -> expr;
    };
}


Composing Optimisations

Multiple optimisation passes compose quite naturally:

public static Expr optimise(Expr expr) {
    Expr result = expr;
    Expr previous;

    // Run until fixed point (no more changes)
    do {
        previous = result;
        result = transformExpr(result, e ->
            simplifyIdentities(foldBinary(e)));
    } while (!result.equals(previous));

    return result;
}


This runs both optimisations repeatedly until the expression stops changing. The immutability of our AST makes equality checking trivial; we can use equals() directly.



Summary

We’ve built a solid foundation for expression language development using Java 25’s data-oriented programming features:


  Sealed interfaces define a closed universe of expression types
  Records provide immutable, transparent data carriers
  Pattern matching enables clean, exhaustive case analysis
  Optics (via Higher-Kinded-J) add composable, bidirectional transformations
  Focus DSL makes optic composition fluent and readable


Optics and the DOP Philosophy

There’s a deeper point here worth pausing on. Critics of pure data-oriented programming sometimes observe that while DOP excels at describing data, it offers less guidance on transforming it. Pattern matching destructures beautifully but reconstructs tediously. The purity of “data separate from behaviour” runs into friction when behaviour must intimately understand data’s shape.

Optics resolve this tension deftly. They’re not methods on your data types (that would embed behaviour). They’re not external functions that pattern-match (that leads to reconstruction cascades). They’re something new: structural correspondences reified as values.

The @GenerateLenses and @GeneratePrisms annotations embody this insight. The structure of a record implies its lenses; the variants of a sealed interface imply its prisms. Higher-Kinded-J makes these implications explicit and composable. You’re not adding behaviour to data; you’re deriving navigation from structure.

This is why optics feel natural alongside DOP rather than against it. Both emphasise that structure should be transparent and operations should compose. Optics simply extends these principles from reading to writing.



What’s Next

There’s a limitation in our current approach: the transformExpr function is hand-written boilerplate. Every time we add a new expression type, we must update it. This violates the DRY principle and risks bugs when someone forgets.

In Part 4, we’ll introduce traversals and the Focus DSL’s TraversalPath: optics that focus on multiple values simultaneously. This is where the Focus DSL becomes powerful:

// Preview: what's coming in Part 4
TraversalPath<Company, Employee> allEmployees = CompanyFocus
    .departments()
    .each()          // Navigate into each department
    .employees()
    .each();         // Navigate into each employee

// Modify all employees in one expression
Company updated = allEmployees.modifyAll(Employee::promote, company);


With the Focus DSL’s collection navigation (each(), at(), atKey()), we can:


  Eliminate manual recursion: TraversalPath.each() replaces switch-based boilerplate
  Collect values fluently: allEmployees.getAll(company) returns all focused values
  Chain arbitrarily deep: Navigate through nested collections with method chains
  Apply conditional updates: modifyWhen() transforms only elements matching a predicate


We’ll also tackle dead code elimination and common subexpression elimination, demonstrating how the Focus DSL scales to real compiler optimisations.



Further Reading

Algebraic Data Types in Java


  
    Brian Goetz, “Towards Better Serialization”: Explores how records and sealed types create ADTs, with implications for pattern matching.
  
  
    Scott Logic, “Algebraic Data Types and Pattern Matching with Java”: A thorough introduction to product types, sum types, and pattern matching in modern Java. Demonstrates how sealed interfaces and records implement ADTs with compile-time safety guarantees.
  


Expression Languages and AST Design


  
    Terence Parr, Language Implementation Patterns (Pragmatic Bookshelf, 2010): The definitive guide to building interpreters and compilers, covering AST design patterns we use here.
  
  
    Robert Nystrom, Crafting Interpreters: A freely available, nicely written guide to interpreter construction. The “tree-walk interpreter” chapters parallel our approach.
  


DOP and the Expression Problem


  
    Philip Wadler, “The Expression Problem” (1998): The classic formulation of the tension between adding new data types versus new operations. DOP with optics offers one resolution.
  
  
    The expression problem asks: can you add both new variants and new operations without modifying existing code? Sealed interfaces make adding operations easy (write a function); optics make the operations themselves composable and reusable.
  


Higher-Kinded-J


  
    Prism Documentation: API reference for working with sum types.
  
  
    Focus DSL Guide: Fluent navigation with FocusPath, AffinePath, and TraversalPath.
  
  
    Effect Path API Guide: Railway-style error handling with MaybePath, EitherPath, and ValidationPath.
  




Next time

Next time in Part 4 we look at how we visit all nodes in the tree, not just the top level. This is where traversals become essential. A traversal focuses on zero or more elements within a structure, making it perfect for recursive tree operations.

Expression Type	Sub-expressions
`Literal`	None
`Variable`	None
`Binary`	`left`, `right`
`Conditional`	`cond`, `then_`, `else_`

Effect Path	Wraps	Use Case
`MaybePath`	`Maybe`	Optional values (might be absent)
`EitherPath`	`Either`	Fail-fast error handling
`ValidationPath`	`Validated`	Error accumulation
`TryPath`	`Try`	Exception handling
`IOPath`	`IO`	Deferred side effects

Architecture	When to use
SaaS Big Data Platform	Many types of data sources and large volumes of data
Lightweight Platform	Single type of data source and small to medium amounts of data

Joiner	Behaviour	Use Case
`allSucceed`	Wait for all, fail-fast on error	Batch operations that must all complete
`anySucceed`	First success wins, cancel others	Redundant calls, fallbacks
`accumulating`	Collect all results OR all errors	Validation, comprehensive reporting

Challenge	Traditional Java	Higher-Kinded-J	Details
Deep updates	Copy-constructor cascade (25+ lines)	Lens composition (1 line)	Part 1
Null handling	`if (x != null)` chains	`MaybePath` makes absence explicit	Part 5
Error handling	Nested try-catch pyramids	`EitherPath` railway model	Part 5
Validation	First-error-only	`ValidationPath` accumulates ALL errors	Part 5
Concurrency	`CompletableFuture` callbacks	`VTaskPath` + `Scope`	Part 5

Capability	What It Does
Zero-config	Add dependency, return `Either`/`Validated` from controllers
Auto status mapping	Error class names map to HTTP status codes
Error accumulation	`Validated, User>` returns ALL validation errors
Async support	`CompletableFuturePath` for non-blocking operations
Actuator metrics	Track success/error rates, latency percentiles
Security integration	`ValidatedUserDetailsService` for functional authentication
JSON serialisation	Configurable formats (TAGGED, UNWRAPPED, DIRECT)

Pattern	Example Types	Generated Optics
Records	Third-party records	Lenses via constructor
Sealed interfaces	Sum types	Prisms for each variant
Enums	Status codes	Prisms for each constant
Wither classes	`LocalDate`, `LocalTime`	Lenses via `withX()` methods
Builders	JOOQ, Protobuf	Via spec interfaces

Pattern	Description	When to Use
Focus-Path-First	Annotate types with `@GenerateFocus`, express operations as paths	Any nested domain model
Effect Stratification	Different phases use different effects (Either for parsing, Validated for checking)	Pipelines with distinct stages
Paths as Configuration	Store paths as values, compose at runtime	Configurable queries, reporting
Validated for Users	Accumulate all errors for user-facing validation	Forms, API requests

Effect Path	Contains	Use Case
`MaybePath`	Value or nothing	Optional data, silent failures
`EitherPath`	Error or value	Fail-fast error handling
`TryPath`	Exception or value	Wrapping throwing code
`ValidationPath`	Errors or value	Accumulating all problems
`IOPath`	Deferred computation	Side effects, resource management
`VTaskPath`	Virtual thread computation	Concurrent operations, parallelism

Aspect	VTaskPath	IOPath
Thread model	Virtual threads	Caller’s thread
Parallelism	Built-in via `Par`, `Scope`	Manual composition
Structured concurrency	Yes, with `Scope`	No
Best for	I/O-bound concurrent work	Single-threaded effects

Level	API	Best For
High	Effect Path API	Most application code, clear intent
Low	`modifyF` with `Kind`	Libraries, custom effects, maximum flexibility