Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/blog-prose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ jobs:
# base version of each changed post.
fetch-depth: 0

- name: Detect changed blog posts
- name: Detect changed blog and social prose
id: detect
run: |
set -euo pipefail
Expand All @@ -52,9 +52,9 @@ jobs:
if [ -z "$mb" ]; then
count=0
else
count="$(git diff --name-only --diff-filter=ACMR "$mb" HEAD -- 'docs/website/content/blog/*.md' | wc -l | tr -d ' ')"
count="$(git diff --name-only --diff-filter=ACMR "$mb" HEAD -- 'docs/website/content/blog/*.md' 'docs/website/social/linkedin/*.md' | wc -l | tr -d ' ')"
fi
echo "Changed blog posts: $count"
echo "Changed blog and social files: $count"
if [ "$count" -gt 0 ]; then
echo "has_posts=true" >> "$GITHUB_OUTPUT"
else
Expand Down
169 changes: 169 additions & 0 deletions docs/website/content/blog/accessibility-semantics.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,169 @@
---
title: "Accessibility Semantics: The UI Tree You Cannot See"
slug: accessibility-semantics
url: /blog/accessibility-semantics/
date: '2026-07-20'
author: Shai Almog
description: "Codename One now builds a portable accessibility semantics tree for its lightweight components and maps it to VoiceOver, TalkBack, UI Automation, AT-SPI, Java accessibility, and web ARIA."
feed_html: '<img src="https://www.codenameone.com/blog/accessibility-semantics.jpg" alt="Accessibility semantics parallel UI tree" /> Codename One now exposes a portable semantics tree to VoiceOver, TalkBack, UI Automation, AT-SPI, Java accessibility, and web ARIA.'
series: ["release-2026-07-17"]
---

![Accessibility Semantics: The UI Tree You Cannot See](/blog/accessibility-semantics.jpg)

Accessibility has become personal for me. I am getting older, and large type is no longer an abstract preference somebody else needs. It is how I read a phone comfortably.

I worked with accessibility experts at Sun Microsystems and learned how deep the problem goes. A label is the easy part. Real accessibility needs roles, values, ranges, actions, traversal order, live announcements, collections, focus, platform conventions, and a way to test all of it. That complexity is why full Codename One accessibility support sat dormant for a decade.

We eventually added `setAccessibilityText()`. It was useful, but it was the poor man's version. [PR #5363](https://github.com/codenameone/CodenameOne/pull/5363) replaces that single-label model with a portable semantics tree we can be proud of.

## Lightweight UI needs a second tree

Codename One paints lightweight components into its own native surface. VoiceOver cannot inspect a `Button` as a UIKit button because there is no UIKit button there. TalkBack cannot walk an Android `View` hierarchy because most of the painted controls are not Android views.

The new accessibility manager builds an immutable virtual tree beside the visual component tree. Standard controls infer their semantics. Custom controls can replace or extend them. Each port exposes that virtual tree through the platform accessibility API.

{{< mermaid >}}
flowchart TD
A["Codename One component tree"] --> B["Portable semantics tree"]
B --> C["VoiceOver<br/>UIAccessibilityElement"]
B --> D["TalkBack<br/>AccessibilityNodeProvider"]
B --> E["Windows<br/>UI Automation"]
B --> F["Linux<br/>ATK / AT-SPI"]
B --> G["Java SE<br/>AccessibleContext"]
B --> H["Web<br/>off-screen ARIA DOM"]
{{< /mermaid >}}

The visual and semantic hierarchies can differ. A card made from five labels might need to read as one item. A chart may paint 200 points from one component, but expose each meaningful point as a virtual child. A renderer-backed list can expose stable rows even though those rows are not component instances.

## Standard components work without annotations

Buttons, checkboxes, radio buttons, sliders, text fields, lists, tables, tabs, labels, dialogs, and containers infer their normal roles, values, states, and actions. Existing `setAccessibilityText()` calls continue to work as a compatibility alias for the semantic label.

You only add code when the inferred result is incomplete or the UI represents something more specific:

```java
Button save = new Button("Save");
save.getSemantics()
.setHint("Saves the edited profile")
.setIdentifier("profile-save");
```

A custom switch can supply its role and checked state:

```java
wifiSwitch.getSemantics()
.setRole(AccessibilityRole.SWITCH)
.setLabel("Wi-Fi")
.setChecked(AccessibilityCheckedState.CHECKED)
.setEnabled(Boolean.TRUE)
.setHint("Double tap to turn Wi-Fi off");
```

Identifiers are for tooling and stable tests. Labels are for people. Keeping them separate avoids tests that break when product copy changes.

## Semantics are more than labels

The API covers the parts a label-only layer cannot express:

- `AccessibilityRange` describes minimum, maximum, current value, step size, and spoken value for sliders and progress controls.
- `AccessibilityAction` exposes standard activation plus named actions such as Archive or Delete.
- `AccessibilityGrouping` merges descendants, treats a container as a group, or hides decorative subtrees.
- Sort keys and traversal constraints change reading order without changing paint order.
- Collection metadata describes row and column counts, spans, position in a set, and selection behavior.
- Live regions announce status changes with polite or assertive priority.
- Virtual child providers expose semantic items that have no component instance.

Here is a chart exposing each data point as an accessible virtual child:

```java
chart.getSemantics().setChildProvider(owner -> {
List<AccessibilityNode> result = new ArrayList<>();
for (ChartPoint point : points) {
AccessibilityNode node = new AccessibilityNode(
"point-" + point.getId());
node.setRole(AccessibilityRole.IMAGE)
.setLabel(point.getLabel())
.setValue(point.getFormattedValue())
.setBounds(point.getBounds());
result.add(node);
}
return result;
});
```

That same tree can represent a list cell, a map marker, a game menu item, or any custom renderer where the meaningful objects do not map one-to-one to components.

## Preferences can change the UI before a screen reader arrives

Accessibility includes users who never enable VoiceOver or TalkBack. The new APIs expose high contrast, reduce motion, reduce transparency, differentiate without color, and known color-vision deficiency preferences.

```java
if (CN.isHighContrastEnabled()) {
chart.setUIID("HighContrastChart");
}
if (CN.isReduceMotionEnabled()) {
chart.putClientProperty("animate", Boolean.FALSE);
}
if (CN.isReduceTransparencyEnabled()) {
chart.setUIID("OpaqueChart");
}
AccessibilityColorVisionDeficiency colorVision =
CN.getColorVisionDeficiency();
if (CN.isDifferentiateWithoutColorEnabled()
|| (colorVision != AccessibilityColorVisionDeficiency.NONE
&& colorVision != AccessibilityColorVisionDeficiency.UNKNOWN)) {
status.setText("Disconnected: action required");
}
```

Color must never be the only signal for important state. The preference is an extra input, not permission to hide the text or icon when the preference is absent.

The simulator now lets you force these states, including combinations that are awkward to reproduce on a physical device.

![Simulator accessibility preferences for motion, transparency, contrast, and color vision](/blog/accessibility-semantics/simulator-preferences.png)

## The inspector audits the resolved tree

The Component Inspector has an Accessibility tab that shows the tree the platform will receive. It flags unlabeled interactive nodes, duplicate identifiers, invalid ranges, contradictory state, traversal cycles, and other machine-detectable failures.

![Component Inspector showing an accessibility audit](/blog/accessibility-semantics/component-inspector-audit.png)

You can put the same checks in a unit or screenshot test:

```java
AccessibilityTreeSnapshot tree =
AccessibilityInspector.snapshot(form);

AccessibilityAssertions.assertNoErrors(tree);
AccessibilityAssertions.assertNoUnlabeledInteractiveNodes(tree);

AccessibilityNodeSnapshot save =
tree.getNodeByIdentifier("profile-save");
if (save.getRole() != AccessibilityRole.BUTTON) {
throw new AssertionError("profile-save must expose the button role");
}
if (save.getAction(AccessibilityAction.ACTIVATE) == null) {
throw new AssertionError("profile-save must expose the activate action");
}
```

The snapshot is immutable and can be serialized as JSON. That makes failures reviewable in CI and gives a bug report something more precise than “VoiceOver skipped my button.”

## Every platform still gets a human pass

An automated audit can prove that an interactive node has a label. It cannot prove that VoiceOver speaks the right sentence, that TalkBack focus recovers after a dialog closes, or that a Windows screen-reader user understands a custom collection.

The final check still uses VoiceOver, TalkBack, Narrator, Orca, Java Access Bridge, or browser accessibility tools. Navigate in both directions. Activate every action. Change adjustable values. Enter and leave collections. Trigger errors and live updates. Confirm focus after navigation, deletion, and modal dialogs.

That is the boundary. The portable semantics tree removes the architectural wall and makes most behavior testable once. Platform assistive technologies still apply their own presentation rules.

The surprise is that this tree also describes the screen to software. Tomorrow's post shows how the same immutable snapshot lets an AI agent inspect and drive the simulator over the Model Context Protocol without relying on screenshot coordinates.

---

## Discussion

_Join the conversation via GitHub Discussions._

{{< giscus >}}
130 changes: 130 additions & 0 deletions docs/website/content/blog/codename-one-mcp-server.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
---
title: "Your Codename One App Can Be an MCP Server"
slug: codename-one-mcp-server
url: /blog/codename-one-mcp-server/
date: '2026-07-21'
author: Shai Almog
description: "The Codename One JavaSE port can expose a running simulator or desktop tool over the Model Context Protocol. An agent reads the accessibility semantics tree, performs UI actions on the EDT, and can call application-defined tools."
feed_html: '<img src="https://www.codenameone.com/blog/codename-one-mcp-server.jpg" alt="Codename One application exposed as an MCP server" /> A running Codename One simulator or JavaSE tool can expose its semantic UI and application tools to coding agents over MCP.'
series: ["release-2026-07-17"]
---

![Your Codename One App Can Be an MCP Server](/blog/codename-one-mcp-server.jpg)

[Yesterday's accessibility work](/blog/accessibility-semantics/) created an immutable tree that describes what is on screen, what each item means, and which actions it supports. VoiceOver and TalkBack consume that tree for people.

[PR #5377](https://github.com/codenameone/CodenameOne/pull/5377) lets an agent consume it too.

The JavaSE port can expose a running simulator or Codename One desktop tool as a local [Model Context Protocol](https://modelcontextprotocol.io/) server. Codex, Claude Code, Claude Desktop, opencode, or another MCP host can inspect the current form, find a field by label, enter text, activate a button, and call tools published by the application.

## The agent reads meaning, not coordinates

Screenshot automation sees colored rectangles and guesses where to click. MCP exposes the same resolved semantics that accessibility technology receives:

```json
{
"id": "profile-save",
"role": "button",
"label": "Save",
"enabled": true,
"actions": ["activate"]
}
```

The agent can ask for `ui_snapshot`, find `profile-save`, then invoke `activate`. It does not need to assume the button stayed at yesterday's x and y coordinates.

{{< mermaid >}}
sequenceDiagram
participant Agent as MCP host
participant Server as Codename One MCP server
participant Tree as Accessibility snapshot
participant EDT as Codename One EDT
Agent->>Server: ui_snapshot
Server->>Tree: build immutable semantics tree
Tree-->>Agent: roles, labels, values, actions
Agent->>Server: ui_set_text(profile-name, "Ada")
Server->>EDT: dispatch action
EDT-->>Server: success + fresh snapshot
Server-->>Agent: updated UI state
{{< /mermaid >}}

The built-in tools are small on purpose:

| Tool | Purpose |
|---|---|
| `ui_snapshot` | Return the current semantic UI tree as JSON |
| `ui_find` | Find nodes by identifier, label, or screen coordinate |
| `ui_perform_action` | Run a semantic action with an optional argument |
| `ui_activate` | Activate a node |
| `ui_set_text` | Set editable text through the UI action model |

Every action runs on the Codename One event dispatch thread. The agent never mutates the live component tree from the MCP transport thread. Each action returns a fresh snapshot so the next decision uses current state.

## Starting a server is explicit

There is no build hint that quietly exposes an application. Calling the API is the switch:

```java
MCP.startSocketServer(8765);
```

The socket mode is useful for a running simulator session a person can watch. A tool launched directly by an MCP host can use standard input and output instead:

```java
MCP.startStdioServer();
```

The JavaSE port owns the stdio transport because process standard input is not available on every Codename One target. While the transport is active, normal application logging is redirected away from standard output so it cannot corrupt the newline-delimited JSON-RPC stream.

## Desktop tools get a menu without application code

The JavaSE port adds an MCP menu to the simulator and Codename One desktop tools, including the new Settings editor. The menu can expose the running tool, detect installed MCP hosts, install or remove the local host registration, and control debug logging.

Registration uses a small bridge. Most local MCP hosts launch servers over stdio. The visible Codename One tool is already running and listens on a loopback socket. `MCPStdioLauncher` relays between the host's stdio connection and that socket.

```text
Coding agent <-- stdio --> MCPStdioLauncher <-- loopback --> running tool
```

This is how an agent can drive the actual Certificate Wizard or Settings window in front of you instead of launching a hidden copy with different state.

## Your app can publish domain tools

UI actions are useful, but some operations should not be simulated as clicks. An application can expose a typed `Tool` with a JSON schema and handler:

```java
MCP.addTool(new Tool(
"current_user",
"Returns the signed in user",
"{\"type\":\"object\",\"properties\":{}}",
argumentsJson -> "{\"name\":\"" + signedInUser + "\"}"
));
```

The server merges application tools with the built-in UI tools. Codename One already uses the same `com.codename1.ai.Tool` contract for model tool calls inside an app, so one definition can serve an in-app model and an external MCP host.

Treat these tools as a privileged API. Do not publish a tool that returns signing passwords, API tokens, or unrestricted file contents because the handler happens to be local. The current server is local, but the agent still receives whatever the tool returns.

## Screenshots remain available

The semantic tree means a vision model does not need a screenshot for routine navigation. Some UI facts remain visual, such as a chart shape or a rendering defect. The server therefore exposes the current form as an optional PNG resource too.

The two sources complement each other. The tree says “this is an enabled Save button with an activate action.” The PNG says “the button overlaps the footer.” A screenshot alone cannot reliably provide the first fact. A semantic tree cannot provide the second.

## The scope is JavaSE today

This release supports the JavaSE port, which covers the simulator and JavaSE-hosted desktop tools. It does not make every packaged Codename One application an MCP server. Packaged executable jars, cloud desktop builds, mobile targets, the JavaScript port, and the native macOS, Linux, and Windows ports do not yet have the launcher and transport plumbing.

If you need MCP in one of those native targets, let us know which port and deployment model you need. The protocol engine and semantic tools are portable. The missing work is the transport, startup, registration, and security boundary for that target.

The PR includes 13 protocol and UI-driving tests plus an end-to-end run against the reference MCP Inspector client. It also builds the core into an iOS application to verify that unused MCP code is pruned and that referenced core classes stay within the ParparVM API surface.

Tomorrow's post covers another machine-readable view of Codename One. The new port status page turns the test suite into a dated support matrix instead of asking you to trust a manually maintained table.

---

## Discussion

_Join the conversation via GitHub Discussions._

{{< giscus >}}
Loading
Loading