BUG REPORT: Process/Steam Isolation fails completely "out-of-the-box" in the new Duo version.

[KampeLoL]

When launching games (e.g., Dying Light The Beast) or Steam on the host and then on the instance, I immediately get a
"Game is already running" Mutex error, and Steam logs out the host session. The isolation is simply not functioning.

Technical Findings (Root Cause):
I inspected the system state to see why the isolation failed, and it appears the installer/DuoService is completely
failing to deploy the verifier.dll payloads to the OS, despite the settings being enabled in the registry.

1. Registry State: Looking at HKLM\SOFTWARE\Duo, isolation is clearly toggled ON:

   • EnableVerifier = 1
   • SteamIsolation = 1
   • EnableProcessPatching = 1
   • VerifierUsesBlacklist = 1

2. The Failure (Payload deployment): Despite the registry indicating that global hooking (Blacklist mode) is active,
   Duo failed to overwrite or place its custom verifier.dll in the system directories. Both
   C:\Windows\System32\verifier.dll and the SysWOW64 equivalent remain the original, signed 400KB Microsoft files.
   There are also no IFEO global keys set to trigger the hook.

Conclusion:
The UI/Registry says isolation is enabled, but the actual deployment of the API hooks (the custom DLLs) to
System32/SysWOW64 is failing silently during installation or service startup. Because the DLLs are missing, no hooking
occurs, global Mutexes bleed across RDP boundaries, and the multi-seat experience breaks.

Is there a permission bug in the new installer script preventing it from copying the payloads to System32?

[KampeLoL]

UPDATE / Follow-up Investigation:

@Black-Seraph After deeper testing of v1.5.7, I realized the missing verifier.dll files were intentional, as you moved
to the Application Compatibility API (ACT shims) for process patching.

However, there is a major architectural flaw with how this new method handles complex games.

The Current Flaw (Shim Escaping):

1. I enabled Isolate Steam in the UI.
2. I verified in the registry (AppCompatFlags\Custom) that the ACT shim is successfully applied to steam.exe.
3. I am sharing a single Steam installation between the Host and the Instance.
4. When I launch a heavy engine game (e.g., Dying Light The Beast) via Steam on the Host, and then on the Instance, I
   still get the "Game is already running" (Mutex crash).

Why it's failing:
While steam.exe is successfully isolated by your shim, the child game processes (like DyingLightGame.exe) are
completely escaping the shim environment. They are creating Global Mutexes (\BaseNamedObjects...) that bleed right
through the RDP session boundaries, breaking the multi-seat setup. The old verifier.dll method successfully hooked the
game executable directly, preventing this.

Proposed Solution:
The new ACT method needs a way to either forcefully cascade the shim to all child processes spawned by Steam, or (more
practically) we desperately need a "Targeted Applications" UI list back so we can manually add specific standalone
game .exe files to the compatibility layer!

[KampeLoL]

[mad0x20wizard]

First some kudos for the thorough analysis. This is very helpful. You identified correctly, that the verifier approach has been dropped with the new release, in favour of the Application Compatibility API. Although you can see individual applications being targeted by the AppCompat database, this doesn't mean that the shim doesn't cascade to child processes. In fact all applications are meant to receive the shim library, which you can verify with a tool like SystemInformer:

When you see the __COMPAT_LAYER = Duo and DuoAcRedVersion environment variable , that means the process received all the available patches successfully.

Duo 1.5.7 primarily focused on applying patches that fix problems that affect users globally (like Steam isolation, Remote session checking, etc.). But each process can use a variety of mechanisms that could potentially break the isolation for any particular game. Most games don't care about this that much, but some do. For example Anno 117 cannot be started concurrently for still unknown reasons.

We have to look at each of these games individually to identify and fix the problems.

The creation of Mutexes could potentially be hooked and isolated properly. The problem here is, that such a fix cannot be applied broadly, because of the unknown side-effects for other programs. Therefore I wonder that this actually worked with 1.5.6. Can you check this?

Also can you elaborate a bit more on the problem:

• Does this bleed provoke a Steam shutdown or is it the game telling you, that it doesn't want to be started twice?
• Do you share the same Steam account or the same Windows user account between the host and the Duo instance?
• Can you identify the problematic Mutex?

The goal here is to fix as many isolation problems as possible, but without deteriorating general stability.

[Black-Seraph]

@mad0x20wizard This would fall neatly in line with the planned AppContainer sandbox re-implementation, as AppContainer silos automatically re-route all global kernel objects.

As such I'm not sure if we should even bother implementing this as an AppCompat shim or not, as this issue could potentially solve itself as part of the already planned feature roadmap.

But implementing this as a shim as a temporary measure could still be beneficial in the short term while we're still busy implementing the new sandbox implementation, as its way less effort to shim it than sandbox it.

@KampeLoL Can you inspect the game's process using System Informer to see if the AppCompat module loaded successfully?

And could you give me the full name of the mutex object it's trying to access?

[KampeLoL]

@mad0x20wizard @Black-Seraph Thank you for the detailed explanation! I have performed the exact diagnostics you
requested on the instance while the game was running. Here are the results:

1. Shim Inheritance Failure (SystemInformer Check):
   I inspected the child game process (DyingLightGame_TheBeast_x64_rwdi.exe) running inside the Duo instance.
   The __COMPAT_LAYER and DuoAcRedVersion environment variables are NOT present in the game's process environment block.
   The AppCompat module is also not loaded into the game's memory space.
   This confirms that while steam.exe successfully receives the Duo shim, the heavy custom engine of Dying Light
   completely drops or escapes the inheritance tree during launch. It launches "naked" without any of Duo's patches.

2. The Bleeding Mutexes:
   Because the game escapes the shim, it creates its locks in the global namespace rather than isolating them to the RDP
   session. The exact Mutex objects causing the "Game is already running" crash are:

• \BaseNamedObjects\DyingLightTheBeast
• \BaseNamedObjects{52813408-3561-4705-820a-2b3b78be92ba}_0
  (I confirmed this by manually closing these specific handles on the host using Sysinternals handle64.exe, which
  immediately allowed the second instance to launch perfectly).

3. My Specific Setup:
   To answer your question about the environment:

• I am sharing a single Steam installation directory on the PC.
• However, we are using two completely different Steam accounts (one logged in on the Host, the other logged in on
  the Duo Instance). Both accounts own their own license of the game.

[KampeLoL]

@Black-Seraph @mad0x20wizard Follow-up: Architectural Suggestions for the ACT Shim Method

Thinking more about why relying on process inheritance from steam.exe is fragile for modern game engines (since they
often strip their environment block or launch via intermediate processes), here are three potential ways to solidify
the ACT shim approach before the AppContainer rewrite is ready:

1. Re-introduce the "Targeted Applications" UI (Easiest Win):
   Bring back the UI list, but instead of using IFEO VerifierFlags like in v1.5.6, use it to write the target game
   .exe paths directly into HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Custom (or Layers). This
   gives power users an immediate escape hatch to manually enforce the shim on stubborn executables without relying on
   Steam's inheritance tree.

2. Dynamic SDB (Shim Database) Generation:
   Instead of relying just on registry keys, DuoService could dynamically generate and install a custom .sdb file via
   sdbinst.exe that applies your isolation shim to any .exe launched from within known Steam Library folders. This
   bypasses the environment variable drop issue entirely.

3. ETW / WMI Process Creation Monitoring:
   If you want to keep it "zero-config" for the user, DuoService could listen to ETW (Event Tracing for Windows) for
   process creation events originating from the isolated steam.exe tree. When a child process spawns, DuoService can
   quickly inject the __COMPAT_LAYER = Duo environment variable into the new process block before its main thread
   resumes.

Hope these insights help solidify the new isolation method!

[Black-Seraph]

@KampeLoL I'm sorry to be the bearer of bad news here, but...

1. would be a bandaid solution at best, but yes, this could potentially solve the problem in your particular case but isn't user friendly.
2. means nothing as that is already what we do, we aren't using registry keys to propagate at all.
3. doesn't work as, by the time we get the notification, we'd already be too late to change anything.

Our best approach to this would be to figure out where exactly we lose the propagation and fix things there.

As I don't own the game I need your help in debugging this a little more in-depth.

The best way to do so would be if we got a direct line of contact to each other, Discord preferably.

My Discord handle is black5eraph, feel free to send a friend request / get in touch.

Do let me know your Discord handle ahead of time though so I can sift you out of the 100+ friend requests.

A good starting lead would be to use process monitor to figure out what the parent process chain looks like.