[BACKPORT] Cygwin: open: Unlock fdtab before open_with_arch()

[lazka]

Since the commit 31bf91f, opening fifo causes a deadlock. This is because, open_with_arch() for fifo can be blocked until the other side of the fifo is opened. The commit 31bf91f moves the creating cygheap_fdnew before open_with_arch() to address the issue: https://cygwin.com/pipermail/cygwin/2026-May/259664.html However, cygheap_fdnew locks fdtab, so open() for the other side of fifo cannot create cygheap_fdnew until fdtab is unlocked. This is the cause of the deadlock.

With this patch, fdtab is unlocked before open_with_arch(), but marked as used using tentative fhandler. The summary of open() is as follows.

1. Lock fdtab.
2. Create new fd.
3. Mark fd as used using tentative fhandler.
4. Unlock fdtab.
5. Call open_with_arch().
6. Set final fhandler to fd.

The important point is that create fd before open_with_arch() to address https://cygwin.com/pipermail/cygwin/2026-May/259664.html, but unlock fdtab before open_with_arch() to address https://cygwin.com/pipermail/cygwin/2026-July/259884.html.

Fixes: 31bf91f ("Cygwin: Ensure unused fd available for open()")
Addresses: https://cygwin.com/pipermail/cygwin/2026-July/259884.html
Reported-by: kikairoya kikairoya@gmail.com

Reviewed-by: Mark Geisert mark@maxrnd.com
(cherry picked from commit 524d75f)

[dscho]

While I have some (not yet sent) concerns about that patch which will probably require follow-up fixes, it has been pushed to cygwin/master already, and therefore is appropriate to backport as-is.