ODS2 COPY of VMS text library from PC to ods2 results in unreadable library file

[vaxorcist]

I use the following commands to transfer the MORIA VMS PASCAL sources (including a MORIAHLP.HLB) from my PC to an RD31 disk image:

SET NOERROR_EXIT
SET VERIFY
INIT/CREATE/MEDIUM_TYPE=RD31/LOG MORIA.RD31 MORIA
SHOW VOL
CREATE/DIRECTORY/LOG [SOURCE]
CREATE/DIRECTORY/LOG [EXECUTE]
CREATE/DIRECTORY/LOG [DOC]

set cwd moria_443
COPY /TO_FILES-11/ASCII build.com [000000].

set cwd doc
COPY /TO_FILES-11/ASCII . [DOC].

set cwd "../execute"
COPY /TO_FILES-11/BINARY/record_format=fixed=512 .hlb [EXECUTE].*

set cwd "../source"
COPY /TO_FILES-11/ASCII .pas [SOURCE].*

CREATE/DIRECTORY/LOG [SOURCE.MACRO]
set cwd "macro"
COPY /TO_FILES-11/ASCII .mar [SOURCE.MACRO].*

CREATE/DIRECTORY/LOG [SOURCE.INCLUDE]
set cwd "../include"
COPY /TO_FILES-11/ASCII . [SOURCE.INCLUDE].

SET DEFAULT [SOURCE.INCLUDE]
DIR /DATE/SIZE
DISMOUNT A:

When I try to read the MORIAHLP.HLB from MORIA (or DCL), I get the following error message:

%HELP-E-OPENIN, error opening DUA1:[EXECUTE]MORIAHLP.HLB; as input
-RMS-F-IRC, illegal record encountered; VBN or record number = 0

The MORIAHLP.HLB characteristics as seen from VMS DCL are:

$ dir/full MORIAHLP.HLB

Directory DUA1:[EXECUTE]

MORIAHLP.HLB;1 File ID: (18,1,0)
Size: 151/151 Owner: [1,1]
Created: 14-AUG-2017 12:08 Revised: 14-AUG-2017 12:08 (0)
Expires: Backup:
File organization: Sequential
File attributes: Allocation: 151, Extend: 0, Global buffer count: 0
Version limit: 0, Contiguous
Record format: Fixed length 512 byte records
Record attributes: None
Journaling enabled: None
File protection: System:RWED, Owner:RWED, Group:RE, World:
Access Cntrl List: None

Total of 1 file, 151/151 blocks.
$

The characteristics of a known working HLB file are:

SDA.HLB;1 File ID: (379,1,0)
Size: 53/54 Owner: [SYSTEM]
Created: 15-JUN-1987 12:17 Revised: 15-JUN-1987 12:17 (1)
Expires: Backup:
File organization: Sequential
File attributes: Allocation: 54, Extend: 0, Global buffer count: 0
No version limit, Contiguous best try
Record format: Fixed length 512 byte records
Record attributes: None
Journaling enabled: None
File protection: System:RWED, Owner:RWED, Group:RWED, World:RE
Access Cntrl List: None

The version of ods2 used is:

ODS2$> sh versi
ODS2 V3.0.2 built Apr 11 2023 20:11:33 with libedit version 2.11
Platform: Linux x86-64 C11
Compiler: gcc, version 10.2.1 20210110
Large files and devices (>2GB) are supported
VHD format image files are supported
ODS2$>

[tlhackque]

I'll try to find some time to look into this, and the other open ODS2 issues.

However, to reproduce this I would need the "MORIA VMS PASCAL sources" that you are working with. Please provide a URL. Also the distribution and version of your Linux host.

Note that .HLB is a binary format. It has an internal index structure, which if corrupt could result in a reference to VBN 0. (VBNs start with 1, so 0 is never a valid address for data.)

ODS2 in binary mode simply moves the bytes that it sees on the PC to (or from) the Files-11 disk. It uses POSIX open in binary ("rb"/"wb") mode and 512 byte freads/fwrites on the host side. On the FILES-11 side, it writes 512 byte records (if necessary, padding the last host read to 512 bytes.) It reads FILES-11 as UDF.

For your example to work, whoever created the kit that you are using would have to have transferred it from a VMS system as a strict binary. Do the sources that you are using indicate how this was done? Do they include the help library source? (If so, the easiest thing to do would be to build the HLB on the OpenSIMH VAX and bring it back to the PC with ODS2. If that is successful and the HLB can stand a round trip back to the VAX, it would indicate that ODS2 is not at fault...)

Moving binary data across-architectures is tricky, even if the data ends up in the same place. Intermediate stops and tools can introduce problems.

ODS2's promise is that if you use it to move binary from a disk to a host, the host file will contain exactly the bytes in the source, such that you can move it back to a Files-11 disk, unchanged. If you move the file from host to host, you are subject to the vagaries of the transport protocol and the hosts' filesystems, which may introduce "helpful" transformations. (E.g. record lengths, CRLF conversions, null-stripping, character set conversions...) In theory, these should all be transparent to ODS2 (which opens the host file in binary mode). But there is a lot of opportunity for error.

[pkoning2]

Could this be a case of RMS attributes being needed but not correct for this case? That's a hard problem with file transfer tools; I have a collection of partial solutions in "flx" for this. Part of the answer is to ignore entirely complex formats like "indexed" or "relative" file organizations. But even with just sequential fixed or variable, getting it right for applications that care is tricky. An option would be to let the user specify the desired output attributes, which of course requires the user knowing what the right answer is.

[tlhackque]

In this case the right attributes are shown in his log. ODS2 supports conversions - it actually has a built-in RMS emulation, so it does sys$parse, sys$connect, sys$put, etc, with FABs, RABs, XABs, NAMs, etc. You can send a text file from the host to the F11 disk & specify pretty much any (sequential) output format. There are switches to specify fixed, variable, carriage control, etc. In the other direction, it converts the RMS attributes to reasonable (in my opinion) stream text. Binary is intended to be a byte stream (RMS UDF). It can get things wrong - and when I took it over, it was woefully incomplete at both the F11 and RMS levels. If you run it, it has DCL style help, and you can see all the switches for specifying attributes. It doesn't support indexed or relative organizations, just sequential. (And unlike real RMS, won't convert those to a sequential view if asked.) But those aren't the issues here.

In this case, the most likely cause is the transfer off whatever system created the distribution that @vaxorcist is trying to restore. It may have been done via TCP or thru another channel such as a simh lpt or ptr in ASCII mode. .HLB is not a common binary extension. In VMS, it's a unique binary indexed file. Not RMS indexed; something unique to the Librarian (LBR in the RSX/RSTS worlds). The 16-bit machines use LBR for macros (text -.MLB) and object libraries (.OLB). VMS also uses it for help text - that's the .HLB. If the source kit is complete, the text files for the help modules can be put on the disk, and a valid library built on VMS when the disk is attached to a simulator. If that can be round-tripped with ODS2, ODS2 is in the clear.

But I don't rule out an ODS2 issue, and can't until the actual files are involved are available. Then they can be dumped on both sides, round-tripped, and if necessary, ODS2 can be debugged.

I've done a lot of work on ODS2, but it still is ugly code.

[vaxorcist]

I did a test using a known good HLB file SDA.HLB. I transferred this file from my VMS ODS2 image to Linux and back using:

MOUNT/WRITE HLBTEST.RD31 HLBTEST 
SHOW VOL
DIR
COPY/BINARY SDA.HLB SDA.HLB
COPY /TO_FILES-11/BINARY SDA.HLB [000000]SDABIN.HLB
COPY /TO_FILES-11/BINARY/record_format=fixed=512 SDA.HLB [000000]SDABINFIX.HLB
DIR /DATE/SIZE 
DISMOUNT A:

The results when having a look at the original file and the two copies:

$ lib/list sda.hlb   
Directory of HELP library DUA2:<000000>SDA.HLB;1 on  3-FEB-2024 09:21:28
Creation date:  23-MAY-1987 15:25:01      Creator:  VAX-11 Librarian V04-00
Revision date:  23-MAY-1987 15:25:03      Library format:   3.0
Number of modules:     19                 Max. key length:  15
Other entries:          0                 Preallocated index blocks:      1
Recoverable deleted blocks:      0        Total index blocks used:        1
Max. Number history records:      20      Library history records:        0
Library is in DCX data reduced format

ATTACH
COPY
DEFINE
EVALUATE
EXAMINE
Execute_Command
EXIT
Expressions
FORMAT
HELP
Initialization
Operation
READ
REPEAT
SEARCH
SET
SHOW
SPAWN
VALIDATE
$
$ lib/list sdabin.hlb
%LIBRAR-F-OPENIN, error opening DUA2:<000000>SDABIN.HLB;1 as input
-LBR-E-ILLFMT, illegal library format
$
$ lib/list sdabinfix.hlb
%LIBRAR-F-OPENIN, error opening DUA2:<000000>SDABINFIX.HLB;1 as input
-RMS-F-IRC, illegal record encountered; VBN or record number = 0
$ 

And voila:
The same error that I encountered trying to transfer MORIAHLP.HLB!

This does not necessarily prove that my MORIAHLP.HLBis o.k., but that there is a problem transferring HLB files to an ODS" volume using ods2.

HLBTEST.RD31.ZIP

This is my Linux system:
Linux think12 6.1.0-17-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux

Thank you very much for your efforts!
Ulli

[vaxorcist]

I can reproduce the same problem transferring a VMS EXE file from my VMS ODS2 image to Linux and back using:

MOUNT/WRITE EXETEST.RD31 EXETEST 
SHOW VOL
DIR
COPY/BINARY MORIA.EXE MORIA.EXE
COPY /TO_FILES-11/BINARY MORIA.EXE [000000]MORIABIN.EXE
COPY /TO_FILES-11/BINARY/record_format=fixed=512 MORIA.EXE [000000]MORIABINFIX.EXE
DIR /DATE/SIZE 
DISMOUNT A:

The results when having a look at the original file and the two copies:

$ MORIA == "$DUA2:[000000]MORIA"
$ moria
Created DUA2:[000000]HOURS.DAT
Created DUA2:[000000]MORIA.DAT
Created DUA2:[000000]MORIACHR.DAT
Created DUA2:[000000]MORIATOP.DAT
Notice: System MORIA wizard should set the protection
        on  files  just created.  See INSTALL.HLP for
        help on setting protection on the files.
Notice: File HOURS.DAT may be edited to set operating
        hours for MORIA.
Notice: File MORIA.DAT may be edited to contain  news
        items, etc...
$ del *.dat;
$
$ MORIA == "$DUA2:[000000]MORIABIN"
$ moria
%DCL-W-ACTIMAGE, error activating image DUA2:[000000]MORIABIN
-CLI-E-IMGNAME, image file DUA2:[000000]MORIABIN.EXE;1
-IMGACT-F-IMG_SIZ, image header descriptor length is invalid
$
$ MORIA == "$DUA2:[000000]MORIABINFIX"
$ moria
%DCL-W-ACTIMAGE, error activating image DUA2:[000000]MORIABINFIX
-CLI-E-IMGNAME, image file DUA2:[000000]MORIABINFIX.EXE;1
-RMS-F-IRC, illegal record encountered; VBN or record number = 0
$ 

So that looks more like a general problem than one of HLB files only, doesn't it?

Ulli