One side is the understanding of the COBOL code; and because that's "mostly plain English" normally works quite well with many LLMs.
The other side is writing valid and working COBOL code (most LLMs fail to write code that even compiles).
But there's even a benchmark for this: https://github.com/zorse-project/COBOLEval, maybe you can try to use that to evaluate XMainframe's capabilities here (and also compare it to mAInframer, which this was used on, comparing to bad results of common general LLMs.
One side is the understanding of the COBOL code; and because that's "mostly plain English" normally works quite well with many LLMs.
The other side is writing valid and working COBOL code (most LLMs fail to write code that even compiles).
But there's even a benchmark for this: https://github.com/zorse-project/COBOLEval, maybe you can try to use that to evaluate XMainframe's capabilities here (and also compare it to mAInframer, which this was used on, comparing to bad results of common general LLMs.