ByteList.toString() assumes the encoding to be ISO-8859-1

[eregon]

ByteList.toString() assumes the encoding to be ISO-8859-1 and does not care about the encoding field!

• Could it decode properly to Java String using the ByteList encoding, like RubyString.toString() does?
• Why was ISO-8859-1 chosen arbitrarily as source encoding? Is is special in Java?

[enebo]

8859 is basically raw bytes as in ascii-8bit. I don't recall the motivation but it has been this way a long time string of all bytes and not necessarily proper chars

[eregon]

I know ISO-8859-1 as latin1, so not so much as a binary encoding in my experience 😉
Is this toString() useful in practice? It seems fairly dangerous.
Beside, ByteList implementing CharSequence is badly defined as charAt() is not consistent with toString().

[enebo]

@eregon Sorry I was using a phone so that was pretty terse :)

8859_1 encoding was only picked because it is a single-byte charset. You will notice we use 8859_1 in many places around the codebase in JRuby. There are a couple of things to say:

1. We don't actually know all consumers of this toString on ByteList (Think: '"foo" + bytelist;) so we get nervous about changing this (you are not first person to ask about it). It is also possible people expect this behavior in extensions. We don't know and it is a little scary.
2. The encoding stored in ByteList is a jcodings Encoding not a Java Charset one. So bytelist can represent strings which are not possible to make a properly encoded Java String. In this sense, I do not think anyone should be using toString for anything other than debugging.

[eregon]

1. Right, but anyway the current behavior is only correct if it already is ISO-8859-1 (fairly uncommon in Ruby) or only ASCII 7-bit chars. Doing the proper decoding would preserve the second property, the first one as long as a real ISO-8859-1 string is tagged as such (otherwise it's a legitimate bug) and fix all the other cases.
2. The couple rare encodings which cannot translate to UTF-16 should raise an error in that case, or provide some output indicating the problem.