escapeIndex() breaks string escaping on big-endian (s390x)

[dmkif]

escapeIndex() breaks string escaping on big-endian (s390x)

Hi, I'm mostly a mainframe guy (s390x, zLinux), not really a Go dev, so please bear with me if I describe this clumsily. I ran into this debugging Portainer on our s390x box, running under zCX (Docker on IBM Z), openSUSE Tumbleweed.

What I actually saw

We build our own container images and some of them have OCI labels set. On s390x, when Portainer has to list containers that carry those labels, the container list just doesn't show up and I get an error instead. Same setup on x86 works fine, only s390x is affected.

Portainer uses this package instead of the standard encoding/json. Looking at the API responses on s390x, the JSON coming back is broken: some characters that should be escaped (backslashes and quotes inside the label values) get written out raw, so it isn't valid JSON. When I rebuilt Portainer against stdlib encoding/json the containers showed up again, which is what pointed me here.

Small reproducer

Smallest thing I could come up with. Run it on a big-endian box (s390x):

package json_test

import (
	"bytes"
	stdjson "encoding/json"
	"testing"

	json "github.com/segmentio/encoding/json"
)

func TestEscapeIndexBigEndian(t *testing.T) {
	cases := []string{
		`C:\Users\admin`, // backslash near the start
		"key\tvalue",     // tab
		"\nfoo_bar_",     // newline at the very start
		`ab"cdefg`,       // quote near the start
	}

	for _, in := range cases {
		type W struct {
			V string `json:"v"`
		}
		var buf bytes.Buffer
		enc := json.NewEncoder(&buf)
		enc.SetAppendNewline(false)
		if err := enc.Encode(W{V: in}); err != nil {
			t.Errorf("%q: encode error: %v", in, err)
			continue
		}
		// parse it back with the stdlib to see if it's even valid JSON
		var got W
		if err := stdjson.Unmarshal(buf.Bytes(), &got); err != nil {
			t.Errorf("%q: got invalid JSON: %v\n  raw: %s", in, err, buf.Bytes())
			continue
		}
		if got.V != in {
			t.Errorf("%q: came back as %q", in, got.V)
		}
	}
}

On amd64 this passes. On s390x it fails like this:

--- FAIL: TestEscapeIndexBigEndian
    "C:\Users\admin": got invalid JSON: invalid character 'U' in string escape code
        raw: {"v":"C:\Users\\admin"}
    "key\tvalue": got invalid JSON: invalid character '\t' in string literal
        raw: {"v":"key<TAB>value"}
    "\nfoo_bar_": got invalid JSON: invalid character '\n' in string literal
        raw: {"v":"<NL>foo_bar_"}
    "ab\"cdefg": got invalid JSON: invalid character 'c' after object key:value pair
        raw: {"v":"ab"cdefg"}

One thing that confused me at first: in C:\Users\admin the first backslash is written raw but the second one is escaped correctly. Seems to depend on where in the string the character sits.

What I think is going on

escapeIndex in json/string.go looks at the string 8 bytes at a time by casting it to a uint64 via stringToUint64 (unsafe pointer cast). It then uses bits.TrailingZeros64(...) / 8 to find which byte in the chunk needs escaping.

As far as I can tell, that math only comes out right if byte 0 of the string ends up in the lowest byte of the uint64, i.e. little-endian layout. On s390x the bytes are the other way around, so it returns the mirrored position (7-k instead of k) within the chunk. When the special character sits in the first half of the 8-byte block, the escaping logic ends up looking past it and never escapes it.

I noticed json/parse.go does something very similar but uses binary.LittleEndian.Uint64 instead of the unsafe cast, and that one seems to work fine on s390x. So maybe string.go was just missed when that pattern was introduced.

I'm happy to open a PR that switches escapeIndex over to binary.LittleEndian.Uint64 like parse.go does, if that's the direction you'd want.