Improved speedups

[thewhaleking]

Came across this older PR: #438 which had no activity in quite some time. Thought the improved speed is a good idea, but I don't really know Rust, so I implemented the changes in C.

The benchmarks seem good:

  ┌──────────────┬─────────┬────────────────┬────────────────┬────────────────┐
  │  Benchmark   │ Native  │ Old C speedups │ New C speedups │ Rust (from PR) │
  ├──────────────┼─────────┼────────────────┼────────────────┼────────────────┤
  │ short escape │ 438 ns  │ ~417 ns        │ 227 ns         │ 522 ns         │
  ├──────────────┼─────────┼────────────────┼────────────────┼────────────────┤
  │ long escape  │ 23.5 µs │ ~7.79 µs       │ 2.60 µs        │ 6.71 µs        │
  ├──────────────┼─────────┼────────────────┼────────────────┼────────────────┤
  │ short plain  │ 333 ns  │ ~349 ns        │ 182 ns         │ 401 ns         │
  ├──────────────┼─────────┼────────────────┼────────────────┼────────────────┤
  │ long plain   │ 23.4 µs │ ~7.77 µs       │ 2.60 µs        │ 6.73 µs        │
  ├──────────────┼─────────┼────────────────┼────────────────┼────────────────┤
  │ long suffix  │ 183 µs  │ ~131 µs        │ 48.4 µs        │ 58.3 µs        │
  └──────────────┴─────────┴────────────────┴────────────────┴────────────────┘

Running on 2024 MacBook Pro M4 Pro
The Rust column is from the PR, as I wasn't actually able to get it to run locally.

I didn't add tests, because it seemed like this behavior is already covered.

[davidism]

Thanks, cool to see that it works well in C too. I still plan to use the Rust version, as I'm also interested in the easier maintainability and safety of it. The speed isn't too important to me at this point, since it's already so fast, we were benchmarking there to make sure it wasn't unreasonably slow. If you're able, try running the rust yourself so the timings are comparable, that would be interesting to see.

[thewhaleking]

Thanks, cool to see that it works well in C too. I still plan to use the Rust version, as I'm also interested in the easier maintainability and safety of it. The speed isn't too important to me at this point, since it's already so fast, we were benchmarking there to make sure it wasn't unreasonably slow. If you're able, try running the rust yourself so the timings are comparable, that would be interesting to see.

That's understandable. I think once Rust matures a bit (3-5 years) it will be really useful in the Python ecosystem. I think the current instability of the Rust-Python workflows (with like PyO3) put it in a maintenance nightmare at this point.

I wasn't able to run the Rust tests myself. I have cargo installed, and build tools with it frequently, but pip install . did not work for me, and I didn't really want to spend time debugging it.