deadlock when using flask and libsql with local db (works fine with stock sqlite3 library)

[Meehai]

minimal repro i could come up with. Tested on python 3.11 (created via conda create -n deadlock python=3.11 anaconda)

run via: python main.py [sqlite3/libsql]

Versions:

(deadlock) mihai[deadlock]$ pip freeze | grep "Flask\|libsql"
Flask==3.0.2
Flask-Cors==4.0.0
libsql-client==0.3.1

import libsql_client
import os
import sys
import sqlite3
from flask.testing import FlaskClient
from flask import Flask, Response, current_app

def get_data() -> Response:
    if isinstance(current_app.db, sqlite3.Connection):
        data = current_app.db.execute("select item from data").fetchall()[0][0]
    else:
        data = current_app.db.execute("select item from data").rows[0][0]
    return {"hello": data}, 200

def _setup_db(db_path: str):
    os.remove(db_path)
    client = libsql_client.create_client_sync(f"file:{db_path}")
    client.execute("create table data (id integer primary key unique, item text)")
    client.execute("insert into data(id, item) values(null, 'world')")
    client.close()

def setup_app(db_path: str, engine: str) -> Flask:
    assert engine in ("sqlite3", "libsql"), engine
    app = Flask(__name__)
    app.add_url_rule("/get_data", "get_data", view_func=get_data, methods=["GET"])
    if engine == "sqlite3":
        app.db = sqlite3.Connection(db_path)
    else:
        app.db = libsql_client.create_client_sync(f"file:{db_path}")
    return app

if __name__ == "__main__":
    _setup_db("test.db")
    app = setup_app("test.db", sys.argv[1])
    app.config.update({"TESTING": True})
    flask_client: FlaskClient = app.test_client()
    data = flask_client.get("get_data").json
    assert data["hello"] == "world"

Upon ctrl+c:

^CException ignored in: <module 'threading' from '/home/mihai/libs/miniconda3/envs/deadlock/lib/python3.11/threading.py'>
Traceback (most recent call last):
  File "/home/mihai/libs/miniconda3/envs/deadlock/lib/python3.11/threading.py", line 1590, in _shutdown
    lock.acquire()
KeyboardInterrupt: 

[Meehai]

nvm, the code is must simpler to deadlock:

import libsql_client
import sys
import sqlite3

if __name__ == "__main__":
    assert sys.argv[1] in ("sqlite3", "libsql"), sys.argv[1]
    if sys.argv[1] == "sqlite3":
        db = sqlite3.Connection("test.db")
        data = db.execute("select item from data").fetchall()[0][0]
    else:
        db = libsql_client.create_client_sync(f"file:test.db")
        data = db.execute("select item from data").rows[0][0]
    assert data == "world"
    print("AAAAAAAAAAAAAAA")

It hangs here:

    async def _dequeue_item(self) -> Optional[_QueueItem]:
        while True:
            with self._lock:
                print("locked _deque_item")
                if len(self._queue) > 0:
                    return self._queue.popleft()
                assert self._waker is None
                waker = self._loop.create_future()
                self._waker = waker
            print("before await waker")
            await waker # < this never ends
            print("unlocked _deque_item")

(web) mihai[dedlock]$ python cmon.py libsql
locked _deque_item
locked _deque_item
before await waker
unlocked _deque_item
locked _deque_item
locked _deque_item
before await waker
AAAAAAAAAAAAAAA

I feel like I'm using the library wrong, i expected just to find & replace my sqlite3.Connection() stuff with libsql (and a bunch of fetchrows() to .rows and remove some cursor which is quite mechanical anyway) and "just work", but apparently it's more than that. I already have a >1k loc flask code that works fine with the stock sqlite, so moving to turso/libsql seems a bit harder than I hoped for.

[Meehai]

seems that using https://github.com/tursodatabase/libsql-experimental-python/ (as per https://docs.turso.tech/sdk/python/quickstart#local-only) provides a much closer API to the stock sqlite3 library and doesn't deadlock. I guess I'll use that but it's a bit confusing to have two of them.

Why not semver and use the same repo? I got a bit confused for sure.