Redis "volatile-lru" cache

[annehaley]

This PR bumps the version of large-image we use so that we can leverage the latest changes to its Redis cache. Since the cache entries will now have a TTL, we can use the "volatile-lru" max memory policy and expect that our redis cache can evict large-image tiles without evicting websocket connections.

See x and x. Resolves #398.

[brianhelba]

Why not just do this through the Django cache framework, which is the default for django-large-image?

[annehaley]

Why not just do this through the Django cache framework, which is the default for django-large-image?

We're currently using the Django cache, but in production when a user tries to visualize the boston orthoimagery layer for our example use cases, the web service with Django runs out of memory: Process running mem=895M(122.6%) Error R14 (Memory quota exceeded).

[brianhelba]

I see that after #404, the Django cache is being used, but it was never actually configured properly.

Per my suggestion, #404 needed to add something like:

LARGE_IMAGE_CACHE_NAME = "tiles"

CACHES = {
    "tiles": {
        "BACKEND": "django.core.cache.backends.redis.RedisCache",
        "LOCATION": env.url("DJANGO_REDIS_URL").geturl(),
        "OPTIONS": {
            # Use database /2 for the tile cache,
            # in case other services use /0 in the future
            "db": "2",
        },
        # An arbitrarily long time
        "TIMEOUT": int(datetime.timedelta(weeks=4).total_seconds()),
    }
}

If we need to balance other cache considerations in the future, the configuration is well-documented by Django and we have all the local knobs (including the actual TIMEOUT) to adjust things.

[brianhelba]

Where do we control the Redis database number here?

We're using Redis for other things, so I think we should have local control of this.

[annehaley]

@brianhelba I tried the above suggestion locally and ran into some problems. First, if I set CACHES directly like that, I get an error about how Django needs a "default" cache. To resolve that, I added the "default" entry like this:

CACHES = {
    "default": {
        "BACKEND": "django.core.cache.backends.locmem.LocMemCache",
    },
    "tiles": {
        "BACKEND": "django.core.cache.backends.redis.RedisCache",
        "LOCATION": env.url("DJANGO_REDIS_URL").geturl(),
        "OPTIONS": {
            # Use database /2 for the tile cache,
            # in case other services use /0 in the future
            "db": "2",
        },
        "TIMEOUT": int(datetime.timedelta(weeks=4).total_seconds()),
    }
}

But the next problem is more prohibitive. After that change, it appeared to work properly, but when I performed minimal load testing, I got consistent crashes. I was watching the logs from the redis container as I loaded our aerial imagery layer on the map. It appeared to be working at first; tile fetching after the first load was snappy and the redis logs reflected the cache hits. But after a few refreshes loading the same tiles on the map, I noticed the following behavior:

1. The django logs show an error about how the web request process "took too long to shut down and was killed".
2. In the web client, tiles are not shown on the map and a generic error dialog is shown (expected behavior for a failed request).
3. The redis logs stop printing output.
4. The performance of my whole machine slows significantly, to the point of my mouse cursor lagging and my Chrome crashing.
5. The redis container shuts down and needs to be restarted manually. The redis logs do not record any errors, so this leads me to believe that it was OOM killed.

[brianhelba]

Just to ensure the problem is truly with the "tiles" cache and Redis, I'd set the "default" cache to django.core.cache.backends.dummy.DummyCache: https://docs.djangoproject.com/en/6.0/topics/cache/#dummy-caching-for-development

[annehaley]

When I run geodatalytics not in a devcontainer (classic docker compose up style), I can get to an error message (instead of just a logless crash): redis.exceptions.OutOfMemoryError: command not allowed when used memory > 'maxmemory'

Changing the "default" cache to a DummyCache doesn't change anything. I get the same behavior for both.

If I increase the maxmemory value in docker-compose.yml (L51) to something large like "40GB", the tiles load successfully and no error is raised. But even "30GB" is still too small and will raise errors.

My expectation was that redis would handle reaching its memory limit gracefully by evicting old keys for tiles (for which we've set a default TTL). I don't know why it would be raising an error like this when the maxmemory-policy is set to volatile-lru.

One additional thing to note is that when the redis container first starts, it takes about 30-60 seconds printing out a bunch of logs that all say * <search> Draining workers thread pool with threshold 100. If any tile requests are made during this time, I get this error: redis.exceptions.BusyLoadingError: Redis is loading the dataset in memory.

[manthey]

I tested volatile-lru behavior with Histomics (not Django) and it properly evicted tiles as expected with voltaile-lru, even for small sizes like 10 MB.