Repository navigation
SafetyMonitor should fail safe (IsSafe=False) when the image is stale or detection keeps failing #120
Description
Summary
IsSafe can keep reporting a stale state indefinitely, including True, when the input image stops updating or can no longer be downloaded. For a SafetyMonitor that drives a roof or dome (for example through N.I.N.A.), a stale True is the worst failure mode. A frozen or missing image should fail safe and report False.
Two failure paths
- The image stops updating, but is still served. For example, the camera software hangs or the step that produces the image fails. The detection loop keeps classifying the same old file every
update_interval. If the last real sky was Clear,IsSafestaysTruewhile clouds roll in. - The image cannot be downloaded. In
_run_single_detection(), ifCloudDetector.detect()raises (for example HTTP errors or timeouts after the retries in_load_image), the exception is only logged (Detection cycle failed).latest_detectionand the debounced state are not updated, sois_safe()keeps returning the last stable state with no time limit.
Proposal
Both checks can be optional and configurable. With the defaults below they would rarely trigger in a healthy setup.
MAX_IMAGE_AGE_SECONDS(e.g. 300): after each detection, read the image age from the HTTPLast-Modifiedheader (oneHEADrequest through the existing session), or from the file mtime forfile://URLs. If the age exceeds the limit, or cannot be determined, treat the instantaneous state as unsafe. It goes through the normal debounce, so returning to safe still needsdebounce_to_safe_sec.MAX_DETECTION_AGE_SECONDS(e.g. 180):is_safe()returnsFalseif the timestamp of the last successful detection is older than the limit.
I have been running this as a local patch of alpaca/device.py. It touches _update_cached_safety(), _run_single_detection() and is_safe(), and adds two small helpers. I tested it both with simulated detections and by stopping the image generation for about 8 minutes: issafe went to False, and came back after the image resumed. Happy to open a PR if you are interested. I noticed that #119 also changes is_safe(), so it may be easier to build on top of that.
Patch against current main (alpaca/device.py)
--- a/alpaca/device.py
+++ b/alpaca/device.py
@@ -5,6 +5,9 @@
import logging
import io
import json
+import os
+import urllib.parse
+from email.utils import parsedate_to_datetime
from collections import deque
from datetime import datetime, timedelta
from typing import Optional, Dict, Any, Tuple, List
@@ -14,6 +17,13 @@
# Watchdog constants
CLIENT_TIMEOUT_SECONDS = 300 # 5 Minutes
+# Fail-safe for stale data: IsSafe is False if the classified image is older
+# than MAX_IMAGE_AGE_SECONDS (Last-Modified header, or mtime for file://) or if
+# the last successful detection is older than MAX_DETECTION_AGE_SECONDS
+# (e.g. the image cannot be downloaded and detect() keeps failing).
+MAX_IMAGE_AGE_SECONDS = int(os.environ.get('MAX_IMAGE_AGE_SECONDS', '300'))
+MAX_DETECTION_AGE_SECONDS = int(os.environ.get('MAX_DETECTION_AGE_SECONDS', '180'))
+
# Import from sibling modules
from .config import AlpacaConfig, get_current_time, ALL_CLOUD_CONDITIONS
# Assuming detect.py is in the root path or installed as a package
@@ -118,6 +128,13 @@
class_name not in self._unsafe_conditions_set
)
+ # A stale image (or one of unknown age) is never safe
+ image_age = detection.get('image_age_seconds')
+ if image_age is None or image_age > MAX_IMAGE_AGE_SECONDS:
+ if is_safe_now:
+ logger.warning(f"Image stale (age={image_age}s, max={MAX_IMAGE_AGE_SECONDS}s): forcing UNSAFE")
+ is_safe_now = False
+
# Step B: Apply Debouncing
if is_safe_now == self._stable_safe_state:
# State matches - reset any pending changes
@@ -226,12 +243,44 @@
logger.error(f"Failed to connect to MQTT broker: {e}")
return None
+ def _get_image_age_seconds(self) -> Optional[float]:
+ """Age of the source image in seconds (None if it cannot be determined)"""
+ image_url = self.detect_config.image_url
+ try:
+ if image_url.startswith("file://"):
+ path_str = urllib.parse.unquote(urllib.parse.urlparse(image_url).path)
+ modified = datetime.fromtimestamp(os.path.getmtime(path_str)).astimezone()
+ else:
+ response = self.cloud_detector.session.head(
+ image_url, timeout=(5, 10), verify=self.detect_config.verify_ssl
+ )
+ response.raise_for_status()
+ last_modified = response.headers.get('Last-Modified')
+ if not last_modified:
+ logger.warning("Image has no Last-Modified header: age unknown")
+ return None
+ modified = parsedate_to_datetime(last_modified)
+ now = get_current_time(self.alpaca_config.timezone)
+ return max(0.0, (now - modified).total_seconds())
+ except Exception as e:
+ logger.warning(f"Cannot determine image age: {e}")
+ return None
+
+ def _detection_is_fresh(self) -> bool:
+ """True if the last successful detection is recent (assumes lock is held)"""
+ timestamp = self.latest_detection.get('timestamp') if self.latest_detection else None
+ if not isinstance(timestamp, datetime):
+ return False
+ age = (get_current_time(self.alpaca_config.timezone) - timestamp).total_seconds()
+ return age <= MAX_DETECTION_AGE_SECONDS
+
def _run_single_detection(self, initial: bool = False):
"""Run a single detection cycle with optimized image handling"""
try:
# Return image so we can process thumbnail
result = self.cloud_detector.detect(return_image=True)
result['timestamp'] = get_current_time(self.alpaca_config.timezone)
+ result['image_age_seconds'] = self._get_image_age_seconds()
# FIX: Normalize class name case to match configuration (Title Case preference)
# This handles cases where detection returns "mostly cloudy" but config expects "Mostly Cloudy"
@@ -429,6 +478,9 @@
return False
with self.detection_lock:
+ # Detection stalled or failing for too long -> unsafe
+ if not self._detection_is_fresh():
+ return False
return self._stable_safe_state
def get_pending_status(self) -> Dict[str, Any]:Thanks for the project. It works very well with a custom-trained model!
Activity
chvvkumar commentedon Oct 9, 2026
@panik988
Good issue. Let me take a look shortly. I think instead of relying purely on timing based checks (which can be a bit iffy depending on factors like clocks being accurate, power disruptions while the image is stale which may never trigger the failsafe etc.). Another point where this can cause problems is if for example, when we set the
I am thinking a robust way to do this would be to check by hashing the image and see if the image hash has changed within some configurable time. Let me think through this a bit.
Thanks for creating this issue and pointing out the gap!
panik988 commentedon Oct 9, 2026
Thanks for the quick reply! I agree: hashing the image and tracking the time of the last content change with a monotonic clock is more robust than relying on
Last-Modified. It doesn't depend on clocks being in sync, and it works forfile://sources and servers that don't send the header.A few thoughts that may help when you design it:
detect()keeps raising (download errors or timeouts), no new hash is ever computed, and today the last stable state is returned indefinitely. Something like "is_safe()returnsFalseif the last successful detection is older than N seconds" covers that path independently of the hash.Last-Modifiedcheck, would cover it.Happy to test a branch on my setup (custom 4-class model, N.I.N.A. driving the roof through the Alpaca SafetyMonitor) and report back.
fix(safety): fail safe on stale image or detection
fix(safety): fail safe on stale image or detection
chvvkumar commentedon Oct 9, 2026
Try the
sndtag, I made a few changes:When the limit is approaching:
If your camera stops capturing for part of the day, increase the Stale Image Limit or set it to 0; otherwise the monitor reports Unsafe for that period.
panik988 commentedon Oct 12, 2026
Tested v0.17 (
:latest) on my setup (Raspberry Pi 4 arm64, custom 4-class Keras model, N.I.N.A. via Alpaca). Withmax_image_age_sec: 300, I stopped the image generator for 8 minutes: the monitor went Unsafe at 314 s with "Stale Image", HA showed the condition sensor as unavailable and the new Safe sensor as off, and it returned to Safe about 64 s after images resumed (debounce). Restart behaviour also works: it holds Unsafe until the first image change. Thanks for the quick fix!chvvkumar commentedon Oct 12, 2026
Nice! Thanks for bringing this up so that the behavior could be modified.