Merge pull request #206 from DomainTools/IDEV-2584

[IDEV-2584] RTTF Support for header authentication & CLI Improvements for download endpoint

Author: briluza

PYTHON_SUPPORT.md

@@ -2,28 +2,20 @@
 
 ## Policy
 
-The DomainTools API library will support all versions of Python that are actively maintained by the Python 
+The DomainTools API library will support all versions of Python that are actively maintained by the Python
 Software Foundation. When a version of Python enters End of Life (EOL), the API library will also end support
-for that version of Python. 
+for that version of Python.
 
-When a version's End of Life date is reached, DomainTools will ensure that a release of the API library that 
+When a version's End of Life date is reached, DomainTools will ensure that a release of the API library that
 contains all changes up to that point in time is available. If a release already exists that has all
-changes at the point of a version's EOL date, no new one will be made. Any changes (features, bugfixes, etc) 
+changes at the point of a version's EOL date, no new one will be made. Any changes (features, bugfixes, etc)
 released after an EOL date will not be tested on the now-unsupported version.
 
-Versions of Python from other organizations (e.g. cython, pypy, jython) will not be actively supported. DomainTools 
-will not develop specifically for those versions of Python, but we welcome community assistance (such as pull 
+Versions of Python from other organizations (e.g. cython, pypy, jython) will not be actively supported. DomainTools
+will not develop specifically for those versions of Python, but we welcome community assistance (such as pull
 requests) to support them.
 
-### Python 2
 
-DomainTools API library support for Python 2.7 (and all Python 2) will end on November 30, 2020. 
+### Python >=3.9
 
-### Python 3.5
-
-DomainTools will continue to support Python 3.5 until November 30, 2020. 
-
-## Upcoming Timeline:
-
-- Support for Python 2.7 will end on Nov 30, 2020
-- Support for Python 3.5 will end on Nov 30, 2020
+DomainTools currently supports Python 3.9 and above.

README.md

@@ -264,7 +264,7 @@ Custom parameters aside from the common `GET` Request parameters:
     api = API(USERNAME, KEY)
     api.nod(endpoint="feed", **kwargs)
     ```
-- `header_authentication`: by default, we're using API Header Authentication. Set this False if you want to use API Key and Secret Authentication. Apparently, you can't use API Header Authentication for `download` endpoints so this will be defaulted to `False` even without explicitly setting it.
+- `header_authentication`: by default, all RTTF endpoints (both `feed` and `download`) use API Header Authentication, sending the API key via the `X-Api-Key` header. Set this to `False` to pass the API key as a query parameter instead.
     ```python
     api = API(USERNAME, KEY, header_authentication=False)
     api.nod(**kwargs)
@@ -275,7 +275,7 @@ Custom parameters aside from the common `GET` Request parameters:
     api.nod(output_format="csv", **kwargs)
     ```
 
-The Feed API standard access pattern is to periodically request the most recent feed data, as often as every 60 seconds. Specify the range of data you receive in one of two ways:
+The `feed` endpoint streams live NDJSON data. The standard access pattern is to poll as often as every 60 seconds. Specify the range of data you receive in one of two ways:
 
 1. With `sessionID`: Make a call and provide a new `sessionID` parameter of your choosing. The API will return the last hour of data by default.
     - Each subsequent call to the API using your `sessionID` will return all data since the last.
@@ -284,6 +284,16 @@ The Feed API standard access pattern is to periodically request the most recent
     - Either an `after=-60` query parameter, where (in this example) -60 indicates the previous 60 seconds.
     - Or `after` and `before` query parameters for a time range, with each parameter accepting an ISO-8601 UTC formatted timestamp (a UTC date and time of the format YYYY-MM-DDThh:mm:ssZ)
 
+The `download` endpoint returns a standard JSON response (not a stream) listing available S3 batch files. Time parameters (`sessionID`, `after`, `before`) are **not** required for download calls.
+
+```python
+api = API(USERNAME, KEY)
+result = api.nod(endpoint="download", limit=5)
+print(result["download_name"])
+for f in result["files"]:
+    print(f["name"], f["url"])
+```
+
 ### Feed parameters
 
 The feed methods accept the following parameters, grouped by purpose. Availability depends on the feed (see the notes below the table).
@@ -325,11 +335,24 @@ The feed methods accept the following parameters, grouped by purpose. Availabili
 
 - `output_format`: `csv` or `jsonl` (default `jsonl`). Not available on the `domainrdap` feed. `csv` is not available for `download` endpoints.
 - `headers`: When `csv` output is used, adds a header row to the first line of the response.
-- `top`: Positive integer from `1` to `1,000,000,000` limiting the number of results in the response payload. 
+- `top`: Positive integer from `1` to `1,000,000,000` limiting the number of results in the response payload. Ignored for the `download` endpoint.
+
+#### Download-only parameters
+
+These parameters are only accepted when `endpoint="download"`. They are ignored for the `feed` endpoint.
+
+- `limit`: Maximum number of files to return in the response.
+- `page`: Zero-indexed page of results to return. Available on `realtime_domain_risk`, `domainhotlist`, `iphotlist`, and `iprisk`.
+- `prefix`: Filter files by date prefix (e.g. `"2026-08-"`). Available on `realtime_domain_risk`, `domainhotlist`, `iphotlist`, and `iprisk`.
+
+```python
+api = API(USERNAME, KEY)
+api.iphotlist(endpoint="download", limit=10, page=0, prefix="2026-08-")
+```
 
-## Handling iterative response from RTUF endpoints:
+## Handling iterative response from RTTF endpoints:
 
-Since we may dealing with large feeds datasets, the python wrapper uses `generator` for efficient memory handling. Therefore, we need to iterate through the `generator` if we're accessing the partial results of the feeds data.
+Since we may be dealing with large feeds datasets, the python wrapper uses `generator` for efficient memory handling. Therefore, we need to iterate through the `generator` if we're accessing the partial results of the feeds data.
 
 ### Single request because the requested data is within the maximum result:
 ```python

domaintools/api.py

@@ -199,6 +199,8 @@ def _handle_api_key_parameters(self, is_rttf_product):
             self.header_authentication = is_rttf_product
 
     def handle_api_key(self, is_rttf_product, path, parameters):
+        if self.header_authentication and not self.always_sign_api_key:
+            return
         if self.https and not self.always_sign_api_key:
             parameters["api_key"] = self.key
         else:
@@ -1204,11 +1206,16 @@ def nod(self, **kwargs) -> FeedsResults:
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
         source = ENDPOINT_TO_SOURCE_MAP.get(endpoint)
-        if (
-            endpoint == Endpoint.DOWNLOAD.value
-            or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value
-        ):
-            # headers param is allowed only in Feed API and CSV format
+
+        if endpoint == Endpoint.DOWNLOAD.value:
+            return self._results(
+                f"newly-observed-domains-feed-({source.value})",
+                f"v1/{endpoint}/nod/",
+                response_path=("response",),
+                limit=kwargs.get("limit"),
+            )
+
+        if kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value:
             kwargs.pop("headers", None)
 
         return self._results(
@@ -1247,11 +1254,16 @@ def nad(self, **kwargs) -> FeedsResults:
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
         source = ENDPOINT_TO_SOURCE_MAP.get(endpoint).value
-        if (
-            endpoint == Endpoint.DOWNLOAD.value
-            or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value
-        ):
-            # headers param is allowed only in Feed API and CSV format
+
+        if endpoint == Endpoint.DOWNLOAD.value:
+            return self._results(
+                f"newly-active-domains-feed-({source})",
+                f"v1/{endpoint}/nad/",
+                response_path=("response",),
+                limit=kwargs.get("limit"),
+            )
+
+        if kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value:
             kwargs.pop("headers", None)
 
         return self._results(
@@ -1291,6 +1303,14 @@ def domainrdap(self, **kwargs) -> FeedsResults:
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
         source = ENDPOINT_TO_SOURCE_MAP.get(endpoint).value
 
+        if endpoint == Endpoint.DOWNLOAD.value:
+            return self._results(
+                f"domain-registration-data-access-protocol-feed-({source})",
+                f"v1/{endpoint}/domainrdap/",
+                response_path=("response",),
+                limit=kwargs.get("limit"),
+            )
+
         return self._results(
             f"domain-registration-data-access-protocol-feed-({source})",
             f"v1/{endpoint}/domainrdap/",
@@ -1327,11 +1347,16 @@ def domaindiscovery(self, **kwargs) -> FeedsResults:
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
         source = ENDPOINT_TO_SOURCE_MAP.get(endpoint).value
-        if (
-            endpoint == Endpoint.DOWNLOAD.value
-            or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value
-        ):
-            # headers param is allowed only in Feed API and CSV format
+
+        if endpoint == Endpoint.DOWNLOAD.value:
+            return self._results(
+                f"real-time-domain-discovery-feed-({source})",
+                f"v1/{endpoint}/domaindiscovery/",
+                response_path=("response",),
+                limit=kwargs.get("limit"),
+            )
+
+        if kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value:
             kwargs.pop("headers", None)
 
         return self._results(
@@ -1370,11 +1395,16 @@ def noh(self, **kwargs) -> FeedsResults:
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
         source = ENDPOINT_TO_SOURCE_MAP.get(endpoint).value
-        if (
-            endpoint == Endpoint.DOWNLOAD.value
-            or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value
-        ):
-            # headers param is allowed only in Feed API and CSV format
+
+        if endpoint == Endpoint.DOWNLOAD.value:
+            return self._results(
+                f"newly-observed-hosts-feed-({source})",
+                f"v1/{endpoint}/noh/",
+                response_path=("response",),
+                limit=kwargs.get("limit"),
+            )
+
+        if kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value:
             kwargs.pop("headers", None)
 
         return self._results(
@@ -1422,11 +1452,18 @@ def realtime_domain_risk(self, **kwargs) -> FeedsResults:
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
         source = ENDPOINT_TO_SOURCE_MAP.get(endpoint).value
-        if (
-            endpoint == Endpoint.DOWNLOAD.value
-            or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value
-        ):
-            # headers param is allowed only in Feed API and CSV format
+
+        if endpoint == Endpoint.DOWNLOAD.value:
+            return self._results(
+                f"real-time-domain-risk-({source})",
+                f"v1/{endpoint}/domainrisk/",
+                response_path=("response",),
+                limit=kwargs.get("limit"),
+                page=kwargs.get("page"),
+                prefix=kwargs.get("prefix"),
+            )
+
+        if kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value:
             kwargs.pop("headers", None)
 
         return self._results(
@@ -1474,11 +1511,18 @@ def domainhotlist(self, **kwargs) -> FeedsResults:
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
         source = ENDPOINT_TO_SOURCE_MAP.get(endpoint).value
-        if (
-            endpoint == Endpoint.DOWNLOAD.value
-            or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value
-        ):
-            # headers param is allowed only in Feed API and CSV format
+
+        if endpoint == Endpoint.DOWNLOAD.value:
+            return self._results(
+                f"real-time-domain-hotlist-({source})",
+                f"v1/{endpoint}/domainhotlist/",
+                response_path=("response",),
+                limit=kwargs.get("limit"),
+                page=kwargs.get("page"),
+                prefix=kwargs.get("prefix"),
+            )
+
+        if kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value:
             kwargs.pop("headers", None)
 
         return self._results(
@@ -1544,11 +1588,18 @@ def iphotlist(self, **kwargs) -> FeedsResults:
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
         source = ENDPOINT_TO_SOURCE_MAP.get(endpoint).value
-        if (
-            endpoint == Endpoint.DOWNLOAD.value
-            or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value
-        ):
-            # headers param is allowed only in Feed API and CSV format
+
+        if endpoint == Endpoint.DOWNLOAD.value:
+            return self._results(
+                f"real-time-ip-hotlist-({source})",
+                f"v1/{endpoint}/iphotlist/",
+                response_path=("response",),
+                limit=kwargs.get("limit"),
+                page=kwargs.get("page"),
+                prefix=kwargs.get("prefix"),
+            )
+
+        if kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value:
             kwargs.pop("headers", None)
 
         return self._results(
@@ -1614,11 +1665,18 @@ def iprisk(self, **kwargs) -> FeedsResults:
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
         source = ENDPOINT_TO_SOURCE_MAP.get(endpoint).value
-        if (
-            endpoint == Endpoint.DOWNLOAD.value
-            or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value
-        ):
-            # headers param is allowed only in Feed API and CSV format
+
+        if endpoint == Endpoint.DOWNLOAD.value:
+            return self._results(
+                f"real-time-ip-risk-({source})",
+                f"v1/{endpoint}/iprisk/",
+                response_path=("response",),
+                limit=kwargs.get("limit"),
+                page=kwargs.get("page"),
+                prefix=kwargs.get("prefix"),
+            )
+
+        if kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value:
             kwargs.pop("headers", None)
 
         return self._results(

domaintools/cli/api.py

@@ -10,6 +10,7 @@
 
 from domaintools.api import API
 from domaintools.constants import Endpoint, RTTF_PRODUCTS_LIST, OutputFormat
+from domaintools.results import FeedsResults
 from domaintools.cli.utils import get_file_extension
 from domaintools.exceptions import ServiceException
 from domaintools._version import current as version
@@ -114,8 +115,10 @@ def args_to_dict(*args) -> Dict:
     def _get_formatted_output(cls, cmd_name: str, response, out_format: str = "json"):
         if cmd_name in ("available_api_calls",):
             return "\n".join(response)
-        if response.product in RTTF_PRODUCTS_LIST:
-            pass  # do nothing
+        if isinstance(response, FeedsResults):
+            pass  # do nothing — streaming output handled in run()
+        elif out_format not in ("json", "xml", "html", "list"):
+            out_format = "json"  # download endpoint returns standard JSON
         return str(getattr(response, out_format) if out_format != "list" else response.as_list())
 
     @classmethod
@@ -225,20 +228,22 @@ def run(cls, name: str, params: Optional[Dict] = {}, **kwargs):
                 params = params | kwargs
 
                 response = dt_api_func(**params)
+                if not isinstance(response, FeedsResults):
+                    response_format = "json"
                 progress.update(
                     task_id,
                     description=f"Preparing results with format of {response_format}...",
                 )
 
-                if name not in ("available_api_calls",) and not getattr(response, "product", None) in RTTF_PRODUCTS_LIST:
+                if name not in ("available_api_calls",) and not isinstance(response, FeedsResults):
                     response.data()
 
                 output = cls._get_formatted_output(
                     cmd_name=name, response=response, out_format=response_format
                 )
 
             if isinstance(out_file, _io.TextIOWrapper):
-                if name not in ("available_api_calls",) and response.product in RTTF_PRODUCTS_LIST:
+                if name not in ("available_api_calls",) and isinstance(response, FeedsResults):
                     for feeds in response.response():
                         print(feeds)
                 else: