EODHD's Financial News API has ineffective pagination behavior.. could use an additional parameter to modify "direction".

[wilson0x4d]

The offset parameter of the API paginates "backwards". If we have a to and from parameter that represent "today" there is no way to use offset to paginate the results to only receive the "latest" or "new" news items.

The problem is as "new" items are available from the API the only way to retrieve them is to specify offset of 0. This manipulates everyone into requesting limit=1000 and offset=0 all day long, and then implementing logic to detect duplicate news items and discard them. This is non-beneficial to EODHD's server capacity (or costs) and has a similar negative impact for customers (because we are forced to re-receive news we've already received, and, we're forced to check if the news was already received before running it through our pipelines.)

What would really help is if there were a new parameter direction that allowed us to reverse the pagination logic. In the following example direction=0 would behave as the current/default behavior, direction=1 would reverse the pagination to be "forward-only"; example:

Current Pagination Behavior (direction=0 -- the default)

offset=0&limit=3&from={today}&to={today}

timestamp	title
5pm	news at 1700
4pm	news at 1600
3pm	news at 1500

offset=1&limit=3&from={today}&to={today}

timestamp	title
4pm	news at 1600
3pm	news at 1500
2pm	news at 1400

Proposed New Pagination Behavior (direction=1)

offset=0&limit=3&from={today}&to={today}&direction=1

timestamp	title
8am	news at 8
9am	news at 9
10am	news at 10

offset=1&limit=3&from={today}&to={today}&direction=1

timestamp	title
9am	news at 9
10am	news at 10
11am	news at 11

Why?

This would allow us to request only "new" news items as the day progresses, and in theory it only requires a sort operation to be implemented by EOD HD.

Again, currently, we are forced to pull "all the news" for the entire day because there's no way to use offset to paginate "forward".

Parting thoughts.. (or "Also, there's another problem..."

Consider this scenario:

1. EODHD has 150 news items available.
2. We request limit=100 with offset=0 for "today".
3. We receive 100 items, so our next request will specify offset=100 in an attempt to get the remaining N items that are available.
4. Before our next request, EODHD's back end receives a new news item for a total of 151 news items available.
5. Our system requests limit=100&offset=100 since it previously received 100 news items.
6. Our system receives a duplicate news item because offset is calculated from newest to oldest by EODHD, and it has received more news.

EODHD could solve this problem by using a bookmark instead of offset, but that complicates things. It would be easier to just add a direction parameter as described above to let us paginate "forward" instead of "backward" (current behavior.)

Thanks.