Limit query responses and document assets

[ejanalysis]

Limit /query responses to 100 rows, return a message when truncation occurs, validate value inputs, and document the static assets route in the README.

[ejanalysis]

/query needs either pagination, a limit on number of row returned, or a way to select fewer columns. Otherwise broad cutoffs like 0.80 can fail with a 500 code from response size

[ejanalysis]

The limit does not have to be 100 -- change it if you think it should be higher, before merging.

[ericnost]

Thanks for this @ejanalysis! Yes, the API needs to be more like an API.... I'm not sure capping at 100 or even 500 rows is a great solution. Pagination is, but it will take some development and testing. This, generated with the help of Gemini through a Google search, seems like the bones of what we want. We'd be adding a page parameter, then on each call to the endpoint, the API would do the whole calculation, returning the results corresponding to the requested page (e.g. if the limit is 100 per page, then page 2 would return results 101-200, or 100-199 depending on indexing).

#* Get a paginated list of items
#* @param page The page number (default 1)
#* @param limit Number of items per page (default 100)
#* @get /data
function(page = 1, limit = 100) {
  
  # Ensure parameters are numeric
  page  <- as.numeric(page)
  limit <- as.numeric(limit)
  
  # Calculate index range
  start_idx <- ((page - 1) * limit) + 1
  end_idx <- page * limit
  
  # Assuming 'my_dataset' is available in your R environment
  total_rows <- nrow(my_dataset)
  
  # Handle out-of-bounds pages gracefully
  if (start_idx > total_rows) {
    return(list(
      data = list(),
      metadata = list(
        current_page = page,
        total_items = total_rows,
        total_pages = ceiling(total_rows / limit)
      )
    ))
  }
  
  # Extract the requested slice of data
  paginated_data <- my_dataset[start_idx:min(end_idx, total_rows), ]
  
  # Return combined data and metadata
  list(
    data = paginated_data,
    metadata = list(
      current_page = page,
      limit = limit,
      total_items = total_rows,
      total_pages = ceiling(total_rows / limit)
    )
  )
}

This example has limit as a parameter, but I might just hard code it to something like 500. Or, leave it as a parameter, but make it no more than 500.