Duplicated Fitbit data on output storage for certain topics

[pvannierop]

Problem

For certain Fitbit topics on a client deployment, selected topics started showing duplicated data rows in output storage analogous to:

key.subject	key.source	key.project	value.time	value.timeRecieved	value.value
my-subject-id	my-source-id	my-project-id	000000001	000000005	20
my-subject-id	my-source-id	my-project-id	000000001	0000000010	20

Note that Fitbit data recorded at the same time appears as a separate line; the only difference is the value.timeReceived.

This was observed for the following Fitbit topics:

connect_fitbit_breathing_rate
connect_fitbit_intraday_heart_rate_variability
connect_fitbit_sleep_stages
connect_fitbit_activity_log
connect_fitbit_skin_temperature
connect_fitbit_sleep_classic

it was noted to NOT occur for the following Fitbit topics:

connect_fitbit_intraday_calories
connect_fitbit_intraday_steps
connect_fitbit_intraday_heart_rate

Some observations

• Data deduplication by output-restructure normally takes all columns into account. For this reason the value.timeReceived is causing the same piece of data to be added as a separate line.
• I suspect that previous from older versions of Fitbit connector, the current connector repeatedly downloads the same date range for the problematic topics. The inclusion of the value.timeReceived column during de-deduplication causes the data to appear multiple times in the output files.
• Many of the problematic topics were updated in PR Update routes to limit range to 30 days #118
• The topics that work well all have a 1d time period hard-coded in the request URL whereas others do not:

FitbitIntradayCaloriesRoute.java:    return baseUrl + "/1/user/%s/activities/calories/date/%s/1d/1min/time/%s/%s.json?timezone=UTC";
FitbitRestingHeartRateRoute.java:    return baseUrl + "/1/user/%s/activities/heart/date/%s/1d.json?timezone=UTC";
FitbitIntradayStepsRoute.java:    return baseUrl + "/1/user/%s/activities/steps/date/%s/1d/1min/time/%s/%s.json?timezone=UTC";
FitbitIntradayHeartRateRoute.java:    return baseUrl + "/1/user/%s/activities/heart/date/%s/1d/1sec/time/%s/%s.json?timezone=UTC";

FitbitSkinTemperatureRoute.java:    return baseUrl + "/1/user/%s/temp/skin/date/%s/%s.json";
FitbitBreathingRateRoute.java:    return baseUrl + "/1/user/%s/br/date/%s/%s/all.json";
FitbitIntradayHeartRateVariabilityRoute.java:    return baseUrl + "/1/user/%s/hrv/date/%s/%s/all.json";
FitbitIntradaySpo2Route.java:    return baseUrl + "/1/user/%s/spo2/date/%s/%s/all.json";
FitbitTimeZoneRoute.java:    return baseUrl + "/1/user/%s/profile.json";
FitbitSleepRoute.java:    return baseUrl + "/1.2/user/%s/sleep/list.json?sort=asc&afterDate=%s&limit=100&offset=0";
FitbitActivityLogRoute.java:    return baseUrl + "/1/user/%s/activities/list.json?sort=asc&afterDate=%s&limit=20&offset=0";