Intelligent chart updating

[Flix6x]

Upon a new time window selection for a chart containing time series data, it is possible to determine what new data is needed and what old data is obsolete, and query and load only the data that is not already there. For instance, if the former selection is 1 year, and the new selection is that same year plus 1 extra day, we'd only need to load the extra day of data.

Relevant code with a first implementation can be found in the empras folder.

[nhoening]

@Flix6x is this still relevant?

[Flix6x]

It is. Somewhat more relevant, even, after my work on showing multiple sensor's on the asset page, and especially the replay functionality.

I should also note that, for a given time window, GETting all beliefs, rather than just the most recent beliefs, may be faster. Just a comment that may be relevant in the context of this PR, but possibly a separate issue (pipeline speedup). A workaround is to get all beliefs for the given period, and use the front-end filtering functionality to get the latest ones.

[nhoening]

I should also note that, for a given time window, GETting all beliefs, rather than just the most recent beliefs, may be faster.

Is this an observation? Am 31. März 2023 14:37:53 MESZ schrieb Felix Claessen ***@***.***>:

It is. Somewhat more relevant, even, after my work on showing multiple sensor's on the asset page, and especially the replay functionality. I should also note that, for a given time window, GETting *all* beliefs, rather than just the most recent beliefs, may be faster. Just a comment that may be relevant in the context of this PR, but possibly a separate issue (pipeline speedup). A workaround is to get all beliefs for the given period, and use the front-end filtering functionality to get the latest ones. -- Reply to this email directly or view it on GitHub: #101 (comment) You are receiving this because you commented. Message ID: ***@***.***>

-- Gesendet von /e/ Mail.

[Flix6x]

Yes, to observe this, too:

1. Load the asset page for an asset with a lot of sensor data
2. Select a date range (with or without sensor data)
3. Hit the replay button

Step 2 GETs only the most recent beliefs. Step 3 GETs all beliefs and filters it in the front-end, and seems a lot faster.

[nhoening]

Closing. I believe this is being picked up in more recent work.

[Flix6x]

Reopening: the intelligent chart updating described here had not actually been implemented. What landed since (the ECharts fast charts, #2234) made rendering much faster, but the fetching side was untouched — changing the selected time range still re-queried the entire range, and only a byte-identical selection was served from memory.

For the record, the empras folder referenced above was removed from main in ba8f900, a couple of months before this issue was filed. The prototype is still there in history, in empras/templates/index.html (updateData()), and its approach — one contiguous loaded interval, trim one side, fetch only the sliver on the other — is what #2433 follows.

Profiling on an asset with 5 sensors of 15-minute data over a year: a chart_data request costs ~105 ms regardless of size, plus ~24 µs per record, and ~93% of the total cost is server time. So extending a 364-day window by a day went from 4.4 s / 14 MiB to 127 ms / 40 KiB, and narrowing the selection and widening it back now costs no request at all.

The second observation in this thread — that fetching all beliefs and filtering in the front end may beat most_recent_beliefs_only=true — is not addressed by #2433, and may well have changed since the search-beliefs post-processing work. Worth re-measuring separately.