Add individual daily transfers. Closes #253 #268
Conversation
| export const fetchDailyRewardTransfers = async (pgPool, filter) => { | ||
| const { rows } = await pgPool.query(` | ||
| SELECT day::TEXT, SUM(amount) as amount | ||
| SELECT day::TEXT, to_address, amount |
There was a problem hiding this comment.
Would it be possible to create some kind of iterator using something like pg-cursor so we could avoid loading all rows into memory in order to do calculation?
There was a problem hiding this comment.
Good point.
I ran this query against the live database, and it returned less than 2000 rows. I think that's because we have data from 2024-12-02 and 2024-12-09 only. When I extrapolate that to the entire year (52 weeks) and account for double network size, we are talking about ~110k rows. Loading that into memory at once seems doable to me. WDYT?
EXPLAIN ANALYZE
SELECT day::TEXT, to_address, amount
FROM daily_reward_transfers
WHERE day >= '2024-12-01' AND day <= '2024-12-31';
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------
Index Scan using daily_reward_transfers_day on daily_reward_transfers (cost=0.29..72.24 rows=1862 width=87) (actual time=0.032..0.586 rows=2189 loops=1)
Index Cond: ((day >= '2024-12-01'::date) AND (day <= '2024-12-31'::date))
Planning Time: 0.106 ms
Execution Time: 0.720 ms
Having wrote that, I have a different concern:
The execution time for two weeks' worth of data is 0.720ms. That would be like 18 seconds to query the entire year. I think we need to impose some limits on the filter interval this API endpoint allows.
There was a problem hiding this comment.
@bajtos Indeed, limiting the range would be more optimal.
| export const fetchDailyRewardTransfers = async (pgPool, filter) => { | ||
| const { rows } = await pgPool.query(` | ||
| SELECT day::TEXT, SUM(amount) as amount | ||
| SELECT day::TEXT, to_address, amount |
There was a problem hiding this comment.
Good point.
I ran this query against the live database, and it returned less than 2000 rows. I think that's because we have data from 2024-12-02 and 2024-12-09 only. When I extrapolate that to the entire year (52 weeks) and account for double network size, we are talking about ~110k rows. Loading that into memory at once seems doable to me. WDYT?
EXPLAIN ANALYZE
SELECT day::TEXT, to_address, amount
FROM daily_reward_transfers
WHERE day >= '2024-12-01' AND day <= '2024-12-31';
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------
Index Scan using daily_reward_transfers_day on daily_reward_transfers (cost=0.29..72.24 rows=1862 width=87) (actual time=0.032..0.586 rows=2189 loops=1)
Index Cond: ((day >= '2024-12-01'::date) AND (day <= '2024-12-31'::date))
Planning Time: 0.106 ms
Execution Time: 0.720 ms
Having wrote that, I have a different concern:
The execution time for two weeks' worth of data is 0.720ms. That would be like 18 seconds to query the entire year. I think we need to impose some limits on the filter interval this API endpoint allows.
Closes #253
With this change, we don't need to manually copy transfers into the rewards releases spreadsheet any more.