inconsistent performance for long running insertions of 10 mil entries

[sprappcom]

i stress tested 10 mil 48bytes long running data entries, i have inconsistent insertion rates.

1. how to improve?
2. it seemed to be running some "compaction" thing behind / wal issue?

Table created successfully
Table verified
Starting insert benchmark...
Inserted: 470000/10000000 (468195.88 inserts/sec)
Inserted: 920000/10000000 (491319.35 inserts/sec)
Inserted: 1400000/10000000 (1963.29 inserts/sec)
Inserted: 1840000/10000000 (488281.54 inserts/sec)
Inserted: 2330000/10000000 (486896.23 inserts/sec)
Inserted: 2810000/10000000 (501378.84 inserts/sec)
Inserted: 3240000/10000000 (509590.83 inserts/sec)
Inserted: 3730000/10000000 (474266.09 inserts/sec)
Inserted: 3930000/10000000 (22011.89 inserts/sec)
Inserted: 4150000/10000000 (1325.40 inserts/sec)
Inserted: 4620000/10000000 (488529.12 inserts/sec)
Inserted: 4960000/10000000 (153780.87 inserts/sec)
Inserted: 5270000/10000000 (470858.65 inserts/sec)
Inserted: 5760000/10000000 (495120.37 inserts/sec)
Inserted: 5800000/10000000 (2199.26 inserts/sec)
Inserted: 6150000/10000000 (489426.22 inserts/sec)
Inserted: 6470000/10000000 (442425.24 inserts/sec)
Inserted: 6730000/10000000 (499604.54 inserts/sec)
Inserted: 7220000/10000000 (494694.60 inserts/sec)
Inserted: 7420000/10000000 (1060.98 inserts/sec)
Inserted: 7880000/10000000 (460902.05 inserts/sec)
Inserted: 8260000/10000000 (101474.55 inserts/sec)
Inserted: 8570000/10000000 (8175.22 inserts/sec)
Inserted: 8840000/10000000 (15937.13 inserts/sec)
Inserted: 8920000/10000000 (494.28 inserts/sec)
Inserted: 9370000/10000000 (473064.40 inserts/sec)
Inserted: 9840000/10000000 (51414.10 inserts/sec)
Insert benchmark completed
Starting query benchmark...
Queries: 2037000 (2088838.29 queries/sec)
Queries: 4143000 (2110613.01 queries/sec)
Queries: 6213000 (2173577.56 queries/sec)
Queries: 8287000 (2174905.94 queries/sec)
Queries: 10378000 (2077404.08 queries/sec)
Queries: 12480000 (2163027.37 queries/sec)
Queries: 14398000 (2166124.41 queries/sec)
Queries: 16438000 (2122236.58 queries/sec)
Queries: 18527000 (2140667.55 queries/sec)
Query benchmark completed

[jcwangxp]

It's because rehash process which will rehash all existing inserted records, so it's slow sometimes. Will use graceful rehash later.

[sprappcom]

@jcwangxp any eta on this?

[jcwangxp]

Could you give your test program and I'd like to reproduce locally to find the root cause of the inconsistent performance.

[sprappcom]

been so long... not sure where... i will try dig it out by next week