Add MySQL / MariaDB / MongoDB option

[mooleshacat]

@leshacat
Member
leshacat commented on 4 Nov 2018 •
I run NOMP on a VPS which has limited memory, and has SWAP mounted to /dev/null

Either init.js or Redis is causing memory to be full.

Can we add in either MySQL / MariaDB / MongoDB option please? It is not good to only have Redis because it is a cache program, NOT a database.

Also there are problems with saving data for reboot. "redis-cli save" does not seem to work. It may be with the way the server just kills the redis process, I will look into it further and post back here.

Easiest fix is to enable more DB options.

Duplicates:
https://github.com/foxer666/node-open-mining-portal/issues/150
zone117x/node-open-mining-portal#644
#140

Thanks

@leshacat leshacat added Enhancement Help Wanted Feature Request v1.1.4 finalization Redis Bug Critical Priority Bug labels on 4 Nov 2018
@leshacat
MemberAuthor
leshacat commented on 4 Nov 2018 •
Enabled AOF (google redis persistance) which causes data to persist across reboots.

Turns out the RDB just starts at low memory (< 5MB) and as shares are submitted, the memory fills up -- this is a security issue, DoS can be achieved by mining enough shares to fill memory.

This is bad design from the beginning, to stuff everything into the memory which most people are limited. Memory has some limit, and thus we should not be using Redis Memory Cache in this way.

Proper database is severely needed.

@leshacat leshacat added the Security Issue label on 4 Nov 2018
@leshacat
MemberAuthor
leshacat commented on 4 Nov 2018 •

Data Sample:

{"time":1541380048,"global":{"workers":2,"hashrate":0},"algos":{"neoscrypt":{"workers":2,"hashrate":744548.4673870506,"hashrateString":"744.55 KH/s"},"x16r":{"workers":0,"hashrate":0,"hashrateString":"0 H/s"}},"pools":{"mogwaicoin":{"name":"mogwaicoin","explorerGetBlock":"http://explorer.mogwaicoin.org/block/","explorerGetBlockJSON":"http://explorer.mogwaicoin.org/api/getblock?hash=","explorerGetTX":"http://explorer.mogwaicoin.org/tx/","symbol":"MOG","algorithm":"neoscrypt","rewardRecipients":{},"poolStats":{"validShares":"1671","validBlocks":"1","invalidShares":0,"totalPaid":0,"networkBlocks":"87763","networkSols":"29564337.28829854","networkSolsString":"29.56 MH/s","networkDiff":"0.8685194020356234","networkConnections":"37","networkVersion":"/Mogwai Core:0.12.2.4/","networkProtocolVersion":"70209"},"blocks":{"pending":1,"confirmed":0,"confirmedData":[],"orphaned":0},"pending":{"blocks":[],"confirms":{}},"confirmed":{"blocks":[]},"payments":[],"currentRoundShares":{"MAKrZXqxfmuzscgsZMknSjMK3Fx2xfhEZc.BOX0":"26569.97493844999999091","MAKrZXqxfmuzscgsZMknSjMK3Fx2xfhEZc.BOX1":"337.42414621000000005"},"workers":{"MAKrZXqxfmuzscgsZMknSjMK3Fx2xfhEZc.BOX1":{"shares":34.211876649999994,"diff":32432,"invalidshares":0,"currRoundShares":337.42414621,"currRoundTime":0,"hashrateString":"14.95 KH/s","luckDays":"824121.677","luckHours":"19778920.245","hashrate":14947.396987562664},"MAKrZXqxfmuzscgsZMknSjMK3Fx2xfhEZc.BOX0":{"shares":1669.92432495,"diff":1954289,"invalidshares":0,"currRoundShares":26569.97493845,"currRoundTime":0,"hashrateString":"729.60 KH/s","luckDays":"16883.848","luckHours":"405212.362","hashrate":729601.070399488}},"hashrate":744548.4673870506,"luckDays":"16544.892","luckHours":"397077.404","workerCount":2,"hashrateString":"744.55 KH/s"},"ravencoin":{"name":"ravencoin","explorerGetBlock":"http://rvnhodl.com/block/","blockTime":60,"explorerGetBlockJSON":"http://rvnhodl.com/api/getrawtransaction?decrypt=1&txid=","explorerGetTX":"http://rvnhodl.com/tx/","symbol":"RVN","algorithm":"x16r","rewardRecipients":{},"poolStats":{"validShares":0,"validBlocks":0,"invalidShares":0,"totalPaid":0,"networkBlocks":"434686","networkSols":"8123667518403.457","networkSolsString":"8.12 TH/s","networkDiff":"104746.0371192352","networkConnections":"8","networkVersion":"/Ravencoin:2.1.0/","networkProtocolVersion":"70016"},"blocks":{"pending":0,"confirmed":0,"confirmedData":[],"orphaned":0},"pending":{"blocks":[],"confirms":{}},"confirmed":{"blocks":[]},"payments":[],"currentRoundShares":{},"workers":{},"hashrate":0,"luckDays":"Infinity","luckHours":"Infinity","workerCount":0,"hashrateString":"0 H/s"}}}
Judging by the amount of, and structure of the table and the data, NOMP should have used a proper database from the beginning. All this share data will eat up limited system memory in no time.

I am just learning NodeJS, but I will look into this for v1.1.4 if not yet completed by someone else. I have a PHP & MySQL background, this should help. Only need to learn how to access MySQL from NodeJS (Google will help)

Adding MySQL and doing a performance/efficiency audit is recommended as others have reported performance issues and this is likely to be the cause.

This is another feature which I will eventually push to NOMP/BootNOMP repos for all to use :)

@leshacat leshacat removed the Feature Request label on 4 Nov 2018
@leshacat leshacat referenced this issue on 18 Dec 2018
Closed
Pool/Miner stats stop working #140

[mooleshacat]

[mooleshacat]

Redis memory issues were resolved (for the most part)

I have other things I need to implement but I will eventually analyze and improve the API more.