Pool/Miner stats stop working

[mooleshacat]

When an invalid miner address makes it into the redis database, the miner stats and pool stats will stop working.

This is because when the browser is making the request to the API for live data, the API stumbles across an invalid address, and attempts to retrieve data for it - causing an exception, and halting the API response dead in it's tracks.

Short term fix is to just clear redis data with redis-cli FLUSHALL; unfortunately this clears all payment data.

Long term, this issue needs to be addressed. https://github.com/foxer666/node-open-mining-portal/issues/143 may fix it. We need a better way to handle exceptions in the API where it fails so that the data is finished outputting in a way the charts still show.

Solution is to validate every worker address that the API requests information for, and only request and return information for valid workers.

Also API data retrieval is very slow. This can be due to https://github.com/foxer666/node-open-mining-portal/issues/143 as well.

Master Issue:
EasyX-Community#140

Duplicate issues:
zone117x/node-open-mining-portal#644
#10

Samples:
https://miningpool.easyx.info/stats/mogwaicoin (charts not loading)
https://miningpool.easyx.info/api/pool_stats (only available until next FLUSHALL)
Uploaded TXT file coming soon...

[mooleshacat]

Turns out I was wrong... But installing RedisMin has brought me closer to the issue.

When I access the "StatHistory" table, it gives an error "ESOCKETOVERFLOW" and the redis server restarts...

Reloading any page with graphs on it, and going back to RedisMin, gives the same error + restart.

Suspecting a corrupted table somehow?

Screenshot attached.

Ideas?

[mooleshacat]

Purging Redis data clears the issue, and at some point it comes back...

This could be due to the memory filling up (because we are using a memory cache like a database)

Solution: Need actual DB (MySQL/MariaDB) to operate properly (and with less memory)

[mooleshacat]

More data: It just happened again. I can access all tables except the last one, statHistory.
There must be invalid data causing a BOF when accessing it, because RedisMin can't even access it.

I will try to flush just that table and see what happens.

[UPDATE] redis-cli DEL statHistory and restarting pool clears the issue temporarily, but statistics data is lost.

More reason why we need MySQL