Bugs from testing

[cannium]

Env:
Test commit 6798275
Object size 10K
Delete ratio 0.5
Cannyls create parameter: capacity 30GiB, journal_ratio 0.1

Bug description:

• lump file stops growing at 8GiB(seen from ll -h)
• lots of error of FileNVM failed to write: write r/8380.lump: no space left on device
• a volume server crash:

2019-08-02 17:57:47.47264403 +0800 CST m=+107.261619150 End to restore index
Index's mem is 0
Min index is 0
Max index is 0
2019-08-02 17:57:47.472721876 +0800 CST m=+107.261696750 :Start to restore allocator
2019-08-02 17:57:47.473016414 +0800 CST m=+107.261991461 :End to restore allocator
I0802 17:57:47 63352 store.go:114] add volume 8750
I0802 19:28:37 63352 needle_parse_multipart.go:47] Reading Content [ERROR] unexpected EOF
I0802 19:28:37 63352 needle_parse_multipart.go:47] Reading Content [ERROR] unexpected EOF
I0802 19:28:37 63352 needle_parse_multipart.go:47] Reading Content [ERROR] unexpected EOF
panic: journal sync failed: FileNVM failed to write: write z/8722.lump: no space left on device

goroutine 18426 [running]:
github.com/thesues/cannyls-go/storage/journal.(*JournalRegion).Sync(0xc19ffe2360)
        /home/can/work/cannyls-go/storage/journal/region.go:255 +0xdc
github.com/thesues/cannyls-go/storage/journal.(*JournalRegion).trySync(...)
        /home/can/work/cannyls-go/storage/journal/region.go:262
github.com/thesues/cannyls-go/storage/journal.(*JournalRegion).appendWithGC(0xc19ffe2360, 0xc0012bb510, 0x1c2b780, 0xc21e7856c0, 0x0, 0x0)
        /home/can/work/cannyls-go/storage/journal/region.go:139 +0x9f
github.com/thesues/cannyls-go/storage/journal.(*JournalRegion).RecordPut(...)
        /home/can/work/cannyls-go/storage/journal/region.go:274
github.com/thesues/cannyls-go/storage.(*Storage).Put(0xc11f150300, 0x103bcf680, 0xc19d34f020, 0x200, 0xc19d34f020, 0xc0de4e42a0)
        /home/can/work/cannyls-go/storage/storage.go:293 +0x110
github.com/journeymidnight/seaweedfs/weed/storage.(*Volume).handleQuery(0xc11f180d20, 0x1, 0xc1e227aea0, 0xc087806420)
        /home/can/work/seaweedfs/weed/storage/volume_read_write.go:75 +0xf6
github.com/journeymidnight/seaweedfs/weed/storage.(*Volume).workerThread(0xc11f180d20)
        /home/can/work/seaweedfs/weed/storage/volume.go:113 +0x3c
created by github.com/journeymidnight/seaweedfs/weed/storage.NewVolume
        /home/can/work/seaweedfs/weed/storage/volume.go:59 +0x205
(END)

[thesues]

1. The error "no space left on device" is reported by linux filesystem instead of the allocator
2. It's the JournalRegion is "out of space"

It looks like "the fallocate" does not work at all. it even can not preallocate enough space for JournalRegion.

and I can not reproduce this bug on ubuntu 18.04 + ext4

What I did is in commit ba8bb96, in branch wip-preallocation

1. fallocate returns an error, if any error happens, return this fatal error.
2. modify cmd/kanils/main.go a bit to reproduce this bug though I can not reproduce this.
   use the cmd below

./kanils WRBench --storage=/home/ubuntu/bug.lusf --count=1572864 --size 10240 --sync=true

This program will try to preallocat 30GB first, and write 10K file, 50% to delete existing item.

[cannium]

So it's about the journal_ratio, could it return an error like "storage journal full" instead of panic?