Issue with IO Queue Full Condition

[pratikarb]

Hi,

I am validating NVMe drive. For validation of that Drive, I have written the Host Test Cases(on Ubuntu 14.04) which calls/uses the DNVMe Low APIs.

Issue Found: Phase-bit toggle issue in IO queues

I have written the test case to validate the Phase-bit toggle condition. For this I have set the IOSQ and IOCQ q_size to 5. As NVMe specs expect it as 0's based value. There are six Entries to process by the Drive. But Host sends 5 commands only. And it expects wrap-around of the Queue at Fifth command only.

On further debug, I found following observations with dnvme Driver:

1. NVME_IOCTL_CREATE_ADMN_Q expects the 1 based for value for the No. Of Elements in the Admin Queues. But when it issues the Write to AQA register, it sends (No. Of Elements - 1) value. Confirmed on the PCIe Analyzer trace and Reading AQA Register in the FW.

For Example, if No. of elements value = 100, the ioctl(file_desc,NVME_IOCTL_CREATE_ADMN_Q,&aq_data ) writes the value 99 to the AQA register for that particular Q. This is what expected by the NVMe Drive. With Admin Queues, Q-Full and Phase toggle test case passes correctly.

2.But in case of NVME_IOCTL_PREPARE_SQ_CREATION &
NVME_IOCTL_PREPARE_CQ_CREATION, driver sends the 1 based value as it is (without decrementing it by 1). This actually conflicts with NVMe Spec. Because of this, On Q_Full condition the driver fails and phase-bit toggling in completion never happens. For this I have to do changes in my FW. i.e. I have to decrement the IO Q size by 1. Then Everything work fines.

Note: This issue can be replicated very easily, by writing the test case for Q-Full condition or to verify Phase-bit toggling. This Issue is with both IO queues only.
Regards,
Pratik