Action server error time to time

[daeunSong]

Hi,

I am currently running KUKA LBR iiwa 7 R800 with ROS Melodic with iiwa_stack.

I get the following error in SmartPad when running the spline motion or setting the end effector tool frame (which are sent via action and service) while having no error logs in the computer (ROS commanding node):

Stopping task due to the following error:
java.lang.IllegalStateException: Error during request GetCurrentConfigurationRequest: Sunrise did not respond to SSR DeviceObserverDFE request within 5 SECONDS.
...

I do not get this error all the time, but from time to time, even when running the same code on with same settings with the same computer. For example, when I run the 100 spline motions, the error occurs sometimes in the 30th motion and sometimes in the 6th motion.

I am suspecting that it could be my network speed issue.. but not so sure.
I wonder if anyone is having a similar issue or has clue about this error.

Thanks !

[exo-core]

Dear Daeun,

unfortunately I have never encountered this particular error before, so I can only guess where the issue might be coming from.

When exactly does this error appear at some point during the motion or a the very beginning of it? Without having seen the full stack trace I would guess it is coming from the real time PC supervising the safety constraints, which is lacking feedback from the control PC...

As you might know it is possible to connect the Sunrise cabinet to an actual screen, mouse and keyboard and log in to the embedded windows running on it. If I remember correctly the login is administrator and 68kuka1secpw59. For a starting point I would take a look at the task manager and check for high CPU load. If this is the case you could try to reduce the spin frequency of the iiwa_stack node, which is set to 100Hz by default. Potentially, updating to a later SunriseOS version might also help.

Regarding the network: The easiest way of verifying would of course be to isolate the robot communication and check whether the issue still appears. You could also try rostopic hz on /iiwa/joint_states or /iiwa/actions/move_along_spline/status (or similar) to see whether the iiwa's communication thread is falling behind. Finally - if I remember correctly - the iiwa's message receiver is running in the main thread, so it might be on option to measure the runtime and see of any of the ActionServers checks for new goals are blocking for too long.

Hope this helps.
Best regards,

Arne

[daeunSong]

Hi Arne,

thank you very much for your kind reply.

When exactly does this error appear at some point during the motion or a the very beginning of it?

Unfortunately, this is random. It sometimes appears right after this log before executing the motion, but sometimes after this log, which it seems like before/after setting the tool frame. (I am setting the tool frame before each spline motion.) I will try to provide the screenshot of the full stack trace for the details.

you could try to reduce the spin frequency of the iiwa_stack node, which is set to 100Hz by default.

I wonder if I got you right. Do you mean the ROSSmartServo node run by the cabinet PC when you mention iiwa_stack node or the node run by a remote PC to command robot?
If former, I wonder where to change this frequency. I see 100 in here and also in line 349, but I see none of them are effecting my application, as I am not using NTP more rosTool.
If latter, just to be sure, do you mean by this ros rate here?

I will also try all your suggestions and let you know.
Thank you very much!

Daeun

[daeunSong]

I checked for the CPU load as you suggested, and surprisingly, when I run the ROSSmartServo application, the CPU usage gets 100%. I did not know about this issue, as the robot motion worked quite well just for simple communication.

Currently we are using Sunrise OS v1.11. Do you have any suggestion for the Sunrise OS version? I plan to ask KUKA for the upgraded version.

Thanks.

[exo-core]

Hi Daeun,

our iiwas are running on Sunrise OS 1.16. However, the fact that we never encountered the issue you describe is of course no guaranty that the problem does not occur in the updated version. It could also just be the case that your splines are simply too complex...
Please also note: For updating Sunrise OS you will need a KUKA Recovery USB flash drive to install the new Windows image they will send you.

Cheers,

Arne

[daeunSong]

Hi Arne, Thank you for the information and the note.

As I have mentioned in the latest comment, our Sunrise OS CPU usage goes to 100% as soon as we run ROSSmartServo, without any other iiwa-related node running. This seems strange.. I wonder if this happened to any others, too.

I thought upgrading the OS worth a try. But, yes, I understand that an OS upgrade will not guarantee the fix.

Daeun