Stability of parellel.foreach inner loop on another foreach

I have written a structure of executing list of command parallelly to multiple devices at the same time.

Parallel.ForEach(this.deviceCmdListMap,//multiple device
new ParallelOptions
{
// multiply the count because a processor has 2 cores
//MaxDegreeOfParallelism = Convert.ToInt32(Math.Ceiling((Environment.ProcessorCount * 0.75) * 2.0))
},
keyValuePair =>
{
......
foreach (string cmd in keyValuePair.Value)//list of command that prepare to send over to a device
{
...
//send command

//receive response from the device
}
}

Most of the the time, the send and receive response message from devices parallelly work normally, however less stability. If you have idea how to resolve the issue, welcome to bid the job.
You are no need to write the send or receive message from device anymore, just need to increase the stability/reliability.