I found the problem. We use a Committed(F/M) signal to indicate the IFU or LSU has an ongoing cache or bus transaction and should not be interrupted. At the time of the mret, the IFU is fetching uncacheable invalid instructions asserting CommittedF. As the IFU finishes the request it unstalls the pipeline but continues to assert CommittedF. (This is not necessary for the IFU). In the same cycle the LSU d cache misses. Because CommittedF is blocking the interrupt the d cache submits a cache line fetch to the EBU.
I am thinking out loud here. At it's core the Committed(F/M) ensure memory operations are atomic and caches don't get into inconsistent states. Once the memory operation is completed the LSU/IFU removes the stall but continues to hold Committed(F/M) because the memory operation has completed and it would be wrong to allow an interrupt to occur with a completed load/store. However this is not true of the IFU. If we lower CommittedF once the operation is complete then this problem is solved. The interrupt won't be masked and the LSU will flush the d cache miss.
This requires a minor change in the cachebusfsm and cachefsm. I will report back after I've confirmed this works.
