I didn't mind it when Japanese people who were trying to help me for whatever reason spoke English to me. It may have been slightly presumptuous of them to assume that I spoke English, but their intentions were always kind. I did mind it when one of a pair of random strangers who were both Japanese and speaking Japanese to each other suddenly started to speak English at the sight of me. I'm not sure what the motivation was as I'm pretty sure it had nothing to do with "helping" me by making sure I understood any overheard conversation. It seemed to me just another variation on "gaijin da!" and it was meant to make sure they knew I recognized my own foreignness. Frankly, it really felt like a way of mocking me and I found it quite rude.
I could be wrong about the motives of such people. Perhaps they were insufferable show-offs who were waiting for a chance to demonstrate their dubious English skills to someone who they believed would appreciate them, but I will not miss Japanese people who speak English to one another upon seeing me.