Hey, I had something very similar happen to me on opening day. I was setup on a ridge, to the north of the turkey's roost. The creek bottom ran north and south, so it's off my right shoulder, and the turkey is directly south of me. When I tree yelped, his hen cut me off with yelping. He gobbled well on the roost, and got silent at flydown. The next time he gobbled, he was already north of me in the creek bottom. He had already gotten past me. I swung around to the far north end of the creek, where I was able to call him in to about 70 yards. He strutted there from 920-1040. He did not answer any call, he did not come to me, the only reaction he gave to my call, was to fluff out more. He simply did not want to leave that hen, no matter what I did.
The turkey that went past you, maybe he was following a hen, or maybe he had somewhere he wanted to go. He will gobble at you, saying, "hey girl, come join my party" but unless you are where he's going anyway, it can be tough.
What I am learning this season. (And I'm still learning, so don't think I'm an expert.) Seems to me that off the roost, those hens have a specific area they want to go to, if you could learn where that is, it may help.