Querying a Contact entity based on AssignedToName

I am writing an API using the CRM SDK to get contacts based on an "assignedToName" which can be a systemuser name or a team name.
I am using QueryExpressions to query CRM. Question is how to create the query expression and the LinkEntities for this?
Currently the QueryExpression for Contact has 2 Link Entities using LeftOuter joins.
One from Contact to SystemUser on the AssignedToID = SystemUserID field
and another from Contact to Team on AssignedToID = TeamID field with a ConditionExpression for the "Name" field = "assignedToName"

The reason for having 2 link entities is because I do not know whether the "assignedToName" belongs to a Team or to a User.
The side effect of this is that now I have false positives in the results where in contact records not matching the "assignedToName" are also being included in the
result set due to the LeftOuter join. What is the correct way of writing the QueryExpression for this?

Sample Code:

 private IEnumerable<Entity> GetContactEntities(string assignedToName, IOrganizationService orgService)
        {
            QueryExpression qe = new QueryExpression("contact");
            LinkEntity linkSystemUser = new LinkEntity("contact", "systemuser", "assignedtoid", "systemuserid", JoinOperator.LeftOuter);
            LinkEntity linkTeam = new LinkEntity("contact", "team", "assignedtoid", "teamid", JoinOperator.LeftOuter);
            linkSystemUser.LinkCriteria.AddCondition(new ConditionExpression("name",ConditionOperator.Equal,assignedToName));
            linkSystemUser.EntityAlias = "linkSystemUser";
            linkTeam.LinkCriteria.AddCondition(new ConditionExpression("name",ConditionOperator.Equal,assignedToName));
            linkTeam.EntityAlias = "linkTeam";
            qe.LinkEntities.Add(linkSystemUser);
            qe.LinkEntities.Add(linkTeam);
            return orgService.RetrieveMultiple(qe).Entities;
        }

-Abhijeet