Scope by custom attribute or excluding data by scope

I have a Model User that has a polymorphic relation to table Relationships with Pivot column relationship_level.

public function activities()
{
    return $this->morphedByMany('App\Activity', 'relationship')->withPivot('relationship_level')->withTimestamps();
}

public function relationships()
{
    return $this->hasMany('App\Relationship');
}

I want one the following:

I want scopeNoCitizens which to exclude everyone who has a relationship_level Citizen no matter what other relationships he has.

I've tried:

scopeNoCitizens($query) {
    $query->whereHas('relationship', function($query) {
        $query->where('relationship_level', '!=', 'Citizen');
    });
}

This doesn't work, maybe because all users have many levels and I want everyone who has even one record among many that has level citizen to be excluded from the collection.

I also have a custom attribute getHighestRoleAttribute, but as far as I understand you can't filter by accessor/custom attribute in a scope.

What I currently do is "filter" the collection after get(), but I have to manually paginate after that and do that on numerous places in the app.

via Chebli Mohamed