Eloquent Relation not working as expected in laravel

I have these two tables:

One is Product Table :

 |id  | Title     | Price|
 -------------------------
 |1   | Title 1   |  5000|
 |2   | Product 2 |  7000|

and the other is product attribute table:

 |id | product_id | attribute_id | attribute_name | value|
 ---------------------------------------------------------
 |1    | 1        |       5      |     Color      |  Red |
 |2    | 1        |       6      |     Size       |  XL  |
 |3    | 2        |       5      |     Color      | Green|

Product and Product attribute is related with following relation (In the product model):

 public function attributes()
    {
        return $this->hasMany(ProductsAttribute::class, 'product_id ');
    }

I'm fetching data like this :

    return Product::with('attributes')
            ->whereHas('attributes', function ($query) use ($attribute_id,$attribute_value){
           if (!empty($attribute_id) && !empty($attribute_value)) {
                $query->whereIn('attribute_id', $attribute_id);
                $query->whereIn('value', $attribute_value);
            }
            })
            ->paginate(10);

There are multiple attributes of a product, even a single match to a given attribute value & attribute id returns the product but i want if any of the attributes doesn't match i.e. its entry is not in the product attribute table then that product should not return in the result.

eg. If user has selected color with attribute id 5 and value red and size of attribute id 6 and value xl then only Product with title 1 should be returned.

Any help is highly appreciated.

via Chebli Mohamed