jeudi 19 novembre 2020

Checking if row exists in Table with 10 million rows

I'm importing XML file values to DB and end up checking if the row already exists, this takes a lot of time as the table has around 10million rows. For a file to import it takes roughly 90 seconds, on empty database 4secs. I'm checking if the row exists with laravel eloquent exists() method. I want to reduce this time as much as possible, if im importing 10k files, it takes DAYS. I was thinking of saving inserted rows in redis cache and checking if row exists from cache, but if for some reason cache is cleared I don't want to make duplicates in database. Is there anything better than redis?



via Chebli Mohamed

Aucun commentaire:

Enregistrer un commentaire