mercredi 30 septembre 2015

Transaction logs with large data processing in mysql

Don't know if I posted in the correct site. But here goes, Im having a problem with database transaction in processing large files. What my program do is I upload a csv file and in that file it can have up to 50,000 records. Right now it worked when I processed 200 rows but when I tried with 20,000 rows it never get to insert in my db (i saw the insertion sql in my logs but when i check it out it never entered). So I'm guessing that my buffer size for the transaction logs hit its limit.

I have this code in my laravel

try {
  DB::beginTransaction();
  process my logic (validation, log, etc..)
  DB::commit()
}catch() {
  log::error();
  DB::rollback;
}

I just added these two config in my /etc/mysql/my.cnf

[innodb]
innodb_buffer_pool_size = 128M
innodb_log_file_size    = 64M

I thought it would fix my problem but sadly not. Please help.



via Chebli Mohamed

Aucun commentaire:

Enregistrer un commentaire