Efficient and Fast Data Import with Laravel Jobs & QueuesLaravel
August 14, 20202 minutesuserMitul Golakiya
Efficient and Fast Data Import with Laravel Jobs & Queues

Recently, we have started working on one CRM System for client, where we offer a user to create an account and then can upload his excel file of Contracts and we will create contracts in the database, and then these contracts will be emailed to both the parties of the contract.

For this, the functionality we want is, we do not let users wait to do these all things, because this process may take longer based on the number of records the user has. Also, this process contains lots of computation like,

  1. Upload an Excel file
  2. Do field mappings
  3. Validate the record of contract
  4. Create a contract in the database along with all other information
  5. Create a PDF of contract (which take significant time)
  6. Email that PDF to both the parties.

Just imagine, if the user has uploaded a file with 10K records in a file it will take an hour to do this in a single process.

Here is where we used the Laravel Jobs & Queues where we can let use relax and we do our processing in backend with multiple Jobs running at the same time.

So we came up with architecture with Several Jobs in place with the following things in mind.

  1. We will have multiple Jobs to perform multiple things to have SRP (Single Responsibility Principle)
  2. When the one Job completes its duty, it will dispatch the next Job with the data that next job needs.
  3. The next job will do that same till the process ends

Another advantage we got here is, we can execute different jobs on different machines/servers based on the resources required. For e.g. We run a Job of Creating PDF on a powerful machine then sending an email.

Also, we can set different priorities for different Job Listeners. For e.g. Sending an Email Job will have a lower priority than others.

So here is the architecture that we came up with and used. I tried to architect the Jobs structure only and haven't mentioned small details of updating and creating records.

Implementing Efficient and Fast Data Import with Laravel Jobs & Queues

Hope it will help you to architect you a better Jobs and Queues architecture where you need to perform multiple things on the backend and do not let your users wait to finish those long-running processes.

Happy & Safe Queuing. :)