I’ve been working on a BuddyPress/BuddyBoss website lately and we recently integrated custom notifications. These trigger whenever a group admin posts a message in a groups feed. All group members receive a notification when this happens.
The website biggest group contains 7000 members, so this process can get quite intensive. Anyway, triggering the notification didn’t use so much resources. Sending out those e-mails though, that’s another cup of tea.
The fun part? Nobody (including me) realised that adding e-mail notifications also means sending out 7000 e-mails at once until we needed the functionality.
So we had to come up with a solution, and quickly as well.
Context
The e-mails are sent via The BuddyBoss/BuddyPress system. We’ve added a new situation using the bp_email_get_type_schema and bp_email_get_schema filters. This makes the e-mails customisable/translatable via the usual E-mails module. The actual e-mail is triggered via the bp_send_email function, which takes 3 parameters: The situation name, the user id and an array of so called ‘tokens’.
The problem
We cannot send 7000 e-mails at once. The browser doesn’t like the duration of this call, the web server doesn’t like this huge task at once and there’s a big chance that the SMTP server will reject messages at some point as well. So, we have to do it in small parts.
The solution
Instead of running the bp_send_email function directly, let’s collect the arguments that the function uses. Whenever a new post is created, collect all (7000) arguments and save them somewhere. Since i’m working in WordPress, I thought it’d be nice to do it the WordPress way So I created a new non-public post type called queued-email for this.
Saving the data
When all arguments are collected, a new post is created and the collection is saved as metadata, since metadata fields in the database allow us to save 4GB of data there is no issue there.
Processing the data
So, the post is submitted and the data is saved. The user can continue with their day and the server can start it’s task.
I’ve added a cron schedule using the cron_schedules filter that runs every 5 minutes. The function that runs when the cron is triggered checks if there’s any published posts in the queued-email post type. If there is we load the collection and get the first 50 rows from the array. We now have the arguments that were saved before and can populate and run the bp_send_email with it.
When that’s done, we update the collection of metadata – minus the first 50 rows – and update the log metadata to see what was processed when.
This process repeats itself every 5 minutes, so a collection of 7000 e-mails take about 11 hours to process.
When the queue is finished? It deletes the post and moves on. 🙂
Leave a Reply