Export Mailgun logs using the Mailgun API

Sometimes there’s just no other way. You need to gather information about e-mails that were sent, but your application just doesn’t have the logging available. In this case, Mailgun does have the information stored that we need, but how do we extract it?

Using the Mailgun API

Mailgun doesn’t allow us to export logs/events using the UI, so we’ll have to do it via API. To get to know the Mailgun API, i’m using Postman the Mailgun API docs.

The events path

([eu.] is for european customers, {domain} is your Mailgun domain, usually something like mail.example.com)

https://api.[eu.]mailgun.net/v3/{domain}/events

Authenticating with the API

Authentication is fairly easy, we set it to basic auth, set the username as api and use the private API key as password.

When firing a GET request, we’ll see the first logs appear.

Filtering the events

In my case, I want to see logs for a specific subject line, to do so, update the request to:

https://api.[eu.]mailgun.net/v3/{domain}/events?subject={Enter your subject here}

Retrieving logs for e-mails that are delivered

By default, Mailgun logs 2 entries per e-mail, “accepted”, and “delivered”. We only want to retrieve logs for the e-mails that were delivered.

https://api.[eu.]mailgun.net/v3/{domain}/events?subject={Enter your subject here}&event=delivered

Done! Oh wait..

We figured out the basics, but, since Mailgun doesn’t return all items (it’d be huge responses) we’ll have to follow the paging provided by the response. You can write this in any language you want. In the example below we’ll use PHP.

Extracting Mailgun logs/events in PHP

Note: You can also use the Mailgun PHP SDK

In this example we’ll stick to ‘manual requests’. In that way the example can be followed for any language. I’ll also make use of vlucas’ phpdotenv so I can store environment variables in a separate file, and publish this example to Github. You can find the Extracting Mailgun logs/events in PHP repository here.

After a composer init, composer require vlucas/phpdotenv, let’s add the .env file:

MAILGUN_DOMAIN=mail.example.com
MAILGUN_API=api.eu.mailgun.net
MAILGUN_API_KEY=PRIVATE_API_KEY
SUBJECT=SUBJECT
MAILGUN_EVENT=delivered

And add some test code:


require 'vendor/autoload.php';

$dotenv = Dotenv\Dotenv::createImmutable(__DIR__);
$dotenv->load();

var_dump($_ENV);

We can then run the command from the command line:

php extracting-events-from-mailgun.php

If all is well, you’ll see the output from your .env file. So, let’s get going!

The first request

$response_json = file_get_contents("https://{$_ENV['MAILGUN_API']}/v3/{$_ENV['MAILGUN_DOMAIN']}/events?subject={$_ENV['SUBJECT']}&event={$_ENV['MAILGUN_EVENT']}");
var_dump($response);

Spaces and other characters in the subject

$subject = urlencode($_ENV['SUBJECT']);

$response_json = file_get_contents("https://{$_ENV['MAILGUN_API']}/v3/{$_ENV['MAILGUN_DOMAIN']}/events?subject={$subject}&event={$_ENV['MAILGUN_EVENT']}");
var_dump($response);

Works! But.. no authentication..

Let the internet tell us how to set authentication with a file_get_contents call. Note: Using the Mailgun PHP SDK would make this a bit easier.

This Stackoverflow topic gives us the solution.

Let’s integrate this into our code:

$auth = base64_encode("api:{$_ENV['MAILGUN_API_KEY']}");
$context = stream_context_create(
    [
    "http" => [
        "header" => "Authorization: Basic $auth"
    ]
    ]
);
$response_json = file_get_contents("https://{$_ENV['MAILGUN_API']}/v3/{$_ENV['MAILGUN_DOMAIN']}/events?subject={$subject}&event={$_ENV['MAILGUN_EVENT']}", false, $context);

And run php extracting-events-from-mailgun.php again.

With a valid response, we can now loop through the pages, until there’s no items left for us.

Processing the items and looping over the pages

In order to know what we’re working with, let’s set some variables:

$response = json_decode($response_json);

$items = $response->items;
$test_item = $response->items[0];
$next_page = $response->paging->next;

var_dump($test_item);

This allows us to learn about what we’re dealing with. Next up is deciding what we need to save from the item and how we’re going to process the next pages.

Let’s start with a function that fires a request to a specific URL and returns the items and next page.

At this stage, it also makes sense to convert our code to a class, since we want to re-use some variables like $context.

Sample, not-useable, request function:

function request($url)
{
    $response_json = file_get_contents($url, false, $context);

    $response = json_decode($response_json);

    $items = $response->items;
    $next_page = $response->paging->next;    
}

Classifying the code

From this point on, I’ll link to Github commits instead of adding the code blocks here. The first version of the class is found here: https://github.com/mklasen/extracting-events-from-mailgun/commit/3af4552e6c4f20a75bb49d3c752344784c9951dc

Looping over all results/pages

First, we want to check if there’s any items left, because if there are not, it doesn’t make sense to continue reading next pages. Relevant commit: https://github.com/mklasen/extracting-events-from-mailgun/commit/618d7f46574c9517ee6c365261467e9dae0e9a91

Reading next pages

Before request all data, let’s do some testing. I’ve added an item count and set it 500. This allows us to process until 500 items and then stop the script. A perfect way to get things running smoothly before processing all data. I’ve also added ID checking, to make sure we’re following the pages correctly.

Relevant commit: https://github.com/mklasen/extracting-events-from-mailgun/commit/0b32a4f4af2f0f56f3d0905de1f3241f03638f9f

Now, I write this script to get insight on which e-mail addresses received multiple e-mails.

Processing the data to get the results

Let’s process the items, an run a report after. Relevant commit: https://github.com/mklasen/extracting-events-from-mailgun/commit/3f610687154e8a2c989224b68ed9fc0cee04eef7

Adding sample data to the duplicate results into valuable data.

Let’s test this before actually processing data from the API. Relevant commit: https://github.com/mklasen/extracting-events-from-mailgun/commit/8eb91f7aa619d156216f8cb69653c0be33c26597

Result:

~ php extracting-events-from-mailgun.php
Addresses: 500
Duplicates: 8
Array
(
    [test@test.test] => 5
    [test2@test.test] => 2
)

Any limits for the Mailgun API?

As Mailgun doesn’t seem to enforce any limit on API requests per second/minute, let’s skip this and see if we run into any issues.

The final run

Let’s remove the test data, remove the 500 item limit and move the result method to the moment where a request does not return any items anymore. Let’s also add some progress reports so we know what’s going on when running the script.

Relevant commits:

Results

Let’s take this one for a real spin. After processing all items, you should get a result like below:

Processing page *
Amount of items processed: *
Duplicates count: 0
Processing page *
Amount of items processed: *
Duplicates count: 0
Processing page *
Amount of items processed: *
Duplicates count: 0
Processing page *
Amount of items processed: *
Duplicates count: *
....
This goes on for a while
....
Processing page *
Amount of items processed: *
Duplicates count: *
Array
(
    [email@example.com] => 12
    [email@example1.com] => 2
    [email@example2.com] => 5
    [email@example3.com] => 5
)

And there we go, a full report of the data we needed!

Happy developing!

Export Mailgun logs using the Mailgun API

Using the Mailgun API

Extracting Mailgun logs/events in PHP

Processing the items and looping over the pages

Results

About Marinus Klasen

Get in touch

Have you read?

Using the Mailgun API

Extracting Mailgun logs/events in PHP

Processing the items and looping over the pages

Results

About Marinus Klasen

Leave a Reply Cancel reply

Get in touch

Have you read?