{"id":832,"date":"2021-12-15T09:07:11","date_gmt":"2021-12-15T09:07:11","guid":{"rendered":"https:\/\/mklasen.com\/?p=832"},"modified":"2022-01-09T13:26:28","modified_gmt":"2022-01-09T13:26:28","slug":"exporting-mailgun-logs","status":"publish","type":"post","link":"https:\/\/mklasen.com\/exporting-mailgun-logs\/","title":{"rendered":"Export Mailgun logs using the Mailgun API"},"content":{"rendered":"\n

Sometimes there’s just no other way. You need to gather information about e-mails that were sent, but your application just doesn’t have the logging available. In this case, Mailgun does have the information stored that we need, but how do we extract it?<\/p>\n\n\n\n

Using the Mailgun API<\/h2>\n\n\n\n

Mailgun doesn’t allow us to export logs\/events using the UI, so we’ll have to do it via API. To get to know the Mailgun API, i’m using Postman<\/a> the Mailgun API docs<\/a>.<\/p>\n\n\n\n

The events path<\/strong> <\/p>\n\n\n\n

([eu.] is for european customers, {domain} is your Mailgun domain, usually something like mail.example.com)<\/p>\n\n\n\n

https://api.[eu.]mailgun.net\/v3\/{domain}\/events<\/code><\/pre>\n\n\n\n
Authenticating with the API<\/strong><\/p>\n\n\n\n
Authentication is fairly easy, we set it to basic auth, set the username as api<\/strong> and use the private API key as password.<\/p>\n\n\n\n
When firing a GET request, we’ll see the first logs appear.<\/p>\n\n\n\n
Filtering the events<\/strong><\/p>\n\n\n\n
In my case, I want to see logs for a specific subject line, to do so, update the request to:<\/p>\n\n\n\n
https:\/\/api.[eu.]mailgun.net\/v3\/{domain}\/events?subject={Enter your subject here}<\/code><\/pre>\n\n\n\n
Retrieving logs for e-mails that are delivered<\/strong><\/p>\n\n\n\n
By default, Mailgun logs 2 entries per e-mail, “accepted”, and “delivered”. We only want to retrieve logs for the e-mails that were delivered.<\/p>\n\n\n\n
https:\/\/api.[eu.]mailgun.net\/v3\/{domain}\/events?subject={Enter your subject here}&event=delivered<\/code><\/pre>\n\n\n\n
Done! Oh wait..<\/strong><\/p>\n\n\n\n
We figured out the basics, but, since Mailgun doesn’t return all items (it’d be huge responses) we’ll have to follow the paging provided by the response. You can write this in any language you want. In the example below we’ll use PHP.<\/p>\n\n\n\n
Extracting Mailgun logs\/events in PHP<\/h2>\n\n\n\n
Note: You can also use the Mailgun PHP SDK<\/a><\/strong><\/em><\/p>\n\n\n\n
In this example we’ll stick to ‘manual requests’. In that way the example can be followed for any language. I’ll also make use of vlucas’ phpdotenv so I can store environment variables in a separate file, and publish this example to Github. You can find the Extracting Mailgun logs\/events in PHP repository here<\/a>.<\/p>\n\n\n\n
After a composer init<\/code>, composer require vlucas\/phpdotenv<\/code>, let’s add the .env file:<\/p>\n\n\n\n
MAILGUN_DOMAIN=mail.example.com\nMAILGUN_API=api.eu.mailgun.net\nMAILGUN_API_KEY=PRIVATE_API_KEY\nSUBJECT=SUBJECT\nMAILGUN_EVENT=delivered<\/code><\/pre>\n\n\n\n
And add some test code:<\/p>\n\n\n\n
\nrequire 'vendor\/autoload.php';\n\n$dotenv = Dotenv\\Dotenv::createImmutable(__DIR__);\n$dotenv->load();\n\nvar_dump($_ENV);<\/code><\/pre>\n\n\n\n
We can then run the command from the command line:<\/p>\n\n\n\n
php extracting-events-from-mailgun.php<\/code><\/pre>\n\n\n\n
If all is well, you’ll see the output from your .env file. So, let’s get going!<\/p>\n\n\n\n
The first request<\/strong><\/p>\n\n\n\n
$response_json = file_get_contents(\"https:\/\/{$_ENV['MAILGUN_API']}\/v3\/{$_ENV['MAILGUN_DOMAIN']}\/events?subject={$_ENV['SUBJECT']}&event={$_ENV['MAILGUN_EVENT']}\");\nvar_dump($response);<\/code><\/pre>\n\n\n\n
Spaces and other characters in the subject<\/strong><\/p>\n\n\n\n
$subject = urlencode($_ENV['SUBJECT']);\n\n$response_json = file_get_contents(\"https:\/\/{$_ENV['MAILGUN_API']}\/v3\/{$_ENV['MAILGUN_DOMAIN']}\/events?subject={$subject}&event={$_ENV['MAILGUN_EVENT']}\");\nvar_dump($response);<\/code><\/pre>\n\n\n\n
Works!<\/strong> But.. no authentication<\/strong>..<\/p>\n\n\n\n
Let the internet tell us how to set authentication with a file_get_contents call. Note: Using the Mailgun PHP SDK would make this a bit easier.<\/em><\/p>\n\n\n\n
This Stackoverflow topic gives us the solution<\/a>.<\/p>\n\n\n\n
Let’s integrate this into our code:<\/p>\n\n\n\n
$auth = base64_encode(\"api:{$_ENV['MAILGUN_API_KEY']}\");\n$context = stream_context_create(\n    [\n    \"http\" => [\n        \"header\" => \"Authorization: Basic $auth\"\n    ]\n    ]\n);\n$response_json = file_get_contents(\"https:\/\/{$_ENV['MAILGUN_API']}\/v3\/{$_ENV['MAILGUN_DOMAIN']}\/events?subject={$subject}&event={$_ENV['MAILGUN_EVENT']}\", false, $context);<\/code><\/pre>\n\n\n\n
And run php extracting-events-from-mailgun.php<\/code> again.<\/p>\n\n\n\n
With a valid response, we can now loop through the pages, until there’s no items left for us.<\/p>\n\n\n\n
Processing the items and looping over the pages<\/h2>\n\n\n\n
In order to know what we’re working with, let’s set some variables:<\/p>\n\n\n\n
$response = json_decode($response_json);\n\n$items = $response->items;\n$test_item = $response->items[0];\n$next_page = $response->paging->next;\n\nvar_dump($test_item);<\/code><\/pre>\n\n\n\n
This allows us to learn about what we’re dealing with. Next up is deciding what we need to save from the item and how we’re going to process the next pages.<\/p>\n\n\n\n
Let’s start with a function that fires a request to a specific URL and returns the items and next page.<\/strong><\/p>\n\n\n\n
At this stage, it also makes sense to convert our code to a class, since we want to re-use some variables like $context.<\/p>\n\n\n\n
Sample, not-useable,<\/strong> request function:<\/p>\n\n\n\n
function request($url)\n{\n    $response_json = file_get_contents($url, false, $context);\n\n    $response = json_decode($response_json);\n\n    $items = $response->items;\n    $next_page = $response->paging->next;    \n}<\/code><\/pre>\n\n\n\n
Classifying the code<\/strong><\/p>\n\n\n\n
From this point on, I’ll link to Github commits instead of adding the code blocks here. The first version of the class is found here: https:\/\/github.com\/mklasen\/extracting-events-from-mailgun\/commit\/3af4552e6c4f20a75bb49d3c752344784c9951dc<\/a><\/p>\n\n\n\n
Looping over all results\/pages<\/strong><\/p>\n\n\n\n
First, we want to check if there’s any items left, because if there are not, it doesn’t make sense to continue reading next pages. Relevant commit: https:\/\/github.com\/mklasen\/extracting-events-from-mailgun\/commit\/618d7f46574c9517ee6c365261467e9dae0e9a91<\/a><\/p>\n\n\n\n
Reading next pages<\/strong><\/p>\n\n\n\n
Before request all data, let’s do some testing. I’ve added an item count and set it 500. This allows us to process until 500 items and then stop the script. A perfect way to get things running smoothly before processing all data. I’ve also added ID checking, to make sure we’re following the pages correctly.<\/p>\n\n\n\n
Relevant commit: https:\/\/github.com\/mklasen\/extracting-events-from-mailgun\/commit\/0b32a4f4af2f0f56f3d0905de1f3241f03638f9f<\/a><\/p>\n\n\n\n
Now, I write this script to get insight on which e-mail addresses received multiple e-mails. <\/p>\n\n\n\n
Processing the data to get the results<\/strong><\/p>\n\n\n\n
Let’s process the items, an run a report after. Relevant commit: https:\/\/github.com\/mklasen\/extracting-events-from-mailgun\/commit\/3f610687154e8a2c989224b68ed9fc0cee04eef7<\/a><\/p>\n\n\n\n
Adding sample data to the duplicate results into valuable data.<\/strong><\/p>\n\n\n\n
Let’s test this before actually processing data from the API. Relevant commit: https:\/\/github.com\/mklasen\/extracting-events-from-mailgun\/commit\/8eb91f7aa619d156216f8cb69653c0be33c26597<\/a><\/p>\n\n\n\n
Result:<\/p>\n\n\n\n
~ php extracting-events-from-mailgun.php\nAddresses: 500\nDuplicates: 8\nArray\n(\n    [test@test.test] => 5\n    [test2@test.test] => 2\n)<\/code><\/pre>\n\n\n\n
Any limits for the Mailgun API?<\/strong><\/p>\n\n\n\n
As Mailgun doesn’t seem to enforce any limit on API requests per second\/minute, let’s skip this and see if we run into any issues. <\/p>\n\n\n\n
The final run<\/strong><\/p>\n\n\n\n
Let’s remove the test data, remove the 500 item limit and move the result method to the moment where a request does not return any items anymore. Let’s also add some progress reports so we know what’s going on when running the script. <\/p>\n\n\n\n
Relevant commits:<\/p>\n\n\n\n