Webhooks Overhaul for Better User Experience

Published on 13 Apr 2021

Product Features

Webhooks have been a key component of the ExaVault platform for several years, and although we loved them, it was time for an update.

Redesigning Webhooks: The Goals

Webhooks are a crucial piece of any automation integration. Thanks to all the feedback we received from our customers, there were two key things holding our webhooks back;

They needed to more reliably be delivered in near real-time
Customers need more control over which paths events were being sent and where

Webhooks endpoint and filters interface.

Reliable & Event-Driven

The first item presented quite a challenge, requiring nearly an entire rewrite of the infrastructure powering webhooks without causing any disruption to the current webhooks when we released. Three different design iterations, and several long conversations with our product and engineering teams, eventually got us there in the end.

The first of two major changes that ended up making our goal possible was changing how webhooks fired. We originally queued webhooks based on actions being written to the activity log. This worked 95% of the time, but added a dependency on other processes and the possibility of falling behind. Now webhooks are event-driven and will fire immediately after an event occurs.

The second change concerns how our retries worked, which was the major culprit of delays, when they occurred. Reties now occur on a totally separate process. The initial webhook firing will never be delayed by a failed attempt.

Customizing Webhooks

Our second goal was simpler, but presented its own challenges along the way. Previously, we only allowed a single webhook on the account which required you to be notified anytime an action occurred anywhere on the account.

Wireframe stage of webhooks overhault project.

We’ve added three new ways to customize your webhooks — choose a specific path for the webhook, create multiple webhooks, and set different endpoint URLs. You no longer need to worry about sorting through notifications you don’t care about to get to what you actually need.

Create new webhook in your ExaVault account.

Polishing the User Experience

With our primary goals met, there was still more we wanted to add and more feedback from customers we wanted to address.

New Webhooks Response Format

A solid API suite is a webhooks best friend. We noticed that a number of customers would make a GET call right after receiving a webhook. It felt like we were missing important information in our webhook.

Our customers were again, invaluable here. We were able to talk to folks and learn more about how they were using our webhooks and what data they found valuable. Now in addition to our v1 response, which is still available, we have a v2 on offer that will return more data about the event, the files affected, and when the event happened. To be more in line with our API v2, we also restructured the response into a JSON friendly format.

{
  "attemptId": "12ngtj-1",
  "accountName": "omorgan",
  "eventTimestamp": "2021-06-18T13:57:48-05:00",
  "ipAddress": "45.18.120.77",
  "protocol": "web",
  "username": "omorgan",
  "event": "resources.download",
  "eventData": {
    "resources": {
      "id": 1894670971,
      "hash": "2a101d7ddb4f8c8e30e4a6782d14d5b4",
      "name": "Biology (1).jpg",
      "extension": "jpg",
      "type": "file",
      "createdBy": "omorgan",
      "uploadDate": "2020-12-09T12:40:59-06:00",
      "createdAt": "2020-12-09T12:40:59-06:00",
      "updatedAt": "2020-12-09T12:41:00-06:00",
      "accessedAt": "2021-06-18T13:57:48-05:00",
      "path": "/Book Photos/test/Biology (1).jpg",
      "size": 8305,
      "fileCount": null,
      "previewable": false
    },
    "transferStatus": "success"
  }
}

Webhook Log Improvements

The last major upgrade we tackled with our new and improved webhooks was updates to the log. Logs should always be the first place you look when having an issue with your webhooks. However, ours weren’t offering much in the way of useful data or tools to troubleshoot, so we had a lot of work to do.

Our log entry details was the first change. Before, you could only see your own response to the webhook. Now you can see the event details and the JSON structured object exactly as we sent it to you.

Thanks to an eleventh hour feedback call with a very sharp customer, we also added a “Resend Webhook” button to the details that refire the webhook exactly as it was sent the first time, including the Attempt ID. We had a long conversation internally on the best way to format the ID for a webhook being manually resent, and ended up using the same ID again, so you have an exact 1:1 match to test with – just with a different timestamp.

Text Filters & Sorting

The final change was intended to be a straightforward change to cap off the project. However, it ended up being the single most problematic piece of work. Yes, more problematic than rewriting our infrastructure.

The ask was simple enough; add text filters and sorting to the log table. First, we tried several different libraries to handle this for us – we’re generally not fans of writing code that someone else has already written – but nothing was working.

Turns out the issue was how we handle searching. We avoid loading the entire log at once since it could be several million entries. And we’d prefer not making our customers wait several minutes to see their logs. Because we search in real-time when changing pages, we couldn’t find a library that could handle search the way we needed to. After over a week of trying and failing to get our logs to work, we ended up building from scratch.

While our search methods are an item we intend to revisit, we did end up adding simple, but performant, filtering. This type of filtering allows you to sort and do text matches to quickly find any entries on the log table. What we intended as about two days of work ended up taking closer to two weeks. We’re happy with the end result.

Advanced filtering for webhook logs.

Next Steps for Webhooks

The rebuild of our webhooks is one of the largest projects we’ve tackled in the last couple of years. It’s a project we’ve been working on since the beginning of November. We want to thank everyone who took the time to talk to us, give us use cases, tell us what they liked, and most importantly, what they hated about our webhooks.

But our work is never done, and our ears are always open. If you think there’s more we could add or improve on, please let us know. It’s your feedback that keeps us building a bigger better product and we’re excited to continue expanding webhooks.

Looking for FTP hosting with event-driven webhooks?
Start your free trial of ExaVault today!