This is a bit of an exploratory question, and I’d love any thoughts you have. I’m posting early in my research because I think this might be interesting to others, and I suspect I may have an XY problem.

Goal

Something a bit like ExVCR, but running in production and normally just proxying the requests.

Background

My team runs a lot of data integrations, where we work with often janky and awkward external datasources. They are important requests, but low volume and raw response time is not a primary concern. The big problem: they frequently break and it’s a pain to debug them.

We’re exploring the idea of a man-in-the-middle proxy, that will simply receive, log, and forward events to our API most of them time.

But when we need to, we can turn on interception for some set of urls. Maybe to respond with a previously saved response, or a modified one. Or to simply direct some set of requests to a different host for dev/debugging.

Here’s a real example:

• We receive POST requests from an oracle service for a kind of SSO
• We process that data and return a redirect url (in XML) to that request

One day, that request starts failing for some of our customers. We later find out that it was because oracle changed how they parsed XML and now needed URLs inside the XML to be URLescaped (not previously the case).

It took a few days of back and forth to solve this, and really what we wanted was the ability to quickly see and mess with requests that were in flight.

We would do this in a staging/preview environment, rarely on production, but there are use cases when we might want to do it in production. For instance, we have a partner who cannot connect to our preview env, they only support a single endpoint (which is prod).

One approach would be something like nginx, with a configurable “forward” address depending on some matchers (e.g. forward some requests to this temporary URL for debugging), but really we’d like some more control than this.

So I’m looking for pointers, and I’m hoping that there is something obvious like a way to cache a plug conn, and then pipe that back into req with some modifications…

for any advice