The Crystal Programming Language Forum

Filter sensitive data from `Log::Emitter`

Is there an easy way to set up a wide-scale (global, even?) filter for sensitive properties for Log::Emitter? Maybe a shard? Or does that need to be handled in application code?

Wondering because I often want to shovel data wholesale into Log.context to be picked up by OpenTelemetry and Honeybadger or even just for old-fashioned logging. Things like HTTP params or headers, but I don’t want sensitive data (passwords, OAuth tokens, cookies, etc) stored by these services.

Manually allowing or filtering keys is getting tedious, so I’m trying to come up with a centralized config for it where either Log::Emitter will skip over it or Log.context wouldn’t store it at all. For example, an ENV var with the names of keys to either allow or filter out, which I could put into a single Kubernetes ConfigMap.

I always thought of that being filtered by the backend directly. Having a FilteredBackend that would act as a decorator is a way to go, you would need to regenerate the Log::Entry on abstract def write(entry : Entry).

But this means that the sensitive data will be in memory.

If you want to avoid that, then the place for this change I think is in the Log::Metadata.setup method: crystal/metadata.cr at 5455e807b9314c82ab16dd8c557f04c3b04a34b5 · bcardiff/crystal · GitHub

There you could check against key names and change the value to a "**FILTERED**" value for example.

Yet metadata can be nested. If we want to filter not only the top level keys, then the change might require to change Log::Value.to_metadata_value to receive the key of the value and apply the filtering logic there. crystal/metadata.cr at 5455e807b9314c82ab16dd8c557f04c3b04a34b5 · bcardiff/crystal · GitHub . There are 4 calls to that (private) method.

1 Like

Yeah we built a log backend that adds filtering for spider-gazelle

You can add additional log backends to a broadcast backend if you need logging going to multiple places too GitHub - place-labs/log-backend: Log backend in use across PlaceOS services

1 Like