Hi Fleet team, it turns out recent we see there are huge number of records in our redis like: 48580) "error{kmRy0XDzl7EQceAvVnCBwF6mFBMCLLSIKNPhR0ZAIRM=}count" 48581) "error{9T NdgJv99PmUmDBPbTjEdkWPlr H5LQtQAhWpGyNLA=}json" 48582) "error{xhlUdro6LedPJI4v8JqF9YKWPO7oakU3Ek t3YHnM6U=}json" 48583) "error{DflsXz xoJAknzh4lRLhoJXeV5ILrnaBnzoyo2LAMZw=}count" 48584) "error{PQkef LQQAvBS0IWBnTdhl8ATm7xABDn5w1YNYFheJA=}json" 48585) "error{caJ auOBSopsh8hH3D1FvFZyL24c8mCbuVIXQFpJVv8=}count" 48586) "error{a8A4nboNO7wbvD lrFuhoFhCex4qeQeWKE SdnBzUiQ=}count" 48587) "error{CJMf9a5X 7JkzFggd5aekkzV5yIB 8lniK Pe1s6uiE=}count" 48588) "error{dJA1EXjtT6xdnwJi1bw2Bq2YXOWbwfYHtGBct OuKKw=}json" 48589) "error{pOS1nouVJjLzGwkdZJeHBtlGYQ3yvlJS9zaw2z9u9tU=}count" 48590) "error{1TnX3ZLdZgInAZNHLWwX1PYIrBjYEJ8KlvAz8YAmvjs=}json" 48591) "error{WncV4rmGDuv5vuwtIq67BkwmsftF3ijHr5JzfwFOQZM=}json" 48592) "error{neaB r4PYDQJ 13M XeC1eA0WgE 617iTFSHluyaXY=}count" 48593) "error{jQgVVidGEzm1pk0bcJYe6 Y88q Z6KbbW 1xJ7NDszg=}count" 48594) "error{6pCYdd3kevQaxk RpcGKAnVLWhX8zusyzb00ZAzCY2o=}count" 48595) "error{wrujIllWIycbLNoGmGN5HQZHqLg LN UoxUer1JVKdk=}count" 48596) "error{fqlkdqt4Eym0yl3unxzDfugd6EbagvzRes0jLnupsHQ=}json" 48597) "error{tr9kgV L07RVeiUy2d0ooETkGBAh2CQ99AgXoWvcz70=}count" 48598) "error{00Uub 2xivsHznhGpMZxghoc7TWzhxeCf3E6HGqdieo=}count" 48599) "error{O8xy4QRYil1LR12s9vdiVpGNjT2TFcLrpKakMPHz0HI=}json" 48600) "error{jWFgJLnSlZ89N4id8rowjjul5VbmwiEKVHBbXWWWJ9I=}json" which running our of resource of redis and cause traffic pile up from fleet side, could u help to explain what are these err coming from? and how to prevent it?

we temporarily store "unique instances" of errors in redis to aid debugging via

fleetctl

, seems like the logic that dedupes errors is not working as expected for one (or more) of the errors you're facing. can you either run

fleetctl debug errors

or try to get the value of one of the

{...}:json

keys so we can pinpoint the culprit? with that we should be able to fix easily