Hi Fleet team, it turns out recent we see there are huge number of records in our redis like: 48580) "error:{kmRy0XDzl7EQceAvVnCBwF6mFBMCLLSIKNPhR0ZAIRM=}:count" 48581) "error:{9T_NdgJv99PmUmDBPbTjEdkWPlr-H5LQtQAhWpGyNLA=}:json" 48582) "error:{xhlUdro6LedPJI4v8JqF9YKWPO7oakU3Ek-t3YHnM6U=}:json" 48583) "error:{DflsXz_xoJAknzh4lRLhoJXeV5ILrnaBnzoyo2LAMZw=}:count" 48584) "error:{PQkef_LQQAvBS0IWBnTdhl8ATm7xABDn5w1YNYFheJA=}:json" 48585) "error:{caJ_auOBSopsh8hH3D1FvFZyL24c8mCbuVIXQFpJVv8=}:count" 48586) "error:{a8A4nboNO7wbvD-lrFuhoFhCex4qeQeWKE_SdnBzUiQ=}:count" 48587) "error:{CJMf9a5X_7JkzFggd5aekkzV5yIB_8lniK_Pe1s6uiE=}:count" 48588) "error:{dJA1EXjtT6xdnwJi1bw2Bq2YXOWbwfYHtGBct-OuKKw=}:json" 48589) "error:{pOS1nouVJjLzGwkdZJeHBtlGYQ3yvlJS9zaw2z9u9tU=}:count" 48590) "error:{1TnX3ZLdZgInAZNHLWwX1PYIrBjYEJ8KlvAz8YAmvjs=}:json" 48591) "error:{WncV4rmGDuv5vuwtIq67BkwmsftF3ijHr5JzfwFOQZM=}:json" 48592) "error:{neaB-r4PYDQJ-13M-XeC1eA0WgE--617iTFSHluyaXY=}:count" 48593) "error:{jQgVVidGEzm1pk0bcJYe6-Y88q_Z6KbbW-1xJ7NDszg=}:count" 48594) "error:{6pCYdd3kevQaxk_RpcGKAnVLWhX8zusyzb00ZAzCY2o=}:count" 48595) "error:{wrujIllWIycbLNoGmGN5HQZHqLg_LN-UoxUer1JVKdk=}:count" 48596) "error:{fqlkdqt4Eym0yl3unxzDfugd6EbagvzRes0jLnupsHQ=}:json" 48597) "error:{tr9kgV-L07RVeiUy2d0ooETkGBAh2CQ99AgXoWvcz70=}:count" 48598) "error:{00Uub-2xivsHznhGpMZxghoc7TWzhxeCf3E6HGqdieo=}:count" 48599) "error:{O8xy4QRYil1LR12s9vdiVpGNjT2TFcLrpKakMPHz0HI=}:json" 48600) "error:{jWFgJLnSlZ89N4id8rowjjul5VbmwiEKVHBbXWWWJ9I=}:json" which running our of resource of redis and cause traffic pile up from fleet side, could u help to explain what are these err coming from? and how to prevent it?

we temporarily store "unique instances" of errors in redis to aid debugging via

fleetctl

, seems like the logic that dedupes errors is not working as expected for one (or more) of the errors you're facing. can you either run

fleetctl debug errors

or try to get the value of one of the

{...}:json

keys so we can pinpoint the culprit? with that we should be able to fix easily