А крашмапа не менялась как раз перед вылетом в инектив?

просто если его живого нет, то ceph -s показывает херню

оно и ща будет тупить, если PMTU выключить или поломать

Hi

ceph -s
 cluster:
   id:     3cf31a85-c84c-49b5-a1e7-82087700d31d
   health: HEALTH_WARN
           Reduced data availability: 4096 pgs inactive

 services:
   mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3
   mgr: support@ceph-mon3(active), standbys: support@ceph-mon1, support@ceph-mon2
   osd: 99 osds: 99 up, 99 in
   rgw: 2 daemons active

 data:
   pools:   14 pools, 6472 pgs
   objects: 24.66M objects, 66.2TiB
   usage:   179TiB used, 140TiB / 319TiB avail
   pgs:     63.288% pgs unknown
            4096 unknown
            2284 active+clean
            65   active+clean+snaptrim_wait
            18   active+clean+scrubbing+deep+repair
            9    active+clean+snaptrim

тогда ХЗ
ceph osd pool ls detail ещё покажи

Что вызвало такую ситуацию?

ceph osd pool ls detail
pool 37 'backup' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 480309 flags hashpspool min_write_recency_for_promote 1 stripe_width 0 application rbd
       removed_snaps [1~3819,3820~7571,ad92~38d,b120~399,b4ba~354,b80f~35c,bb6c~36b,bed8~3af,c288~39c,c625~3be,c9e4~393,cd78~1713,e48c~c4f4,1a981~a9,1aa2b~b9,1aae5~b4,1ab9a~ad,1ac48~b1,1acfa~a9,1ada4~9d,1ae42~a4,1aee7~ca,1afb2~d3,1b086~bf,1b146~c1,1b208~bf,1b2c8~24f4,1d7bd~1,1d7bf~7b5,1df75~6bc2,24b38~c99,257d2~2acb,2829e~13c,283db~d1,284ad~51,284ff~e3,285e3~d6,286ba~a1,2875c~4efd,2d65a~1421,2ea7c~3c18,32695~139,327cf~140,32910~13f,32a50~139,32b8a~4e08,37993~f,379a3~12c,37ad0~4,37ad5~14b,37c21~6,37c28~137,37d60~d,37d6e~127,37e96~9,37ea0~130,37fd1~c,37fde~2cc,382ab~414e,3c3fa~373d,3fb38~1,3fb3c~2,3fb3f~1,3fb41~387,3fec9~3d3,4029d~158,403f6~145,4053c~15c,40699~143,407dd~144,40922~3,40926~85,409ac~32,409df~2d,40a0d~11,40a1f~53,40a73~9d,40b11~d,40b1f~a0,40bc0~14f,40d10~a4,40db5~98,40e4e~35a,411a9~106,412b0~1,412b2~51,41304~16,4131b~a6,413c2~1f,413e2~25,41408~17,41420~11,41432~fe,41531~f,41541~13,41555~1,41557~1,41559~7,41561~19,4157b~f8,41674~b,41680~13,41694~7,4169c~d,416aa~1,416ac~11,416be~fc,417bb~13,417cf~b,417db~7,417e3~f,417f3~3,417f7~f,41807~fb,41903~15,41919~11,4192b~27,41953~445,41d99~e58,42bf2~2e2,42ed5~7,42edd~424,43302~59,4335c~d,4336a~e4,4344f~35,43485~23,434a9~113,435bd~a,435c8~138,43701~1,43703~3b,4373f~5,43745~ea,43830~11,43842~11b,4395e~23,43982~1e0,43b63~112,43c76~7eb,44462~80e,44c71~7f9,4546b~837,45ca3~22f,45ed3~111,45fe5~144,4612a~30,4615b~fe,4625a~15,46270~117,46388~17,463a0~100,464a1~14,464b6~3c,464f3~131,46625~80,466a6~1b,466c2~1,466c4~11b,467e0~15,467f6~10e,46905~19,4691f~119,46a39~19,46a53~de,46b32~6,46b39~50,46b8a~15,46ba0~38,46bd9~a7,46c81~17d,46dff~5ea,473ea~21,4740c~131,4753e~d9,47618~61,4767a~149,477c4~12a,478ef~128,47a18~136,47b4f~10a,47c5a~22,47c7d~139,47db7~9d,47e55~3,47e59~3a,47e94~6a,47eff~140,48040~12c,4816d~136,482a4~137,483dc~111,484ee~16,48505~222,48728~7f,487a8~11,487ba~39,487f4~4c5,48cba~1cd,48e88~e6,48f6f~1ee,4915e~181,492e0~ff,493e0~7,493e8~10e,494f7~d2,495ca~45,49610~46,49657~21,49679~17,49691~2b,496bd~64,49722~ff,49822~10f,49932~1,49934~1,49936~1,49938~1,4993a~1,4993c~1,4993e~5,49944~1,49947~2,4994a~1,4994c~1,4994e~1,49951~2,49954~1,49956~1,49958~1,4995a~1,4995c~1,4995e~1,49960~a2,49a03~1,49a05~4,49a0c~10,49a1d~5,49a23~5,49a29~1,49a2b~5,49a31~1,49a33~1,49a35~5,49a3b~3,49a3f~1,49a41~3,49a45~1,49a47~2,49a4a~3,49a4e~1,49a51~2,49a54~3,49a58~2,49a5b~1,49a5d~1,49a5f~2,49a63~2,49a66~1,49a68~1,49a6a~1,49a6c~1,49a6f~2,49a72~9,49a7c~a,49a87~4,49a8c~9,49a96~1,49a98~3,49a9c~2c,49ac9~3,49acd~e,49adc~1d,49afa~10,49b0b~5,49b11~1,49b13~e,49b22~5,49b28~9,49b32~1,49b34~1,49b36~7,49b3e~1,49b40~1,49b42~4,49b47~2,49b4a~3,49b4e~1,49b50~1,49b52~1,49b54~3,49b58~1,49b5a~1,49b5c~3,49b60~1,49b62~1,49b64~1,49b66~3,49b6a~1,49b6c~1,49b6e~1,49b70~1,49b72~1,49b74~1,49b76~2,49b79~2,49b7c~1,49b7e~1,49b80~7,49b89~c,49b96~8,49b9f~4,49ba4~3,49ba8~30,49bd9~7,49be1~22,49c04~7,49c0c~5,49c12~9,49c1c~a,49c27~1,49c29~5,49c2f~3,49c33~3,49c37~1,49c39~1,49c3b~3,49c3f~1,49c41~3,49c45~3,49c49~1,49c4b~1,49c4e~2,49c51~1,49c53~5,49c59~1,49c5b~1,49c5d~1,49c5f~1,49c61~1,49c63~1,49c65~1,49c67~1,49c69~1,49c6b~1,49c6d~1,49c6f~1,49c71~1,49c73~1,49c75~1,49c77~8,49c80~4,49c85~3,49c89~15,49c9f~1,49ca1~1,49ca3~1a,49cbe~23,49ce2~19,49cfc~b,49d08~1b,49d24~9,49d2e~a,49d39~5,49d3f~8,49d48~7,49d50~4,49d55~2,49d58~7,49d60~3,49d64~1,49d66~1,49d69~2,49d6c~7,49d74~1,49d76~1,49d78~1,49d7a~1,49d7c~1,49d7e~1,49d80~3,49d84~1,49d86~1,49d88~1,49d8a~1,49d8c~1,49d8e~1,49d90~1,49d92~1,49d94~1,49d96~1,49d98~1,49d9a~1,49d9c~1,49d9e~1,49da0~1,49da2~7,49daa~15,49dc0~3,49dc4~3,49dc8~5,49dce~2,49dd1~4f,49e21~d,49e2f~1,49e31~4,49e36~9,49e40~d,49e4e~3,49e53~2,49e56~7,49e5e~9,49e68~1,49e6b~4,49e70~7,49e78~6,49e7f~8,49e88~3,49e8c~1,49e8e~7,49e96~3,49e9a~1,49e9c~1,49e9e~1,49ea1~5,49ea7~4,49eac~1,49eae~1,49eb0~1,49eb2~1,49eb4~1,49eb6~3,49eba~1,49ebc~1,49ebe~1,49ec0~1,49ec2~1,49ec4~1,49ec6~1,49ec8~1,49eca~1,49ecc~1,49ece~1,49ed0~1,49ed2~1,49ed4~1

49ed6~1,49ed8~9,49ee2~8,49eeb~3,49eef~8,49ef8~12,49f0b~5,49f11~9,49f1b~46,49f62~7,49f6a~1,49f6c~b,49f78~1,49f7a~3,49f7e~9,49f88~3,49f8c~8,49f95~2,49f98~7,49fa1~6,49fa8~2,49fab~2,49fae~5,49fb5~2,49fb8~1,49fba~5,49fc0~1,49fc2~7,49fca~3,49fce~3,49fd2~1,49fd4~3,49fd8~1,49fda~3,49fde~1,49fe0~1,49fe2~1,49fe4~1,49fe6~1,49fe8~1,49fea~1,49fec~1,49fee~1,49ff0~1,49ff2~1,49ff4~1,49ff6~1,49ff8~3,49ffc~2,49fff~4,4a004~1,4a006~3,4a00a~5,4a010~7,4a018~9,4a022~1,4a024~5,4a02a~19,4a044~3c,4a081~3,4a085~1,4a087~a,4a092~1,4a094~2,4a098~9,4a0a2~1,4a0a4~7,4a0ac~2,4a0af~1,4a0b2~c,4a0bf~3,4a0c3~1,4a0c5~1,4a0c7~8,4a0d0~1,4a0d2~5,4a0d8~3,4a0dc~3,4a0e0~1,4a0e2~5,4a0e8~5,4a0ee~1,4a0f0~1,4a0f2~1,4a0f4~1,4a0f6~3,4a0fa~1,4a0fc~1,4a0fe~1,4a100~3,4a104~3,4a108~1,4a10a~1,4a10c~1,4a10e~1,4a110~1,4a112~1,4a114~1,4a116~1,4a118~1,4a11a~1,4a11c~1,4a11e~1,4a120~1,4a122~a,4a12d~1,4a130~2,4a133~8,4a13c~1,4a13e~1d,4a15c~50,4a1ad~4,4a1b2~1,4a1b4~c,4a1c1~d,4a1cf~1,4a1d1~3,4a1d5~5,4a1db~3,4a1e0~4,4a1e5~5,4a1ed~6,4a1f4~a,4a200~5,4a206~4,4a20b~1,4a20d~3,4a211~1,4a213~1,4a215~1,4a217~3,4a21b~1,4a21d~1,4a220~3,4a224~2,4a227~1,4a229~1,4a22b~1,4a22d~1,4a22f~1,4a231~3,4a235~2,4a238~2,4a23b~1,4a23d~1,4a240~2,4a243~1,4a245~1,4a247~1,4a249~1,4a24b~1,4a24d~1,4a24f~1,4a251~1,4a253~1,4a255~1,4a257~1,4a259~5,4a260~2,4a263~9,4a26d~13,4a281~5,4a287~3,4a28b~d,4a299~2,4a29c~12,4a2af~36,4a2e6~4,4a2eb~1,4a2ed~2,4a2f0~7,4a2f8~1,4a2fa~b,4a306~3,4a30b~4,4a310~6,4a317~4,4a31c~8,4a325~1,4a327~1,4a329~2,4a32c~4,4a331~7,4a339~2,4a33d~4,4a342~1,4a344~1,4a346~3,4a34a~1,4a34c~3,4a350~1,4a352~1,4a354~1,4a356~1,4a358~3,4a35c~1,4a35e~3,4a362~1,4a364~1,4a366~1,4a368~3,4a36c~1,4a36e~3,4a372~1,4a374~1,4a376~1,4a378~1,4a37a~1,4a37c~1,4a37e~1,4a380~1,4a382~1,4a385~2,4a388~1,4a38a~1,4a38c~1,4a38e~1,4a390~1,4a392~1,4a394~5,4a39a~2,4a39d~1,4a39f~4,4a3a4~7,4a3ad~2,4a3b0~3,4a3b4~1,4a3b6~3,4a3ba~1,4a3bc~1,4a3be~1,4a3c1~4,4a3c6~1,4a3c8~1,4a3ca~7,4a3d2~3,4a3d6~f,4a3e6~5,4a3ec~1,4a3ee~7,4a3f6~d,4a404~1,4a406~8,4a40f~18,4a428~3,4a42c~1,4a42f~1,4a431~3,4a435~5,4a43b~1,4a43d~1,4a43f~5,4a445~1,4a447~5,4a44e~2,4a451~6,4a458~4,4a45d~2,4a462~4,4a469~1,4a46c~5,4a472~7,4a47a~4,4a47f~5,4a485~1,4a487~3,4a48b~3,4a48f~1,4a492~2,4a495~1,4a497~1,4a499~1,4a49b~3,4a49f~2,4a4a2~2,4a4a5~1,4a4a7~1,4a4a9~1,4a4ab~3,4a4b0~2,4a4b3~1,4a4b6~2,4a4b9~1,4a4bb~1,4a4bd~1,4a4bf~1,4a4c1~1,4a4c3~1,4a4c5~1,4a4c7~1,4a4c9~1,4a4cb~1,4a4cd~1,4a4cf~1,4a4d1~1,4a4d3~1,4a4d5~1,4a4d8~1,4a4db~1,4a4dd~b,4a4e9~5,4a4f0~6,4a4f7~3,4a4fb~1,4a4fd~1,4a4ff~1,4a501~1,4a503~1,4a505~1,4a507~5,4a50d~9,4a518~4,4a51d~1,4a51f~3,4a523~1,4a525~21,4a547~3,4a54b~9,4a555~d,4a563~3,4a567~1,4a569~1,4a56b~1,4a56d~1,4a56f~3,4a573~5,4a579~1,4a57b~1,4a57d~3,4a581~3,4a585~1,4a587~3,4a58b~1,4a58e~2,4a591~1,4a593~1,4a595~3,4a59a~2,4a59e~2,4a5a1~2,4a5a6~4,4a5af~3,4a5b4~c,4a5c1~1,4a5c3~1,4a5c5~5,4a5cb~1,4a5cd~3,4a5d1~1,4a5d4~2,4a5d7~1,4a5d9~1,4a5db~1,4a5dd~1,4a5df~1,4a5e1~3,4a5e6~2,4a5e9~1,4a5eb~1,4a5ed~1,4a5ef~1,4a5f1~1,4a5f3~1,4a5f5~1,4a5f7~1,4a5f9~3,4a5fd~1,4a5ff~1,4a601~1,4a603~1,4a605~1,4a607~1,4a609~1,4a60b~3,4a60f~1,4a611~1,4a613~3,4a617~1,4a619~1,4a61b~1,4a61d~3,4a621~1,4a623~7,4a62b~1,4a630~1,4a632~8,4a63b~3,4a63f~1,4a641~1,4a643~1,4a645~3,4a649~1,4a64b~1,4a64d~1,4a64f~1,4a651~1,4a653~1,4a655~1,4a658~c,4a665~1,4a667~1,4a669~3,4a66d~5,4a673~1,4a676~2,4a679~3,4a67d~9,4a687~22,4a6aa~1,4a6ac~1,4a6ae~1,4a6b0~1,4a6b2~1,4a6b4~1,4a6b6~3,4a6ba~1,4a6bc~1,4a6be~1,4a6c0~5,4a6c6~1,4a6c8~1,4a6ca~1,4a6cc~1,4a6ce~1,4a6d0~2,4a6d3~2,4a6d6~1,4a6d8~1,4a6da~1,4a6dc~3,4a6e0~1,4a6e2~3,4a6e8~1,4a6ea~4,4a6f0~2,4a6f6~1,4a6f8~5,4a6fe~1,4a700~5,4a707~2,4a70a~5,4a710~1,4a712~3,4a716~1,4a718~1,4a71a~1,4a71c~1,4a71e~1,4a720~1,4a722~1,4a724~1,4a726~1,4a728~3,4a72d~2,4a730~1,4a732~1,4a734~1,4a736~1,4a739~2,4a73c~1,4a73e~1,4a740~1,4a742~1,4a744~1,4a746~1,4a748~1,4a74a~1,4a74c~1,4a74e~1,4a750~1,4a752~1,4a754~1,4a756~1,4a758~1,4a75a~1,4a75c~1,4a75e~1,4a760~3,4a765~1,4a768~1,4a76a~2,4a76d~2,4a770~9,4a77a~1,4a77c~1,4a77e~1,4a781~2,4a784~1,4a786~3,4a78b~2,4a78e~1,4a790~3,4a794~1,4a796~1,4a798~1,4a79a~1,4a79c~1,4a79e~1,4a7a0~1,4a7a2~6,4a7a9~1,4a7ab~1,4a7ae~4,4a7b3~3,4a7b7~3,4a7bc~4,4a7c1~1,4a7c3~1,4a7c5~17,4a7dd~b,4a7

9~1,4a7eb~1,4a7ed~1,4a7ef~1,4a7f1~1,4a7f3~1,4a7f5~1,4a7f7~1,4a7f9~1,4a7fb~1,4a7fd~1,4a7ff~1,4a801~1,4a805~3,4a809~1,4a80b~1,4a80d~1,4a80f~1,4a811~3,4a815~1,4a817~1,4a819~1,4a81b~7,4a82b~2,4a82e~2,4a831~3,4a835~5,4a83b~1,4a83d~1,4a83f~1,4a841~3,4a845~1,4a847~3,4a84b~1,4a84d~1,4a84f~3,4a853~1,4a855~1,4a857~1,4a859~1,4a85b~1,4a85d~1,4a85f~1,4a861~1,4a864~2,4a867~1,4a869~1,4a86c~2,4a86f~1,4a871~1,4a873~1,4a875~1,4a877~1,4a879~1,4a87b~1,4a87e~2,4a881~1,4a883~1,4a885~1,4a887~1,4a889~1,4a88b~1,4a88d~1,4a88f~1,4a891~1,4a893~1,4a895~1,4a897~1,4a899~1,4a89b~1,4a89d~1,4a89f~1,4a8a4~2,4a8a7~3,4a8ac~3,4a8b0~2,4a8b3~1,4a8b5~1,4a8b7~1,4a8b9~1,4a8bb~1,4a8bd~1,4a8bf~1,4a8c1~1,4a8c3~1,4a8c5~2,4a8c8~2,4a8cb~1,4a8cd~1,4a8cf~1,4a8d1~3,4a8d5~1,4a8d7~1,4a8da~2,4a8de~3,4a8e2~1,4a8e4~1,4a8e6~3,4a8ea~1,4a8ed~1,4a8f0~1,4a8f2~1,4a8f4~3,4a8f8~1,4a8fa~1,4a8fc~1,4a8fe~3,4a902~1b,4a91e~7,4a926~1,4a928~1,4a92a~1,4a92c~1,4a92e~1,4a930~1,4a932~1,4a934~1,4a936~1,4a938~1,4a93a~1,4a93c~1,4a93e~1,4a940~1,4a942~1,4a944~1,4a946~1,4a948~1,4a94a~1,4a94c~1,4a94e~1,4a950~1,4a952~1,4a954~1,4a956~1,4a958~1,4a95a~1,4a95d~2,4a961~2,4a965~1,4a969~1,4a96e~1,4a970~1,4a972~1,4a974~2,4a978~2,4a97b~6,4a982~1,4a984~1,4a986~1,4a988~1,4a98a~1,4a98c~1,4a98e~1,4a990~1,4a992~1,4a994~1,4a996~1,4a998~1,4a99a~1,4a99c~1,4a99e~1,4a9a0~1,4a9a2~1,4a9a4~1,4a9a6~1,4a9a8~1,4a9aa~1,4a9ac~1,4a9ae~1,4a9b0~1,4a9b2~1,4a9b4~1,4a9b6~1,4a9b8~1,4a9ba~1,4a9bc~1,4a9be~1,4a9c0~1,4a9c2~1,4a9c4~1,4a9c6~1,4a9c8~1,4a9cb~2,4a9ce~1,4a9d0~1,4a9d2~1,4a9d4~1,4a9d6~1,4a9d8~1,4a9da~1,4a9dc~1,4a9de~1,4a9e0~1,4a9e2~1,4a9e6~1,4a9e8~1,4a9eb~1,4a9f1~1,4a9f3~3,4a9f7~5,4a9fd~2,4aa00~1,4aa02~1,4aa04~1,4aa07~2,4aa0a~1,4aa0c~1,4aa0e~1,4aa10~1,4aa12~1,4aa14~1,4aa16~1,4aa18~1,4aa1a~1,4aa1c~1,4aa1e~1,4aa21~1,4aa24~2,4aa29~3,4aa2d~1,4aa2f~1,4aa31~1,4aa33~1,4aa35~1,4aa37~1,4aa39~1,4aa3b~1,4aa3d~1,4aa3f~1,4aa41~1,4aa43~1,4aa45~1,4aa47~1,4aa49~1,4aa4b~1,4aa4d~1,4aa4f~1,4aa51~1,4aa53~1,4aa55~1,4aa57~1,4aa59~1,4aa5b~1,4aa5d~1,4aa5f~1,4aa61~3,4aa65~1,4aa67~1,4aa69~1]
pool 38 'iscsi' replicated size 2 min_size 2 crush_rule 3 object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 430543 lfor 430542/430542 flags hashpspool min_write_recency_for_promote 1 stripe_width 0 application rbd
       removed_snaps [1~7]
pool 42 '.rgw.root' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341896 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 43 'default.rgw.control' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341896 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 44 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341896 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 45 'default.rgw.log' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341896 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 46 'eu-west-avz2.rgw.control' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341896 flags hashpspool stripe_width 0 application rgw
pool 47 'eu-west-avz2.rgw.meta' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341896 flags hashpspool stripe_width 0 application rgwpool 48 'eu-west-avz2.rgw.log' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341896 flags hashpspool stripe_width 0 application rgw
pool 49 'eu-west-avz2.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341896 flags hashpspool stripe_width 0 application rgw
pool 50 'eu-west-avz2.rgw.buckets.data' replicated size 2 min_size 2 crush_rule 3 object_hash rjenkins pg_num 128 pgp_num 128 last_change 478014 lfor 0/478011 flags hashpspool stripe_width 0 application rgw

pool 51 'eu-west-avz2.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341896 flags hashpspool stripe_width 0 application rgw
pool 64 'rbd-cold' replicated size 2 min_size 2 crush_rule 3 object_hash rjenkins pg_num 128 pgp_num 128 last_change 430132 lfor 430131/430131 flags hashpspool stripe_width 0 application rbd
       removed_snaps [1~3]
pool 66 'rbd-ssd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 4096 pgp_num 4096 last_change 480419 flags hashpspool stripe_width 0 application rbd

йопт. а можно пожалуйста всё это снести и на пастбин

кастую Ирека @irekfasikhov к человеку по ходу пришёл снаптрим!

=))

ну что я могу сказать, можно поздравить :))

судя по карте дофига снапшотов?

Жопа пришла да?

Да снапшотов очень много

слипуй снаптим и уменьшай количество снапов

ну вот эти 4096 unknown - это видимо пул rbd-ssd, а вот почему она unknown я так сходу хз