Hopefully, if you have followed best practices, your cluster is running with three replicas and is not configured with any dangerous configuration options. Ceph, in most cases, should be able to recover from any failure.
However, in the scenario where a number of OSDs go offline, a number of PGs and/or objects may become unavailable. If you are unable to reintroduce these OSDs back into the cluster to allow Ceph to recover them gracefully, the data in those PGs is effectively lost. However, there is a possibility that the OSD is still readable to use the objectstore tool to recover the PGs contents. The process involves exporting the PGs from the failed OSDs and then importing the PGs back into the cluster. The objectstore tool requires that the OSDs' internal metadata is still in a consistent state, so full recovery is not guaranteed.
In...
