Recently I happened to come across an interesting problem. I had put up ELK stack for monitoring services such as system health(cpu, logged in users, disk space etc) and logs monitoring.

After a server upgrade, I had to restart it, and after restart, logstash wasn't able to read the logs. There was some error coming like:

[2019-06-20T12:23:00,624][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})

To solve it, I had to reset read_only_allow_delete in elastic search. In order to do so, we need to put forward a curl request like below:

curl -X PUT http://localhost:9200/*/_settings -d "{ \"index.blocks.read_only_allow_delete\": null }" -H "Content-Type: application/json"

And voila, that was it! Logstasher was up and running again.
I still haven't been able to figure out what went wrong in the first place, but now it's working atleast. :D