We use Kibana at work (which is a nice UI to sit on top of logstash – I think logstash has adopted Kibana now that I check up on it – which also makes use of elasticsearch) as our visual tool to dig into our system logs from instances we host on AWS. To my dismay I received the alert this morning that we were running out of disk space on our dedicated logging box (it aggregates rsyslog data, with our other production boxes pumping information through to logstash). Mind you, it has been running for close to a year without missing a beat so it was time for some maintenance.
Unfortunately I hadn’t setup a trimming cron job to deal with the eventuality of the logs blowing out on disk space so there was very little disk space left when I finally decided to go take a look. Continue reading “Resolving elasticsearch exception for Parse Failure [No mapping found for [@timestamp] in order to sort on]”