Anand Sudhanaboina

Distributed Log Search Using GNU Parallel

GNU parallel is a shell tool for executing jobs in parallel using one or more computers. If you have a set of servers to ssh into and run a command in parallel, this tool will help you.

Assuming you have an architecture where several cloud instances are behind LB, but doesn’t have a centralized logging sync (Logging into a centralized server or service, like Splunk):

architecture

If you need to search the log files across all the servers with one command, GNU Parallel comes very handy saving a lot of time. Here’s how it works:

  1. Install GNU Parallel. (Below command for Ubuntu)

    sudo apt-get install parallel

  2. Run the tool:

    echo “command” | parallel –onall –slf servers.txt

A few other way to run the command:

  • echo “fgrep -Rl /var/log/” | parallel –onall –slf servers.txt
  • echo “grep ~/log.txt” | parallel –onall –slf servers.txt

Comments