Extract email addresses from a large JSON file

Some requests to API endpoints may generate a very large output in JSON format.

As an example, the crAPI vulnerable web app that I have been practicing with lately has an endpoint that lists the details of all recent posts in the community section of the app. These details happen to include the e-mail addresses of the registered users that have posted in the forum.

Now suppose you want to extract these e-mail addresses to test your password bruteforcing skills, without having to read through all the JSON syntax and potentially miss out some data.

Here’s a quick, dirty and very effective way to speed things up.

Export a JSON file

If you are testing the API in Postman and got a large JSON output as a response to one of the requests you sent, go to the Save Response menu at the top right of the response section and choose Save to a file.

You will obtain a file called response.json.

If you are using another tool to probe your target API, you can just cut and paste the JSON output into a text file.

Use a regex to search the file

Now cd into the directory where you saved the file and run:
cat response.json | grep -oe "[a-zA-Z0-9._]\+@[a-zA-Z]\+.[a-zA-Z]\+"

The regex we are using will allow the grep command to extract from the file all the strings in an e-mail address format.

Get a clean list

You will see in the output that some addresses may appear several times. To get a clean list without duplicates,  just pipe the output into a sort -u command:
cat response.json | grep -oe "[a-zA-Z0-9._]\+@[a-zA-Z]\+.[a-zA-Z]\+" | sort -u

Voila! You have a list of addresses ready for a bruteforce attempt.

This little trick is part of the APIsec University lesson on API authentication attacks. Check out the video here.

Also, you will notice in the video that Corey Ball is using this slightly different command:
grep -oe "[a-zA-Z0-9._]\+@[a-zA-Z]\+.[a-zA-Z]\+" response.json

This syntax works just as well. Enjoy!