How to Parse JSON in Python

Introduction to JSON in Python

Before we dive into parsing JSON in Python, let's understand what JSON is and how to use it in Python.

What is JSON?

JSON, short for JavaScript Object Notation, is a lightweight data interchange format. It is easy for humans to read and write, and easy for machines to parse and generate. This makes it one of the most popular data formats. In particular, JSON has become the "language of the web" as it is widely used to transmit data between servers and web applications via APIs.

Below is an example of JSON:

{
"name": "Maria Smith",
"age": 32,
"isMarried": true,
"hobbies": ["reading", "jogging"],
"address": {
"street": "123 Main St",
"city": "San Francisco",
"state": "CA",
"zip": "12345"
},
"phoneNumbers": [
{
"type": "home",
"number": "555-555-1234"
},
{
"type": "work",
"number": "555-555-5678"
}
],
"notes": null
}

As you can see, JSON consists of key-value pairs. Each key is a string, and each value can be a string, number, boolean, null, array, or object. Even though JSON resembles a JavaScript object, it can be used in any programming language, including Python.

How to Work with JSON in Python

Python supports JSON through the json module, which is part of the Python standard library. This means you don't need to install any additional library to work with JSON in Python. You can import json as follows:

import json

The built-in Python json library provides a full API for working with JSON. In particular, it has two key functions: loads and load. The loads function allows you to parse JSON data from a string. Note that despite its name appearing plural, the ending "s" stands for "string". Thus, it should be read as "load-s". On the other hand, the load function is designed for parsing JSON data from bytes.

With these two methods, json allows you to convert JSON data into equivalent Python objects, such as dictionaries and lists, and vice versa. Additionally, the json module allows you to create custom encoders and decoders for working with specific data types.

Read on to learn how to use the json library to parse JSON data in Python!

Parsing JSON Data with Python

Let's look at a few real-world examples and learn how to parse JSON data from various sources into different Python data structures.

Converting a JSON String to a Python Dictionary

Suppose you have some JSON data stored in a string and you want to convert it to a Python dictionary. Here's what the data looks like in JSON format:

{
"name": "iPear 23",
"colors": ["black", "white", "red", "blue"],
"price": 999.99,
"inStock": true
}

And here is its string representation in Python:

smartphone_json = '{"name": "iPear 23", "colors": ["black", "white", "red", "blue"], "price": 999.99, "inStock": true}'

For storing long multi-line JSON strings, you should use the Python triple-quote convention.

You can verify that smartphone contains a valid Python string with the following line:

print(type(smartphone))

This will output:

<class 'str'>

str stands for "string" and means that the smartphone variable is of type text sequence.

Parse the JSON string contained in smartphone into a Python dictionary using the json.loads() method as follows:

import json

# JSON string
smartphone_json = '{"name": "iPear 23", "colors": ["black", "white", "red", "blue"], "price": 999.99, "inStock": true}'
# from JSON string to Python dict
smartphone_dict = json.loads(smartphone_json)

# verify the type of the resulting variable
print(type(smartphone_dict)) # dict

If you run this snippet, you'll get:

<class 'dict'>

Fantastic! Now smart_dict contains a valid Python dictionary!

Thus, to convert a JSON string to a Python dictionary, simply pass a valid JSON string to json.loads().

Now you can access the fields of the resulting dictionary as usual:

product = smartphone_dict['product'] # smartphone
priced = smartphone['price'] # 999.99
colors = smartphone['colors'] # ['black', 'white', 'red', 'blue']

Keep in mind that the json.loads() function does not always return a dictionary. Specifically, the returned data type depends on the input string. For example, if the JSON string contains a flat value, it will be converted to the corresponding Python primitive:

import json

json_string = '15.5'
float_var = json.loads(json_string)

print(type(float_var)) # <class 'float'>

Similarly, a JSON string containing an array will become a Python list:

import json

json_string = '[1, 2, 3]'
list_var = json.loads(json_string)
print(json_string) # <class 'list'>

Take a look at the conversion table below to see how JSON values are converted to Python data using json:

JSON Value	Python Data
string	str
number (integer)	int
number (real)	float
true	True
false	False
null	None
array	list
object	dict

Converting an API JSON Response to a Python Dictionary

Suppose you need to interact with an API and convert its JSON response to a Python dictionary. In the example below, we'll call the following API endpoint from the {JSON} Placeholder project to get some fake JSON data:

https://jsonplaceholder.typicode.com/todos/1

This RESTful API returns the JSON response shown below:

{
"userId": 1,
"id": 1,
"title": "delectus aut autem",
"completed": false
}

You can call this API using the urllib module from the standard library and convert the resulting JSON to a Python dictionary as follows:

import urllib.request
import json

url = "https://jsonplaceholder.typicode.com/todos/1"

with urllib.request.urlopen(url) as response:
body_json = response.read()

body_dict = json.loads(body_json)
user_id = body_dict['userId'] # 1

urllib.request.urlopen() makes the API call and returns an HTTPResponse object. Its read() method is used to get the response body body_json, which contains the API response as a JSON string. Finally, this string can be parsed into a Python dictionary using json.loads() as described above.

Similarly, you can achieve the same result using requests:

import requests
import json

url = "https://jsonplaceholder.typicode.com/todos/1"
response = requests.get(url)

body_dict = response.json()
user_id = body_dict['userId'] # 1

Note that the .json() method automatically converts the response object containing JSON data into the corresponding Python data structure.

Great! Now you know how to parse an API JSON response in Python using urllib and requests.

Loading a JSON File into a Python Dictionary

Suppose you have some JSON data stored in a file named smartphone.json, as shown below:

{
"name": "iPear 23",
"colors": ["black", "white", "red", "blue"],
"price": 999.99,
"inStock": true,
"dimensions": {
"width": 2.82,
"height": 5.78,
"depth": 0.30
},
"features": [
"5G",
"HD display",
"Dual camera"
]
}

Your goal is to read the JSON file and load it into a Python dictionary. You can achieve this with the snippet below:

import json

with open('smartphone.json') as file:
smartphone_dict = json.load(file)

print(type(smartphone_dict)) # <class 'dict'>
features = smartphone_dict['features'] # ['5G', 'HD display', 'Dual camera']

The built-in open() function lets you load the file and obtain a corresponding file object. Then the json.read() method deserializes the text or binary file containing a JSON document into an equivalent Python object. In this case, smartphone.json becomes a Python dictionary.

Excellent, parsing a JSON file in Python takes just a few lines of code!

From JSON Data to a Custom Python Object

Now you need to parse some JSON data into a custom Python class. Here's what your custom Python class Smartphone looks like:

class Smartphone:
def __init__(self, name, colors, price, in_stock):
self.name = name
self.colors = colors
self.price = price
self.in_stock = in_stock

Here the task is to convert the following JSON string into a Smartphone instance:

{
"name": "iPear 23 Plus",
"colors": ["black", "white", "gold"],
"price": 1299.99,
"inStock": false
}

To solve this, you need to create a custom decoder. To do this, extend the JSONDecoder class and set the object_hook parameter in the __init__ method. Assign it the name of a class method containing the custom parsing logic. In this parsing method, you can use the values from the standard dictionary returned by json.read() to instantiate a Smartphone object.

Define the custom SmartphoneDecoder as shown below:

import json

class SmartphoneDecoder(json.JSONDecoder):
def __init__(self, object_hook=None, *args, **kwargs):
# set the custom object_hook method
super().__init__(object_hook=self.object_hook, *args, **kwargs)

# class method containing the
# custom parsing logic
def object_hook(self, json_dict):
new_smartphone = Smartphone(
json_dict.get('name'),
json_dict.get('colors'),
json_dict.get('price'),
json_dict.get('inStock'),
)

return new_smartphone

Note that to read dictionary values in the custom object_hook() method, you should use the get() method. This helps avoid KeyErrors if a key is missing from the dictionary; instead, None values will be returned.

Now you can pass the SmartphoneDecoder class to the cls parameter in json.loads() to convert the JSON string into a Smartphone object:

import json

# class Smartphone:
# ...

# class SmartphoneDecoder(json.JSONDecoder):
# ...

smartphone_json = '{"name": "iPear 23 Plus", "colors": ["black", "white", "gold"], "price": 1299.99, "inStock": false}'

smartphone = json.loads(smartphone_json, cls=SmartphoneDecoder)
print(type(smartphone)) # <class '__main__.Smartphone'>
name = smartphone.name # iPear 23 Plus

Similarly, you can use SmartphoneDecoder with json.load():

smartphone = json.load(smartphone_json_file, cls=SmartphoneDecoder)

Now you know how to parse JSON data into custom Python objects!

Converting Python Data to JSON

You can also go the other way and convert Python data structures and primitives to JSON. This is possible using the json.dump() and json.dumps() functions, which follow the conversion table below:

Python Data	JSON Value
str	string
int	number (integer)
float	number (real)
True	true
False	false
None	null
list	array
dict	object
Null	None

json.dump() allows you to write a JSON string to a file, as in the following example:

import json

user_dict = {
"name": "John",
"surname": "Williams",
"age": 48,
"city": "New York"
}

# serializing the sample dictionary to a JSON file
with open("user.json", "w") as json_file:
json.dump(user_dict, json_file)

This snippet serializes the Python variable user_dict to a file named user.json.

Similarly, json.dumps() converts a Python variable to its equivalent JSON string:

import json

user_dict = {
"name": "John",
"surname": "Williams",
"age": 48,
"city": "New York"
}

user_json_string = json.dumps(user_dict)

print(user_json_string)

Note that you can also specify a custom encoder, but showing how to do that is beyond the scope of this article. For more details, refer to the official documentation.

Is the Standard json Module the Best Resource for Parsing JSON in Python?

As with data parsing, parsing JSON comes with challenges that cannot be overlooked. For example, in the case of invalid, incomplete, or nonstandard JSON, the Python json module will prove ineffective.

Additionally, you need to be cautious when parsing JSON data from untrusted sources. This is because a malicious JSON string could break the parser or consume a large amount of resources. This is just one of the issues a Python JSON parser must consider.

To address these issues, you can use custom logic. However, this can be time-consuming and lead to complex, unreliable code. Therefore, you should consider a commercial tool that simplifies JSON parsing, such as Web Scraper IDE.

Web Scraping IDE is designed specifically for developers and offers a wide range of features for parsing JSON content and more. This tool will help you save a lot of time and ensure the security of your JSON parsing process.

Conclusion

Python allows you to parse JSON data using the standard json module. It provides a powerful API for serializing and deserializing JSON content. In particular, it offers json.read() and json.reads() methods for working with JSON files and JSON strings, respectively. Here, through several real-world examples, you have seen how to use them to parse JSON data in Python. At the same time, you have also understood the limitations of this approach.