Writing Data to Files on the Raspberry Pi

Single-board computers (SBCs) have opened many application areas that were otherwise impossible in the past. Developers have used SBCs for easy tasks, such as controlling LEDs, to complex tasks such as artificial intelligence in robotics. One application area is the use of SBCs for data-logging. This is where developers collect and record sensor data over time to examine the trends visually. In this way, the SBC measures the sensor data and saves it in a file for processing.

In this tutorial, we will use the Raspberry Pi (RPi) to learn how to create files and write sensor readings both on the RPi and on an external storage device. In combination with the DHT22 sensor, we will create a sample project to store the temperature and humidity values in a text file and a CSV file for offline processing.

Project Components

Here’s what we’ll need. Please note that I do not have a monitor connected to the RPi. Instead, I use an Ethernet cable and PUTTY to access the RPi terminal from my laptop.

• DHT22 temperature and humidity sensor
• Raspberry Pi Model B Rev 2
• USB Flash stick

Uses for writing data to files on the RPi

A file on the RPi provides a means for writing and storing data within the RPi environment. This can either be on the RPi’s memory card or an external USB stick connected to the RPi. On these files, users can write words, numbers, as well as any information which the RPi can understand and process. For the RPi to process these files, users need to install dedicated programs to enable reading from and writing to such files. There are many uses for writing data to files on a computing platform, not just on the RPi. As an example, we have mentioned earlier that we may need to save sensor data for further processing. Here is a list of what else we can do:

• Data logging
• RPi firmware
• Logging error and status messages
• Storing sensor commands

File Types

A file type or file format on the RPi is a conventional way of encoding the storage data in a digital medium. This is characterized by a period (.) and two or three characters identifying the file type. Some file types require dedicated programs to interact with them. For example, .wav files are used to store audio files only in a digital environment. On the other hand, other file types can contain text information but with different styles. For example, .cpp, .php, .html, .mat, .py, among others. They contain text, but they require different interpreters to work with them.

In the diagram below, we have categorized the most common RPi files into 6 groups. These groups are images, programs, archives, audio, documents, and video. They may contain a lot of file types, but here are the common ones:

How to Make a New File

There are two ways of creating and working with files on the RPi. The first method uses the RPi terminal to open a command-line text editor (CLI). The common CLI text editors that can be accessed through this method are GNU nano, Vim, vi, and EMACS. The second method uses graphical desktop editors or graphical user interface-based text editors. Examples of GUI text editors are Geany, Leafpad, Bluej, Adafruit, WebIDE, and Code::Blocks IDE. If you have a monitor connected to your RPI, you can create and edit files using GUI-based text editors. Alternatively, we can use CLI based text editors if we are accessing the RPi remotely. To create a new file using the RPi terminal, we do the following:

1. Open the terminal.
2. Navigate to the directory where you want to store the file cd /directory.
3. If you are using GNU Nano text editor, use the command sudo nano my_file.txt.
4. Here, we are creating a new text file called "my_file" and has a (.txt) extension. If we want a Python extension, we simply use sudo nano my_file.py. We can create files with different extensions this way.

On the other hand, using GUI-based text editors is pretty straightforward. To create a new file, double click on the Leafpad icon to launch the program. Click on “save as” and enter the file name and extension you want. Click “OK” when complete.

How to Write to a File Using Python

In certain circumstances, we may need to create and write to files using a high-level programming language. We can use other languages such as C/C++, JAVA, or C#, but in this tutorial, we will use Python to create a text file and write to it. To open a file in Python, we need to use the "open" function:

file_Object = open("file_Name","mode")

To write data to files, we declare the variable file_Obect to open a file with the name "file_Name". We use the built-in open function, and then Python opens the file with the permissions specified by the "mode" parameter. Different modes exist for the open function, and these are:

• ‘r’ for reading files.
• ‘w’ if we want to open the file for writing. If the specified file_Name is not available, Python creates a new file with the same name. However, if the file is available in that directory, Python opens it for writing.
• ‘a’ for appending data to files.
• ‘w+’ for reading and writing data to a file

The full implementation in this context will be f = open("myFile.txt","w+"). We will open a text file called myFile. If it is not available in that directory, we will create it. The file will have both read and write permissions, as defined by "w+".

Sample Project 1: Creating Text Files

In this first example project, we will write the humidity and temperature values from a DHT22 sensor to a text file. We will use the RPi’s system time to add a date and time stamp for each sensor read.

Components

• Raspberry Pi
• DHT22 Humidity and Temperature sensor
• 10K resistor
• Breadboard
• Jumper cables

Code

import Adafruit_DHT
from datetime import datetime
DHT_SENSOR = Adafruit_DHT.DHT22
DHT_PIN = 4 humidity, temperature = Adafruit_DHT.read_retry(DHT_SENSOR, DHT_PIN) if humidity is not None and temperature is not None: file = open("log.txt","a") file.write("Temp={0:0.1f}*C Humidity={1:0.1f}%".format(temperature, humidity))
else: file = open("log.txt","a") file.write("NAN ") file.write(datetime.today().strftime('%Y-%m-%d %H:%M:%S')+"\n")
file.close()

Code Description

The code in the section above reads the temperature and humidity values from a DHT22 sensor. The code is configured to use the Adafruit DHT22 sensor library, naming pin 4 as the input pin for reading the values. We import the datetime module because this will enable us to extract the date and time parameters from the data logging system.

Next, we use the If-else loop to check if we have a signal from the sensor. If we cannot connect to the sensor (for various reasons), we write the value "NAN" and the corresponding date and time stamp. This means that we failed to record the sensor readings at that time, which can be used to record sensor downtime.

Once the sensor is connected, we use the attribute "a" to append the sensor readings to a new line of a text file called "log.txt". The line file.write(datetime.today().strftime('%Y-%m-%d %H:%M:%S')+"\n") is used for extracting the date and time values. Lastly, we close the file with file.close(). The result is shown below.

How to Format the Data in the Text File

Plain text is one of the most efficient ways to store text. In most circumstances, we may need to enhance the visual appearance of our text. Additional characters such as headers, alignment, or indentation are often added to our original text for formatting the data. To do that, we use escape sequences in our code. These are characters that do not represent themselves when used inside string literals in our Python code. Here is a list of the most common escape characters:

• "\t" horizontal tab
• "\b" backspace
• "\n" new line
• "\r" carriage return
• "\\" backslash character

We can also use conversion specifiers to format the data in our text files.

• "%d" prints an integer as a signed decimal number
• "%u" unsigned integer
• "%e" lower case floating-point number
• "%E" upper case floating-point number
• "%f" prints number as a floating-point

In our code for writing temperature and humidity values to a text file, we used "Temp={0:0.1f}*C Humidity={1:0.1f}%".format(temperature, humidity). This formats the temperature and humidity values to floating-point numbers with one decimal place. You can also replace “f” with “e” to change the output of the values or add additional decimal places as you please.

How to Write Sensor Data to a CSV File with Python

Writing to a CSV file with Python is no different from writing to a text file. What we only need to change is the file extension. In this case, we use a .csv extension. CSV files use commas to specify the boundaries of the data in our file. We write plain text into the files, and we can use any spreadsheet application to interact with the CSV files. Here is an example of writing data to a CSV file.

import Adafruit_DHT
from datetime import datetime
DHT_SENSOR = Adafruit_DHT.DHT22
DHT_PIN = 4 humidity, temperature = Adafruit_DHT.read_retry(DHT_SENSOR, DHT_PIN) if humidity is not None and temperature is not None: file = open("log.csv","a") file.write("Temp={0:0.1f}*C Humidity={1:0.1f}%".format(temperature, humidity) + ",") else: file = open("log.csv","a") file.write("NAN " + ",") file.write(datetime.today().strftime('%Y-%m-%d' + "," '%H:%M:%S')+"\n")
file.close()

Most remains unchanged from the previous code except for the following:

1. Creating a .csv file extension.
2. Adding the "," escape character between the sections we want to separate, and we get the following output:

Sample Project 2: Creating CSV files

For our second project, we are going to write the temperature and humidity values from a DHT22 sensor to a CSV file. We are also going to add a date and timestamp for each sensor read.

How to Make Columns

At this point, we need to see how we can read from a CSV file and make modifications to the text. We can use any CSV file we have, but for conformity, we are going to use the same file that we have been using in this tutorial. Here is the code for creating and writing sensor readings to a standard CSV file.

import Adafruit_DHT
from datetime import datetime
DHT_SENSOR = Adafruit_DHT.DHT22
DHT_PIN = 4 humidity, temperature = Adafruit_DHT.read_retry(DHT_SENSOR, DHT_PIN) if humidity is not None and temperature is not None: file = open("log.csv","a") file.write("{0:0.2f}".format(temperature)+","+"{0:0.2f}".format(humidity)+",") else: file = open("log.csv","a") file.write("NAN "+",") file.write(datetime.today().strftime('%Y-%m-%d'+"," '%H:%M:%S')+"\n")
file.close()

The functionality of this code has not changed. We are still writing the timestamped temperature and humidity values to a CSV file. However, this time, we separated the temperature and humidity readings with a comma. We also removed the "Temp" and "Humidity" headings from the output string. Finally, our sensor readings now have two decimal places. Running the code above will produce the output below.

How to Label the Columns

To make a new column and label it, we are going to open our CSV file and add a new column to the left. This will be the first column of the output CSV file and it will record the log index (LOG ID) of each sensor reading. Here is the code:

import csv
num = 1 with open('log.csv','r') as file: with open('final.csv','w') as output: reader = csv.reader(file) output.write("Log ID"+","+"Temperature"+","+"Humidity"+","+"Date"+","+"Time"+"\n") writer = csv.writer(output, lineterminator='\n') for row in reader: writer.writerow([num]+row) num = num+1

Code Description

1. with open('log.csv','r') as file: We create a new Python file with the name readData.py. In this file, we write code to open our CSV file'log.csv', which contains the temperature and humidity readings. We use the open function and pass the filename and the read attribute'r'as parameters to this function.
2. Also, we create a new CSV file to store the results of our operation. As we have discussed before, we need to add a new column to an existing CSV file and label it. To do that, we create a new CSV file with the name 'final.csv', and we use the 'w' attribute to write to our new file. Here is the code: with open('final.csv','w') as output:
3. output.write("Log ID"+","+"Temperature"+","+"Humidity"+","+"Date"+","+"Time"+"\n") We use this code to create the first row, which contains the column labels.
4. num = 1, writer.writerow([num]+row), num=num+1 We create a variable with the name num. This variable will keep track of the number of rows in the CSV file, assigning each row a number that can be used to refer to the log index of the timestamped sensor readings.

How to Save the Text and CSV files

The data logging exercise’s final task is to store the values in a file for further processing. We may need to store the file either to a local drive or to an external storage device. If you are using a desktop version, simply plug in the USB stick and save the document as you would on your desktop computer or any computing platform. Here are the steps we need to take to save the files to an external storage device.

1. Plugin the USB stick to the RPi.
2. Use this command ls -l /dev/disk/by-uuid to check the ID which has been assigned to your USB stick. In my situation, I have been assigned AAD6-2A81. It is always next to ../../sda1, so you cannot miss it.

3. To create a mount point, we create a new directory by using the following command: sudo mkdir /media/usb. You can use any name to identify your new directory.
4. Next, we are going to give Pi the right to own this directory with this command: sudo chown -R pi:pi /media/usb.
5. Finally, we are going to use sudo mount /dev/sda1 /media/usb -o uid=pi,gid=pi to manually mount the drive.
6. After successfully mounting the drive, we can use the following command to check if our device is ready to use. We use the df -h command. Basically, this command displays the available space of file systems and we get the following:

And that’s it! We are now ready to copy or move files from our drive to the RPi or vice versa. To copy the log.csv file from the RPi to the USB stick, we can navigate to the directory of the file and use the following command: cp log.csv /media/usb.