IBMi (AS400) fans only ' Detecting and Handling Non-Printable Characters in DB2 SQL Using LOCATE() and REPLACE() Functions

When working with DB2 SQL on the IBMi platform, one of the challenges we might encounter is dealing with non-printable characters in our data. These are characters that don't have a direct representation on a standard keyboard and can cause display programs to crash if not handled properly.

For this example, we'll be working with the CLANA00F file, which contains customer data. You can find details on how to create this file on my blog post here.

Non-printable characters can sneak into our records for various reasons, and to prevent issues, we can run an SQL query that uses the LOCATE() function. This function returns the position of the first occurrence of a substring within a string, allowing us to detect these problematic characters.

Suppose we have a customer name with a non-printable character, specifically the EBCDIC '0D' character, which corresponds to the "Carriage Return" (CR) control character in ASCII. To find this character in our records, we can use the following SQL query:

SELECT CLANN0, CLCCL0, CLCIB0, CLNOM0
FROM CLANA00F                        
WHERE LOCATE(x'0D', CLNOM0) > 0;

In this query, x'0D' represents the EBCDIC Carriage Return character. The LOCATE function searches for this character in the CLNOM0 field and returns its position. If the position is greater than 0, it means the character is present.

Additionally, another non-printable character that might cause issues is the EBCDIC '25' character, which corresponds to the Line Feed (LF) control character in ASCII. To find this character, we can use a similar SQL query:

SELECT CLANN0, CLCCL0, CLCIB0, CLNOM0
FROM CLANA00F                        
WHERE LOCATE(x'25', CLNOM0) > 0;

In this query, x'25' represents the EBCDIC Line Feed character. The LOCATE function helps us detect its presence in the CLNOM0 field.

To test for multiple non-printable characters at once, we can combine the conditions using the OR operator. Here's an SQL query that checks for both the Carriage Return and Line Feed characters:

SELECT CLANN0, CLCCL0, CLCIB0, CLNOM0
FROM CLANA00F                        
WHERE LOCATE(x'0D', CLNOM0) > 0
   OR LOCATE(x'25', CLNOM0) > 0;

In this query, we use the OR operator to check if either the Carriage Return (x'0D') or the Line Feed (x'25') character is present in the CLNOM0 field. If either condition is true, the query returns the record.

Now, for the icing on the cake, here's an SQL query that replaces any non-printable characters found in the CLNOM0 field with a blank space:

UPDATE CLANA00F                                              
SET CLNOM0 = REPLACE(REPLACE(CLNOM0, x'0D', ' '), x'25', ' ')
WHERE   LOCATE(x'25', CLNOM0) > 0                            
    OR  LOCATE(x'0D', CLNOM0) > 0;

This query uses the REPLACE function to substitute the non-printable characters '0D' (Carriage Return) and '25' (Line Feed) with a blank space in the CLNOM0 field. The WHERE clause ensures that only records containing these characters are updated.

Here’s how you can use these queries to detect and handle non-printable characters in your data:

Identify Records with Non-Printable Characters:

Run the queries above to find all records in CLANA00F where the CLNOM0 contains the Carriage Return character (x'0D') or the Line Feed character (x'25').

Review and Clean Data:

Once you've identified the problematic records, you can manually review and clean the data. For example, you might replace the non-printable characters with spaces or other appropriate characters.

Prevent Future Issues:

Implement data validation rules to prevent non-printable characters from being entered into your database in the future. This can be done at the application level or by using database constraints and triggers.

By running these SQL queries periodically or incorporating them into your data validation processes, you can ensure that your data remains clean and free of non-printable characters that could cause issues in your applications.

That’s it! If you have any comments or suggestions, please feel free to share them on this blog.