IBMi (AS400) fans only ' Detecting and Handling Non-Printable Characters in DB2 SQL Using LOCATE() and REPLACE() Functions

By -





SQL Using LOCATE() and REPLACE() Functions


When working with DB2 SQL on the IBMi platform, one of the challenges we might encounter is dealing with non-printable characters in our data. These are characters that don't have a direct representation on a standard keyboard and can cause display programs to crash if not handled properly.


For this example, we'll be working with the CLANA00F file, which contains customer data. You can find details on how to create this file on my blog post here.


Non-printable characters can sneak into our records for various reasons, and to prevent issues, we can run an SQL query that uses the LOCATE() function. This function returns the position of the first occurrence of a substring within a string, allowing us to detect these problematic characters.


Suppose we have a customer name with a non-printable character, specifically the EBCDIC '0D' character, which corresponds to the "Carriage Return" (CR) control character in ASCII. To find this character in our records, we can use the following SQL query:


SELECT CLANN0, CLCCL0, CLCIB0, CLNOM0
FROM CLANA00F                        
WHERE LOCATE(x'0D', CLNOM0) > 0;

In this query, x'0D' represents the EBCDIC Carriage Return character. The LOCATE function searches for this character in the CLNOM0 field and returns its position. If the position is greater than 0, it means the character is present.


Additionally, another non-printable character that might cause issues is the EBCDIC '25' character, which corresponds to the Line Feed (LF) control character in ASCII. To find this character, we can use a similar SQL query:


SELECT CLANN0, CLCCL0, CLCIB0, CLNOM0
FROM CLANA00F                        
WHERE LOCATE(x'25', CLNOM0) > 0;

In this query, x'25' represents the EBCDIC Line Feed character. The LOCATE function helps us detect its presence in the CLNOM0 field.


To test for multiple non-printable characters at once, we can combine the conditions using the OR operator. Here's an SQL query that checks for both the Carriage Return and Line Feed characters:


SELECT CLANN0, CLCCL0, CLCIB0, CLNOM0
FROM CLANA00F                        
WHERE LOCATE(x'0D', CLNOM0) > 0
   OR LOCATE(x'25', CLNOM0) > 0;

In this query, we use the OR operator to check if either the Carriage Return (x'0D') or the Line Feed (x'25') character is present in the CLNOM0 field. If either condition is true, the query returns the record.


Now, for the icing on the cake, here's an SQL query that replaces any non-printable characters found in the CLNOM0 field with a blank space:


UPDATE CLANA00F                                              
SET CLNOM0 = REPLACE(REPLACE(CLNOM0, x'0D', ' '), x'25', ' ')
WHERE   LOCATE(x'25', CLNOM0) > 0                            
    OR  LOCATE(x'0D', CLNOM0) > 0;

This query uses the REPLACE function to substitute the non-printable characters '0D' (Carriage Return) and '25' (Line Feed) with a blank space in the CLNOM0 field. The WHERE clause ensures that only records containing these characters are updated.


Here’s how you can use these queries to detect and handle non-printable characters in your data:


Identify Records with Non-Printable Characters:

Run the queries above to find all records in CLANA00F where the CLNOM0 contains the Carriage Return character (x'0D') or the Line Feed character (x'25').


Review and Clean Data:

Once you've identified the problematic records, you can manually review and clean the data. For example, you might replace the non-printable characters with spaces or other appropriate characters.


Prevent Future Issues:

Implement data validation rules to prevent non-printable characters from being entered into your database in the future. This can be done at the application level or by using database constraints and triggers.


By running these SQL queries periodically or incorporating them into your data validation processes, you can ensure that your data remains clean and free of non-printable characters that could cause issues in your applications.


That’s it! If you have any comments or suggestions, please feel free to share them on this blog. 

Comments

Post a Comment

Popular posts from this blog

IBMi (AS400) fans only ' Efficient WRKSPLF with WSF - How to Search string into spooled files, Sort, and Generate PDFs on IBMi

By -
Image
Searching Spooled Files Made Easy: A Unique Solution Managing spooled files efficiently is crucial for users dealing with numerous rows of print data. IBM's WRKSPLF command provides a standard way to interact with spooled files, but it lacks certain features that can significantly enhance user experience. In this article, we'll delve into a powerful functionality that IBM doesn't offer — the ability to search among spooled file lists. Our goal is to highlight the advantages of this capability and introduce a practical solution. The Limitations of Standard WRKSPLF While the WRKSPLF command is a handy tool for working with spooled files, it falls short in certain aspects. One prominent limitation is the inability to search efficiently through the list of spooled file rows. Users often find it challenging to quickly locate specific files based on criteria like creation date, status, or content. Introducing a Unique Solution: WSF (WorkSpooledFiles) To address the limitations of
Read more

IBMi (AS400) fans only - How to Sniff User Access

By -
Image
If you need to monitor user access on your IBM i system, you can use SQL to query the history log. Note: At the end of this post, you'll find a request directly addressed to you, the readers. Here’s a simple example of how you can do this: SELECT MESSAGE_ID,        MESSAGE_TIMESTAMP,        FROM_USER,        MESSAGE_TEXT     FROM TABLE (             QSYS2.HISTORY_LOG_INFO()         )     WHERE message_id IN ('CPF1164', 'CPF1124')           AND FROM_USER IN ('QSECOFR', 'QSYS', 'QSYSOPR')     ORDER BY MESSAGE_TIMESTAMP In this query: - QSYS2.HISTORY_LOG_INFO(): This function retrieves information from the history log. -  message_id IN ('CPF1164', 'CPF1124') : Filters the results to include only specific message IDs, which indicate sign-on and sign-off events. - FROM_USER in ('QSECOFR', 'QSYS', 'QSYSOPR'): Filters the results to include only messages from specific users. Update 28/07/2024: A Smart Enhancem
Read more

IBMi (AS400) fans only ' SQLCODE values: Dear readers, unleash your suggestions!

By -
Image
This RPG code uses a DOU loop to fetch data from a cursor named C1 until the SQLCODE is not equal to 0, indicating the end of the recordset. Inside the loop, it clears a data structure (RecordDsC1), fetches data into it, and then checks if the fetch was successful. If successful, it assigns values and writes to an output file. After the loop, it checks if the SQLCODE is not 100, indicating unexpected termination, and displays a warning message if needed. // Loop until SQLCODE is not equal to 0 dow sqlCode = 0 ;   // Clear the data structure RecordDsC1   clear RecordDsC1;   // Fetch data into RecordDsC1 using cursor C1   exec sql fetch C1 into :RecordDsC1;   // If fetch was successful (SQLCODE = 0)   if sqlcode = 0 ;     // Assign values and write to output     W1QUEU = rDsW1QUEU;     write W1SF1;   endif; enddo; // Check if SQLCODE is not 100 (indicating unexpected termination) if sqlCode <> 100;   // Build a warning message   wMsg='Warning: +    The selection has terminated
Read more