Overview

The customer reports that on an IPLMS server there are a lot of files that appear to be stuck for a particular service in this example Service No. 7705:

[crestel@hydcol2 input]$ ls |wc -l
114165
[crestel@hydcol2 input]$ pwd
/data/distribution_root/input

The CMS command line utility shows pending files

CMS> COUNTERSTATUS
localhost:7705
=========For Service =======IPLOG_PARSING_SERVICE-000======
Total Files :: 17431
Total Success Files :: 17431
Total Failure Files :: 0
Total Pending Files :: 63028
Total In Process Files :: 3
Total Partially Successful Files :: 0
Total malform flowsets :: 0
Total Packets :: 0
Total Records :: 1364262397
Average TPS :: 93919

The below steps explain ways to clear the backlog and process the backlog without impacting other services.

Solution

Step-1: Fixing the current Service flow 

• Ask the Customer/Managed Services or L1/L2 Support team of the customer to take a backup for these files and remove them from the input path.
• After that login to CMS utility as crestel user and restart the service
• To restart the service first go to $CRESTEL_P_ENGINE/module/mediation/bin/ and do the following:
• 
• Now start the service using the command from the Master.sh script
• 

Step-2: Handle the backlog of stuck files

• Create a script such as this to move the stuck files back to the processing directory
• In this example, the script processes 200 files at a time
• 
• Create a cronjob to execute this script hourly until the backlog is cleared
• 

Verification

• Keep monitoring the input directory and verify there are pending files
• Keep checking backlog processing directory and verify that backlog file count is reducing on a hourly basis.