PerfectSW - Searcher

Searcher is a free application that identifies files that may contain certain types of data such as social security number, credit card number or any data pattern users wish to search for. The main purpose of this application is to find files that contain confidential information. In fact, it contains a basic module that searches for pattern of numbers that resemble Social Security number (ssn). Searcher is a java application which runs on any computer platforms (Windows, Mac, Linux etc.) that support java 1.5 or higher. In addition, because of its pluggable design for search engines construction, Searcher's functionality can be easily extended to handle files that are encoded or compressed. Seacher also creates a log file that lists all the files identified as containing data (confidential) you wish to search for. The person who uses this application can examine this log file and take actions, accordingly or send this log file to a designated http server to be examined by IT security officer, later.

The following documents show the installation process and use of application's features:

Searcher's basic ssn engine (included in this distribution package) may incorrectly identify certain types of files as containing confidential data. This problem can be resolved in the future as more accurate engines are developed and added to this application. However, every effort should be made to verify this application's scan result.

Note: the basic ssn engine (included in this distribution package) only identifies strings similar to ssn from a byte stream of file without understanding the file structure. For example, 800-88-8888 (not valid ssn) is saved in Microsoft Excel file as a number, 800888888, but formatted as 800-88-8888 in a cell for display purpose only. Since this is not a string, it cannot be detected using this basic ssn search module (which uses regular expression). You can create custom plug-in classes that understand certain file structures by simply implementing IPluggable interface. The process will be explained later in "How to use Seacher" section.

Requirements: any operating system (O.S.) platform with Java Runtime Environment (JRE) 1.5 or higher installed. To ensure you have the correct version of java, check the box below:

If the currently installed version of java does not meet the minimum requirement, go to http://java.sun.com/ and select the latest of version of JRE available.

1. Download Searcher installation package. Click here.

2. Decompress the installation package. You will see a folder called Searcher. The main program and related search plug-ins are located inside Searcher folder.

3. If you are installing this application in unix-like operating system (Linux, Mac etc.), make sure you grant execution permission to Searcher.sh file.

[root@test-nb1 Searcher]$ chmod 755 Searcher.sh

4. Make sure you grant appropriate read/write permissions to Searcher.ini file and logs folder (including its contents, if any). The following are examples of security setting for Unix-like and Windows XP operating system.

Read/Write Permissions to Regular Users

Unix-like Operating System:
[root@test-nb1 Searcher]$ chmod -R 666 Searcher.ini

Windows XP Operating System:

Read/Write Permissions to Only Owner and Assigned Group

Unix-like Operating System:
[root@test-nb1 Searcher]$ chmod -R 660 logs

Windows XP Operating System:

A. Scan selected drive or directory

1. Run Searcher.sh (unix-like O.S.) or Searcher.bat (Windows) file to launch Searcher application.

2. Click the Scan button. You will be prompted to select drive and directory you wish to search for. If your selection (directory) contains a very deep hierarchical folder structure (like root of your main drive), Searcher may take a long time to run.

3. Click the Select button and then, Searcher will begin to search files. If you want to cancel the search operation, click the Stop button.

Note: if you want to enable or disable certain plug-ins, select File->Plug-ins and click any plug-in (check box) you wish to enable or disable. If enabled, you can see a check mark next to a plug-in item as shown below.

B. Save (or send via http) the report of scanned files

Upon completion of the scan operation, select File->Save. Searcher will create the scanned report in logs folder. Make sure that write/read permissions are granted to logs folder. (see How to Install Searcher section).

In addition, you can send the scanned report to a designated http server via HTTP POST method.
Click File->Send results to ... You will be prompted to enter User Name and HTTP Address as shown below.

After entering User Name and Address (required), click the Send button.

Note: you can use any server-side scripting methods (asp, aspx, php, jsp etc.) as long as it can process the multipart post method with two fields, one for string and another for file. The string field must be of type "text" and be called "name". Here is an example of equivalent client-side form with multi-part post method. You should make a server-side script that can process this form:

<html>
<body>
<form method="post" enctype="multipart/form-data" action="fileUploader.aspx">
Name: <input type="text" name="name"
<br> 
File: <input type="file" name="upfile" />
<br>
<input type="submit" value="Submit" />
</form>
</body>
</html>

In this example, fileUploader.aspx was used for server-side processing. Below is a sample ASP.NET code.

using System;
using System.Collections.Specialized;
using System.Web;

public partial class fileUploader : System.Web.UI.Page

{

    protected void Page_Load(object sender, EventArgs e)

    {
	// Process String Part
        NameValueCollection userPostedParams = Request.Params;

        for (int j = 0; j < userPostedParams.Count; j++)
        {
            if (userPostedParams.GetKey(j) == "name")
                Response.Write(userPostedParams[j].ToString()+"<br>");
        }

       
	// Process File Part
        HttpFileCollection userPostedFiles = Request.Files;

        for (int i = 0; i < userPostedFiles.Count; i++)
        {

            HttpPostedFile file = userPostedFiles[i];

            try
            {

                if (file.ContentLength > 0)
                {
                    file.SaveAs(Server.MapPath("files\\"+System.IO.Path.GetFileName(file.FileName)));
                }

                Response.Write("Completed successfully");

            }
            catch (Exception ex)
            {
                Response.Write("WARNING: an error has occurred while uploading your file.<br>"+ex.ToString());
            }
        }
    }
}

C. Change the application behavior by modifying Searcher.ini

By adding some settings in Searcher.ini, you can make Searcher to automatically save or send scanned report when you exit the application.

To send scanned report to a designed http server upon exit, add the following keys in Searcher.ini:

REPORT_URL=http://yourdomain.com/fileUploader.aspx
SENDUPONEXIT=true

To save scanned report in logs folder upon exit, add the key below in Searcher.ini:

SAVEUPONEXIT=true

D. How to create a custom search engine

Any custom plug-in class you wish to create must implement all methods in IPluggable interface. The following example is given to you as a starting point for your custom search engine construction. I will suggest creating search engines that understand archived or compressed file structure (e.g., zip. cab, tar, etc.), or a combination of both (tar.bz).

package interfaces;

public interface IPluggable {
	int scan(String str); // 1: found; 0: not found; -1: error
	String getName();
	String getLasterror();
	String version();
}

This interface is very simple, but must be in interfaces package as indicated in this example. The most important method is int scan(String str). The parameter of this function is a string (absolute file path) that indicates a file to be inspected, but its returned value is an integer which can be interpreted as 1 for "pattern not found", -1 "I/O error" and 0 for "pattern found".

The example below shows how to create a search engine that scans for Credit Card Number using java regex engine.

BasicCreditCardFilter.java

package plugins;

import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.io.UnsupportedEncodingException;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import interfaces.IPluggable;

public class BasicCreditCardFilter  implements IPluggable
{
	private String error_msg;
	
	public BasicCreditCardFilter()
	{
		error_msg = "";
	}
	
	public String getLasterror() {
		// TODO Auto-generated method stub
		return error_msg;
	}

	public String getName() {
		// TODO Auto-generated method stub
		return "Basic Credit Card Number Search Engine - VISA, AMEX & MASTER CARD Only";
	}

	public int scan(String str) {
		// TODO Auto-generated method stub
		int ret = 0;
		
		switch(regMatch(str))
		{
			case 1:
				ret = 1;
				break;
			case 0:
				ret = 0;
				break;
			case -1:
				error_msg = "It cannot either IO error or permission issue.";
				ret = -1;			
		}
		
		return ret;
	}

	public String version() {
		// TODO Auto-generated method stub
		return "1.0.0";
	}
	
	public static int regMatch(String path)
	{

		RandomAccessFile is;
		byte bytes[] = new byte[1024];
		int numRead;
		long file_total_size;
		long file_cur_pos=0;
		int  ret = 0;
		// 1: matched; 0: not matched; -1: error; 
		
		try {
			is = new RandomAccessFile(path, "r");
			is.seek(0);
			
			file_total_size = is.length();
			
			file_cur_pos = 0;
			numRead=0;
									
			for(int i=0; i<1024; i++)
				bytes[i] = 0x00;			
			
			numRead = is.read(bytes);
			if(numRead <= 1024)
				file_cur_pos = file_cur_pos + numRead;
			else
				file_cur_pos = file_cur_pos + 768; //numRead/2;				
			
			if(numRead != -1)
				is.seek(file_cur_pos);
			
			
			if(numRead < bytes.length){
				// entire file was read.
				if(numRead != -1){
					if(foundVISACCN(bytes)){
						ret = 1;
					}
					
					if(foundAMEXCCN(bytes)){
						ret = 1;
					}					
				
					if(foundMCCN(bytes)){
						ret = 1;
					}
					
				}
				
			}else{
				
				// iterate through file
				while(numRead != -1)
				{
					if(foundVISACCN(bytes)){
						ret = 1;
						break;
					}										
					
					if(foundAMEXCCN(bytes)){
						ret = 1;
						break;
					}					
				
					if(foundMCCN(bytes)){
						ret = 1;
						break;
					}
					
					for(int i=0; i<1024; i++)
						bytes[i] = 0x00;
					
					numRead = is.read(bytes);
					
					if(numRead <= 1024)
						file_cur_pos = file_cur_pos + numRead;
					else
						file_cur_pos = file_cur_pos + 768; //numRead/2;
					
					if(numRead != -1)
						is.seek(file_cur_pos);
					
				}
								
			}
			
			is.close();
			
			
		} catch (FileNotFoundException e) {
			// TODO Auto-generated catch block
			ret = -1;
		} catch (IOException e) {
			// TODO Auto-generated catch block
			ret = -1;
		}
		
		return ret;
		
	}

	public static boolean foundVISACCN(byte[] str)
	{
		boolean ret = false;
		// VISA
		String pattern = "4\\d{3}([-. ]?)\\d{4}([-. ]?)\\d{4}([-. ]?)\\d{4}";
		
		Pattern regPat = Pattern.compile(pattern, Pattern.CASE_INSENSITIVE);		
		String text2;
		
		try {
			text2 = new String(str, "US-ASCII");
						
			Matcher matcher = regPat.matcher(text2);
			
			while(matcher.find())
			{
						
				String matchedText = matcher.group();			
				
				ret=true;			
				//break;
			}
			
		} catch (UnsupportedEncodingException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
				
		return ret;
		
	}
	
	public static boolean foundMCCN(byte[] str)
	{
		boolean ret = false;
		
		// MASTER CARD
		String pattern = "5[1-5]\\d{2}([-. ]?)\\d{4}([-. ]?)\\d{4}([-. ]?)\\d{4}";
		
		Pattern regPat = Pattern.compile(pattern, Pattern.CASE_INSENSITIVE);		
		String text2;
		
		try {
			text2 = new String(str, "US-ASCII");
						
			Matcher matcher = regPat.matcher(text2);
			
			while(matcher.find())
			{
						
				String matchedText = matcher.group();			
				
				ret=true;			
				//break;
			}
			
		} catch (UnsupportedEncodingException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
				
		return ret;
		
	}
	
	public static boolean foundAMEXCCN(byte[] str)
	{
		boolean ret = false;
		
		// AMEX
		String pattern = "3[4,7]\\d{2}([-. ]?)\\d{6}([-. ]?)\\d{5}";
		
		Pattern regPat = Pattern.compile(pattern, Pattern.CASE_INSENSITIVE);		
		String text2;
		
		try {
			text2 = new String(str, "US-ASCII");
						
			Matcher matcher = regPat.matcher(text2);
			
			while(matcher.find())
			{
						
				String matchedText = matcher.group();			
				
				ret=true;			
				//break;
			}
			
		} catch (UnsupportedEncodingException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
				
		return ret;
		
	}
	
}

To compile this example, follow the steps below:

Note: IPluggable.java and BasicCreditCardFilter.java are located under ~/src folder in this example.

[root@test-nb1 src]$ mkdir interfaces
[root@test-nb1 src]$ mkdir plugins
[root@test-nb1 src]$ mv IPluggable.java interfaces/.
[root@test-nb1 src]$ mv BasicCreditCardFilter.java plugins/.
[root@test-nb1 src]$ javac interfaces/IPluggable.java
[root@test-nb1 src]$ javac plugins/BasicCreditCardFilter.java

Under ~src/plugins folder, you can locate BasicCreditCardFilter.class. Place this file under Searcher/plugins folder. All the search engines (plug-ins) located in this folder will be loaded when Searcher is launched.