This is another blog post in the FTimes series showcasing various aspects and controls that can be utilized within the FTimes framework. This blog post will focus on using file hooks, a feature that offers the ability to run external programs or scripts on matching files during dig, map, or mad stages.

Enabling file hooks (via the FileHooks control) makes it possible for ftimes to run external programs or scripts that can perform additional tasks on matching files during dig, map, or mad stages.

This blog post gives an overview of adding a FileHook entry to a map configuration and provides a few relevant examples. The final example shows how a file hook can be used to locate SQLite databases and subsequently map their internal structure with the help of a Python script.

With FTimes, file hooks are implemented using KoreLogic Expression Language (KLEL) guarded commands. According to KLEL documentation, a guarded command consists of a boolean guard, an eval call, and an optional set of expected return codes. With FTimes, the KLEL eval function supports two default interpreters ('exec' and 'system') and three optional interpreters ('lua', 'perl', and 'python'). Under the hood, the default interpreters are implemented using the well-known execv(3) and system(3) function calls. While extremely useful (virtually any external program or script can be invoked), it can be expensive in terms of process execution time. In the most extreme case (i.e., a guarded command whose expression always evaluates to true), a new process would be launched for every file encountered. That's a lot of overhead, especially if you are mapping millions of files in a single pass. Fortunately, the three embedded interpreters eliminate that problem. By embedding an interpreter such as Lua, Perl, or Python, we gain the ability to run scripts written in those languages from within the running ftimes process, and that can result in huge performance gains.

This blog post assumes that you've read the previous blog post about compiling FTimes with Python. It also assumes that you have met the OS requirements from that post (i.e., that you are using Kali Linux).

First, we begin by compiling libklel. As of the writing of this blog post, the current version of KLEL is 1.2.0, which is available here.

tar -zxf libklel-1.2.0.tar.gz
cd libklel-1.2.0
mkdir b
cd b
../configure && make && sudo make install

This will install the header files and libraries with a prefix of '/usr/local'.

N.B. If you want to change where this software is installed, use the '--prefix' option when running the configure command above.

Next, we compile FTimes with file hooks and embedded Python support. The latest FTimes tar ball is available here. Additional details pertaining to embedded Python support are available here.

tar -xzf ftimes-3.13.0.tgz
cd ftimes-3.13.0
mkdir b
cd b
../configure --enable-file-hooks --with-python=`which python3` --with-all-tools --with-klel=/usr/local
make && sudo make install

Once the build completes and the software is installed, run 'ftimes --version'; it should produce output similar to the following:

ftimes 3.13.0 64-bit klel(1.2.0),pcre(8.39),python(3.7.5),filters(pcre),hooks,xmagic

Your Python and PCRE versions could be different depending on the installed versions of those libraries.

Now the fun part: utilizing the new KLEL features and taking advantage of file hooks.

Example #1

This example shows the usage of a basic file hook. First, we create a map configuration (map.cfg) in the current working directory that contains a simple file hook entry (i.e., FileHook).

BaseName=-
OutDir=.
FieldMask=none

FileHook=echo : if (true) then eval("exec", "/bin/echo", "%{f_name}") pass [0]

The above file hook entry breaks down as follows:

FileHook=echo

This is defining a FileHook with a designator (or tag) of 'echo'. This tag can be any string that matches the following regular expression:

[A-Za-z][0-9A-Za-z_]+

if (true)

This is the boolean guard. As you can see, it always evaluates to true. This implies that the guarded command will be invoked for every file ftimes encounters.

eval("exec", "/bin/echo", "%{f_name}")

This is the guarded command, an eval statement that executes '/bin/echo' (via 'exec') passing 'f_name' as its only parameter. In this case, the value of 'f_name' (i.e., '%{f_name}') is the current file's full path.

pass [0]

This is the set of expected return codes. If the command being executed does not return one of the expected return codes, its execution is deemed a failure, and ftimes will report an error. In this particular case, there is only one expected return code: zero.

FTimes exposes a number file attributes to KLEL. This means, for example, you could create hooks that match on a file's size (f_size), cryptographic hash (f_md5, f_sha1, f_sha256), or even some combination of any available file attributes.

An important note to remember is that anything printed to STDOUT by an executing hook can pollute the ftimes output stream unless care is taken to format the data in a way that is consistent with normal ftimes output. Consequently, this may become an issue if other FTimes tools are subsequently used to process the output.

Example #2

This example shows a file hook that fires based on file type as determined by XMagic. If the 'magic' attribute (designated as 'f_magic' in the guarded command) matches our defined file type (i.e., 'database/sqlite'), then it will again echo the current file's full path.

BaseName=-
OutDir=.
FieldMask=none+magic

FileHook=echo : if (f_magic =~ ("database/sqlite")) then eval("exec", "/bin/echo", "%{f_name}") pass [0]

Thus far, the guarded commands haven't been very useful (i.e., they simply echo the filename to STDOUT whenever the hook's guard evaluates to true). Let's make things more interesting by actually doing something to/on the files being matched by the guard shown in Example #2.

Example #3

This example shows how to use the embedded Python interpreter and the script provided below to map out the contents of identified SQLite databases.

Place the following Python script (map_sqlite.py) in your current working directory:

import sqlite3
import sys
from ftimes import neuter_string

if len(sys.argv) < 1:
    print("Need a SQLite database.")
    sys.exit(2)

InFile = sys.argv[1]
NeuteredInfile = neuter_string(InFile)

conn = sqlite3.connect(InFile)
c = conn.cursor()
c.execute("SELECT name FROM sqlite_master WHERE type IN ('table','view') AND \
          name NOT LIKE 'sqlite_%' UNION ALL SELECT name FROM \
          sqlite_temp_master WHERE type IN ('table','view') ORDER BY 1")

tables = []
rowcount = 1

for row in (c.fetchall()):
    tables.append(row)

for mytable in tables:
    NeuteredTableName = neuter_string(mytable[0])
    TableLine = '"' + NeuteredInfile + '{' + NeuteredTableName + '}"|' + str(len(NeuteredTableName)) + "|table"
    print(TableLine)
    TableNames = c.execute("PRAGMA table_info('%s')" % mytable[0]).fetchall()
    c.execute("SELECT * FROM '%s'" % mytable[0])
    for row in (c.fetchall()):
      for name in TableNames:
        if len(str(row[name[0]])) == 0:
          rowprint = '"'  + NeuteredInfile + '{' + NeuteredTableName + '/' + str(rowcount) + '/' + neuter_string(name[1]) + '/}"|' \
                     + str(len(row[name[0]])) + '|'
        else:
          rowprint = '"'  + NeuteredInfile + '{' + NeuteredTableName + '/' + str(rowcount) + '/' + neuter_string(name[1]) + '/}"|' \
                     + str(len(str(row[name[0]]))) + '|' + neuter_string(str(row[name[0]]))
        print(rowprint)
      rowcount += 1

c.close()
conn.close()

Below is the map configuration (map.cfg) to use for this example. Place it in along side the Python script in your current working directory.

BaseName=-
OutDir=.
FieldMask=none+size+magic

FileHook=sqlite : if ((f_magic =~ "database/sqlite")) then eval("python", "map_sqlite.py", "%{f_name}") pass [0]

Using the following as input, we will create a test SQLite database that can be mapped using the above script.

CREATE TABLE sample
(
   fnum INTEGER NOT NULL PRIMARY KEY
  ,mode INTEGER
  ,name TEXT NOT NULL UNIQUE
  ,size INTEGER
);
INSERT INTO sample VALUES(21,644,'/etc/passwd',1505);
INSERT INTO sample VALUES(42,755,'/etc',12288);
INSERT INTO sample VALUES(7,755,'/sbin',12288);
INSERT INTO sample VALUES(3,600,'/etc/shadow',1005);
INSERT INTO sample VALUES(9,777,'/tmp',4096);
INSERT INTO sample VALUES(31,700,'/opt/lost+found',16384);
INSERT INTO sample VALUES(39,700,'/opt/My Documents',4096);

Save the above SQL statements into a file called create_test_db.sql. Then, run the following command to create the database:

sqlite3 /tmp/test.db < create_test_db.sql

N.B. This database was created under /tmp for the purposes of this example only. Indiscriminate creation and/or use of files in /tmp can fall victim to abuse (e.g., race conditions, data-driven attacks, etc.). Make sure you understand the security implications of /tmp before using it for any purpose.

Next, we will use ftimes along with map.cfg and map_sqlite.py to map this database.

ftimes --map map.cfg -l 6 /tmp/test.db

Below is a sample of what the output should look like.

name|size|magic
"/tmp/test.db"|12288|database/sqlite: version="3";
"/tmp/test.db{sample}"|6|table
"/tmp/test.db{sample/1/fnum/}"|1|3
"/tmp/test.db{sample/1/mode/}"|3|600
"/tmp/test.db{sample/1/name/}"|11|/etc/shadow
"/tmp/test.db{sample/1/size/}"|4|1005
"/tmp/test.db{sample/2/fnum/}"|1|7
"/tmp/test.db{sample/2/mode/}"|3|755
"/tmp/test.db{sample/2/name/}"|5|/sbin
"/tmp/test.db{sample/2/size/}"|5|12288
"/tmp/test.db{sample/3/fnum/}"|1|9
"/tmp/test.db{sample/3/mode/}"|3|777
"/tmp/test.db{sample/3/name/}"|4|/tmp
"/tmp/test.db{sample/3/size/}"|4|4096
"/tmp/test.db{sample/4/fnum/}"|2|21
"/tmp/test.db{sample/4/mode/}"|3|644
"/tmp/test.db{sample/4/name/}"|11|/etc/passwd
"/tmp/test.db{sample/4/size/}"|4|1505
"/tmp/test.db{sample/5/fnum/}"|2|31
"/tmp/test.db{sample/5/mode/}"|3|700
"/tmp/test.db{sample/5/name/}"|15|/opt/lost%2bfound
"/tmp/test.db{sample/5/size/}"|5|16384
"/tmp/test.db{sample/6/fnum/}"|2|39
"/tmp/test.db{sample/6/mode/}"|3|700
"/tmp/test.db{sample/6/name/}"|17|/opt/My+Documents
"/tmp/test.db{sample/6/size/}"|4|4096
"/tmp/test.db{sample/7/fnum/}"|2|42
"/tmp/test.db{sample/7/mode/}"|3|755
"/tmp/test.db{sample/7/name/}"|4|/etc
"/tmp/test.db{sample/7/size/}"|5|12288

This output shows a single map record for the SQLite database along with a number of pseudo map records that reveal the contents of the 'sample' table. Note that the magic field has been overloaded (i.e., it now holds actual content for each database record/field mapped). Also note that the magic values are FTimes neutered (i.e., various special and non-printable characters have been encoded).

So there you have it. Using this new found feature, you can come up with various file hooks that can be utilized to do whatever you want (e.g., map a SQLite database, unpack a ZIP archive and map its contents, perform searches or binary analysis on bzip files etc.). The possibilities are endless ...

That's all for now on this blog post. Stay tuned for additional FTimes features blog posts.