Finding duplicated code with CPD

Overview

Duplicate code can be hard to find, especially in a large project. But PMD’s Copy/Paste Detector (CPD) can find it for you! CPD has been through three major incarnations:

• First we wrote it using a variant of Michael Wise’s Greedy String Tiling algorithm (our variant is described here).

• Then it was completely rewritten by Brian Ewins using the Burrows-Wheeler transform.

• Finally, it was rewritten by Steve Hawkins to use the Karp-Rabin string matching algorithm.

Each rewrite made it much faster, and now it can process the JDK 1.4 java.* packages in about 4 seconds (on my workstation, at least).

Note that CPD works with Java, JSP, C, C++, C#, Fortran and PHP code and some more languages. For the full list, see below Supported Languages. Your own language is missing? See how to add it here.

CPD is included with PMD, which you can download here.

CLI options

Examples

Note: The following example use the Linux start script. For Windows, just replace “./run.sh cpd” by “cpd.bat”.

Minimum required options: Just give it the minimum duplicate size and the source directory:

$ ./run.sh cpd --minimum-tokens 100 --files /usr/local/java/src/java

You can also specify the language:

$ ./run.sh cpd --minimum-tokens 100 --files /path/to/c/source --language cpp

You may wish to check sources that are stored in different directories:

$ ./run.sh cpd --minimum-tokens 100 --files /path/to/other/source  --files /path/to/other/source --files /path/to/other/source --language fortran

There should be no limit to the number of ‘–files’, you may add… But if you stumble one, please tell us !

And if you’re checking a C source tree with duplicate files in different architecture directories you can skip those using –skip-duplicate-files:

$ ./run.sh cpd --minimum-tokens 100 --files /path/to/c/source --language cpp --skip-duplicate-files

You can also specify the encoding to use when parsing files:

$ ./run.sh cpd --minimum-tokens 100 --files /usr/local/java/src/java --encoding utf-16le

You can also specify a report format - here we’re using the XML report:

$ ./run.sh cpd --minimum-tokens 100 --files /usr/local/java/src/java --format xml

The default format is a text report, and there’s also a csv report.

Note that CPD is pretty memory-hungry; you may need to give Java more memory to run it, like this:

$ export HEAPSIZE=512m
$ ./run.sh cpd --minimum-tokens 100 --files /usr/local/java/src/java

In order to change the heap size under Windows, you’ll need to edit the batch file cpd.bat set the “OPTS” variable to -Xmx512m.

If you specify a source directory but don’t want to scan the sub-directories, you can use the non-recursive option:

$ ./run.sh cpd --minimum-tokens 100 --non-recursive --files /usr/local/java/src/java

Exit status

Please note that if CPD detects duplicated source code, it will exit with status 4 (since 5.0). This behavior has been introduced to ease CPD integration into scripts or hooks, such as SVN hooks.

Supported Languages

• Apex
• C#
• C/C++
• EcmaScript (JavaScript)
• Fortran
• Go
• Groovy
• Java
• Jsp
• Matlab
• Objective-C
• Perl
• PHP
• PL/SQL
• Python
• Ruby
• Scala
• Swift
• Visualforce

Available report formats

• text : Default format
• xml
• csv
• csv_with_linecount_per_file
• vs

Ant task

Andy Glover wrote an Ant task for CPD; here’s how to use it:

    <target name="cpd">
        <taskdef name="cpd" classname="net.sourceforge.pmd.cpd.CPDTask" />
        <cpd minimumTokenCount="100" outputFile="/home/tom/cpd.txt">
            <fileset dir="/home/tom/tmp/ant">
                <include name="**/*.java"/>
            </fileset>
        </cpd>
    </target>

Attribute reference

Also, you can get verbose output from this task by running ant with the -v flag; i.e.:

ant -v -f mybuildfile.xml cpd

Also, you can get an HTML report from CPD by using the XSLT script in pmd/etc/xslt/cpdhtml.xslt. Just run the CPD task as usual and right after it invoke the Ant XSLT script like this:

    <xslt in="cpd.xml" style="etc/xslt/cpdhtml.xslt" out="cpd.html" />

GUI

CPD also comes with a simple GUI. You can start it via some scripts in the bin folder:

For Windows:

cpdgui.bat

For Linux:

./run.sh cpdgui

Here’s a screenshot of CPD after running on the JDK 8 java.lang package:

Suppression

Arbitrary blocks of code can be ignored through comments on Java, C/C++, Javascript, Matlab, Objective-C, PL/SQL and Python by including the keywords CPD-OFF and CPD-ON.

    public Object someParameterizedFactoryMethod(int x) throws Exception {
        // some unignored code

        // tell cpd to start ignoring code - CPD-OFF

        // mission critical code, manually loop unroll
        goDoSomethingAwesome(x + x / 2);
        goDoSomethingAwesome(x + x / 2);
        goDoSomethingAwesome(x + x / 2);
        goDoSomethingAwesome(x + x / 2);
        goDoSomethingAwesome(x + x / 2);
        goDoSomethingAwesome(x + x / 2);

        // resume CPD analysis - CPD-ON

        // further code will *not* be ignored
    }

Additionally, Java allows to toggle suppression by adding the annotations @SuppressWarnings("CPD-START") and @SuppressWarnings("CPD-END") all code within will be ignored by CPD.

This approach however, is limited to the locations were @SuppressWarnings is accepted. It’s legacy and the new comment’s based approch should be favored.

    //enable suppression
    @SuppressWarnings("CPD-START")
    public Object someParameterizedFactoryMethod(int x) throws Exception {
        // any code here will be ignored for the duplication detection
    }
    //disable suppression
    @SuppressWarnings("CPD-END)
    public void nextMethod() {
    }

Other languages currently have no support to suppress CPD reports. In the future, the comment based approach will be extended to those of them that can support it.