Add Java-CPG language module #1659

robinmaisch · 2024-03-19T18:29:59Z

This Draft PR contains the new "Java Code Property Graph" module.
It uses a CPG library and graph transformations to normalise substructures of the code prior to tokenization, which should lead to a higher resilience against refactoring attacks.

It also serves to fix SonarCloud errors before a proper PR is opened.

Content: - A new CPG language frontend for JPlag - An interface to transform submissions into CPGs - An interface to transform CPGs into token lists - A Graph Transformation Engine (to be extended) . interfaces representing node and graph patterns, matches of these patterns, transformations . an isomorphism detector . a transformation algorithm - Some graph transformations (to be extended)

…rmation pass

- implemented multi-root graph patterns - implemented searching for "all matches at once" - GraphOperations should leave EOG intact - implemented new kinds of edges and properties for graph patterns - tokenization works well - implemented DFG sort pass -- this requires specialized treatment for all kinds of language features. Surely the considered feature set is incomplete.

It was designed for local use.

Add many comments and put a file in the 'passes' package. When JavaDoc cannot find a Java file in there, it quits.

TwoOfTwelve

Just a couple of pointers concerning your code style. I only added the comments for one occurrence each. It would be great if you could fix those things to make the style more consistent with the rest of the code.

language-api/src/main/java/de/jplag/Language.java

language-api/src/main/java/de/jplag/TokenType.java

languages/java-cpg/src/main/java/de/jplag/java_cpg/JavaCpgLanguage.java

TwoOfTwelve · 2024-03-26T09:00:31Z

languages/java-cpg/src/main/java/de/jplag/java_cpg/passes/ATransformationPass.kt

+ * A ATransformationPass is an abstract transformation pass. All transformation passes function the same way, but need
+ * to be separate classes to work with the CPG pipeline.
+ */
+abstract class ATransformationPass(ctx: TranslationContext) : TranslationResultPass(ctx) {


Please do not use Kotlin, as using more languages complicates the process of compiling JPlag and make the code harder to understand for new contributors.

The implementation of the CPG library requires that Kotlin is used, any subclass of Pass written in Java leads to an exception, unfortunately. I could ask the authors to change that, if that is an absolute requirement.

What's the exception? Do you remember this?

languages/java-cpg/src/main/java/de/jplag/java_cpg/token/IVisitorExitor.java

languages/java-cpg/src/main/java/de/jplag/java_cpg/transformation/Role.java

Kr0nox · 2024-03-28T17:46:44Z

Please also add the language to the report viewer here: https://github.com/jplag/JPlag/blob/develop/report-viewer/src/model/Language.ts

Also set a language for highlighting here: https://github.com/jplag/JPlag/blob/develop/report-viewer/src/utils/CodeHighlighter.ts

report-viewer/src/model/Language.ts

sonarcloud · 2024-04-03T19:43:41Z

Quality Gate passed for 'JPlag Plagiarism Detector'

Issues
0 New issues
1 Accepted issue

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

tsaglam

Do not forget to add a language module readme. Especially to explain the differences to the other Java modules. It should probably both include information for users and for developers due to the complexity of the PR.

On another note: We should just include Kotlin in the spotless config: https://github.com/diffplug/spotless/tree/main/plugin-maven#kotlin. We can add the config directly, but I would wait to apply the formatting after the reviews are done and the feedback addressed.

Finally, some questions for our next meeting:
This is a LOT of code, I initially hoped more functionality would be provided by the API itself. In your opinion, what is the main cause for the size of the PR? And why does the CPG API require so much boilerplate code?

cli/pom.xml

languages/java-cpg/pom.xml

tsaglam · 2024-04-09T09:11:14Z

languages/java-cpg/src/main/java/de/jplag/java_cpg/CpgAdapter.java

+            clearTransformations();
+            setReorderingEnabled(false);
+        }
+        // TokenizationPass sets tokenList


leftover comment?

The field tokenList is set to null a few lines before. I thought I'd include that comment to clarify that the tokenList is not supposed to still be null when it is returned in line 49, but it is set by the TokenizationPass via the method CpgAdapter.setTokenList(List<Token>).

Ah, okay, this was not clear to me, thanks.

tsaglam · 2024-04-09T09:12:37Z

languages/java-cpg/src/main/java/de/jplag/java_cpg/CpgAdapter.java

+        } catch (InterruptedException e) {
+            throw e;
+        } catch (ExecutionException | ConfigurationException e) {
+            throw new ParsingException(List.copyOf(files).getFirst(), e);


Why the copy when only one element is passed?

Because any one element can not be gotten from the Set<File> files. I agree that this is not very elegant.
Maybe replace it with files.stream().findAny().orElse(null)?

Use files.iterator().next(); instead.

languages/java-cpg/src/main/java/de/jplag/java_cpg/JavaCpgLanguage.java

tsaglam · 2024-04-09T09:50:20Z

languages/java-cpg/src/main/java/de/jplag/java_cpg/transformation/matching/edges/CpgEdge.java

+/**
+ * This is a wrapper for a graph edge (with a 1:1 relation).
+ * @param <T> The type of the source node
+ * @param <R> The type of the related node
+ */
+public class CpgEdge<T extends Node, R extends Node> extends AEdge<T, R> {


Unclear to me: Why are we wrapping the AEdge which is just a base class?

If you refer to IEdge vs. AEdge:
IEdge existed before AEdge, and I did not realize that it could actually just fully replace it. I'll remove the IEdge interface, then.

If you refer to the phrase "wrapper for a graph edge", then let me explain it in a bit more detail.
To summarize: The Edge classes make up for the fact that in the CPG library, edges are represented inconsistently.
Either:

the edge target node is a field of the source (e.g., an IfStatement node has the fields condition, thenStatement, elseStatement); the AST edge itself is not modeled
--- or, if it is a 1:n relation ----

that relation is represented as a list of nodes (e.g. a FunctionType has the field parameters)

that relation is represented as a list of edges (e.g. a Block node has the field statementEdges)

The edge class of the CPG library is called PropertyEdge.

For the transformations, I need to get and the target(s) of edges and eventually set them to new values. My Edge classes save the appropriate getter and setter functions for all three cases.
In the class de.jplag.java_cpg.transformation.matching.edges.Edges, you can see all the kinds of edges I use, and how I save the appropriate getter and setter functions from the node classes in them.

tsaglam · 2024-04-09T09:50:55Z

...ges/java-cpg/src/main/java/de/jplag/java_cpg/transformation/matching/edges/CpgMultiEdge.java

+import de.jplag.java_cpg.transformation.matching.pattern.NodePattern;
+
+/**
+ * This is a wrapper for a graph edge (1:n relation).


Can you elaborate here?

tsaglam · 2024-04-09T09:52:41Z

languages/java-cpg/src/main/java/de/jplag/java_cpg/transformation/matching/edges/IEdge.java

+/**
+ * This serves as an interface to wrap any kind of {@link PropertyEdge}.
+ * @param <T> the source node type
+ * @param <R> the related node type
+ */
+public interface IEdge<T extends Node, R extends Node> {


To be honest, I am kind of lost in the edge hierarchy.

tsaglam · 2024-04-09T09:55:23Z

...cpg/src/main/java/de/jplag/java_cpg/transformation/matching/pattern/GraphPatternBuilder.java

+        }
+    }
+}


Again, very large class, with many nested classes. In the long term, maintaining this language module will be very tricky.

I can move them out, that's not a problem.

tsaglam · 2024-04-09T09:57:22Z

...a-cpg/src/main/java/de/jplag/java_cpg/transformation/matching/pattern/relation/Relation.java

+import de.jplag.java_cpg.transformation.matching.pattern.Match;
+import de.jplag.java_cpg.transformation.matching.pattern.NodePattern;
+
+public abstract sealed class Relation<T extends Node, R extends Node, V> permits OneToNRelation, RelatedNode {


uuqjz

Please have a lock at the things i marked, I've not checked tests

languages/java-cpg/src/main/java/de/jplag/java_cpg/CpgAdapter.java

uuqjz · 2024-04-11T06:40:52Z

languages/java-cpg/src/main/java/de/jplag/java_cpg/JavaCpgLanguage.java

+ */
+@MetaInfServices(de.jplag.Language.class)
+public class JavaCpgLanguage implements Language {
+    private static final int DEFAULT_MINIMUM_TOKEN_MATCH = 9;


You could take the dafault mtm of the java module instead of setting it to 9

That is possible, but it would require a module dependency on the Java frontend.
Do you think that would be justified?

No, that would not be justified.

uuqjz · 2024-04-11T06:44:29Z

languages/java-cpg/src/main/java/de/jplag/java_cpg/passes/DfgSortPass.kt

+        node: Statement,
+        essentialNodes: MutableList<Statement>,
+    ): Boolean {
+        /*


Do you think this is necessary? Or can it be formatted so it doesn't span 10 lines?

languages/java-cpg/src/main/java/de/jplag/java_cpg/passes/JTokenizationPass.java

...ages/java-cpg/src/main/java/de/jplag/java_cpg/transformation/operations/InsertOperation.java

...ages/java-cpg/src/main/java/de/jplag/java_cpg/transformation/operations/RemoveOperation.java

uuqjz · 2024-04-11T08:51:24Z

...s/java-cpg/src/main/java/de/jplag/java_cpg/transformation/operations/TransformationUtil.java

+     * @param astRoot the root of the sub-AST
+     * @return the EOG {@link SubgraphWalker.Border} of the AST
+     */
+    public static SubgraphWalker.Border getEogBorders(Node astRoot) {


What does EOG stand for?

That is the Evaluation Order Graph, a concept introduced by the authors of the CPG library, where the term is also often used. The EOG is a type of Control Flow Graph, but instead of blocks, it operates on individual Nodes.
For example, in a binary operation expression, the left operand is evaluated first (except if it is an assignment), then the right operand is evaluated, and then the operation itself. An EOG of a binary operation expression would contain a directed edge from the left operand node to the right operand node and from the right operand to the operation node to reflect that. In control statements, the EOG may split.
In the CPG library, the EOG is the basis for complex name analysis, data flow analysis, and other types of analyses.
I use the EOG as the basis for CPG Linearization.

languages/java-cpg/src/main/java/de/jplag/java_cpg/visitor/MethodOrderStrategy.java

languages/java-cpg/src/main/java/de/jplag/java_cpg/visitor/NodeOrderStrategy.java

languages/java-cpg/src/main/java/de/jplag/java_cpg/CpgAdapter.java

languages/java-cpg/pom.xml

robinmaisch and others added 17 commits December 29, 2023 11:55

Fixed ReplaceOperation to support swapping elements

6ae3fc5

Reorder passes; Fixed Token generation; Extended NodePattern relations

0c6be35

Bump CPG dependencies from 8.0.0-alpha-2 to 8.0.0

185b311

Add patterns to match one edge of a multi-edge; Introduce 2nd transfo…

7ab9b0f

…rmation pass

Fixed old tests and some code problems indicated by the IDE

4447224

Merge branch 'jplag:main' into cpg-dev

1b900a8

Fix errors from merging JPlag v5.0.0

722b256

Feature-complete version (before Sonar)

4ed18ea

Apply Spotless

5536a3f

Remove CompareFrontendsTest.java

95fcb9e

It was designed for local use.

Fix tests

640e1fa

Fix tests

eb592f0

Fix JavaDoc

88dd331

Add many comments and put a file in the 'passes' package. When JavaDoc cannot find a Java file in there, it quits.

Add more documentation

fc3d079

Merge remote-tracking branch 'origin/develop' into cpg-dev

83995bf

tsaglam added enhancement Issue/PR that involves features, improvements and other changes major Major issue/feature/contribution/change language PR / Issue deals (partly) with new and/or existing languages for JPlag labels Mar 20, 2024

robinmaisch added 3 commits March 23, 2024 18:05

Fix Sonar remarks

9ad7388

Add missing files

9c25f9f

Fix Regex

b2fc2f1

robinmaisch marked this pull request as ready for review March 23, 2024 20:22

TwoOfTwelve requested changes Mar 26, 2024

View reviewed changes

Improve documentation and consistency of type parameter naming

2e1e333

This comment was marked as outdated.

Sign in to view

robinmaisch added 2 commits April 2, 2024 14:28

Revert changes to base code

50b83ea

Revert changes to base code – No.2

1acd991

This comment was marked as outdated.

Sign in to view

Add support for Java-CPG module in report viewer

7946823

tsaglam requested review from tsaglam, uuqjz and a team April 2, 2024 14:56

tsaglam reviewed Apr 3, 2024

View reviewed changes

report-viewer/src/model/Language.ts Outdated Show resolved Hide resolved

Fix missing comma

60427fd

tsaglam requested changes Apr 9, 2024

View reviewed changes

uuqjz reviewed Apr 11, 2024

View reviewed changes

dfuchss reviewed Apr 13, 2024

View reviewed changes

languages/java-cpg/src/main/java/de/jplag/java_cpg/CpgAdapter.java Show resolved Hide resolved

dfuchss reviewed Apr 13, 2024

View reviewed changes

languages/java-cpg/pom.xml Show resolved Hide resolved

dfuchss reviewed Apr 13, 2024

View reviewed changes

languages/java-cpg/pom.xml Show resolved Hide resolved

tsaglam changed the title ~~Add Java-CPG frontend~~ Add Java-CPG language module Jul 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Java-CPG language module #1659

Add Java-CPG language module #1659

robinmaisch commented Mar 19, 2024

TwoOfTwelve left a comment

TwoOfTwelve Mar 26, 2024

robinmaisch Mar 26, 2024 •

edited

Loading

dfuchss Apr 13, 2024

This comment was marked as outdated.

Kr0nox commented Mar 28, 2024

This comment was marked as outdated.

sonarcloud bot commented Apr 3, 2024 •

edited

Loading

tsaglam left a comment

tsaglam Apr 9, 2024

robinmaisch Apr 13, 2024

tsaglam Apr 16, 2024

tsaglam Apr 9, 2024

robinmaisch Apr 13, 2024

tsaglam Apr 15, 2024

tsaglam Apr 9, 2024

robinmaisch Apr 13, 2024

tsaglam Apr 9, 2024

tsaglam Apr 9, 2024

tsaglam Apr 9, 2024

robinmaisch Apr 13, 2024

tsaglam Apr 9, 2024

uuqjz left a comment

uuqjz Apr 11, 2024

robinmaisch Apr 14, 2024

tsaglam Apr 16, 2024

uuqjz Apr 11, 2024

uuqjz Apr 11, 2024

robinmaisch Apr 14, 2024

Add Java-CPG language module #1659

Are you sure you want to change the base?

Add Java-CPG language module #1659

Conversation

robinmaisch commented Mar 19, 2024

TwoOfTwelve left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robinmaisch Mar 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as outdated.

Kr0nox commented Mar 28, 2024

This comment was marked as outdated.

sonarcloud bot commented Apr 3, 2024 • edited Loading

Quality Gate passed for 'JPlag Plagiarism Detector'

tsaglam left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uuqjz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robinmaisch Mar 26, 2024 •

edited

Loading

sonarcloud bot commented Apr 3, 2024 •

edited

Loading