3.1

Code snippets extraction

We represent the source code as vectors that can include more contextual semantic relations. However, directly using the entire source code is not a good choice because there is much irrelevant information. To make our system do well, we first propose transforming programs into a representation of code snippets. We observe that two reasons cause most vulnerabilities in smart contracts: (1) incorrect operations of variables, like integer overflow; (2) improper use of API function calls, like timestamp dependency and reentrancy. For example, incorrect add operations to a uint type variable may cause integer overflow and improper uses of call.value (a Solidity API) may cause reentrancy. In addition, we believe that the statements of Library and Event in Solidity source codes will not cause vulnerabilities, so we just ignore them.

Based on this fact, we design the work flow of code snippets extraction. Figure 2 shows an example of that. Firstly, we extract multiple program slices for each global variable and its corresponding functions. In a sense, this kind of program slice can reveal more semantic information. Secondly, we generate the code snippets from the extracted program slices. We notice that most functions and variables’ identifier naming is meaningless and has nothing to do with vulnerability. Therefore, we normalize the program slices by mapping user-defined variables and functions to symbolic names (e.g., “VAR1”, “FUNC1”) one by one. Thirdly, we label the vulnerable code snippet as “1” and label the non-vulnerable code snippets as “0”. If a code snippet contains a vulnerable line, it is defined as vulnerable; otherwise, it is non-vulnerable. The data of vulnerable lines can be found in the training dataset.

Figure 2.

An example of work flow of code snippets extraction.

3.2

Deep learning model

We adopt Word2Vec¹³ for the extracted code snippets to encode their tokens into feature vectors. The reason why we choose Wrod2Vec is its widespread use and excellent performance in the field of text classification¹⁴. It can convert a token to a fixed-length vector. Moreover, we set a hyperparameter τ as the fixed size of vectors corresponding to code snippets for subsequent model training. When the number of tokens in the code snippet is greater than τ, it will be truncated; otherwise, it will be filled with 0.

To learn richer contextual information in the code snippets, we introduce a model based on Transformer-Encoder as the next stage. The encoder is composed of six identical layers. Each layer has two sub-layers. The first is a multi-head selfattention mechanism, and the second is a position-wise fully connected feed-forward network¹⁵. Transformer has been proven to achieve surprising results on multiple natural language processing tasks¹⁶. Source code vulnerability detection can also be seen as an NLP task. After that, we use a fully connected layer to map high-dimensional vectors to lowdimensional ones for the final classification task.

3.3

Detector for unknown source code

Our system aims to determine whether the unknown Solidity source code is vulnerable. So, after the training phase, we design the detection phase. During the detection phase, we use an unknown Solidity source code as input. Same as the training phase, this source code is also extracted into multiple code snippets. Each of them will be tested through the trained neural network and then get a result. In other words, one program will get a list of results. If all the results are non-vulnerable, the source code is non-vulnerable. But even if there is only one vulnerable result, the source code is vulnerable.

4.1

Experimental settings

Dataset description. The first step of our experiment is to collect enough data. We use two datasets, including Smartbugs and Smartbugs-wild.1) Smartbugs. Smartbugs is a curated dataset that contains 143 annotated contracts with 208 tagged vulnerabilities that can be used to evaluate the accuracy of analysis tools. 2) Smartbugs-wild. Smartbugs-wild is a dataset with 47,398 unique contracts from the Ethereum network. But these smart contacts are not labelled. To solve this problem, we use Oyente to pseudo-label this dataset.

Implementation details. All experiments are conducted on a computer equipped with an Nvidia GeForce RTX 3090 GPU. The code snippets extraction is implemented with C++ (GCC version 9.2), while the neural networks are implemented with Python 3.7 and Pytorch 1.7.0. For training, we use 40,000 smart contracts from Smartbugs-wild and split them into 80% as a training set, 20% as a validation set. For evaluation, we use 4,000 smart contracts from Smartbugs-wild and all smart contracts from Smartbugs as the testing set. We use the Adam optimizer and cross-entropy for the classification loss. More details of experimental parameters can be found in our released source code.

4.2

Performance comparison

4.2.1

Experiments for answering RQ1.

In order to evaluate how effective the code snippets extraction module is. We conduct two experiments on the Smartbugs-wild dataset respectively. One contains the code snippets extraction module. Besides, for the sake of controlling the number of code snippets and maintaining the balance between positive and negative samples, we limit the generation of negative samples (i.e., do not contain vulnerabilities). In total, we extracted 62,523 code snippets from 40,000 source codes, consisting of 32,362 positive samples and 30,161 negative samples. The other just use the source code files directly.

We use the widely used Precision, Recall, F1-score, and Accuracy metric as the evaluation. As shown in Table 1, the code snippets extraction module can make the SVChecker more effective. The improvement in each of the metrics is substantial: 1.2% in Precision, 7% in Recall, 4% in F1-score, and 1.8% in Accuracy. Confusingly, our two experiments detected different numbers of positive and negative samples for the same 4,000 samples. In order to find out this reason, we conducted an in-depth analysis.

Table 1.

Performance comparison of SVChecker with code snippets extraction or not.

System	Number (P | N)	Precision	Recall	F1	Accuracy
SVChecker (without CSE)	4,000 (2,841 | 1,159)	91.36%	90.29%	90.90%	92.53%
SVChecker (with CSE)	4,000 (2,235 | 1,765)	92.53%	97.23%	94.82%	94.35%

Finally, we observe that the Oyente tool incorrectly judged the add function in SafeMath Library as vulnerable (Figure 3). However, our code snippets extraction module will not extract these pieces of code at all. This fact shows that this module in our system is very useful from another point of view.

Figure 3.

Possible bug found by Oyente.

4.2.2

Experiments for answering RQ2 and RQ3.

In order to answer RQ2, we compare the effectiveness of SVChecker with other vulnerability detection tools. At the same time, we use the Smartbugs dataset to test these tools. This dataset provides a collection of vulnerable Solidity smart contracts according to the DASP taxonomy and contains ten different types of vulnerabilities. We select six types of vulnerabilities for testing shown in Table 2. The first three types of vulnerabilities appeared in our training set, while the latter three did not. In this way, we can also answer RQ3. We use the report provided by Durieux⁵ as the results of Oyente, Securify, Slither and Smartcheck. We illustrate the performance of different tools in Table 2. It should be pointed out that the number of samples is different because of the upgrade of Smartbugs.

Table 2.

Performance of different tools on the Smartbugs dataset.

As we can see, the SVChecker has the best experimental result in detecting various types of vulnerabilities. And detection rate is far ahead of other detection tools. We make following observations. First, the deep learning-based system SVChecker outperforms the other pattern-based static analysis detection systems. Second, for the types of vulnerabilities that Oyente can detect, the detection rate of the SVChecker exceeds 80%. Even for the types of vulnerabilities that have not appeared in the training set, the SVChecker has a certain detection rate. We think it benefits from the powerful learning ability of the Transform-Encoder. In contrast, the detection effects of other detection tools perform poorly on Smartbugs.