A misleading published paper on Elixir security

Sad to see a paper like this.
Awesome reply and counterpoints to it!

What the hell is that paper about, the quality of the content is worse than my laboratory work in the first year of the university, just repeating 100 times that we extracted commits from some random projects and found X vulnerabilities, without even showing a single example of a vulnerability.

With their methodology, I could hire 2 kids from school and search for words in commits with the same result:

image912×467 25.9 KB

This is just embarrassing

And by the way, when vulnerabilities are fixed, they are not marked as CRITICAL VULNERABILITY FIXED, HERE IS AN EXAMPLE if you want to replicate it, as vulnerabilities are things that can happen in any language/library.

Having quickly read the article cited above, my opinion is the following.

The research question is narrow and does appear to be part of a broader, established research field. It is unclear which conclusions for Elixir, the field of computer science or for security research the answering of the research question yields.

The paper does not seem to follow established methodology. What are the established metrics for measuring code quality and security issues ? What prior research exist in that field ? No papers on established methodology are quoted. Normally, a paper does either apply established methodology to a new dataset, or it applies new methodology to an already studied dataset. This study does not explain, which one it does.

The whole paper is short. It does not explore the topic in its depth nor in its breadth. Additionally, the few pages written are bloated with unnecessary information. This paper is not research, this is undergraduate homework.

How can the authors address these issues? I see two ways. Either (i) broaden the scope and explore the security aspect of Elixir in a comparative study with other languages. Does Elixir code show statistically different levels of security issues than other code ? Are there differences in the types of security issues ? Do code bases with high levels of security issues have further differences compared to code bases with lower levels ? Are these differences also observable for Elixir code ? Or (ii) deepen the research and explore which security problems can be identified for the specific case of Elixir code. What type of issues can be identified ? How extensive are they ? What can these issue say about the language and tools used ? What suggestion for improved code can be derived ?

Relax, some people write papers because they are mandatory to finish the university to get a grade. Whole academia is more about getting a degree, not about science or anything really useful.

I don’t think that these students are wrong. In my opinion, they are 100% correct: nobody cares about academic papers or their quality, young people just need a degree to have more opportunities in the future. I think that there should be more papers like this. This would destroy authority of universities in terms of defining competence

Nice to see this on HN. Not sure what kind of discussion it’ll spark but also nice it’s kept Elixir on the front page for another day