Porovnanie verzií ChatGPT pre analýzu kódu

Od posledného článku, kedy sme sa venovali využitiu ChatGPT na analýzu kódu prešiel takmer rok. Medzitým sa technológia posunula a novšia verzia je vo viacerých smeroch lepšia. Je to ale tak aj pri analýze kódu?

V predchádzajúcom článku (Využitie nástroja ChatGPT na Secure Code Review) sme sa venovali možnému využitiu ChatGPT v 3.5 (z 3.8.2023) na vykonávanie bezpečnostných analýz kódu. Vysvetlili sme si, ako analýza kódu funguje, čo je ChatGPT a na čo sa používa. Na naše testovanie sme si vybrali framework ASP.NET (primárne C#). Na zistenie použiteľnosti ChatGPT na analýzu kódu sme použili viaceré útržky kódu, ktoré obsahovali často sa vyskytujúce zraniteľnosti. Napriek tomu, že ChatGPT častokrát správne dané zraniteľnosti nachádzal, mal aj určité nedostatky, ako napr. false positive nálezy, ale poukázali sme aj na etické problémy a iné. Naše zhodnotenie bolo nasledovné: „Vzhľadom na naše zistenia myslíme, že momentálne existujú lepšie riešenia ako ChatGPT na Secure Code Review.“

Od našeho článku prešiel takmer rok, a ChatGPT sa značne posunul. Najnovšia verzia (4) dokáže už vytvárať a interpretovať obrázky, a všeobecne sa javí ako lepšia. Pre zaujímavosť, ChatGPT 4 sme sa opýtali, v čom je v analýze kódu lepší ako jeho predošlá verzia. Jeho odpoveď bola nasledovná:

ChatGPT v3.5 and the latest version based on the GPT-4 architecture have several key differences that impact their capabilities, particularly in specialized tasks like secure code review. Here's a detailed breakdown of these differences and how they relate to secure code review:

Model Size and Training Data

  • GPT-3.5: Smaller than GPT-4, with fewer parameters. It was trained on a diverse range of internet text up to its cutoff in 2021.
  • GPT-4 (ChatGPT): Larger in terms of parameters, which enables it to understand and generate more complex texts with higher accuracy. It was trained on a broader dataset including more recent data up to its cutoff in late 2023.

Impact on Secure Code Review:

  • GPT-4: Better understanding of nuances in code and potential security vulnerabilities due to more training data and advanced language capabilities. Can generate more precise recommendations and recognize subtle security issues.
  • GPT-3.5: May not capture as many details or recognize complex vulnerabilities as effectively.

Contextual Understanding and Handling

  • GPT-3.5: Handles context well but has limitations in maintaining long context strings, which can be a challenge in understanding long scripts or codes.
  • GPT-4: Improved contextual memory and can maintain context over longer interactions. This is crucial in understanding entire codebases or lengthy scripts without losing track of earlier parts.

Impact on Secure Code Review:

  • GPT-4: More effective in reviewing large codebases or scripts that require understanding the context established much earlier in the text. This helps in identifying security issues that depend on multi-part interactions within the code.

Training Techniques

  • GPT-3.5: Trained using older versions of deep learning techniques.
  • GPT-4: Uses more refined training techniques, potentially including techniques like reinforcement learning from human feedback (though OpenAI has not detailed all specific advancements), which helps the model better align with user expectations and output more useful responses.

Impact on Secure Code Review:

  • GPT-4: Likely to produce more aligned and actionable advice on security practices, code smells, and potential vulnerabilities. Its responses are typically more tailored to user queries, providing more relevant and context-aware insights.

Error Rates and Reliability

  • GPT-3.5: Generally reliable but has a higher error rate in complex tasks compared to GPT-4.
  • GPT-4: Demonstrates reduced error rates across a range of tasks and offers more nuanced understanding, which is essential in security-sensitive areas like secure code review.

Impact on Secure Code Review:

  • GPT-4: Reduced likelihood of overlooking subtle security risks. Offers more reliable and accurate security insights, especially in complex code environments.

Specificity and Detail in Responses

  • GPT-3.5: Provides detailed responses but may not always tailor its advice to the specificities of a query as effectively as GPT-4.
  • GPT-4: Improved ability to generate specific and highly detailed responses, crucial for addressing specific security concerns in code review.

Impact on Secure Code Review:

  • GPT-4: Better at providing specific guidance and explanations for security vulnerabilities, including how to fix them, which significantly aids in the secure code review process.

Overall, GPT-4's advancements make it a more robust tool for secure code review, enhancing its ability to understand complex code structures, maintain context, and provide accurate, detailed security insights.

ChatGPT svoju novú verziu veľmi vychválil. Nás zaujímajú avšak hlavne nasledujúce body:

  • ... generate more complex texts with higher accuracy. It was trained on a broader dataset including more recent data up to its cutoff in late 2023 ...
  • likely to produce more aligned and actionable advice on security practices, code smells, and potential vulnerabilities
  • reduced error rates across a range of tasks

Tieto body naznačujú, že nová verzia by mala byť oveľa schopnejšia rozoznávať zraniteľnosti v kóde, a takisto k nim uvádzať lepšie sumarizácie. Najjednoduchšie, ako otestovať, či sa tieto tvrdenia zakladajú na pravde alebo nie, je otestovať novú verziu ChatGPT na rovnakých útržkoch zraniteľného kódu, aké sme použili v predošlom článku.

ChatGPT 4 takisto obsahuje jednu novú funkcionalitu, ktorá sa pre potreby analýzy kódu javí ako veľmi prínosná – custom GPT. Custom GPT umožňuje si „naprogramovať“ svoj vlastný model. Týmto dokážeme docieliť, aby bol úplne zameraný na analýzu zdrojového kódu.

Úloha znie jednoducho – vytvoriť si vlastného „secure code review analytika“. Požadujeme jeho nasledovné správanie:

  • žiadny text navyše
  • vypíš všetky zraniteľnosti (aj menej dôležité) – aby sme dosiahli konzistentné vypisovanie odpovedí, na túto pomlčku sme museli dať veľký dôraz. V opačnom prípade dochádzalo akoby k náhodnému výberu zraniteľností, ktoré vypíše.
  • k zraniteľnostiam maximálne 2 krátke vety, kde sú opísané
  • žiadne odporúčania (nie sú potrebné, pretože ich vieme vytvoriť sami)
  • požadovaný jazyk je C#, framework ASP.net
  • každý útržok kódu je osobitný celok a teda predchádzajúci naň nemôže mať vplyv

Vytvorenie daného modelu podľa nami určených pravidiel bol v princípe trial & error. Existujú „slová“, ktoré dokážu viac ovplyvniť daný model, aby robil to čo potrebujete, ale stále je potrebné testovať jeho výstup, kým nebude požadovanej kvality. Častokrát sa stáva, že aj keď je zadané, že ku každej zraniteľnosti chceme vypísať maximálne 2 krátke vety, toto pravidlo úplne odignoruje. V tom prípade sa nám ukázalo ako prínosné opísať tento problém a spýtať sa ho, ako by mal byť prompt upravený, aby sa už daný problém neopakoval.

Po vytvorení a otestovaní nášho custom GPT teda nasledovalo samotné veľké testovanie. V nasledujúcich častiach sa povenujeme porovnaniu samotných verzií s ohľadom na analýzu kódu. Do nášho modelu pošleme rovnaký kód ako do pôvodného a následne porovnáme výstupy jednotlivých verzií. Toto porovnanie bude vo forme tabuľky a vysvetlenia, ak bude potrebné. Vysvetlením jednotlivých zraniteľností sa nebudeme v tomto článku zaoberať, keďže boli opísané v predchádzajúcom článku.

Input Validation - SQL Injection

Describe any security vulnerabilities in the code below:
string strcon = ConfigurationManager.ConnectionStrings["users"].ConnectionString;
SqlConnection con = new SqlConnection(strcon);
string qry = "SELECT * FROM TABLE WHERE Email='" + Login1.Email + "'and Password='" + Login1.Password + "' ";
adpt = new SqlDataAdapter(qry, con);
test = new DataTable();
adpt.Fill(test);
if (test.Rows.Count >= 1) {
  Response.Redirect("admin.aspx");
}
  1. SQL Injection: The query construction uses string concatenation incorporating user input (Login1.Email and Login1.Password), which makes it vulnerable to SQL injection. An attacker could manipulate the input to execute arbitrary SQL commands.
  2. Sensitive Data Exposure: The application handles sensitive data (passwords) and might be transmitting or storing it as plain text, which is insecure. Passwords should be hashed and compared using secure methods.
  3. *Use of SELECT : The SQL query uses SELECT *, which is inefficient as it retrieves all columns from the table regardless of the actual need. This can lead to unnecessary data exposure and increased load on the database server.
  4. Lack of Input Validation: There's no visible input validation for Login1.Email and Login1.Password. Validating inputs can mitigate several issues, including SQL injection and malformed input data.
  5. Error Handling: The code does not handle potential exceptions that could be thrown while accessing the database or during the query execution, such as SqlException. This could lead to unhandled exceptions causing application crashes or inappropriate error messages displayed to users.
  6. Direct Database Connection: The application creates a direct connection to the database using connection strings, which can be less secure if not handled properly (e.g., not using connection pooling, exposing connection details).
  7. Redirection Handling: The use of Response.Redirect("admin.aspx") based solely on a row count check could be insecure if not combined with proper authentication checks, potentially allowing unauthorized access if the condition is bypassed or manipulated.
  8. Hardcoded SQL Command: The query string is hardcoded with dynamic parts appended, which not only makes it prone to SQL injection but also makes the code harder to maintain and audit.

V nasledujúcej tabuľke porovnávame nálezy zistené obidvoma verziami:

3.5 4
SQL Injection
Sensitive Data Exposure
Use of Select ×
Lack of Input Validation
Error Handling
Direct Database Connection ×
Redirection Handling ×
Hardcoded SQL Command ✔ (FP) ✔ (FP)
Lack of Authentication and Authorization ×

Ako môžeme vidieť, novšia verzia (4) pridala nálezy ako Use of Select, Direct DB Connection, ktoré predchádzajúca verzia nespomenula. Kupodivu avšak nespomenula nález hovoriaci o autorizácii, ktorý bol spomenutý len v staršej verzii. Obidve verzie hovoria o Hardcoded SQL Command, ktorá je v našom prípade False Positive.

Describe any security vulnerabilities in the code below:
string[] bl = new string[] {
  "<SCRIPT>",
  "--",
  "'"
};
string password = Login1.password;
string email = Login1.email;
foreach(var bl_input in bl) {
  if (password.ToUpper().Contains(bl_input) || email.ToUpper().Contains(bl_input)) {
    Response.Redirect("default.aspx");
  }
}
string strcon = ConfigurationManager.ConnectionStrings["users"].ConnectionString;
SqlConnection con = new SqlConnection(strcon);
string qry = "SELECT * FROM TABLE WHERE Email='" + email + "'and Password='" + password + "' ";
adpt = new SqlDataAdapter(qry, con);
test = new DataTable();
adpt.Fill(test);
if (test.Rows.Count >= 1) {
  Response.Redirect("index.aspx");
}
  1. Ineffective Blacklisting: The code attempts to block SQL and XSS injections by checking if the password or email includes certain characters or strings (like <SCRIPT>, --, "). However, this method is ineffective as it is case-sensitive and can be easily bypassed by changing the case or using different but equivalent encodings or variations.
  2. SQL Injection: Despite the attempt at blacklisting certain characters, the email and password are directly concatenated into the SQL query, making it vulnerable to SQL injection if the attacker uses characters or strings not included in the blacklist.
  3. Sensitive Data Exposure: Handling passwords directly within the query suggests that they might be stored in plaintext in the database, which is a severe security risk as it exposes sensitive data to anyone who can access the database.
  4. *Use of SELECT : The SQL query uses SELECT *, which is inefficient as it retrieves all columns from the table regardless of the actual need. This practice can lead to unnecessary data exposure and increased load on the database server.
  5. Lack of Input Validation: The code lacks effective input validation mechanisms, relying only on a weak blacklisting approach that is easy to bypass. Proper validation and sanitization of inputs are necessary to prevent SQL injection and other forms of input manipulation.
  6. Error Handling: The code does not handle potential exceptions that could be thrown while accessing the database or during the query execution, such as SqlException. This can lead to unhandled exceptions that may disrupt application flow or reveal sensitive information.
  7. Direct Database Connection: The direct usage of database connections can lead to issues if not properly managed, such as resource leaks if the connection is not closed after use, or if it exposes connection strings containing sensitive information.
  8. Redirection Logic: The logic for redirection based on the presence of blacklisted characters or successful authentication is simplistic and does not account for more sophisticated access control and session management mechanisms, potentially allowing for unauthorized access.
  9. Hardcoded SQL Command: Similar to the previous issue, the hardcoded SQL command with appended user inputs makes the system vulnerable and difficult to maintain or secure effectively.

V nasledujúcej tabuľke porovnávame nálezy zistené obidvoma verziami:

3.5 4
Ineffective Blacklisting
SQL Injection ✔ (FP) ✔ (FP)
Sensitive Data Exposure ✔ (FP) ✔ (FP)
Use of SELECT ×
Lack of Input Validation
Error Handling
Direct Database Connection ×
Redirection Logic ×
Hardcoded SQL Command

Aj v tomto prípade môžeme vidieť, že novšia verzia opäť odhalila viacero zraniteľností, a tým pádom poskytla lepší výsledok ako predchádzajúca verzia. Jedná sa o viacmenej rovnaké nálezy ako v predchádzajúcom bode.

Sensitive information leakage

Describe any security vulnerabilities in the code below:
private void Page_Error(object sender, EventArgs e) {
  Exception ex = Server.GetLastError();
  Response.Write("Following error has occured: " + ex.StackTrace);
  Context.ClearError();
}

private void Page_Error(object sender, EventArgs e) {
  Response.Write("Following error has occured: " + Server.GetLastError().StackTrace);
  Context.ClearError();
}

private void Page_Error(object sender, EventArgs e) {
  Exception ex = Server.GetLastError();
  Response.Write("Following error has occured: " + ex.ToString());
  Context.ClearError();
}
  1. Information Disclosure: All three versions of the Page_Error method expose detailed error information to the user by writing the stack trace or exception details directly to the response. This can reveal sensitive application or system information to potential attackers, which could be used to craft further attacks.
  2. Lack of Proper Error Handling: The methods capture and display errors without any logging or more secure handling, such as redirecting to a generic error page. This approach not only risks exposing details about the underlying architecture but also fails to provide a means to review and analyze the errors systematically for ongoing security or functionality improvements.
  3. Redundant Code/Functionality: The presence of multiple versions of the Page_Error handler suggests redundant or overlapping functionality that can lead to confusion and maintenance issues. Additionally, this might indicate inconsistencies in how errors are handled across different parts of the application.
  4. Inconsistent Error Management: In the last version of the method, the entire exception object is converted to a string and written to the response, which can include even more detailed information than the stack trace alone, such as exception messages and inner exception details. This inconsistency in error handling could lead to varying levels of information exposure depending on which method executes.
  5. Security Misconfiguration: The error handling methods suggest a potential misconfiguration or inadequate secure coding practices, as they directly output error details to the end user. A better practice would be to handle errors in a way that informs the user of the error without revealing any technical details.

V nasledujúcej tabuľke porovnávame nálezy zistené obidvoma verziami:

3.5 4
Information Disclosure
Lack of Proper Error Handling
Redundant Code/Functionality ×
Inconsistent Error Management ×
Security Misconfiguration ×
Premature Error State Clearing ×

V tomto prípade môžeme vidieť, že verzia 4 našla oveľa viac zaniteľností. Po ich prečítaní avšak zistíme, že nálezy Redundant Code/Functionality, Inconsistent Error Management a Security Misconfiguration existujú len preto, že bolo spojených viacero funkcií spolu. V reálnom svete by sa takýto kód nevyskytol, a teda to nepovažujeme za vadu. Je však dobré, že si to ChatGPT všimol.

Describe any security vulnerabilities in the code below:
<?xml version="1.0" encoding="UTF-8"?><configurationxmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform"><runtime><ThrowUnobservedTaskExceptionsenabled="false"/></runtime><system.web><customErrorsdefaultRedirect="GenericError.htm"mode="Off"xdt:Transform="Replace"><errorstatusCode="500"redirect="InternalError.htm"/></customErrors></system.web></configuration>
  1. Improper Error Handling Configuration: The customErrors mode is set to "Off", which typically means that detailed error messages, including stack traces, may be displayed to end-users. This setting can reveal sensitive application details that could assist an attacker.
  2. Security Misconfiguration: The ThrowUnobservedTaskExceptions setting in the <runtime> section is disabled (enabled="false"). This configuration can potentially suppress important exceptions that should be logged and reviewed, leading to unnoticed issues that might impact application stability or security.
  3. Inconsistent Use of Custom Error Pages: Although a default redirect for custom errors is specified (defaultRedirect="GenericError.htm"), the explicit configuration under customErrors to redirect to InternalError.htm for a 500 status code error might be ignored due to the mode="Off" setting. This inconsistency could lead to a confusing user experience and improper error handling.
  4. Potential XML Namespace Issue: The configuration tag appears to contain a typo or formatting issue with its namespace definition (xmlns:xdt). If malformed, this could lead to issues parsing the XML, particularly when transformations are supposed to be applied (xdt:Transform="Replace"), which might not work as expected.

V nasledujúcej tabuľke porovnávame nálezy zistené obidvoma verziami:

3.5 4
Improper Error Handling Configuration
Security Misconfiguration
Inconsistent Use of Custom Error Pages
Potential XML Namespace Issue ×
Lack of Input Validation ×
XDT Transformation Security ×

Pri tomto kóde sa prvýkrát stretávame s tým, že staršia verzia odhalila viacero zraniteľností. Avšak tieto nálezy, ktoré sú navyše sú veľmi diskutabilné, rovnako ako nález „Potential XML Namespace Issue“, ktorý vznikol len kvôli chybe v kóde.

Cryptographic Flaws

Describe any security vulnerabilities in the code below:
 string key = "hashpassword";
 byte[] keybytes = UTF8Encoding.UTF8.GetBytes(key);
 SymmetricAlgorithm enc = SymmetricAlgorithm.Create("DES");
 enc.Key = keybytes;
 enc.Mode = CipherMode.ECB;
 enc.Padding = PaddingMode.Zeros;
 enc.KeySize = 32;
 enc.BlockSize = 32;
  1. Weak Encryption Algorithm: The code uses DES (Data Encryption Standard) which is considered outdated and insecure due to its short key length. DES is vulnerable to brute-force attacks because of its 56-bit key size, making it unsuitable for secure applications.
  2. Improper Key Size and Block Size Configuration: The KeySize and BlockSize are set to 32 bits, which is incorrect for DES. DES uses a 64-bit block size (56-bit key size technically, with 8 bits used for parity). Setting these properties incorrectly can lead to runtime errors or the framework defaulting to values that may not be secure.
  3. ECB Mode Usage: The code configures the cipher mode to ECB (Electronic Codebook), which is insecure for encrypting data that spans more than one block. ECB mode does not use an initialization vector (IV) and encrypts identical plaintext blocks into identical ciphertext blocks, making it vulnerable to pattern analysis and revealing data patterns.
  4. Insecure Padding Scheme: Padding mode is set to PaddingMode.Zeros, which can lead to security weaknesses in certain scenarios. Zero padding can be problematic if the plaintext message ends with zero bytes because it might be stripped incorrectly during decryption.
  5. Improper Usage of Encoding for Key: The key is derived directly from a UTF-8 encoded string without a secure key generation mechanism like hashing or a key derivation function, leading to potential weaknesses in the cryptographic strength of the key.

V nasledujúcej tabuľke porovnávame nálezy zistené obidvoma verziami:

3.5 4
Weak Encryption Algorithm
Improper Key Size and Block Size Configuration
ECB Mode Usage
Insecure Padding Scheme
Improper Usage of Encoding for Key
Lack of Initialization Vector (IV) ×
Missing Error Handling ×
Deprecated API Usage ×

V tomto prípade viac nálezov poskytla staršia verzia ChatGPT. Aj keď nie sú kriticky závažné, je zaujímavé, že aj napriek tomu, že bolo vyslovene od novšej verzie žiadané vypísanie všetkých nálezov, tak tieto nespomenula.

Malicious File execution

Describe any security vulnerabilities in the code below:
protected void Upload_Click(object sender, EventArgs e) {
  if (file1.HasFile) {
    file1.SaveAs(Server.MapPath("~/upload/") + file1.FileName);
    file1Label.Text = "Uploaded file: " + file1.PostedFile.FileName;
  } else {
    file1Label.text = "No file chosen."
  }
}
  1. Arbitrary File Upload Vulnerability: The code saves files uploaded by users directly to a server directory using the original file name (file1.FileName). This practice can lead to arbitrary file uploads where an attacker might upload a malicious file (like a script or executable) that could be executed on the server or serve as a vector for further attacks.
  2. Path Traversal/Directory Traversal: By directly appending the user-provided filename to a directory path, the application is vulnerable to path traversal attacks. An attacker could potentially manipulate the filename to include directory traversal characters (../) to access or overwrite files in unauthorized directories.
  3. Lack of File Type Validation: The code does not perform any checks on the type of the file being uploaded. Without validation on the MIME type or file extension, malicious files could be uploaded, posing a significant security risk.
  4. Potential Disclosure of Internal Path: The label file1Label is updated to display the file name of the uploaded file, which could inadvertently expose the internal path or structure of the file system if manipulated. This information could be used by an attacker to further explore potential vulnerabilities.
  5. No Size Limitation on Uploads: The code does not check the size of the files being uploaded. This omission can lead to denial-of-service attacks if large files are uploaded, potentially exhausting server resources.

V nasledujúcej tabuľke porovnávame nálezy zistené obidvoma verziami:

3.5 4
Arbitrary File Upload Vulnerability ×
Path Traversal/Directory Traversal
Lack of File Type Validation
Potential Disclosure of Internal Path
No Size Limitation on Uploads
Missing Authentication and Authorization ×
Missing Error Handling ×

Kým v predchádzajúcej verzii bol výstup akýmsi sklamaním, pretože zraniteľnosť Remote Code Execution bola spomenutá až nepriamo v 5. bode a to v spojení nevalidovania typu súboru. Novšia verzia túto zraniteľnosť spomenula hneď ako prvú, čo sme si aj predstavovali. Sklamaním bolo naopak avšak nespomenutie autentizácie, autorizácie a spravovania chýb, ktoré staršia verzia správne odhalila.

Iné zraniteľnosti

Describe any misconfigurations that could lead to security vulnerabiltiies on the web.config file below:‎ 
<?xml version="1.0" encoding="UTF-8"?><configuration><configSections><sectionname="entityFramework"type="System.Data.Entity.Internal.ConfigFile.EntityFrameworkSection, EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"requirePermission="false"/></configSections><system.web><httpRuntimerequestValidationMode="2.0"/><pagesvalidateRequest="false"/><customErrorsmode="Off"defaultRedirect="Default.aspx"/><traceenabled="true"localOnly="false"/><compilationdebug="true"targetFramework="3.0"/><pagesvalidateRequest="false"/><authenticationmode="Forms"><formsname=".ASPXAUTH"loginUrl="login.aspx"defaultUrl="test.aspx"requireSSL="false"cookieless="UseUri"slidingExpiration="true"timeout="1000"protection="None"><credentialspasswordFormat="Clear"><username="AEC1"password="verystrong"/></credentials></forms></authentication><membershipdefaultProvider="SqlProvider"/><httpCookieshttpOnlyCookies="false"requireSSL="false"/><roleManagercookieRequireSSL="false"/><sessionStateregenerateExpiredSessionId="false"/><machineKey>validationKey="AutoGenerate,IsolateApps" decryptionKey="Autogenerate,IsolateApps" validation="MD5" decryption="DES"</machineKey><compilationdebug="true"targetFramework="4.8"/><httpRuntimetargetFramework="4.8"/><pagesvalidateRequest="false"><namespaces><addnamespace="System.Web.Optimization"/></namespaces><controls><addassembly="Microsoft.AspNet.Web.Optimization.WebForms"namespace="Microsoft.AspNet.Web.Optimization.WebForms"tagPrefix="webopt"/></controls></pages><compilationdebug="true"/></system.web><runtime><assemblyBindingxmlns="urn:schemas-microsoft-com:asm.v1"><dependentAssembly><assemblyIdentityname="Antlr3.Runtime"publicKeyToken="eb42632606e9261f"/><bindingRedirectoldVersion="0.0.0.0-3.5.0.2"newVersion="3.5.0.2"/></dependentAssembly><dependentAssembly><assemblyIdentityname="Microsoft.Web.Infrastructure"publicKeyToken="31bf3856ad364e35"/><bindingRedirectoldVersion="0.0.0.0-2.0.1.0"newVersion="2.0.1.0"/></dependentAssembly><dependentAssembly><assemblyIdentityname="Newtonsoft.Json"publicKeyToken="30ad4fe6b2a6aeed"/><bindingRedirectoldVersion="0.0.0.0-12.0.0.0"newVersion="12.0.0.0"/></dependentAssembly><dependentAssembly><assemblyIdentityname="WebGrease"publicKeyToken="31bf3856ad364e35"/><bindingRedirectoldVersion="0.0.0.0-1.6.5135.21930"newVersion="1.6.5135.21930"/></dependentAssembly></assemblyBinding></runtime><system.codedom><compilers><compilerlanguage="c#;cs;csharp"extension=".cs"type="Microsoft.CodeDom.Providers.DotNetCompilerPlatform.CSharpCodeProvider, Microsoft.CodeDom.Providers.DotNetCompilerPlatform, Version=2.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"warningLevel="4"compilerOptions="/langversion:default /nowarn:1659;1699;1701"/><compilerlanguage="vb;vbs;visualbasic;vbscript"extension=".vb"type="Microsoft.CodeDom.Providers.DotNetCompilerPlatform.VBCodeProvider, Microsoft.CodeDom.Providers.DotNetCompilerPlatform, Version=2.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"warningLevel="4"compilerOptions="/langversion:default /nowarn:41008 /define:_MYTYPE=\&quot;Web\&quot; /optionInfer+"/></compilers></system.codedom><entityFramework><providers><providerinvariantName="System.Data.SqlClient"type="System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer"/></providers></entityFramework></configuration>
  1. Custom Error Mode: The customErrors mode is set to "Off", which can lead to detailed error messages being displayed to the end users. This can reveal sensitive information about the application, its structure, and underlying technologies which can be used by attackers.
  2. Tracing Enabled: The configuration has tracing enabled (traceenabled="true" with localOnly="false"). This could potentially expose sensitive information about the application's internal workings and user data if accessed remotely.
  3. Debug Mode: The compilation section has debug="true" enabled, which should be disabled in a production environment. Debug mode can significantly impact performance and may reveal detailed error messages and other sensitive debugging information.
  4. Weak Cryptography: The machineKey section is using outdated cryptographic algorithms (validation="MD5" and decryption="DES"). MD5 and DES are considered weak due to vulnerabilities that allow them to be compromised relatively easily.
  5. Forms Authentication Weaknesses: The forms authentication is configured with several insecure settings:
    • protection="None" which means that the authentication cookie is not protected against tampering.
    • requireSSL="false" which allows authentication cookies to be transmitted over non-HTTPS connections, susceptible to interception (e.g., Man-in-the-Middle attacks).
    • cookieless="UseUri" which can lead to session fixation attacks as the session identifier is exposed in the URL.
    • passwordFormat="Clear" stores credentials in clear text within the configuration file, which is insecure.
  1. HTTP Cookies Security: The httpCookies configuration is insecure with httpOnlyCookies="false" and requireSSL="false", which makes cookies susceptible to client-side scripts (increasing XSS risks) and interception over insecure connections, respectively.
  2. Role Manager Configuration: The roleManager is configured with cookieRequireSSL="false", which could lead to the role cookie being transmitted insecurely.
  3. Session Management: The sessionState configuration has regenerateExpiredSessionId="false" which can contribute to session fixation vulnerabilities.
  4. Repeated Insecure Directives: There are multiple instances where validateRequest="false" is set, which disables ASP.NET's built-in request validation for detecting potentially dangerous inputs from being processed, increasing the risk of XSS attacks.

V nasledujúcej tabuľke porovnávame nálezy zistené obidvoma verziami:

3.5 4
Custom Error Mode
Tracing Enabled ×
Debug Mode
Weak Cryptography
Forms Authentication Weaknesses
HTTP Cookies Security
Role Manager Configuration
Session Management ×
Repeated Insecure Directives
Outdated and Vulnerable Dependencies ×
Obsolete CodeDom Compiler Configuration ×
Disclosure of Entity Framework Configuration ×

Aj v tomto prípade sa stalo podobné ako v predchádzajúcich. Novšia verzia odhalila zraniteľnosti, s ktorými sme mali problém a ktoré sme očakávali, že odhalené budú, ale zabudla na nejaké, ktoré staršia verzia vypísala

Záver

V tomto článku sme porovnali ChatGPT verzie 3.5 a 4 s ohľadom využitia na secure code review. Môžeme konštatovať, že novšia verzia častokrát poskytla presnejšie a faktickejšie výsledky. Avšak vo viacerých prípadoch prekvapila naopak staršia verzia, ktorá nálezov vypísala viac. Počas písania článku sa nám nepodarilo prinútiť novšiu verziu naozaj vypísať všetky nálezy. Stretávali sme sa s prípadom, že keď sme sa spýtali, či na daný nález nezabudol, tak ho tam pridal, avšak pri analýze kódu je potrebné, aby to trafil na prvýkrát. Takisto napriek tomu, že sme mali vytvorený vlastný custom GPT, odpovede sa nedajú považovať za úplne konzistentné. Úprava promptu a výslovné pokyny tento problém dosť zlepšili, ale stále tam vidíme priestor na zlepšenie.

Ostáva zodpovedať otázku, ktorú sme si kládli aj v prvom článku - oplatí sa už využívať ChatGPT na automatizovanú analýzu kódu? Aj keď môžeme povedať, že sa vo všeobecnosti model zlepšil, ostatné problémy využitia ChatGPT na analýzu kódu teda zostávajú viacmenej rovnaké - limit na počet znakov, false positive, zdieľanie kódu. Čo sa ale týka len technickej stránky, novšia verzia je pre analýzu kódu lepšia aj napriek miernym nedostatkom. Stále si avšak myslíme, že sú lepšie a konzistentnejšie metódy.