Enabling a true “shift-left” in software security by empowering DevOps and security teams to work together

In some recent articles I shared how you as a developer can add security to your skillset by using New Relic capabilities. I also dug deeper into ways on how to mitigate hidden security risks of open source software libraries. Both of these blogs focused on third-party code and how it can impact the security of your software applications. In this article I will focus on the security of your own custom code, i.e. the code that you write yourself.

In general, there are various methods you can employ to ensure the security of your first-party code. One common approach involves utilizing static code analysis tools, which scan your source code prior to deployment in different environments like testing, QA, and production. This practice proves valuable as it prevents the inclusion of hard-coded secrets in your repository or software builds. However, such techniques only offer insight into the static aspects of your application.

To address this limitation, the industry often relies on dynamic application security tools, which perform black-box tests by examining your application from an external perspective. While this approach provides an additional perspective, it still only reveals one facet of the overall security situation. The crux of the matter is that neither of these solutions offers a comprehensive view.

New Relic Interactive Application Security Testing (IAST) is precisely designed to address this issue. With IAST, you can enhance your observability as New Relic’s tools already provide exceptional visibility into the dynamic aspects of your application from within. The nature of Application Performance Monitoring (APM) is that we observe the code as it’s executing on the systems. That means we not only see the performance and errors, but we see which version of code has been deployed to each instance of the service and which methods are actually being triggered based on the real time activity in the application itself.

In the observation of your services, application security concerns, arising from both first-party and third-party sources, are increasingly crucial. New Relic IAST plays a key role in this context, as it effectively eliminates ambiguity by rapidly detecting vulnerabilities in your applications. It provides conclusive evidence of exploits and their precise locations within your code. Additionally, the tool offers actionable and repeatable testing procedures and mitigation measures, empowering you to efficiently address security issues in pre-production or development environments before they become exploitable. Let me give you an example of what we mean by actionable and repeatable: for every security test, there’s an associated curl command (or other form of test script) so that you can go back and re-test as you fix your code. The same test script can then be used to verify it’s truly fixed without running the entire set of tests all over again.

As the first step towards provable security, New Relic IAST enables a true “shift-left” in software security by empowering DevOps and security teams to work together to accelerate CI/CD pipelines by empowering organizations to focus on innovation while securing your applications early in the development lifecycle.

Another key benefit of focusing on exploitable vulnerabilities is that traditional application security tools often lack context, leading to an overwhelming number of alerts that can confuse DevOps and security teams, making it difficult to identify immediate security risks and verify security improvements. Basically, New Relic IAST eliminates the noise (false positives) found in many other solutions and provides proof-of-exploit for validation.

I said a lot of things, so let me now show you how New Relic IAST works in practice.

Please note: Before I go ahead and share how to get started with New Relic IAST, I want to point out an important aspect. New Relic IAST simulates a battery of real-world attacks within your application. This means that test data will be used to simulate and test for exploitable vulnerabilities. For that reason, New Relic IAST should be used in pre-production environments only.

New Relic language agents

The newest New Relic APM agents now include the IAST capability. We all know how easy it is to add APM agents to an app to identify bugs and performance issues. Have a look at this intuitive example for a .NET environment. Now, with no extra effort, we can also test your custom code for OWASP exploitable vulnerabilities, such as injection attacks. In my opinion, this is an incredible benefit of New Relic IAST and many stakeholders will love it:

IT, because they do not have to deploy yet another agent Security teams, because this turns development teams into a security force multiplier You as a developer, because it is already part of your tool chain you already use. This capability is very easy to implement. In my example, I am using the OWASP Juice Shop application again as a demo environment.

Juice Shop is built with Node.js and so I go ahead and add the latest New Relic Node.js agent (IAST was added in version v10.3.0) to my application.

We follow the well-known approach of running the following command inside of our repository:

npm install newrelic

This installs the New Relic agent library into our application. The easiest way to configure the agent is by defining these environment variables:

NEW_RELIC_APP_NAME=juice-shop-mysql
NEW_RELIC_LICENSE_KEY=<your New Relic license key>

You can now install the rest of the dependencies of Juice Shop with npm install and run the application by npm start. After a short while, data will appear in your New Relic account in the familiar curated experience for APM.

Without any further configuration, the agent will capture all of the libraries used within your application and check for any known vulnerabilities. This data is published into the Vulnerability Management section of your service in New Relic.

Enable New Relic IAST

Now that we have the agent up and running and already reporting vulnerabilities for third-party libraries, let’s continue with enabling the IAST capabilities.

I stop the application and I will go ahead and add two additional environment variables:

NEW_RELIC_SECURITY_ENABLED=true
NEW_RELIC_SECURITY_AGENT_ENABLED=true

The first environment variable determines whether the security data is sent to New Relic or not. When this is disabled and NEW_RELIC_SECURITY_AGENT_ENABLED is true, the security module will run but data will not be sent. The second allows you to enable or disable all security functionality in general.

After we have defined these, there is nothing else for us to configure. The agent will handle the rest for us once the application is executed again.

Simulation of real-world attacks

So now that the application is running again (by executing npm start again), we can execute our regular unit tests. Within the Juice Shop repository, all of the unit tests are located in the test folder. For example, you can execute the API tests by running npm run test:api. Just by triggering these tests, the agent will identify the context of the request, i.e. a transaction in the New Relic context including all details of parameters, payload, etc.

For example, our Juice Shop includes an API for logging in the users of the application. This endpoint receives some payload including the user credentials for the authentication of the user.

Let’s use the application and try to login to the application with an existing user.

After a short while you will see some activity from our agent hitting your application. If an issue is identified by the agent, it will add a vulnerability to New Relic’s Vulnerability Management section. The big difference is that it will only do so, if the vulnerability is actually exploitable.

Let’s have a look at one example.

You can identify exploitable vulnerabilities with an exclamation point in the Exploitable column. This means that we not only found an issue, but we also have identified a proof-of-exploit for these. Another key aspect when analyzing the vulnerabilities is that this list of vulnerabilities is prioritized based on severity.

You click on the SQL Injection issue for more information on the issue and New Relic’s unique capability to provide proof of exploitability. Here, users will find a detailed description of the vulnerability and specific instances of this issue within your first-party code.

By clicking on one of the SQL Injection instances, you will get further information about the location within your source code and how to fix it.

Fixing an Exploitable Vulnerability

One of the key features of New Relic IAST is the provided replay test that New Relic provides. In the Test section of the screen you will find a test script that you can use to prove the exploitability. In our case, we will run this script to prove that SQL Injection is actually a real and exploitable issue in Juice Shop.

Let’s open up a terminal and run the following command that we copy from the New Relic UI. Please note: I added the full URL in the first line of the script. I also slightly adjusted the data-raw parameter.

curl --location --request POST 'https://wxmqzkpgeu.us-east-1.awsapprunner.com/rest/user/login' \
--header 'x-forwarded-proto:https' \
--header 'user-agent:Mozilla/5.0 (Macintosh;Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.67;' \
--header 'accept:application/json, text/plain, */*' \
--header 'sec-ch-ua-platform:"macOS"' \
--header 'sec-ch-ua-mobile:?0' \
--header 'sec-fetch-site:same-origin' \
--header 'accept-encoding:gzip, deflate, br' \
--header 'accept-language:de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6;' \
--header 'host:wxmqzkpgeu.us-east-1.awsapprunner.com' \
--header 'cookie:language=en;welcomebanner_status=dismiss;cookieconsent_status=dismiss;' \
--header 'origin:https://wxmqzkpgeu.us-east-1.awsapprunner.com' \
--header 'x-envoy-expected-rq-timeout-ms:120000' \
--header 'content-type:application/json' \
--header 'sec-fetch-dest:empty' \
--header 'x-request-id:dfc1b83b-5e24-4cb6-877b-20913a9cd54f' \
--header 'sec-ch-ua:"Not.A/Brand";v="8", "Chromium";v="114", "Microsoft Edge";v="114";' \
--header 'sec-fetch-mode:cors' \
--header 'x-forwarded-for:79.252.203.104' \
--header 'referer:https://wxmqzkpgeu.us-east-1.awsapprunner.com/' \
--header 'x-envoy-external-address:79.252.203.104' \
--data-raw '{"email": "1'"'"' OR TRUE; #","password":"12345"}'

After this script is executed, you will notice that we get some JSON data back with information from the Juice Shop admin user. This is of course not what the application should provide and proves that we actually have a critical issue and risk associated with it.

The vulnerability location provided by New Relic IAST shows us that one of the SQL Injection issues is located in /rest/user/login.

Let’s have a look into the source code of the application and open the server.ts file (in the root of the repository). Somewhere around line 543, you will see what method is actually being called when someone hits the /rest/user/login endpoint. The method login() is executed and this can be found in /routes/login.ts. As a developer, I can see the SQL query being concatenated by adding the user input directly into the SQL statement (line 36).

models.sequelize.query(`SELECT * FROM Users WHERE email = '${req.body.email || ''}' AND password = '${security.hash(req.body.password || '')}' AND deletedAt IS NULL`, { model: UserModel, plain: true })

This is definitely not best practice and developers should avoid building SQL queries like that.

Ideally you would use parameterized queries. In the sequelize framework this is referred to as replacements. So ideally we should use parameters in the query itself and replace them with the content using the replacements concept. The updated code should look something like this:

models.sequelize.query(`SELECT * FROM Users WHERE email = :email AND password = :password AND deletedAt IS NULL`,
      {
        model: UserModel,
        plain: true,
        replacements: {
          email: req.body.email || '',
          password: security.hash(req.body.password || '')
        }
      })

Let’s go ahead, fix this code issue and redeploy the application. Now that our new release of our app is up and running and data is reported into New Relic, we can re-run our test curl script again.

Once this is finished executing, we do not get the same result from before, i.e. the Juice Shop admin user credentials. But, we get a message that we provided an Invalid email or password. This is actually what we are expecting and proves that our fix within our first-party source code actually fixed the bug.

If we now switch back to our New Relic user interface and browse to the Vulnerability Management section of our application, you will notice that we still identify the two exploitable vulnerabilities. However, we also notice that the SQL Injection issue was last detected about 4 hours ago, but the Reflected XSS issue still exists since it was again reported after our deployment.

Conclusion

What I showed you in this example is a full cycle of running an interactive application security scan by looking at your application and executing tests to find exploitable vulnerabilities.

We not only identified an issue, but also received certain instances with test scripts that we could replay to actually prove that identified issues are in fact exploitable. After fixing the issue in our first-party code, we built a new version of our application and deployed it. This time the tests of our New Relic IAST agent were not able to successfully execute for this specific area within our application. I was able to prove the successful fix by running the same test script and validate that this is no longer an issue in our application.

Next steps

Try New Relic IAST today. If you have a free New Relic account, you already have access to New Relic IAST if you’re using a supported agent. Read the docs to see how to get started.

Don’t have a New Relic account yet? Sign up for free today. Your free account includes 100 GB/month of data ingest and one full user.

• monitoring
• newrelic
• observability
• security
• vulnerability management
• iast
• devsecops
• appsec