Graham King

Solvitas perambulum

Decrypt your HTTPS traffic with mitmproxy

software

tl;dr Use the mitmproxy doc, return here if trouble.

I am porting a server from Java to Go, and need to watch the traffic it receives. The clients include OSX and Windows desktop apps, talking to the server over HTTPS. Here’s how I did that with mitmproxy and iptables. We will setup a transparent proxy.

Instructions are for Ubuntu (arguably the most popular server operating system), but mitmproxy will also run on OSX (anecdotaly the most popular developer laptop) using pf instead of iptables.

The client is on one machine (OSX or Windows). It talks to the server on another machine (192.168.1.10). We are going to set iptables on the server to redirect incoming traffic for port 443 (https) to port 8080 (default mitmproxy port). mitmproxy will impersonate the the server, do the SSL negotiation with the client using a certificate it generates (and signs) for the requested host. mitmproxy will then make a new connection to the real server, fetch whatever content the client wanted, and forward it, decrypting and re-encrypting it with it’s own cert. This will allow us to inspect the traffic as it passes through.

Note that mitmproxy is designed for HTTP. It doesn’t handle websockets or http2, although it does have a TCP mode. http2 support is under active development.

Setup

First, make sure everything works unintercepted. You probably have DNS rules on the client machine to point server.example.com to 192.168.1.100. The server probably has a self-signed cert, any kind of cert is fine. mitmproxy docs will ask you to set the default host on the client, but because we have the DNS rules we won’t need to do that.

Install mitmproxy 

sudo apt-get install python-pip python-dev libffi-dev libssl-dev libxml2-dev libxslt1-dev
sudo pip install mitmproxy

Ubuntu 14.04 (TLS) had an old pyasn1, so I had to also: sudo pip install pyasn1

Trust mitmproxy’s root certificate

For the dynamic certificate generation / signing to work, the client must trust mitmproxy’s root certificate. Follow the instructions here: Install mitmproxy root certificate. If your client is the web browser, the mitm.it trick is the easiest setup. For desktop clients, use the manual setup. It’s easy.

DNS

On OSX after you edit /etc/hosts, you might need to flush the DNS cache: dscacheutil -flushcache;sudo killall -HUP mDNSResponder. In OSX El Capitan you may need to disable rootless (WTF Apple). On Windows the hosts file is in C:\Windows\System32\drivers\etc\hosts, and you need to be Administrator. In Cygwin you run vim as Administrator with cygstart --action=runas vim hosts.

iptables

First, enable IP Forwarding in the kernel: sudo sysctl -w net.ipv4.ip_forward=1.

iptables is very powerful, but has a steep learning curve. Here’s what you need to know for our use case:

  1. All rules belong in a table. The default table is filter, and it is mainly for firewall rules. You can list the default table rules with sudo iptables -L -v. We need to affect routing, not firewall rules, so we will be using the nat table instead: sudo iptables -t nat -L -v. It is called nat because it is for Network Address Translation. It has the ability to rewrite TCP fields (we want to change the port) before the packet gets routed.
  2. Inside each table there are pre-defined chains, which you attach rules to. We are going to append some rules to the PREROUTING chain. That chain runs before the Linux kernel routes the packet.
  3. A rule is a set of things to match (this protocol, on this interface, etc)
  4. A rule ends with a target (REDIRECT in our case).
  5. iptables has a set of modules documented in man iptables-extensions. Each module adds command line options and targets. The module matching the protocol name is loaded automatically, so some of what we’re going to use is in the tcp module.

And with that, here is the rule:

sudo iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 8080
  • -t nat: Use the nat table.
  • -A PREROUTING: Append this rule to the PREROUTING chain.
  • -i eth0: Rule only applies to packets coming in over the eth0 network interface. Maybe you want wlan0 instead. Importantly, this won’t affect localhost packets, so it won’t affect the connection from mitmproxy to the server, only the connection from the client to mitmproxy. That also means you do need two machines to try this. There’s a sidebar on doing this all on localhost later on.
  • -p tcp: Rule only applies to TCP protocol packets. This also loads the tcp iptables module (technically it’s lazy loaded).
  • –dport 443: Rule applies to packets for port 443. dport is in the tcp iptables module.
  • -j REDIRECT: Re-write the destination address (without changing the port) to the local machine’s address, so that we will process it locally. In our case the packet was already headed to this machine, but, read on..
  • –to-port 8080: Also re-write the port, changing it to 8080.

The result is any packets coming in to our machine headed for port 443, will instead go to port 8080. It is useful to also see the unencrypted HTTP traffic, and my server listens on a couple of other ports too, so I also have these rules using the multiport module:

sudo iptables -t nat -A PREROUTING -i eth0 -p tcp -m multiport --dports 80,443,9000,9443 -j REDIRECT --to-port 8080

I put these rules in a script, which ends with a reset, erasing all the rules on the nat table:

sudo iptables -t nat -F
sudo sysctl -w net.ipv4.ip_forward=0

localhost too

If you want to run the client on the same machine as the server (you’re in a coffee shop on your laptop; have a pastry for me), we can’t use the PREROUTING table, because it only applies to packets coming from outside. What we can do is modify the destination port on packets OUTPUT by our client process. The catch is that it will also affect packets output by mitmproxy, and we’ll get into a routing loop.

There are probably several ways to solve this, but the one that worked for me was running mitmproxy as root, and making the iptables rule not apply to root-owned processes.

sudo iptables -t nat -A OUTPUT -p tcp -m owner ! --uid-owner root --dport 443 -j REDIRECT --to-port 8080
  • -m owner: Load the owner module.
  • ! –uid-owner root: Rule does not apply to root-owned processes

Remember that in this case you’ll run mitmproxy as root. This will also log all your https web browser traffic. Add -m multiport and replace --dport with --dports to intercept multiple ports (or just repeat the line with a different port).

mitmproxy incantation (explained)

Start mitmproxy:

mitmproxy -T --host --ssl-port 443 --ssl-port 9443
  • -T: Transparent proxy mode. mitmproxy has many modes.
  • –host: Use the Host header to construct URLs for display.
  • –ssl-port: mitmproxy expects SSL traffic on ports 443 and 8443, only. If it receives an SSL handshake on a different port, it will interpret it as HTTP, and give you an error about Bad HTTP request line. Use this flag to tell it to expect SSL on other ports.

At this point everything should work. Requests from the client machine should appear in mitmproxy. If not, read on.

No cipher overlap

If your client’s list of supported ciphers doesn’t overlap with mitmproxy’s, you will see an error in mitmproxy’s event log (press e) about possibly the client not trusting our certificate. Your client might not display mitmproxy’s returned HTML, which can make it hard to see the real error. tcpdump to the rescue.

Dump tcp traffic, coming in on eth0 (-i eth0), in ascii (-A), without resolving IP address or port/service names (-nn):

sudo tcpdump -i eth0 -nnA tcp

If you really need to dig into the TCP traffic, dump it to a pcap file and open that in Wireshark:

sudo tcpdump -i eth0 -nnvA -w tcpdump.pcap
wireshark tcpdump.pcap

The solution is to add –ciphers-client to tell mitmproxy which ciphers to allow during the SSL handshake. You need ssldump to find these, which is the topic of the next section. If you’re in a hurry, you can just put a decent selection in, and it will probably work. This is what the next version of mitmproxy does, I’m including it’s default list since 0.13.1. Prior to that it lets openssl choose, which should include almost all the ciphers, but strangely doesn’t work for me.

mitmproxy -T --host --ssl-port 443 --ssl-port 9443 --ciphers-client ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA

ssldump

Here’s how to find the minimum list of ciphers to put in --ciphers-client.

ssldump is, as the name suggests, an SSL/TLS protocol analyzer. It will show you the SSL handshake in detail. Originally by Eric Rescorla in 2001, I found a much updated version on github:

git clone https://github.com/adulau/ssldump.git && cd ssldump
./configure --prefix=/usr --with-pcap-lib=/usr/lib/x86_64-linux-gnu --with-openssl-lib=/usr/lib/x86_64-linux-gnu
make
sudo make install

Run it: sudo ssldump -i eth0 -d

Run your setup without mitmproxy interception (i.e. without the iptables rules). This will show you which cipher the client and server agree on. Look in the ServerHello message for a line like cipherSuite TLS_DHE_RSA_WITH_AES_256_CBC_SHA.

Next, you need to translate the TLS name to an openssl name. Search for the TLS name here (scroll to Cipher Suite Names): Cipher Suite Names. That’s the name you need to pass to mitmproxy’s --ciphers-client.

Other debugging tools include openssl s_client -connect <host:port> to see what an SSL/TLS server responds, and openssl ciphers -v to show which ciphers your openssl supports.

Happy interception!