Dec 18, 2015 - mitmproxy - Watch webservices at work

Usually, when I find a new interesting tool, I leave a bookmark in my “Tools”-folder, forget it and after some months I’ll miss it… So I thought I start a series of blog posts of the tools I use as a reminder for myself and of course as suggestions for the reader. It was pretty quiet in this blog anyway :)

mitmproxy

mitmproxy (“Man-In-The-Middle proxy”, Github) is a small HTTP(s)-proxy, that allows you to observe, inspect and manipulate requests. So every time you wonder, what you your application and/or services send and receive to and from other services you can startup mitmproxy and configure the proxy.

mitmproxy starts curses-like commandline UI. There is also mitmdump, which provides “tcpdump-like functionality”. In other words, mitmproxy is the interactive ui, whereas mitmdump is a programmatic helper.

This post is a quick introduction. mitmproxy can do much more for you, like manipulating request, avoid any caching, …. I suggest to read the documentation.

Installation

Arch provides mitmproxy in its Community repository. The version is 0.15 (2015-12-05).

pacman -S mitmproxy

Debian has mitmproxy in their official repositories too

apt-get install mitmproxy

However, the version 0.10 in Jessie is obviously older. The same for ubuntu: Wily comes with 0.11 and there is no official package for Precise (the LTS).

So, why do I tell you about the available versions? It seems this tools is under heavy development and the Arch-version I use has some options and probably abilities the other versions don’t have. It makes sense to always try the latest version.

Startup

To start mitmproxy, just … well, start it.

mitmproxy --port 8000

This will let mitmproxy listen on Port 8000.

On the commandline for most tools you can set the environment variables HTTP_PROXY and HTTPS_PROXY. For all popular browser there are proxy-switcher plugins, that let you switch between different proxy profiles. More interesting are programmatic clients, that are used within applications to communicate with other services. All of them should provide a “proxy”-setting somehow somewhere. For example Guzzle let you set a proxy during for a request, or as default during instanciation.

There are also several ways to let mitmproxy watch the traffic, if you have to handle more complex infrastructures

Watch

You should see some requests coming in now. For example when I request kingcrunch.eu/ I can see this:

Overview

This is the overview. You can already see some useful infos, but especially you can see all requests made since startup. With the arrow keys you can now select one entry and with Enter open the details. Use Tab to switch between Request- and Response-Infos and details about the connection.

Request

Response

Details

HTTPS

You may notice, that https://kingcrunch.eu/ should be SSL-encrypted, but mitmproxy is still able to track it. To do so mitmproxy actually performs an “Mitm-Attack” by creating a custom certificate for the requested domain on-the-fly. The tool creates a special CA-certificate for your local machine. To avoid “unsecure connection”-warnings you can trust this certificate, but You really should keep the private key private, else it makes you vulnerable for real mitm-attacks.

You can find the self-generated CA-certificate in ~/.mitmproxy/mitmproxy-ca-cert.*. There is one .p12 for use with windows, one .pem for use with non-windows systems, and one .crt, which is actually a .pem, but with a different extension for use with android. You can also visit mitm.it. This will show you several download links, when the traffic is passed through the proxy. Else you’ll a simple static page from the real mitm.it-domain telling you, that you setup your proxy first.

In Arch installing a certificate uses trust.

cp ~/.mitmproxy/mitmproxy-ca-cert.pem /etc/ca-certificates/trust-source/anchors/
trust extract-compat

Debian makes use of the probably more widely known update-ca-certificates

$ cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/
$ update-ca-certificates

In Android after downloading the certificate via http://mitm.it the OS already asks, if it should install it.

Summary

For me mitmproxy already served well. The setup is pretty simple, because HTTP-clients from “real” clients like browser, over programmatic browser in form of libraries, to embedded clients (think of android apps) should be able to send its traffic through a proxy.

The other (bigger) benefit is, that once you have it up and running you can see all requests/responses made since start. When you see something strange, you don’t have to redo everything and watch logs. I also saw requests made be the application, that shouldn’t happen at all, or requests were made in the wrong order.

Sep 24, 2014 - XML-issue with HHVM 3.3

I tried to get Symfony running on HHVM 3.3, because 3.2 caused some annoying issues. However, 3.3 didn’t run out of the box neither, because now it refused to parse DIC-XMLs. I’ve found the solution in one ticket, that I cannot find anymore. I found the explanation in the “inconsistencies”-file instead.

(7) Loading of external entities in the libxml extension is disabled by default for security reasons. It can be re-enabled on a per-protocol basis (file, http, compress.zlib, etc…) with a comma-separated list in the ini setting hhvm.libxml.ext_entity_whitelist.

To sum it up: To get it Symfony working again, just add this to your php.ini

hhvm.libxml.ext_entity_whitelist = file

The security implications should be relatively small, because usually you don’t have any malicious entity-files on your local disc.

For those, who want to know: file is enough, because Symfony rewrites the XML-files, so that they don’t refer to the remote locations, but to local ones within the project itself (vendor/symfony/..), because the remote ones doesn’t match the required versions and it prevents one from downloading the file each and every time the container gets rebuild.

Aug 20, 2014 - Setting up SSI on nginx to work with Symfony 2.6

With the upcoming Symfony 2.6 SSI support is directly built-in. For me this is pretty exciting, because it is the first merged PR, that is more than a minor addition, or bugfix. However, setting up nginx wasn’t that flawless at the end. More about that later in this post.

server {
    server_name domain.tld www.domain.tld;
    root /var/www/project/web;

    location / {
        # try to serve file directly, fallback to app.php
        try_files $uri /app.php$is_args$args;
    }

    location ~ ^/(app|app_dev|config)\.php(/|$) {
        fastcgi_pass unix:/var/run/php5-fpm.sock;
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param HTTPS off;
    }

    error_log /var/log/nginx/project_error.log;
    access_log /var/log/nginx/project_access.log;
}

The SSI-implementation behaves like the long existing ESI-implementation, which means you have to enable it on server-side too 1. As long as the server doesn’t tell the Symfony2-stack, that it is able to handle SSI-tags (or ESI-tags), it falls back to the InlineRenderer, that renders the response of the action directly into the document 2. To do this, we add a header.

In the response the application itself adds a header. The intention of this header is the exact inverse idea of the header we send to the application: It allows the server to decide, whether, or not substitution is required. This post only covers unconditional SSI: It tries to find and replace SSI-tags, even if there is none just to keep it simple. However, we still don’t want anybody outside to see this, so we will remove it.

location ~ ^/(app|app_dev|config)\.php(/|$) {
    ssi on;

    # Other options

    fastcgi_param HTTP_SURROGATE_CAPABILITY symfony2="SSI/1.0";
    fastcgi_hide_header SURROGATE_CONTROL;
}

While this was as simple as it could be, now here comes the ugliness: It doesn’t work! If there is a SSI-tag, it will end up in an infinite loop…

The problem is not directly visible in our configuration, because it comes from the default fastcgi_params. This file contains (beside others) this line:

# [..]
fastcgi_param REQUEST_URI $request_uri;
# [..]

Digging further we can find something in the nginx Manual, that describes, what this means:

$request_uri
    full original request URI (with arguments)
[..]
$uri
    current URI in request, normalized
    The value of $uri may change during request processing,
    e.g. when doing internal redirects, or when using index files.

The definition of “full original request URI” is interpreted pretty strict by nginx, because it always and in every case contains the initial URI. For nginx everything, that somehow influences the used URI is treated as an internal subrequest and to perform a subrequest nginx updates $uri and starts processing this new URI from the beginning. That includes rewrite of course, try_files and even SSI, even if it is actually something slightly different than the previous mentioned rewrites.

For our configuration above this means, that every SSI-subrequest back to the Symfony2-application will have the same and identical REQUEST_URI-parameter. Of course the result will contain the SSI-tag again. I spent a long time investigating the best solution to work around this, that both “works” and is not too complex. At the end I’ve found my solution, that at least works. The downside is, that now URIs like example.com/app.php/foo/bar (–> note the app.php/) doesn’t work anymore.

server {
    server_name domain.tld www.domain.tld;
    root /var/www/project/web;

    location / {
        set $orig_uri $uri; # <-- Remember uri during first iteration of every (sub)request
        try_files $uri /app.php$uri$is_args$args;
    }

    location ~ ^/(app|app_dev|config)\.php(/|$) {
        ssi on;

        fastcgi_pass unix:/var/run/php5-fpm.sock;
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param HTTPS off;

        fastcgi_param REQUEST_URI $orig_uri$is_args$args; # <-- Use $orig_uri instead

        fastcgi_param HTTP_SURROGATE_CAPABILITY symfony2="SSI/1.0";
        fastcgi_hide_header SURROGATE_CONTROL;
    }

    error_log /var/log/nginx/project_error.log;
    access_log /var/log/nginx/project_access.log;
}

I had something in mind with a named location, but I hadn’t followed it any further for now. I’ll probably come back to this later.

  1. Read more about Edge Architecture Specification, primary initiated by Akamai Technologies.

  2. Although SSI itself doesn’t mention to use headers to control it’s behaviour I’ve decided to follow the ESI-specifications for the Symfony2-implementation too to avoid confusion.