Apr 12, 2017 - Understood Symfony environments

I can imagine this blog post will leave developers with the feeling, that they read something very obvious. That’s reasonable. From a specific point of view at least. But my work life with (in the meantime) many different “Symfony developers” showed me, that I am by far not alone.

I thought I understood Symfony Environments. I used to think the environments represent the different stages an application gets deployed to, or is used in. For example test, acceptance, staging, production, preview, but also unit tests. It always seemed a little bit flawed to me. Why does the Symfony standard edition only provides frontcoller for dev (app_dev.php) and prod (app.php), but not for test by default? Why should I use the test-environment for both the test-server and the unit tests [2]? Why do I need to take care to setup the webservers correctly only to point to the right front-controller (environment)? And others. Small details, that felt like either I, or the entire community missed something.

And then it hit me. I got it in the exact moment I told somebody else “Even prod should be runnable on your local machine without incidents”.

You indeed should run every stage with the prod-environment. The environments are not about the stage an application is in, but about the behaviour it should show. Deploying an application on the server used for acceptance tests means, that I want to show, how it will look like in production. Of course it should behave like production. Exactly. In every single detail.

On the other hand during development, testing, or for example – as mentioned in the Symfony documentation – benchmarking you want different behaviours. You want something different from production, that helps you to achieve your goal. Benchmarking? You need a profiler. Development? You want some more insight into the applications architectural details. Testing? You need something you can verify your assertions against.

Still, there are differences between the stages. I’ll just summarize some points I imagine colleagues will ask me.

Whats about … Logging?

Questions: Do you really need specific log levels during acceptance, when you don’t need them in production? Do you actually look at them? Do you really need debug logs on the test server? Why and what do you debug there, what you cannot debug on your development platform? But that aside. It’s just that maybe different log levels on some servers are not worth the hassle to handle different environments. Or even doesn’t any benefit at all. I’ve seen teams, that never look at the logs on the test server, but frequently changed the config on production.

If you still need them, consider introducing a log_level parameter in parameters.yml. As a side effect you can change the log-level of deployed applications without re-deployment and even on production, when and if needed. Maybe introduce multiple parameters for different loggers. I see this frequently with java applications and property-files. I guess it’s not too bad.

Whats about … APIs?

Integrating external APIs into an application is just integrating another external system. You thought the database is part of the application? It isn’t. It is something the application uses [1]. The connection settings for the database are already in the default parameters.yml, so the settings for every other external should be there. As a side effect you prevent yourself from accidentally accessing production APIs from your development platform just because you hardcoded the URLs into the config_*.yml and did something wrong with the environment.

Whats about … Mails?

Of course you don’t want to send mails from your test server to your costumers. It happened to me once and theoretically I ordered a “test”-vaccum-cleaner. But to be fair, the bug I reported was about my email address, thus I guess they just needed an example email address to work with. I’ll dedicate a separate post for this.

At the end: Instead of saying “the difference between the stages should be minimal”, you should say “There is no difference at all”. You only need one environment.

[1] You can say, that the database is something the applications “owns” and “controls” in a certain way, but it is not within the application and there may be situations, were others (DBA?) access them. It’s not “yours”, it’s just something somebody allowed you to use.

[2] Even the documentation mentions

The test environment is used when writing functional tests and is not accessible in the browser directly via a front controller. In other words, unlike the other environments, there is no app_test.php front controller file.

Yeah, I did it wrong all the time.

Dec 18, 2015 - mitmproxy - Watch webservices at work

Usually, when I find a new interesting tool, I leave a bookmark in my “Tools”-folder, forget it and after some months I’ll miss it… So I thought I start a series of blog posts of the tools I use as a reminder for myself and of course as suggestions for the reader. It was pretty quiet in this blog anyway :)

mitmproxy

mitmproxy (“Man-In-The-Middle proxy”, Github) is a small HTTP(s)-proxy, that allows you to observe, inspect and manipulate requests. So every time you wonder, what you your application and/or services send and receive to and from other services you can startup mitmproxy and configure the proxy.

mitmproxy starts curses-like commandline UI. There is also mitmdump, which provides “tcpdump-like functionality”. In other words, mitmproxy is the interactive ui, whereas mitmdump is a programmatic helper.

This post is a quick introduction. mitmproxy can do much more for you, like manipulating request, avoid any caching, …. I suggest to read the documentation.

Installation

Arch provides mitmproxy in its Community repository. The version is 0.15 (2015-12-05).

pacman -S mitmproxy

Debian has mitmproxy in their official repositories too

apt-get install mitmproxy

However, the version 0.10 in Jessie is obviously older. The same for ubuntu: Wily comes with 0.11 and there is no official package for Precise (the LTS).

So, why do I tell you about the available versions? It seems this tools is under heavy development and the Arch-version I use has some options and probably abilities the other versions don’t have. It makes sense to always try the latest version.

Startup

To start mitmproxy, just … well, start it.

mitmproxy --port 8000

This will let mitmproxy listen on Port 8000.

On the commandline for most tools you can set the environment variables HTTP_PROXY and HTTPS_PROXY. For all popular browser there are proxy-switcher plugins, that let you switch between different proxy profiles. More interesting are programmatic clients, that are used within applications to communicate with other services. All of them should provide a “proxy”-setting somehow somewhere. For example Guzzle let you set a proxy during for a request, or as default during instanciation.

There are also several ways to let mitmproxy watch the traffic, if you have to handle more complex infrastructures

Watch

You should see some requests coming in now. For example when I request kingcrunch.eu/ I can see this:

Overview

This is the overview. You can already see some useful infos, but especially you can see all requests made since startup. With the arrow keys you can now select one entry and with Enter open the details. Use Tab to switch between Request- and Response-Infos and details about the connection.

Request

Response

Details

HTTPS

You may notice, that https://kingcrunch.eu/ should be SSL-encrypted, but mitmproxy is still able to track it. To do so mitmproxy actually performs an “Mitm-Attack” by creating a custom certificate for the requested domain on-the-fly. The tool creates a special CA-certificate for your local machine. To avoid “unsecure connection”-warnings you can trust this certificate, but You really should keep the private key private, else it makes you vulnerable for real mitm-attacks.

You can find the self-generated CA-certificate in ~/.mitmproxy/mitmproxy-ca-cert.*. There is one .p12 for use with windows, one .pem for use with non-windows systems, and one .crt, which is actually a .pem, but with a different extension for use with android. You can also visit mitm.it. This will show you several download links, when the traffic is passed through the proxy. Else you’ll a simple static page from the real mitm.it-domain telling you, that you setup your proxy first.

In Arch installing a certificate uses trust.

cp ~/.mitmproxy/mitmproxy-ca-cert.pem /etc/ca-certificates/trust-source/anchors/
trust extract-compat

Debian makes use of the probably more widely known update-ca-certificates

$ cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/
$ update-ca-certificates

In Android after downloading the certificate via http://mitm.it the OS already asks, if it should install it.

Summary

For me mitmproxy already served well. The setup is pretty simple, because HTTP-clients from “real” clients like browser, over programmatic browser in form of libraries, to embedded clients (think of android apps) should be able to send its traffic through a proxy.

The other (bigger) benefit is, that once you have it up and running you can see all requests/responses made since start. When you see something strange, you don’t have to redo everything and watch logs. I also saw requests made be the application, that shouldn’t happen at all, or requests were made in the wrong order.

Sep 24, 2014 - XML-issue with HHVM 3.3

I tried to get Symfony running on HHVM 3.3, because 3.2 caused some annoying issues. However, 3.3 didn’t run out of the box neither, because now it refused to parse DIC-XMLs. I’ve found the solution in one ticket, that I cannot find anymore. I found the explanation in the “inconsistencies”-file instead.

(7) Loading of external entities in the libxml extension is disabled by default for security reasons. It can be re-enabled on a per-protocol basis (file, http, compress.zlib, etc…) with a comma-separated list in the ini setting hhvm.libxml.ext_entity_whitelist.

To sum it up: To get it Symfony working again, just add this to your php.ini

hhvm.libxml.ext_entity_whitelist = file

The security implications should be relatively small, because usually you don’t have any malicious entity-files on your local disc.

For those, who want to know: file is enough, because Symfony rewrites the XML-files, so that they don’t refer to the remote locations, but to local ones within the project itself (vendor/symfony/..), because the remote ones doesn’t match the required versions and it prevents one from downloading the file each and every time the container gets rebuild.