This is what causes XXE vulnerabilities

XML external entity (XXE) attacks are one of the OWASP Top Ten security risks. XXE attacks are caused by XML parsers that have entity processing enabled. Here’s an example of a simple Ruby program that has entity processing enabled in Nokogiri, its XML parser:

This allows our XML parser to read the contents of our local filesystem, the key point being that this occurs because the NOENT flag is enabled. When we run the program, we see contents of /etc/passwd (limited to the first 10 lines for brevity):

$ ruby xxe.rb | head -n 10
<?xml version="1.0"?>
<!DOCTYPE root [
<!ENTITY users SYSTEM "file:///etc/passwd">
]>
<root>
    <child>##
# User Database
# 
# Note that this file is consulted directly only when the system is running
# in single-user mode.  At other times this information is provided by

If we ran the program with NOENT disabled, we’d see the following:

$ ruby xxe.rb | head -n 10
<?xml version="1.0"?>
<!DOCTYPE root [
<!ENTITY users SYSTEM "file:///etc/passwd">
]>
<root>
    <child>&users;</child>
</root>

In this case, we see that there’s still a reference to the users entity, and we haven’t read the contents of our local filesystem.

This raises a question: what does NOENT actually mean?

At first glance, the naming is a bit counterintuitive. NOENT looks like it means something like “no entities,” but we are processing our users external entity when that flag is enabled.

Luckily, we don’t have to search far in Nokogiri’s source code to see how NOENT is used. Nokogiri is partially implemented in Java, and we can find this code snippet in its XmlDomParserContent.java source file:

In the same file, we find FEATURE_NOT_EXPAND_ENTITY defined like so:

To summarize what we’ve discovered so far: when NOENT is enabled, the FEATURE_NOT_EXPAND_ENTITY feature is turned off, and this is when we see our entity expanded with contents from the local filesystem. When NOENT is disabled, the FEATURE_NOT_EXPAND_ENTITY feature is turned on, and we don’t read contents from the local filesystem.

That’s a lot of consecutive negatives! Let’s reword it for clarity: when our flag is enabled, the feature which expands entities is turned on. Put this way, the behaviour is a bit more clear – we see the contents of the local filesystem because our entity-expanding feature is enabled.

Still, this doesn’t answer our original question – why the name NOENT? To answer that, we can look at Apache documentation related to the FEATURE_NOT_EXPAND_ENTITY definition shown previously. Under the definition of the http://apache.org/xml/features/dom/create-entity-ref-nodes feature, we expect the following behaviour when FEATURE_NOT_EXPAND_ENTITY is set to true:

Create EntityReference nodes in the DOM tree. The EntityReference nodes and their child nodes will be read-only.

And when it’s set to false:

Do not create EntityReference nodes in the DOM tree. No EntityReference nodes will be created, only the nodes corresponding to their fully expanded sustitution [sic] text will be created. 

In other words, when NOENT is enabled, it means that we don’t expect to see an EntityReference node in our parsed content, and our parser should replace an entity with its definition (in the case of our example, replace the users node with /etc/passwd). If NOENT is disabled, it means we do expect to see our entity in our parsed content, and so we still see a reference to users in the output of our parser.

In conclusion: the NOENT flag does mean “no entities”, as in, “no references to entities should exist in our parsed XML.” This is why our parser replaces it with the contents of /etc/passwd. This naming convention leaves plenty of room for confusion, which is why fixing the names of parser flags is actually on the Nokogiri roadmap!

Start your journey towards writing better software, and watch this space for new content.

Don’t iterate to convert built-in Ruby data types

Both the Ruby language and the Ruby on Rails framework provide a lot of convenience methods compared to other technology stacks, improving ease of use and developer productivity. However, it’s difficult when starting out with Ruby to know what ease-of-use functions are available to you as a programmer.

Here’s a heuristic to help with that: if you’re writing a Ruby program, and you find yourself writing code that uses iteration to convert one built-in data type to another, there’s probably a language feature that could simplify your code. Let’s go through a few examples of this that I’ve encountered since I started coding in Ruby.

Generating an array of numbers between 1 and N

Suppose you want to generate an ordered array of integers between 1 and N. We might be tempted to use the .times method on our integer N, convert that enumerator to an array, then map each number to be increased by 1 (since .times provides us numbers from 0 to N – 1). For example, if N is 5, we can use the following code:

5.times.to_a.map{|n| n + 1}
=> [1, 2, 3, 4, 5] 

This works, but Ruby provides simpler ways for us to achieve the same goal using upto or a Range:

1.upto(5).to_a
 => [1, 2, 3, 4, 5] 

(1..5).to_a
 => [1, 2, 3, 4, 5] 

Convert the values of a hash into an array

Suppose you want an array containing the values of a Hash, and don’t need to know what the keys are. We might be tempted to call .map on the Hash, and then return the value in that block:

{foo: 1, bar: 2, car: 3}.map{ |_key, value| value }
 => [1, 2, 3] 

With far fewer keystrokes, .values lets us do the same thing:

{foo: 1, bar: 2, car: 3}.values
 => [1, 2, 3] 

(Of course, if we’re interested in the keys and not the values, Ruby provides a .keys method as well.)

Combining two hashes

Suppose have two Hash variables, h0 and h1, and want to create a new Hash containing the keys and values in both of them. In the case of duplicate keys between h0 and h1, we’ll take the value from h1 (this is arbitrary; using the value from h0 would be valid too). We could write a method to iterate over two hashes like so:

However, the far simpler approach is to use the .merge method provided by Ruby:

{foo: 1, bar: 2}.merge({bar: 3, car: 4})
 => {:foo=>1, :bar=>3, :car=>4} 

Conclusion

Ruby (and Ruby on Rails) provides us with a lot more convenience methods than other popular languages like Go and JavaScript. Learning all of them requires reading the language documentation, along with simply practicing with the language over time. However, these convenience methods aren’t created at random; there are patterns in them that we can identify to become more effective Ruby developers.

Start your journey towards writing better software, and watch this space for new content.

Use RSpec to improve performance via precomputation

Suppose that we have a Ruby class called SomeClass, and SomeClass has an array class constant called SOME_ARRAY. SomeClass also has a method called same_elements? that checks if an input array has the same elements as SOME_ARRAY without regard to ordering. We may be tempted to write code that looks like this:

This code works, but it’s an inefficient solution. We sort SOME_ARRAY every time same_elements? is called, even though it’s a constant that we defined ourselves!

We could define SOME_ARRAY as ['foo', 'bar', 'car'].sort, and that gets us most of the way to improved performance. We only expect Rails to load our class definition for SomeClass once when we start our application, so even if we’re creating many instances of SomeClass (eg. once per request to a REST endpoint), we’re still only calling .sort on SOME_ARRAY once.

However, if we want to be even more efficient with our compute power at runtime, we can simply define SOME_ARRAY to be sorted when we write our code, which lets us avoid calling .sort when we load the class definition.

This revised code has even fewer calls to .sort than when we defined SOME_ARRAY as as ['foo', 'bar', 'car'].sort. However, this introduces a new problem into our code: if we ever rewrite or modify SOME_ARRAY in the future, we need to ensure that that constant remains sorted.

For example, a future commit could change SOME_ARRAY to ['new element', 'bar', 'car', 'foo'], which would cause a bug in our same_elements? method. Hopefully our code review process and tests on same_elements? would stop us from introducing that bug.

However, through RSpec, we can actually enforce that SOME_ARRAY remains sorted. We introduce a test like so into our SomeClass test suite:

This ensures that SOME_ARRAY remains ordered when we introduce new elements to it. By precomputing the sorted order of SOME_ARRAY in our testing and continuous integration environments, we’re able to do fewer .sort computations at runtime in our application.

Note that we should only use this method when there’s no impact to SOME_ARRAY‘s readability. If SOME_ARRAY is defined as ['foo', 'bar', 'car'] (ie. random elements where we don’t care about the ordering), making its definition alphabetical in the source code makes sense. Using this method also makes sense when SOME_ARRAY is something we already expect to be alphabetized (eg. ['alpha', 'bravo', 'charlie']). However, if we expect some other sort of non-alphabetized ordering of SOME_ARRAY, then making our code less readable in order to save compute power would be the wrong tradeoff. For example, redefining ['one', 'two, 'three'] to the sorted ['one', three', 'two'] makes our code less readable, which isn’t worth the slight performance gains.

Start your journey towards writing better software, and watch this space for new content.

This is how percentages work in Flipper

Flipper is a Ruby library that implements feature flags. We use it to turn features on and off via configuration data that our application reads at runtime, meaning we can enable and disable features without modifying our actual Ruby code. Flipper has the concept of a Gate to determine whether or not a feature is enabled – for example, the simplest Gate is a boolean that globally determines whether a feature is turned on/off for the whole application. However, Flipper also lets us turn a feature on/off for a certain percentage of actors. Within this blog post, I’ll treat a “user” and “actor” as being interchangeable concepts.

If we want to gradually roll out a new feature to our application’s users, we may start by only enabling a Flipper Gate for 1% of users – then gradually increase that percentage to 2%, then 3%, and so on until we reach 100%.

This raises multiple questions, though:

  • Which 1% of users do we enable this feature for? Are these users selected randomly or deterministically?
  • When we increase that percentage from 1% to 2%, are the users from the 1% cohort a subset of the 2% cohort? We’d hope that Flipper is designed this way, since we don’t want a user to see a new feature in our application toggled on/off during a gradual rollout.
  • Are we introducing additional state to a user when a feature is enabled for them?

Let’s find out by reading the percentage_of_actors.rb file in the Flipper source code. open? looks like a promising method to read in that file:

We can avoid diving deeply into how Types::Actor.wrappable? works, since it’s an internal implementation detail of Flipper that is outside of the scope of this blog post. We can assume that actor.value returns the unique flipper_id that we see examples of in Flipper’s documentation. For example, if we’re dealing with a feature flag called foo_enabled for a user with an ID of 123, actor.value should be "User;#123", meaning the local variable id in the above snippet is "foo_enabledUser;123".

We then calculate the CRC32 of that id. CRC32 is an algorithm designed to check data integrity, but it works pretty well as a hash function if we’re not using it in a security-related context. This means that the output of CRC32 is pseudorandom and fairly uniformly distributed, even though it’s calculated deterministically. So, it should look like the users who have a feature enabled were selected at random, even though their eligibility is deterministic.

The inequality below (taken from the code snippet) will tell us whether or not cohorts for smaller percentages will be subsets of cohorts for larger percentages:

Zlib.crc32(id) % (100 * scaling_factor) < percentage * scaling_factor

percentage is the variable that we defined to be strictly increasing from 1% to 100%. scaling_factor is hardcoded to 1000, meaning percentage * scaling_factor is strictly increasing as well. Ruby’s implementation of CRC32 also interprets its output as an unsigned (ie. positive) integer. This means that all of the IDs for whom the inequality held true with a given percentage will also hold true for a larger percentage.

Let’s use an example: if percentage is 1, then any users where Zlib.crc32(id) % (100 * scaling_factor) is less than 1000 will have the foo_enabled feature turned on for them. Roughly 99% of our users will have a value of over 1000 in this check, so this feature will be turned off for 99% of users. However, for user 24 (ie. with id value set to "foo_enabledUser;24“, the CRC of their ID is 278800337, and 278800337 % (100*scaling_factor) returns 337.

The key point here is that this will always be 337 for user 24, regardless of what our percentage is set to! 337 is less than 1000, and if we strictly increase percentage to values such as 2 and 3, this inequality also holds true for their corresponding values for percentage * scaling_factor (such as 2000 and 3000).

To conclude:

  • We can expect our users to be sampled more-or-less at random by Flipper, even though their eligibility for a feature is calculated deterministically via CRC32. This is because CRC32 can be used as a hash function in this context.
  • If we’re strictly increasing our percentage of users enabled, then all users from the cohort of smaller percentages will also be included in the cohorts of larger percentages too.
  • Calculating a user’s eligibility for a feature via CRC32 doesn’t add additional state to the user.

Start your journey towards writing better software, and watch this space for new content.