Introducing opinel: Scout2's favorite tool
03 Aug 2015 - Loïc Simon
With boto3 being stable and generally available1, NCC took the opportunity
to migrate Scout2 and AWS-recipes to boto3. As part of that migration
effort, we decided to publish the formerly-known-as AWSUtils repository – used
by Scout2 and AWS-recipes – as a python package required by these tools,
rather than requiring users to work with Git submodules. We’ve also added more
flexibility when working with MFA-protected API calls and improved versioning
across the project.
opinel
To avoid name conflicts, we decided to rename the shared AWSUtils code to a
less misleading name: opinel. The opinel package is published on PyPI, and thus can
be installed using pip and easy_install. The corresponding source code is still
open-sourced on Github at https://github.com/iSECPartners/opinel.
As a result, Scout2 and AWS-recipes have been modified to list opinel as a
requirement, which significantly simplifies installation and management of this
shared code.
Support for Python 2.7 and 3.x
Because boto3 supports both Python2 and Python3, we decided to make sure that
the code we build on top of that package has similar properties. As a result,
the latest versions of Scout2 and AWS-recipes support Python 2.7 and 3.x.
Note that opinel will NOT work with Python 2.6.
Modification of the MFA workflow
As requested by a user of AWS-recipes2, we modified the workflow when
using MFA-protected API access to no longer store the long-lived credentials
in a separate file. As a result, the .aws/credentials.no-mfa file is no
longer supported and all credentials are stored in the standard AWS credentials
file under .aws/credentials. Usage of the existing tools remains unchanged,
but the long-lived credentials are now accessible via a new profile name:
profile_name-nomfa. This allows users to work with both STS and long-lived
credentials if need be.
If you already had configured your environment to work with MFA-protected API
access, you will need to copy your long-lived credentials back to the
.aws/credentials file. This can be done with a simple command such as the
following:
cat ~/.aws/credentials.no-mfa | sed -e 's/]$/-nomfa]/g' >> ~/.aws/credentials
Support to use assumed-role credentials
With this new workflow implemented, we created a new recipe that allows
configuration of role-credentials in the .aws/credentials file. When the following
command is run, it uses the credentials associated with the isecpartners
profile to request role credentials for the IAM-Scout2 role. The role
credentials are then written in the .aws/credentials file in a new profile
named isecpartners-Scout2, which is the profile name appended by the role
session name.
$ ./aws_recipes_assume_role.py --profile isecpartners --role-arn arn:aws:iam::AWS_ACCOUNT_ID:role/IAM-Scout2 --role-session-name Scout2
Users can then use their favorite tools that support profiles. For example,
Scout2 could be run with the following command line:
$ ./Scout2.py --profile isecpartners-Scout2
Note that this recipe supports MFA if the assumed role requires it:
- If you never configured your environment to work with MFA, you can provide your MFA serial number (ARN) and current token code as arguments.
- If you already configured your environment to work with MFA and stored your MFA serial in the .aws/credentials file, you just need to pass your token code as an additional argument.
- Finally, if you already initiated an STS session, you do not need to provide a new token code and can run the command as above.
Conclusion
With the release of opinel, we hope to simplify distribution and management of
the code shared between Scout2 and AWS-recipes. Additionally, we
significantly modified the workflow and credentials storage when working with
MFA-protected API calls, which allows users to use both their long-lived and STS
credentials.
IAM user management strategy (part 2)
09 Jun 2015 - Loïc Simon
The previous [IAM user management strategy]
(/aws/2015/02/24/iam_user_management.html) post discussed how
usage of IAM groups enables AWS administrators to consistently grant privileges
and enforce a number of security rules (such as MFA-protected API access). This
blog post will build on this idea by introducing category groups and
documenting new tools to improve IAM user management.
Categorize your IAM users
For a variety of reasons, applying a single set of security rules to all IAM
users is not always practical. For example, because many applications running
in AWS predate IAM roles, numerous environments still rely on the existence of
headless IAM users. Additionally, third parties may be granted access to an AWS
account for a number of reasons but may not be able to comply with the same set
of security rules that employees follow. For this reason, NCC recommends using
category groups to sort IAM users and reliably enforce appropriate security
measures. For example, one group for all human users and a second for all headless users may be
created: MFA-protected API access and password management are not relevant for
headless users. Furthermore, human users may be categorized into several groups
such as employees and contractors: API access can be restricted to the
corporate IP range for employees but might not be achievable for contractors.
Note 1: The set of category groups should define all types of IAM users that
may exist in your AWS account and each IAM user should belong to one – and
only one – category group (they may belong to other groups though).
Note 2: The common group and category groups should be used to enable enforcing
security in one’s AWS environment. Policies attached to these groups should be
carefully reviewed and grant the minimum set of privileges necessary for this
type of IAM user (e.g. credential management for humans).
Example of category groups
The rest of this article describes a number of tools developed and used by
NCC to help implement this IAM user management strategy. These tools can be found
in the AWS-Recipes repository. We will
use our test AWS environment as an example, in which we use three category groups in
addition to the AllUsers common group:
- AllHumans, the group all employees must belong to.
- AllHeadlessUsers, the group all headless IAM users must belong to.
- AllMisconfiguredUsers, a placeholder for sample misconfigured users.
We also have an IAM user naming convention that requires usernames to match the
following schema:
- Employees: firstname initial appended with lastname
- Headless user: name of the service prefixed with HeadlessUser-
- Misconfigured: description of the misconfiguration prefixed with MisconfiguredUser-
Based on these rules, we created a configuration file stored under
.aws/recipes/isecpartners.json, with isecpartners matching the profile’s
name. If you do not use profiles, the configuration will be under
.aws/recipes/default.json.
{
"common_groups": [ "AllUsers" ],
"category_groups": [
"AllHumanUsers",
"AllHeadlessUsers",
"AllMisconfiguredUsers"
],
"category_regex": [
"",
"^Headless-(.*)",
"^MisconfiguredUser-(.*)"
],
"profile_name": [ "isecpartners" ]
}
This configuration file declares the name of the common IAM group and two lists
related to the categorization of IAM users:
- A list of category groups.
- A list of regular expressions matching our naming convention.
Note 1: If you do not have a naming convention in place to distinguish the
type of user, remove the category_regex attribute from your configuration
file.
Note 2: If a regular expression is only applicable to a subset of category
groups, you must ensure that both lists have the same length and use an empty
string for groups that cannot be automatically associated (see the
AllHumanUsers group in our example).
Note 3: Use of a configuration file is not necessary as all values may be
passed as command line arguments. If a configuration file exists and a value is
passed as an argument, the value passed via the command line will be used.
The purpose of this tool is to create IAM groups whose name matches the common
and category groups specified in the above configuration file. Running the
following command results in four new groups being created if they did not
already exist.
./aws_iam_create_default_groups.py --profile isecpartners
This tool iterates through all IAM users and attempts to automatically detect
the IAM groups each user should belong to. For convenience, we recommend adding
the following to your AWS recipes configuration files:
"aws_sort_users.py": {
"create_groups": false,
},
"force_common_group": true
This specifies default values for additional arguments to be set when running
aws_iam_sort_users.py. Specifically, with these values, running this tool
will automatically add all IAM users to the common group AllUsers and will
not attempt to create the default groups (not necessary as we already did
this). Additionally, this tool checks that each IAM user belongs to one of the
category groups. If this is not the case and the username matches a regular
expression, the user is automatically added to the matching category group. Otherwise, a
multi-choice prompt appears to allow manual selection of the appropriate
category group.
Additional advantages of configuration files
Besides helping with simplification of these tools’ usage, this new AWS-recipe
configuration file can be used across tools, allowing for more consistent
rule enforcement. For example, the
aws_iam_create_user.py.
tool uses this configuration file and applies the same business logic to add
users to the common group and appropriate category group at user creation time. In
our test environment, for example, running the following command automatically
added the new user to the MisconfiguredUser group:
$ ./aws_iam_create_user.py --profile isecpartners --users MisconfiguredUser-BlogPostExample
Creating user MisconfiguredUser-BlogPostExample...
Save unencrypted value (y/n)? y
User 'MisconfiguredUser-BlogPostExample' does not belong to the mandatory common group 'AllUsers'. Do you want to remediate this now (y/n)? y
User 'MisconfiguredUser-BlogPostExample' does not belong to any of the category group (AllHumanUsers, AllHeadlessUsers, AllMisconfiguredUsers). Automatically adding...
Enabling MFA for user MisconfiguredUser-BlogPostExample...
Conclusion
While efficient and reliable management of IAM users can be challenging, using
the right strategy and tools significantly simplifies this process. Creation
and use of a naming convention for IAM users enables
automated user management and enforcement of security rules.
iSEC audit of MediaWiki
21 Apr 2015 - Valentin Leon
iSEC Partners is happy to announce the public release of our latest project with the Open Technology Fund: the review of Wikimedia Foundation’s MediaWiki. The Open Technology Fund engaged iSEC Partners to perform a source-code assisted security review of MediaWiki, the wiki engine behind Wikipedia, for a duration of two weeks at the very beginning of this year.
MediaWiki is a PHP application that evolved through a long history of patches and code rewrites. The MediaWiki engine not only drives public wikis such as Wikipedia, but it also powers many private or corporate wikis, where only a limited set of users can read and edit, therefore page content must be protected securely. The Wikimedia Foundation also seeks to ensure that their readers cannot be de-anonymized, including controls such as preventing an attacker from correlating which articles a victim reads or whether or not the victim has registered.
Most of the outward-facing attack surfaces have already been reviewed for security flaws due to the exposure of Wikipedia to the Internet. iSEC focused on traditional web vulnerabilities and on eight areas of concern prioritized by the Wikimedia Foundation. The iSEC consultants were able to find a total of fourteen issues, including two of high severity. Most of the high and medium severity vulnerabilities are related to data validation and allow for various common attacks including XSS, DoS, and CSRF. Detailed descriptions of the vulnerabilities, including proof-of-concepts, can be found in the complete report.
The Wikimedia Foundation released an article covering the research. You can also find the complete, public version of the report on our GitHub repository. We would like to thank the Open Technology Fund for making this engagement possible, and the Wikimedia Foundation team for their help and support. iSEC hopes this audit will help MediaWiki continue to bring content securely to countless readers in the future.
Work daily with enforced MFA-protected API access
03 Apr 2015 - Loïc Simon
AWS Security Token Service
The AWS Security Token Service (STS) is the gateway used to create sessions
when MFA-protected API access is enabled. This service allows IAM users to
retrieve short-lived credentials (i.e access key ID, secret access key, and
session token) in exchange for their long-lived credentials (i.e. AWS access
key ID and secret key) and their current authentication code. When enforcing
MFA-protected API access, as recommended in the previous Use and enforce
Multi-Factor
Authentication post,
IAM users must use these short-lived credentials to access other AWS services.
Challenges with MFA-protected API access
When MFA-protected API access is enforced, managing AWS access keys becomes
challenging because configuration files that contain these credentials must be
updated regularly. Users must also ensure that they do not lose their
long-lived credentials when modifying the configuration files to write their
short-lived credentials. In order to help with this workflow, iSEC wrote and
released several simple tools in the
AWS-recipes repository.
The collection of tools that we will discussed below uses the “new and
standardized way to manage credentials in the AWS
SDKs”,
meaning that SDKs are expecting to read credentials from the .aws/credentials
file under the user’s home or profile directory.
The
aws_recipes_configure_iam.py
tool allows users to configure and store their long-lived credentials in a new,
non-standard, .aws/credentials.no-mfa file. In addition to prompting for the
AWS access key ID and secret key, this tool also prompts for the MFA device
serial number because this information must be provided when making calls to
the STS API. Similar to the AWS CLI and SDKs, it supports profile names. The
following code snippet is an example of calling this tool to configure a new
profile called isecpartners:
$ ./aws_recipes_configure_iam.py --profile isecpartners
AWS Access Key ID: AWS_KEY_ID
AWS Secret Access Key: AWS_SECRET_KEY
AWS MFA serial: arn:aws:iam::AWS_ACCOUNT_ID:mfa/USER_NAME
When looking at the .aws folder, we can see that a credentials.no-mfa file
exists and that it contains the credentials that were just entered:
$ ls -l ~/.aws
total 4
-rw-r--r-- 1 loic loic 93 Apr 3 14:00 credentials.no-mfa
$ cat ~/.aws/credentials.no-mfa
[isecpartners]
aws_access_key_id = AWS_KEY_ID
aws_secret_access_key = AWS_SECRET_KEY
aws_mfa_serial = arn:aws:iam::AWS_ACCOUNT_ID:mfa/USER_NAME
Now that long-lived credentials are configured, we can use the next tool to
call the AWS STS API and request short-lived credentials that will be used
to access other AWS services.
Note: This workflow was modified since the publication of this blog post
and the .aws/credentials.no-mfa file is no longer used, refer to this new
blog post
for further details.
aws_recipes_init_sts_session.py
The
aws_recipes_init_sts_session.py
tool reads long-lived credentials configured in the .aws/credentials.no-mfa
file, prompts users for their MFA code, and retrieves STS credentials (AWS
access key ID, AWS secret key, and session token). The short-lived credentials
are then saved under the standardized .aws/credentials file to be accessible
to the AWS CLI and other tools built with the AWS SDKs. The following code
snippet demonstrates calling this tool to request an STS session token:
$ ./aws_recipes_init_sts_session.py --profile isecpartners
Enter your MFA code: 123456
Successfully configured the session token for profile 'isecpartners'.
When looking at the .aws folder, we can see that a standard credentials
file now exists as well and that it contains the short-lived credentials:
$ ls -l ~/.aws
total 8
-rw-r--r-- 1 loic loic 576 Apr 3 14:14 credentials
-rw-r--r-- 1 loic loic 179 Apr 3 14:00 credentials.no-mfa
$ cat ~/.aws/credentials
[isecpartners]
aws_access_key_id = STS_KEY_ID
aws_secret_access_key = STS_SECRET_KEY
aws_mfa_serial = arn:aws:iam::AWS_ACCOUNT_ID:mfa/USER_NAME
aws_session_token = AWS//////////SESSION_TOKEN
Now that the short-lived credentials are configured, we can use the AWS CLI or
other tools built with the AWS SDKs that read credentials from this standard
location. When the STS session expires, users just need to re-run the
aws_recipes_init_sts_session.py
tool and the standard credentials file will be updated with new valid
short-lived credentials.
Note: This workflow was modified since the publication of this blog post
and the .aws/credentials.no-mfa file is no longer used, refer to this new
blog post
for further details.
Conclusion
By using this pair of tools to manage their AWS access keys, IAM users can
easily use the AWS CLI and other tools built with various AWS SDKs in
environments that have been secured and enforce MFA-protected API access.
Use and enforce Multi-Factor Authentication
02 Apr 2015 - Loïc Simon
What is Multi-Factor Authentication?
When enabled, Multi-Factor Authentication (MFA) provides strong
defense-in-depth against compromises of credentials. MFA-enabled users have a
device that periodically generates a new authentication code (i.e. one-time
password); they need to enter the current authentication code along with their
static credentials (i.e. username and password) in order to successfully
authenticate. In addition to supporting MFA when accessing the web console
(i.e. password-based authentication), AWS also offers MFA-protected API
access for users who work with AWS access keys. Through the Security Token
Service (STS), IAM users can request temporary credentials in exchange for
their long-lived credentials (i.e. AWS access key ID and secret key) and
their current authentication code.
Why should one use and enforce MFA?
For companies deploying their application in the cloud, a breach that results
in unauthorized access to the management console — or API — is the
worst-case scenario. While a number of AWS administrators have realized the
importance of enabling MFA when they access the web console, a limited number
of them enforce MFA-protected API access. This represents a huge gap in one’s
security posture because AWS access keys do not come with as many security
features as passwords do:
- AWS administrators can enforce password expiration; this is currently not
possible for AWS access keys.
- While it is probably safe to assume that most AWS administrators do not store
their password in plaintext, most of them use AWS access keys. By design, these
keys are meant to be stored in plaintext files that are accessed by tools built
with the various AWS SDKs.
- A lost password is a forgotten password; a lost key is a key stored in a lost
file, which may be on an unencrypted storage device (e.g. hard drive or USB Flash
drive).
Because AWS access keys are long-lived credentials that are stored in plaintext
files, they are more susceptible to compromise than passwords. It is therefore
necessary to enable MFA when the AWS API is accessed using these keys and not
only when users sign in using their passwords.
How can one enforce MFA?
Unfortunately, at time of writing, AWS does not offer an option to enforce
MFA-protected API access via a global setting. Therefore, AWS account
administrators must carefully manage their IAM users and develop a strategy to
reliably achieve this. In order to enforce MFA-protected API access, iSEC
recommends the following:
- Create a common IAM group that all IAM users belong to, as discussed in the previous IAM user management strategy post.
- Add the following policy (also available on Github) to enforce MFA for all users who belong to this group.
This policy will enforce MFA regardless of how the IAM user authenticated with
AWS; it will be effective whether they use password-based or key-based
authentication.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Action": "*",
"Resource": "*",
"Condition": {
"Null":{"aws:MultiFactorAuthAge":"true"}
}
},
{
"Effect": "Deny",
"Action": "*",
"Resource": "*",
"Condition": {
"NumericGreaterThan":{"aws:MultiFactorAuthAge":"28800"}
}
}
]
}
The first statement in the above policy denies all actions if the
aws:MultiFactorAuthAge key is not present; this key only exists if MFA is
used1.
The second statement verifies that the validation of the MFA code was performed
less than eight hours ago. Temporary credentials may be valid for a duration
between fifteen minutes and thirty-six hours2. iSEC recommends requiring
users to initiate a new session at least once a day.
Note: An “explicit deny” means that, regardless of other policies granted
to a user, this deny rule will prevail. More information about the IAM policy
evaluation logic can be found in the AWS documentation at
https://docs.aws.amazon.com/IAM/latest/UserGuide/AccessPolicyLanguage_EvaluationLogic.html.
Use AWS Scout2 to detect users without MFA
The default ruleset used by AWS
Scout2 includes a rule that checks for
IAM users who have password-based authentication enabled but do not have an MFA
device configured. If Scout2 detects IAM users with password-based
authentication enabled and no MFA device, it will document a “Lack of MFA”
security risk in the IAM menu dropdown, as illustrated in the below screenshot.
When clicked, this “Lack of MFA” link filters the list of IAM users to display
those who have password-based authentication enabled but no MFA device
configured. The red “No” following “Multi-Factor enabled” indicates a danger
tied to that particular IAM user.
Users of the AWS CLI (and other command line tools) have several methods to
configure their credentials, such as environment variables, configuration
files, or command line arguments. However, updating these settings on a daily
basis when MFA-protected API access is enabled is inconvenient. To help
facilitate this work flow, iSEC has created a set of Python tools and released
them in the AWS-recipes
repository. Further details about these tools will be published in the next
blog post.
Additional information about MFA with AWS is available in the AWS
documentation at
https://docs.aws.amazon.com/IAM/latest/UserGuide/Using_ManagingMFA.html.
Conclusion
Enforcing Multi-Factor Authentication for all IAM users is extremely important
in order to mitigate the risks of credentials compromise (especially the AWS
access key ID and secret). This aspect of security is commonly overlooked and
may result in catastrophic damages. By using a strict strategy for management
of IAM users and the above IAM policy, AWS administrators may significantly
reduce risks of account compromise.