With boto3 being stable and generally available¹, NCC took the opportunity to migrate Scout2 and AWS-recipes to boto3. As part of that migration effort, we decided to publish the formerly-known-as AWSUtils repository – used by Scout2 and AWS-recipes – as a python package required by these tools, rather than requiring users to work with Git submodules. We’ve also added more flexibility when working with MFA-protected API calls and improved versioning across the project.

opinel

To avoid name conflicts, we decided to rename the shared AWSUtils code to a less misleading name: opinel. The opinel package is published on PyPI, and thus can be installed using pip and easy_install. The corresponding source code is still open-sourced on Github at https://github.com/iSECPartners/opinel. As a result, Scout2 and AWS-recipes have been modified to list opinel as a requirement, which significantly simplifies installation and management of this shared code.

Support for Python 2.7 and 3.x

Because boto3 supports both Python2 and Python3, we decided to make sure that the code we build on top of that package has similar properties. As a result, the latest versions of Scout2 and AWS-recipes support Python 2.7 and 3.x. Note that opinel will NOT work with Python 2.6.

Modification of the MFA workflow

As requested by a user of AWS-recipes², we modified the workflow when using MFA-protected API access to no longer store the long-lived credentials in a separate file. As a result, the .aws/credentials.no-mfa file is no longer supported and all credentials are stored in the standard AWS credentials file under .aws/credentials. Usage of the existing tools remains unchanged, but the long-lived credentials are now accessible via a new profile name: profile_name-nomfa. This allows users to work with both STS and long-lived credentials if need be.

If you already had configured your environment to work with MFA-protected API access, you will need to copy your long-lived credentials back to the .aws/credentials file. This can be done with a simple command such as the following:

cat ~/.aws/credentials.no-mfa | sed -e 's/]$/-nomfa]/g' >> ~/.aws/credentials

Support to use assumed-role credentials

With this new workflow implemented, we created a new recipe that allows configuration of role-credentials in the .aws/credentials file. When the following command is run, it uses the credentials associated with the isecpartners profile to request role credentials for the IAM-Scout2 role. The role credentials are then written in the .aws/credentials file in a new profile named isecpartners-Scout2, which is the profile name appended by the role session name.

$ ./aws_recipes_assume_role.py --profile isecpartners --role-arn arn:aws:iam::AWS_ACCOUNT_ID:role/IAM-Scout2 --role-session-name Scout2

Users can then use their favorite tools that support profiles. For example, Scout2 could be run with the following command line:

$ ./Scout2.py --profile isecpartners-Scout2

Note that this recipe supports MFA if the assumed role requires it:

• If you never configured your environment to work with MFA, you can provide your MFA serial number (ARN) and current token code as arguments.
• If you already configured your environment to work with MFA and stored your MFA serial in the .aws/credentials file, you just need to pass your token code as an additional argument.
• Finally, if you already initiated an STS session, you do not need to provide a new token code and can run the command as above.

Conclusion

With the release of opinel, we hope to simplify distribution and management of the code shared between Scout2 and AWS-recipes. Additionally, we significantly modified the workflow and credentials storage when working with MFA-protected API calls, which allows users to use both their long-lived and STS credentials.

The previous [IAM user management strategy] (/aws/2015/02/24/iam_user_management.html) post discussed how usage of IAM groups enables AWS administrators to consistently grant privileges and enforce a number of security rules (such as MFA-protected API access). This blog post will build on this idea by introducing category groups and documenting new tools to improve IAM user management.

Categorize your IAM users

For a variety of reasons, applying a single set of security rules to all IAM users is not always practical. For example, because many applications running in AWS predate IAM roles, numerous environments still rely on the existence of headless IAM users. Additionally, third parties may be granted access to an AWS account for a number of reasons but may not be able to comply with the same set of security rules that employees follow. For this reason, NCC recommends using category groups to sort IAM users and reliably enforce appropriate security measures. For example, one group for all human users and a second for all headless users may be created: MFA-protected API access and password management are not relevant for headless users. Furthermore, human users may be categorized into several groups such as employees and contractors: API access can be restricted to the corporate IP range for employees but might not be achievable for contractors.

Note 1: The set of category groups should define all types of IAM users that may exist in your AWS account and each IAM user should belong to one – and only one – category group (they may belong to other groups though).

Note 2: The common group and category groups should be used to enable enforcing security in one’s AWS environment. Policies attached to these groups should be carefully reviewed and grant the minimum set of privileges necessary for this type of IAM user (e.g. credential management for humans).

Example of category groups

The rest of this article describes a number of tools developed and used by NCC to help implement this IAM user management strategy. These tools can be found in the AWS-Recipes repository. We will use our test AWS environment as an example, in which we use three category groups in addition to the AllUsers common group:

1. AllHumans, the group all employees must belong to.
2. AllHeadlessUsers, the group all headless IAM users must belong to.
3. AllMisconfiguredUsers, a placeholder for sample misconfigured users.

We also have an IAM user naming convention that requires usernames to match the following schema:

1. Employees: firstname initial appended with lastname
2. Headless user: name of the service prefixed with HeadlessUser-
3. Misconfigured: description of the misconfiguration prefixed with MisconfiguredUser-

Based on these rules, we created a configuration file stored under .aws/recipes/isecpartners.json, with isecpartners matching the profile’s name. If you do not use profiles, the configuration will be under .aws/recipes/default.json.

{
    "common_groups": [ "AllUsers" ],
    "category_groups": [
        "AllHumanUsers",
        "AllHeadlessUsers",
        "AllMisconfiguredUsers"
    ],
    "category_regex": [
        "",
        "^Headless-(.*)",
        "^MisconfiguredUser-(.*)"
    ],
    "profile_name": [ "isecpartners" ]
}

This configuration file declares the name of the common IAM group and two lists related to the categorization of IAM users:

1. A list of category groups.
2. A list of regular expressions matching our naming convention.

Note 1: If you do not have a naming convention in place to distinguish the type of user, remove the category_regex attribute from your configuration file.

Note 2: If a regular expression is only applicable to a subset of category groups, you must ensure that both lists have the same length and use an empty string for groups that cannot be automatically associated (see the AllHumanUsers group in our example).

Note 3: Use of a configuration file is not necessary as all values may be passed as command line arguments. If a configuration file exists and a value is passed as an argument, the value passed via the command line will be used.

Create your default groups with aws_iam_create_default_groups.py

The purpose of this tool is to create IAM groups whose name matches the common and category groups specified in the above configuration file. Running the following command results in four new groups being created if they did not already exist.

./aws_iam_create_default_groups.py --profile isecpartners

(Automatically) sort IAM users with aws_iam_sort_users.py.

This tool iterates through all IAM users and attempts to automatically detect the IAM groups each user should belong to. For convenience, we recommend adding the following to your AWS recipes configuration files:

"aws_sort_users.py": {
    "create_groups": false,
},
"force_common_group": true

This specifies default values for additional arguments to be set when running aws_iam_sort_users.py. Specifically, with these values, running this tool will automatically add all IAM users to the common group AllUsers and will not attempt to create the default groups (not necessary as we already did this). Additionally, this tool checks that each IAM user belongs to one of the category groups. If this is not the case and the username matches a regular expression, the user is automatically added to the matching category group. Otherwise, a multi-choice prompt appears to allow manual selection of the appropriate category group.

Additional advantages of configuration files

Besides helping with simplification of these tools’ usage, this new AWS-recipe configuration file can be used across tools, allowing for more consistent rule enforcement. For example, the aws_iam_create_user.py. tool uses this configuration file and applies the same business logic to add users to the common group and appropriate category group at user creation time. In our test environment, for example, running the following command automatically added the new user to the MisconfiguredUser group:

$ ./aws_iam_create_user.py --profile isecpartners --users MisconfiguredUser-BlogPostExample
Creating user MisconfiguredUser-BlogPostExample...
Save unencrypted value (y/n)? y
User 'MisconfiguredUser-BlogPostExample' does not belong to the mandatory common group 'AllUsers'. Do you want to remediate this now (y/n)? y
User 'MisconfiguredUser-BlogPostExample' does not belong to any of the category group (AllHumanUsers, AllHeadlessUsers, AllMisconfiguredUsers). Automatically adding...
Enabling MFA for user MisconfiguredUser-BlogPostExample...

Conclusion

While efficient and reliable management of IAM users can be challenging, using the right strategy and tools significantly simplifies this process. Creation and use of a naming convention for IAM users enables automated user management and enforcement of security rules.

iSEC Partners is happy to announce the public release of our latest project with the Open Technology Fund: the review of Wikimedia Foundation’s MediaWiki. The Open Technology Fund engaged iSEC Partners to perform a source-code assisted security review of MediaWiki, the wiki engine behind Wikipedia, for a duration of two weeks at the very beginning of this year.

MediaWiki is a PHP application that evolved through a long history of patches and code rewrites. The MediaWiki engine not only drives public wikis such as Wikipedia, but it also powers many private or corporate wikis, where only a limited set of users can read and edit, therefore page content must be protected securely. The Wikimedia Foundation also seeks to ensure that their readers cannot be de-anonymized, including controls such as preventing an attacker from correlating which articles a victim reads or whether or not the victim has registered.

Most of the outward-facing attack surfaces have already been reviewed for security flaws due to the exposure of Wikipedia to the Internet. iSEC focused on traditional web vulnerabilities and on eight areas of concern prioritized by the Wikimedia Foundation. The iSEC consultants were able to find a total of fourteen issues, including two of high severity. Most of the high and medium severity vulnerabilities are related to data validation and allow for various common attacks including XSS, DoS, and CSRF. Detailed descriptions of the vulnerabilities, including proof-of-concepts, can be found in the complete report.

The Wikimedia Foundation released an article covering the research. You can also find the complete, public version of the report on our GitHub repository. We would like to thank the Open Technology Fund for making this engagement possible, and the Wikimedia Foundation team for their help and support. iSEC hopes this audit will help MediaWiki continue to bring content securely to countless readers in the future.

AWS Security Token Service

The AWS Security Token Service (STS) is the gateway used to create sessions when MFA-protected API access is enabled. This service allows IAM users to retrieve short-lived credentials (i.e access key ID, secret access key, and session token) in exchange for their long-lived credentials (i.e. AWS access key ID and secret key) and their current authentication code. When enforcing MFA-protected API access, as recommended in the previous Use and enforce Multi-Factor Authentication post, IAM users must use these short-lived credentials to access other AWS services.

Challenges with MFA-protected API access

When MFA-protected API access is enforced, managing AWS access keys becomes challenging because configuration files that contain these credentials must be updated regularly. Users must also ensure that they do not lose their long-lived credentials when modifying the configuration files to write their short-lived credentials. In order to help with this workflow, iSEC wrote and released several simple tools in the AWS-recipes repository.

The collection of tools that we will discussed below uses the “new and standardized way to manage credentials in the AWS SDKs”, meaning that SDKs are expecting to read credentials from the .aws/credentials file under the user’s home or profile directory.

aws_recipes_configure_iam.py

The aws_recipes_configure_iam.py tool allows users to configure and store their long-lived credentials in a new, non-standard, .aws/credentials.no-mfa file. In addition to prompting for the AWS access key ID and secret key, this tool also prompts for the MFA device serial number because this information must be provided when making calls to the STS API. Similar to the AWS CLI and SDKs, it supports profile names. The following code snippet is an example of calling this tool to configure a new profile called isecpartners:

$ ./aws_recipes_configure_iam.py --profile isecpartners
AWS Access Key ID: AWS_KEY_ID
AWS Secret Access Key: AWS_SECRET_KEY
AWS MFA serial: arn:aws:iam::AWS_ACCOUNT_ID:mfa/USER_NAME

When looking at the .aws folder, we can see that a credentials.no-mfa file exists and that it contains the credentials that were just entered:

$ ls -l ~/.aws
total 4
-rw-r--r-- 1 loic loic 93 Apr  3 14:00 credentials.no-mfa
$ cat ~/.aws/credentials.no-mfa
[isecpartners]
aws_access_key_id = AWS_KEY_ID
aws_secret_access_key = AWS_SECRET_KEY
aws_mfa_serial = arn:aws:iam::AWS_ACCOUNT_ID:mfa/USER_NAME

Now that long-lived credentials are configured, we can use the next tool to call the AWS STS API and request short-lived credentials that will be used to access other AWS services.

Note: This workflow was modified since the publication of this blog post and the .aws/credentials.no-mfa file is no longer used, refer to this new blog post for further details.

aws_recipes_init_sts_session.py

The aws_recipes_init_sts_session.py tool reads long-lived credentials configured in the .aws/credentials.no-mfa file, prompts users for their MFA code, and retrieves STS credentials (AWS access key ID, AWS secret key, and session token). The short-lived credentials are then saved under the standardized .aws/credentials file to be accessible to the AWS CLI and other tools built with the AWS SDKs. The following code snippet demonstrates calling this tool to request an STS session token:

$ ./aws_recipes_init_sts_session.py --profile isecpartners
Enter your MFA code: 123456
Successfully configured the session token for profile 'isecpartners'.

When looking at the .aws folder, we can see that a standard credentials file now exists as well and that it contains the short-lived credentials:

$ ls -l ~/.aws
total 8
-rw-r--r-- 1 loic loic 576 Apr  3 14:14 credentials
-rw-r--r-- 1 loic loic 179 Apr  3 14:00 credentials.no-mfa
$ cat ~/.aws/credentials
[isecpartners]
aws_access_key_id = STS_KEY_ID
aws_secret_access_key = STS_SECRET_KEY
aws_mfa_serial = arn:aws:iam::AWS_ACCOUNT_ID:mfa/USER_NAME
aws_session_token = AWS//////////SESSION_TOKEN

Now that the short-lived credentials are configured, we can use the AWS CLI or other tools built with the AWS SDKs that read credentials from this standard location. When the STS session expires, users just need to re-run the aws_recipes_init_sts_session.py tool and the standard credentials file will be updated with new valid short-lived credentials.

Note: This workflow was modified since the publication of this blog post and the .aws/credentials.no-mfa file is no longer used, refer to this new blog post for further details.

Conclusion

By using this pair of tools to manage their AWS access keys, IAM users can easily use the AWS CLI and other tools built with various AWS SDKs in environments that have been secured and enforce MFA-protected API access.