Blog

How to Run Queries in AWS Lambda Serverless for Troubleshooting, Control, and Reporting



How to run real-time queries for troubleshooting, cost and resource control, and management reporting

Introduction

The purpose of this blog post is to illustrate how one can use AWS lambda functions to securely query an environment and send the output to slack.

I realize that most readers will not be users of Ravello. But the overall process can be used for any type of system query.

For example, just remove the Ravello portions in the handler function (below) and substitute your own code to query a system of your choice

Ex.

In addition to this post, all the code described herein can be found on GitHub at https://github.com/Solinea/lambda-ravello. The code works and currently runs in production.

Between the GitHub repository contents and the descriptions below, you should get a fairly complete picture of how useful lambdas can be for everyday chores.

Prerequisites

This assumes you have a working AWS ID, have installed the AWS Commandline Tools and Python 2.7.

Background

We have a test environment called Ravello in which we have several VMs. The VMs are supposed to run only during normal business hours. We take them down at the end of each day to avoid additional costs.

The problem was detecting when VMs were left running. There was no notification mechanism. We simply did not know when a VM was accidently left active for hours or even days. While this seems trivial, the costs mount up and can be substantial.

What would a notification process look like? We use Slack for all our internal communications. Posting a message there would be ideal. Naturally, all of this would have to be secure. We would not want to expose user IDs or passwords anywhere.

Therefore, we have 3 requirements:

  1. That we are notified of any running VMs outside business hours
  2. That these notifications go to Slack.
  3. That communication be secure

The Solution

This is where AWS lambda functions come in handy. We can use a lambda function to periodically query Ravello (#1 below), report the output on Slack (#2), and show it on one of our channels (#3).

Creating The AWS Lambda Role

Before we can upload a new lambda function to AWS, we need to create an IAM role to support it. This is called an IAM Execution Role. Here are the steps

  1. Sign in to the IAM console at https://console.aws.amazon.com/iam/
  2. Follow the steps in “Creating a Role for an AWS Service” in the IAM User Guide to create an IAM role (execution role). As you follow the steps to create a role, note the following:
    1. In Role Name, use a name that is unique within your AWS account (for example, lambda_ravello).
    2. In Select Role Type, choose AWS Service Roles, and then choose AWS Lambda. This grants the AWS Lambda service permissions to assume the role.
    3. In Attach Policy, choose AWSLambdaBasicExecutionRole.

Creating The Slack Incoming Web Hook

In order to post messages to Slack, we need to define an “Incoming Webhook”. This is basically a special URL used for posting messages to a channel. To create a webhook, do the following

  1. On Slack, go to the channel where you would like your lambda messages to be posted
  2. Click on the “Conversation Settings” icon (it looks like a little cog)
  3. Select “Add an app or integration”. This will take you to a web page.
  4. Search for and select “Incoming Webhooks”
  5. Click on the green “Add Configuration” button
  6. Pick the channel you want and then click the green “Add Incoming Webhooks Integration”
  7. COPY AND SAVE the Webhook URL displayed
  8. Select the green “Save Setting” button at the bottom
  9. That’s it. You now have an incoming webhook for that channel.

Source Code

For the lambda function itself, all you need is a directory and a couple source files.

Make a directory somewhere on your local machine called “audit”. In it, create 2 files, audit.py and lambda.json

Set the content of lambda.jason to the following. It is used primarily by the lambda-uploader utility described below.

Notes:

  1. Make sure you set the region to match your own
  2. Substitute your account number for “999999999999”

Set the contents of audit.py to the following. This is the actual lambda function used in AWS.

Simple, right? Well, not completely. There are a couple security issues we have to address.
Notice there are 3 variables which aren’t initialized in the above source. They are ENCRYPTED_RAVELLO_USER, ENCRYPTED_RAVELLO_PASSWORD and ENCRYPTED_SLACK_HOOK_URL.
To supply these values, we have to do some work on AWS first.

Creating The AWS Encryption Key

To encrypt our secrets, we first need to create a key for them on AWS. To do that

  1. Sign in to the IAM console at https://console.aws.amazon.com/iam/
  2. Select Encryption Keys
  3. Press the Create key button
  4. Give the new key an alias (e.g. lambda_key)
  5. On the Key Administrator and Key User panels, select the Lambda Role lambda_ravello we created (above).
  6. Select Next until completed
  7. You should now have a new entry called lambda_key in your list of keys

Using The AWS Encryption Key

What we’re going to do now is encrypt our Ravello user id/password and as well as our Slack webhook url. This will allow us to finally add values to the uninitialized variables in our audit.py file above.

  1. For each of the entities Ravello User ID, Ravello Password, and Slack Webhook URL, do the following
    1. Go to your local command line and encrypt your strings using the newly-created key like this:
    2. The output should look something like this:
    3. The resulting output in CiphertextBlob will be used for the content of the uninitialized variables in our audit.py source code above. For example if the plaintext content string is the Ravello User Id, then the CiphertextBlob content will be used for ENCRYPTED_RAVELLO_USER.

Upload Utility

Now we’re almost ready to upload our new lambda function. To make lambda creation and uploading easier, let’s install a python utility called lambda-uploader. We will use this to stage our lambda function on AWS.

Upload The New Lambda Function

To upload our lambda function, go to the parent directory of the audit directory. Then issue the command

If successful, the output should look like this:

Associating The Lambda Function With Our Encryption Key

Even though we now have successfully uploaded our lambda function, we still have more to do. First, we need to tell the function which key to use for deciphering the encrypted strings in our audit.py file. Do the following

  1. Sign in to the Lambda console at https://console.aws.amazon.com/lambda/
  2. Click on our function audit
  3. Select “Configuration” from the tabs listed on the page
  4. Select the “Advanced” fold-down menu
  5. Scroll to the bottom of the page until you see the “KMS Key” field
  6. Select our key lambda_key from the fold-down list of keys.
  7. Press the “Save” button at the top of the page.
  8. At this point, if you press “Test” or “Save And Test”, you should see output go to your Slack channel.

Setting The Lambda Function To Execute On A Timed Basis

Now we need to define when our function should be executing.

  1. Sign in to the Lambda console at https://console.aws.amazon.com/lambda/
  2. Click on our function audit
  3. Select “Triggers” from the tabs listed
  4. Click on “Add Trigger”
  5. Click in the dashed icon and you will get a list of options
  6. Select “Cloudwatch Events”
  7. Select “Create A New Role”
  8. Set the new role’s name (e.g. ravello)
  9. Set the “Schedule Expression” in the bottom field (e.g.cron(0 17 ? * MON-FRI *)). This tells the trigger when to execute the lambda function.
  10. Click “Submit”
  11. You’re Done! Now all you have to do is wait until the timer trips and you should see output on your Slack channel!


Contact Solinea for More Information



Author: Tennis Smith, Senior DevOps Engineer, Solinea