Query AWS ES cluster by signing http requests with AWS IAM roles (python)

Written by mannem on . Posted in Elasticsearch

The AWS public facing documentation provides some python examples to sign the http reqests with IAM users’s to access other AWS resources. In this case, AWS ES cluster whose access policies are restricted to those IAM users.

If you wish to restrict the access to ES cluster with IAM roles instead, the signing process is a bit different.

The document (http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html) seem to be only for IAM users but not for IAM roles.


Changing the signed header

Signing requests with IAM roles need additional header called ‘session token’ added the request header using a header name of ‘x-amz-security-token’.

So, in the ESrequest.py replacing this line:

headers = {'x-amz-date':amzdate, 'Authorization':authorization_header}

With the following should work for signing requests with IAM roles cred’s.

headers = {'x-amz-date':amzdate, 'Authorization':authorization_header, 'x-amz-security-token':token}

Where, the session ‘token’ string should be obtained from the corresponding IAM role. (These credential trio for roles are rotated frequently and they have an expiration date. So make sure you are using unexpired token)


Obtaining ‘token’ string from IAM Roles

If an EC2 instance is assuming a role, you can get it with

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/MyEc2Role
(http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials)

In python, all these credentials can be obtained with ‘requests’ module and parsing them accordingly.

The token can be obtained similarly from other services assuming IAM roles like Lambda etc.


Find the end-to-end code here : https://github.com/mannem/elasticsearchDev/blob/master/ES_V4Signing_IAMRoles

Trackback from your site.

Leave a comment

  • cloudformation

    cloudformation

    pipeline

    Data-pipelines

    directoryservice

    directoryservicez

    cloudtrail

    cloudtrail

    config

    config

    trustedadvisor

    Trustedadvisor

  • snap

    Snapshot

    glacier

    Glacie

    storagegw

    Storage Gatewa

    s3

    S3

    cloudFront

    Cloud Front

  • r53

    Route 53

    lambda

    lambd

    directConnect

    DirectConnect

    vpc

    VPC

    kinesis

    Kinesis

    emr

    Emr

  • sns

    SNS

    transcoder

    Transcoder

    sqs

    SQS

    cloudsearch

    Cloud Search

    appstream

    App Stream

    ses

    SES

  • opsworks

    opsworks

    cloudwatch

    Cloud Watch

    beanstalk

    Elastic Beanstalk

    codedeploy

    Code Deploy

    IAM

    IAM

  • dynamodb

    dynamodb

    rds

    RDS

    elasticache

    ElastiCache

    redshift

    Redshift

    simpledb

    simpledb