Using Kinesis agent to push data to multiple streams on multiple AWS accounts

Written by mannem on . Posted in Kinesis

A single Kinesis Agent cannot push data to multiple accounts. So, we need to run multiple independent Agents , one Agent for every account.

This post will discuss about Kinesis agent and guides you run multiple agents on Amazon Ec2 instance. It also have some sample scripts to build and run your own Agent from source.

When you install Kinesis agent as outlined in the following documentation , it runs as a service. kinesis

http://docs.aws.amazon.com/streams/latest/dev/writing-with-agents.html#agent-config-settings

The service executes a JAVA class “com.amazon.kinesis.streaming.agent.Agent” with a command like :

A help on this command or AgentOptions would provide the options that you can specify on the agent. So, you could have multiple agents , each running with different configurations. Currently(Oct 2016), each Agent can only have one authentication configuration set. So, each Agent process can only write to streams on just one account. So, with separate config files you can send data to different accounts. The authentication for each agent can be using a IAM user credentials or with an Assumed Role.

Example agent configurations and how to run them :


Building Kinesis Agent :

Instead of running agent as service , If you wish to build and use your own Agent , you can use https://github.com/awslabs/amazon-kinesis-agent .

A sample script to do so.

Trackback from your site.

Leave a comment