Using Kinesis agent to push data to multiple streams on multiple AWS accounts
A single Kinesis Agent cannot push data to multiple accounts. So, we need to run multiple independent Agents , one Agent for every account.
When you install Kinesis agent as outlined in the following documentation , it runs as a service.
http://docs.aws.amazon.com/streams/latest/dev/writing-with-agents.html#agent-config-settings
The service executes a JAVA class “com.amazon.kinesis.streaming.agent.Agent” with a command like :
1 |
java -server -Xms256m -Xmx512m -XX:OnOutOfMemoryError="/bin/kill -9 %p" -cp /usr/share/aws-kinesis-agent/lib:..: com.amazon.kinesis.streaming.agent.Agent |
A help on this command or AgentOptions would provide the options that you can specify on the agent. So, you could have multiple agents , each running with different configurations. Currently(Oct 2016), each Agent can only have one authentication configuration set. So, each Agent process can only write to streams on just one account. So, with separate config files you can send data to different accounts. The authentication for each agent can be using a IAM user credentials or with an Assumed Role.
Example agent configurations and how to run them :
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
agent1.json -> { "awsAccessKeyId":"DEV ACCOUNT ", "awsSecretAccessKey":"ExxxxxxxxxxxxxxxxxxxxxxxVH", "cloudwatch.emitMetrics": true, "firehose.endpoint": "https://firehose.us-west-2.amazonaws.com" , "kinesis.endpoint": "https://kinesis.us-west-2.amazonaws.com" , "flows": [ { "filePattern": "/app/errors*.log", "kinesisStream": "stream_one" }, { "filePattern": "/tmp/app2.log*", "deliveryStream": "firehosedeliverystream_1" } ] } agent2.json -> { "awsAccessKeyId":"QA ACCOUNT ", "awsSecretAccessKey":"Ezzzzzzzzzzzzzzzzzzzzzz", "cloudwatch.emitMetrics": true, "firehose.endpoint": "https://firehose.us-west-2.amazonaws.com" , "kinesis.endpoint": "https://kinesis.us-west-2.amazonaws.com" , "flows": [ { "filePattern": "/app/errors*.log", "kinesisStream": "stream_two" } ] } |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
Usage: aws-kinesis-agent [options] Options: --configuration, -c Path to the configuration file for the agent. Default: /etc/aws-kinesis/agent.json --help, -h Display this help message --log-file, -l Path to the agent's log file. --log-level, -L Log level. Can be one of: TRACE,DEBUG,INFO,WARN,ERROR. # Sample commands # to run multiple agent daemons using aws-kinesis-agent-manual-run.sh # and different Agent options sudo nohup /bin/sh -c /usr/bin/aws-kinesis-agent-manual-run.sh -c /etc/aws-kinesis/agent1.json -l /var/log/aws-kinesis-agent/aws-kinesis-agent1.log -L DEBUG & sudo nohup /bin/sh -c /usr/bin/aws-kinesis-agent-manual-run.sh -c /etc/aws-kinesis/agent2.json -l /var/log/aws-kinesis-agent/aws-kinesis-agent2.log -L DEBUG & |
Building Kinesis Agent :
Instead of running agent as service , If you wish to build and use your own Agent , you can use https://github.com/awslabs/amazon-kinesis-agent .
A sample script to do so.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# Build and run Kinesis Agent sudo yum update -y echo "install dev packages" sudo yum -y install wget tar git subversion gcc gcc-c++ make cmake ant fuse autoconf automake libtool sharutils xmlto lzo-devel zlib-devel fuse-devel openssl-devel python-devel libxml2-devel libxslt-devel cyrus-sasl-devel sqlite-devel mysql-devel openldap-devel rpm-build createrepo redhat-rpm-config echo "installing Maven" sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo sudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo sudo yum install -y apache-maven git clone https://github.com/awslabs/amazon-kinesis-agent.git cd /home/ec2-user/amazon-kinesis-agent/amazon-kinesis-agent # Build Kinesis agent which generates a JAR /target/amazon-kinesis-agent-1.1.jar mvn clean install -Dgpg.skip=true dependency:copy-dependencies # use aws-kinesis-agent-manual-run.sh script runner to run multiple agents with optional config/LogFile/Log settings. sudo nohup /bin/sh /home/ec2-user/amazon-kinesis-agent/bin/aws-kinesis-agent-manual-run.sh -c /home/ec2-user/agent1.json -l /var/log/aws-kinesis-agent/aws-kinesis-agent1.log -L DEBUG & sudo nohup /bin/sh /home/ec2-user/amazon-kinesis-agent/bin/aws-kinesis-agent-manual-run.sh -c /etc/aws-kinesis/agent2.json -l /var/log/aws-kinesis-agent/aws-kinesis-agent2.log -L DEBUG & # Alternatively , you can run this with JAVA command. sudo nohup java -server -Xms256m -Xmx512m -XX:OnOutOfMemoryError="/bin/kill -9 %p" -cp /home/ec2-user/amazon-kinesis-agent/target/amazon-kinesis-agent-1.1.jar:/home/ec2-user/amazon-kinesis-agent/target/dependency/* com.amazon.kinesis.streaming.agent.Agent -c /etc/aws-kinesis/agent.json -l /var/log/aws-kinesis-agent/aws-kinesis-agent.log -L DEBUG & |