How to send log data to AWS S3 server by FluentD

1. First you need S3 and IAM on AWS. Let's create new user on IAM:















2. I will not cover every single step, since it is easy just follow their instructions.
Be sure to download user credentials(which has aws_key_id, aws_sec_key) upon creating it(it will be used on td-agent.conf). Assign your created user to any group, if does not have group, then create it. After all attach policy(permission) to your group:


3. Create the bucket on S3 named "generjet" for instance :

4. Install td-agent on your server, and configure it as follows:
  <source>
    @type tail
    format none
    path /home/ubuntu/onepage/log/development.log #Change it to your log path
    pos_file /var/log/td-agent/access.log.pos
    refresh_interval 10
    tag s3.send
  </source>
  <match s3.send>
    @type s3
    aws_key_id xxxxxx # Change it(Copy from credentials.csv downloaded from IAM when created new user)
    aws_sec_key xxxxxx
# Change it(Copy from credentials.csv downloaded from IAM when created new user)
    s3_bucket generjet # Bucket name on S3
    s3_region us-west-2 # AWS region
    path logs/ # Path on S3 bucket
    buffer_path /var/log/td-agent/s3 # Buffering path on your server
    time_slice_format %Y%m%d%H
    time_slice_wait 10m # It seems minimum 10m can work properly
    utc
    buffer_chunk_limit 256m
  </match>

5. $ sudo service td-agent restart
that's all