Skip Ribbon Commands
Skip to main content
Navigate Up
Sign In

Quick Launch

Average Rating:

facebook Twitter
Print Bookmark Alert me when this article is updated


Streaming Ingestion Service creating one file per message when S3 is picked as target
Problem Description
When an S3 v2 connection is used as target and a Kafka Topic as source , we see that we have one file in S3 target per message in Kafka. 
​This is the default behavior of streaming ingestion service, to avoid this, its recommended to use a combiner transformation as shown: 


Minimum number Events= No of events/messages it stores per file
Time Limit= If you donot receive the previously said no of messages in the mentioned amount of time, still want to create the file 
Delimiter = Delimiter in the S3 file (As fo R32 it cant be "\n"

​Please use combiner transformation as mentioned above to downsize the number fo files created in S3.
More Information
Applies To
Product: Cloud Data Integration
Problem Type: Configuration
User Type: Architect
Project Phase: Configure
Product Version:
Operating System:
Other Software:

Last Modified Date:9/9/2019 9:15 PMID:589269
People who viewed this also viewed


Did this KB document help you?

What can we do to improve this information (2000 or fewer characters)