Skip Ribbon Commands
Skip to main content
Navigate Up
Sign In

Quick Launch

Average Rating:

facebook Twitter
Email
Print Bookmark Alert me when this article is updated

Feedback

Streaming Ingestion Service creating one file per message when S3 is picked as target
Problem Description
When an S3 v2 connection is used as target and a Kafka Topic as source , we see that we have one file in S3 target per message in Kafka. 
Cause
​This is the default behavior of streaming ingestion service, to avoid this, its recommended to use a combiner transformation as shown: 

Combiner.JPG 

Minimum number Events= No of events/messages it stores per file
Time Limit= If you donot receive the previously said no of messages in the mentioned amount of time, still want to create the file 
Delimiter = Delimiter in the S3 file (As fo R32 it cant be "\n"

Solution
​Please use combiner transformation as mentioned above to downsize the number fo files created in S3.
More Information
Applies To
Product: Cloud Data Integration
Problem Type: Configuration
User Type: Developer
Project Phase: Configure
Product Version:
Database:
Operating System:
Other Software:

Reference
Attachments
Last Modified Date:9/9/2019 9:15 PMID:589269
People who viewed this also viewed

Feedback

Did this KB document help you?



What can we do to improve this information (2000 or fewer characters)