Failed to parse billing file

[sayan-rudder]

Failed to parse billing data config: <s3_bucket>/<account_id>-aws-billing-detailed-line-items-with-resources-and-tags-2020-05.csv.zip

ice_1 | | Error unacceptable character '' (0x3) special characters are not allowed
ice_1 | in "'reader'", position 2
ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.snakeyaml.reader.StreamReader.checkPrintable(StreamReader.java:93)
ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:192)
ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.snakeyaml.reader.StreamReader.(StreamReader.java:60)
ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.YAMLParser.(YAMLParser.java:147)
ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.YAMLFactory._createParser(YAMLFactory.java:426)
ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:327)
ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:300)
ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:14)
ice_1 | | Error at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2726)
ice_1 | | Error at com.netflix.ice.processor.config.BillingDataConfig.(BillingDataConfig.java:148)
ice_1 | | Error at com.netflix.ice.processor.ProcessorConfig.readBillingDataConfig(ProcessorConfig.java:605)
ice_1 | | Error at com.netflix.ice.processor.ProcessorConfig.processBillingDataConfig(ProcessorConfig.java:525)

Is this expected, any pointers will be helpful

[nfonrose]

It looks like a configuration error inside your ice.properties file. The line of code failing is trying to read a YAML based configuration file. https://github.com/jimroth/ice/blob/master/src/java/com/netflix/ice/processor/config/BillingDataConfig.java#L148 You probably need to post the configuration here if you need more help. Nicolas

…

-- Nicolas Fonrose | Teevity | Founder +33.6.61.35.43.31 https://teevity.com - Cloud Costs Analytics for AWS, GCP and Azure powered by NetflixOSS Ice twitter - @nfonrose / @Teevity

On Tue, Jun 2, 2020 at 9:38 AM sayan-rudder ***@***.***> wrote: Failed to parse billing data config: <s3_bucket>/<account_id>-aws-billing-detailed-line-items-with-resources-and-tags-2020-05.csv.zip ice_1 | | Error unacceptable character '' (0x3) special characters are not allowed ice_1 | in "'reader'", position 2 ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.snakeyaml.reader.StreamReader.checkPrintable(StreamReader.java:93) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:192) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.snakeyaml.reader.StreamReader.(StreamReader.java:60) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.YAMLParser.(YAMLParser.java:147) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.YAMLFactory._createParser(YAMLFactory.java:426) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:327) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:300) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:14) ice_1 | | Error at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2726) ice_1 | | Error at com.netflix.ice.processor.config.BillingDataConfig.(BillingDataConfig.java:148) ice_1 | | Error at com.netflix.ice.processor.ProcessorConfig.readBillingDataConfig(ProcessorConfig.java:605) ice_1 | | Error at com.netflix.ice.processor.ProcessorConfig.processBillingDataConfig(ProcessorConfig.java:525) Is this expected, any pointers will be helpful — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#304>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAJJJPNWO6NBXMAVHXDWJLRUSUBHANCNFSM4NQO7LBQ> .

[sayan-rudder]

Sorry for my low understanding.

is a config file different from the actual aws provided billing data file?
The above error is requiring a config file ? Where to find it ?

or should I post the billing data file itself?

Thanks for all the help!

@nfonrose

[sayan-rudder]

I think, it processed one of the billing files now but NPE in saving the data :(

ice_1 | 2020-06-02 05:23:16,630 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.DetailedBillingReportProcessor - processed 500000 lines...
ice_1 | 2020-06-02 05:23:17,489 [com.netflix.ice.processor.BillingFileProcessor] INFO basic.BasicLineItemProcessor - account_id-aws-billing-detailed-line-items-with-resources-and-tags-2020-05.csv Support: com.netflix.ice.processor.DetailedBillingReportLineItem@1a49e344
ice_1 | 2020-06-02 05:23:17,490 [com.netflix.ice.processor.BillingFileProcessor] INFO basic.BasicLineItemProcessor - account_id-aws-billing-detailed-line-items-with-resources-and-tags-2020-05.csv Support: com.netflix.ice.processor.DetailedBillingReportLineItem@1a49e344
ice_1 | 2020-06-02 05:23:17,490 [com.netflix.ice.processor.BillingFileProcessor] INFO basic.BasicLineItemProcessor - account_id-aws-billing-detailed-line-items-with-resources-and-tags-2020-05.csv Support: com.netflix.ice.processor.DetailedBillingReportLineItem@1a49e344
ice_1 | 2020-06-02 05:23:17,490 [com.netflix.ice.processor.BillingFileProcessor] INFO basic.BasicLineItemProcessor - account_id-aws-billing-detailed-line-items-with-resources-and-tags-2020-05.csv Support: com.netflix.ice.processor.DetailedBillingReportLineItem@1a49e344
ice_1 | 2020-06-02 05:23:17,530 [com.netflix.ice.processor.BillingFileProcessor] INFO basic.BasicLineItemProcessor - account_id-aws-billing-detailed-line-items-with-resources-and-tags-2020-05.csv Support: com.netflix.ice.processor.DetailedBillingReportLineItem@1a49e344
ice_1 | | Error 2020-06-02 05:23:17,558 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.DetailedBillingReportProcessor - Estimated,account_id,,Rounding,,,,,,,,,,This line contains rounding error due to the consolidated billing and hourly line item calculation processes.,,,,,0.0095870483,,0.0100143818,,
ice_1 | Message: Invalid format: ""
ice_1 | Line | Method
ice_1 | ->> 747 | parseMillis in org.joda.time.format.DateTimeFormatter
ice_1 | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ice_1 | | 67 | getStartMillis in com.netflix.ice.processor.DetailedBillingReportLineItem
ice_1 | | 159 | getUsageInterval . . . . in com.netflix.ice.basic.BasicLineItemProcessor
ice_1 | | 210 | process in ''
ice_1 | | 234 | processOneLine . . . . . in com.netflix.ice.processor.DetailedBillingReportProcessor
ice_1 | | 195 | processBillingFile in ''
ice_1 | | 152 | processBillingZipFile . in ''
ice_1 | | 128 | processReport in ''
ice_1 | | 295 | downloadAndProcessReport in ''
ice_1 | | 145 | processReports in com.netflix.ice.processor.BillingFileProcessor
ice_1 | | 81 | poll . . . . . . . . . . in ''
ice_1 | | 50 | doWork in com.netflix.ice.common.Poller
ice_1 | | 28 | access$000 . . . . . . . in ''
ice_1 | | 88 | run in com.netflix.ice.common.Poller$1
ice_1 | ^ 748 | run . . . . . . . . . . in java.lang.Thread
ice_1 | | Error 2020-06-02 05:23:17,653 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.DetailedBillingReportProcessor - Estimated,account_id,,InvoiceTotal,,,,,,,,,,Total amount for invoice Estimated,,,,,0.01,,0.01,,
ice_1 | Message: Invalid format: ""
ice_1 | Line | Method
ice_1 | ->> 747 | parseMillis in org.joda.time.format.DateTimeFormatter
ice_1 | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ice_1 | | 67 | getStartMillis in com.netflix.ice.processor.DetailedBillingReportLineItem
ice_1 | | 159 | getUsageInterval . . . . in com.netflix.ice.basic.BasicLineItemProcessor
ice_1 | | 210 | process in ''
ice_1 | | 234 | processOneLine . . . . . in com.netflix.ice.processor.DetailedBillingReportProcessor
ice_1 | | 195 | processBillingFile in ''
ice_1 | | 152 | processBillingZipFile . in ''
ice_1 | | 128 | processReport in ''
ice_1 | | 295 | downloadAndProcessReport in ''
ice_1 | | 145 | processReports in com.netflix.ice.processor.BillingFileProcessor
ice_1 | | 81 | poll . . . . . . . . . . in ''
ice_1 | | 50 | doWork in com.netflix.ice.common.Poller
ice_1 | | 28 | access$000 . . . . . . . in ''
ice_1 | | 88 | run in com.netflix.ice.common.Poller$1
ice_1 | ^ 748 | run . . . . . . . . . . in java.lang.Thread
ice_1 | | Error 2020-06-02 05:23:17,659 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.DetailedBillingReportProcessor - ,account_id,454531037350,AccountTotal,,,,,,,,,,Total for linked account# 454531037350,,,,,-6.8566690473,,-0.0000044282,,
ice_1 | Message: Invalid format: ""
ice_1 | Line | Method
ice_1 | ->> 747 | parseMillis in org.joda.time.format.DateTimeFormatter
ice_1 | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ice_1 | | 67 | getStartMillis in com.netflix.ice.processor.DetailedBillingReportLineItem
ice_1 | | 159 | getUsageInterval . . . . in com.netflix.ice.basic.BasicLineItemProcessor
ice_1 | | 210 | process in ''
ice_1 | | 234 | processOneLine . . . . . in com.netflix.ice.processor.DetailedBillingReportProcessor
ice_1 | | 195 | processBillingFile in ''
ice_1 | | 152 | processBillingZipFile . in ''
ice_1 | | 128 | processReport in ''
ice_1 | | 295 | downloadAndProcessReport in ''
ice_1 | | 145 | processReports in com.netflix.ice.processor.BillingFileProcessor
ice_1 | | 81 | poll . . . . . . . . . . in ''
ice_1 | | 50 | doWork in com.netflix.ice.common.Poller
ice_1 | | 28 | access$000 . . . . . . . in ''
ice_1 | | 88 | run in com.netflix.ice.common.Poller$1
ice_1 | ^ 748 | run . . . . . . . . . . in java.lang.Thread
ice_1 | | Error 2020-06-02 05:23:17,662 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.DetailedBillingReportProcessor - ,account_id,account_id,AccountTotal,,,,,,,,,,Total for linked account# account_id,,,,,6.8570819990,,-0.0000099536,,
ice_1 | Message: Invalid format: ""
ice_1 | Line | Method
ice_1 | ->> 747 | parseMillis in org.joda.time.format.DateTimeFormatter
ice_1 | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ice_1 | | 67 | getStartMillis in com.netflix.ice.processor.DetailedBillingReportLineItem
ice_1 | | 159 | getUsageInterval . . . . in com.netflix.ice.basic.BasicLineItemProcessor
ice_1 | | 210 | process in ''
ice_1 | | 234 | processOneLine . . . . . in com.netflix.ice.processor.DetailedBillingReportProcessor
ice_1 | | 195 | processBillingFile in ''
ice_1 | | 152 | processBillingZipFile . in ''
ice_1 | | 128 | processReport in ''
ice_1 | | 295 | downloadAndProcessReport in ''
ice_1 | | 145 | processReports in com.netflix.ice.processor.BillingFileProcessor
ice_1 | | 81 | poll . . . . . . . . . . in ''
ice_1 | | 50 | doWork in com.netflix.ice.common.Poller
ice_1 | | 28 | access$000 . . . . . . . in ''
ice_1 | | 88 | run in com.netflix.ice.common.Poller$1
ice_1 | ^ 748 | run . . . . . . . . . . in java.lang.Thread
ice_1 | | Error 2020-06-02 05:23:17,665 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.DetailedBillingReportProcessor - ,account_id,,StatementTotal,,,,,,,,,,Total statement amount for period 2020-05-01 00:00:00 - 2020-05-31 23:59:59,,,,,0.01,,0.01,,
ice_1 | Message: Invalid format: ""
ice_1 | Line | Method
ice_1 | ->> 747 | parseMillis in org.joda.time.format.DateTimeFormatter
ice_1 | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ice_1 | | 67 | getStartMillis in com.netflix.ice.processor.DetailedBillingReportLineItem
ice_1 | | 159 | getUsageInterval . . . . in com.netflix.ice.basic.BasicLineItemProcessor
ice_1 | | 210 | process in ''
ice_1 | | 234 | processOneLine . . . . . in com.netflix.ice.processor.DetailedBillingReportProcessor
ice_1 | | 195 | processBillingFile in ''
ice_1 | | 152 | processBillingZipFile . in ''
ice_1 | | 128 | processReport in ''
ice_1 | | 295 | downloadAndProcessReport in ''
ice_1 | | 145 | processReports in com.netflix.ice.processor.BillingFileProcessor
ice_1 | | 81 | poll . . . . . . . . . . in ''
ice_1 | | 50 | doWork in com.netflix.ice.common.Poller
ice_1 | | 28 | access$000 . . . . . . . in ''
ice_1 | | 88 | run in com.netflix.ice.common.Poller$1
ice_1 | ^ 748 | run . . . . . . . . . . in java.lang.Thread
ice_1 | 2020-06-02 05:23:17,669 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.DetailedBillingReportProcessor - done processing account_id-aws-billing-detailed-line-items-with-resources-and-tags-2020-05.csv.zip, end is 2020-06-01 00:00:00
ice_1 | 2020-06-02 05:23:17,669 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - Process reservations pulled from the accounts
ice_1 | 2020-06-02 05:23:17,676 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.DetailedBillingReservationProcessor - ---------- Process Non-resource
ice_1 | 2020-06-02 05:23:17,677 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - adding savings data for 2020-05-01T00:00:00.000Z...
ice_1 | | Error 2020-06-02 05:23:17,696 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.BillingFileProcessor - Failed to process reports: java.lang.NullPointerException
ice_1 | | Error java.lang.NullPointerException
ice_1 | | Error at com.netflix.ice.processor.BillingFileProcessor.addSavingsData(BillingFileProcessor.java:273)
ice_1 | | Error at com.netflix.ice.processor.BillingFileProcessor.processReports(BillingFileProcessor.java:214)
ice_1 | | Error at com.netflix.ice.processor.BillingFileProcessor.poll(BillingFileProcessor.java:81)
ice_1 | | Error at com.netflix.ice.common.Poller.doWork(Poller.java:50)
ice_1 | | Error at com.netflix.ice.common.Poller.access$000(Poller.java:28)
ice_1 | | Error at com.netflix.ice.common.Poller$1.run(Poller.java:88)
ice_1 | | Error at java.lang.Thread.run(Thread.java:748)