FATAL: sorry, too many clients already

[visch]

Recieved a bunch of messages for one stream Schema has changed for stream 'campaign_performance_stats'. Mapping definitions will be reset.

The schema messages here were the same so it's not clear why this was happening.

The tap here shouldn't be sending so many schema messages but the target shouldn't say the schema has changed when it hasn't I believe.

Example messages that cause this

{"type": "SCHEMA", "stream": "campaign_performance_stats", "schema": {"properties": {"Name": {"type": ["string", "null"]}, "Id": {"type": ["string", "null"]}, "Date": {"format": "date", "type": ["string", "null"]}, "Clicks": {"type": ["integer", "null"]}, "Impressions": {"type": ["integer", "null"]}, "Conversions": {"type": ["integer", "null"]}, "CurrencyCode": {"type": ["string", "null"]}, "Cost": {"type": ["number", "null"]}, "OrganicClicks": {"type": ["integer", "null"]}, "OrganicImpressions": {"type": ["integer", "null"]}, "Applystarts": {"type": ["integer", "null"]}, "OrganicApplystarts": {"type": ["integer", "null"]}, "_sdc_employer_id": {"type": ["string", "null"]}, "_sdc_campaign_id": {"type": ["string", "null"]}}, "type": "object"}, "key_properties": ["Id", "Date"], "bookmark_properties": ["Date"]}
{"type": "SCHEMA", "stream": "campaign_performance_stats", "schema": {"properties": {"Name": {"type": ["string", "null"]}, "Id": {"type": ["string", "null"]}, "Date": {"format": "date", "type": ["string", "null"]}, "Clicks": {"type": ["integer", "null"]}, "Impressions": {"type": ["integer", "null"]}, "Conversions": {"type": ["integer", "null"]}, "CurrencyCode": {"type": ["string", "null"]}, "Cost": {"type": ["number", "null"]}, "OrganicClicks": {"type": ["integer", "null"]}, "OrganicImpressions": {"type": ["integer", "null"]}, "Applystarts": {"type": ["integer", "null"]}, "OrganicApplystarts": {"type": ["integer", "null"]}, "_sdc_employer_id": {"type": ["string", "null"]}, "_sdc_campaign_id": {"type": ["string", "null"]}}, "type": "object"}, "key_properties": ["Id", "Date"], "bookmark_properties": ["Date"]}
{"type": "SCHEMA", "stream": "campaign_performance_stats", "schema": {"properties": {"Name": {"type": ["string", "null"]}, "Id": {"type": ["string", "null"]}, "Date": {"format": "date", "type": ["string", "null"]}, "Clicks": {"type": ["integer", "null"]}, "Impressions": {"type": ["integer", "null"]}, "Conversions": {"type": ["integer", "null"]}, "CurrencyCode": {"type": ["string", "null"]}, "Cost": {"type": ["number", "null"]}, "OrganicClicks": {"type": ["integer", "null"]}, "OrganicImpressions": {"type": ["integer", "null"]}, "Applystarts": {"type": ["integer", "null"]}, "OrganicApplystarts": {"type": ["integer", "null"]}, "_sdc_employer_id": {"type": ["string", "null"]}, "_sdc_campaign_id": {"type": ["string", "null"]}}, "type": "object"}, "key_properties": ["Id", "Date"], "bookmark_properties": ["Date"]}

[visch]

Verified that this opens up 100 connections in my use case. This seems like multiple issues.

1. Pooling doesn't seem to be working properly / setup properly
2. Schema messages that are the same shouldn't trigger a "schema has changed" message

[visch]

Related to #140 and #103

[visch]

I thought we solved it in #153 but I just hit this running a seperate job in a production setting. Going to dive in

Found that the original cause for this issue (the schema messages in the order sent) no longer cause this. Not sure what causes my setup to fail with this yet.

Number of connections just goes up, it looks like connections are not torn down right now properly or atleast not reused as expected

[visch]

I seem to hit this in 2 occasions

1. Sending a lot of SCHEMA messages on existing sinks
2. Running pytest (Call select count(*) from pg_stat_activity; as the tests are running and you'll get up to 40-80 connections) . This may be due to new Targets being initialized for every test.

[BuzzCutNorman]

If you would please give this sdk pr a try, meltano/sdk#1864, and see if it reduces the connection count during a run.

[amotl]

Hi @visch, @BuzzCutNorman, and @edgarrmondragon,

it looks like this one is tripping again. I am observing it on my workstation, and on CI, like at GH-250. Do you think that would qualify to re-open this ticket?

@visch said:

Verified that this opens up 100 connections in my use case. This seems like multiple issues.

1. Pooling doesn't seem to be working properly / setup properly

That would also be my personal feeling about the situation. However, I haven't been able to take a closer look up until now, and thought it would be a good idea to talk to you beforehand.

With kind regards,
Andreas.