Implements RetryConnectionMiddleware for Oracle - #1303
Conversation
| /** | ||
| * Retries transient Oracle listener failures during connection establishment. | ||
| * | ||
| * Oracle may temporarily return ORA-12516 when the listener has not yet |
There was a problem hiding this comment.
Are you sure? IIRC this issue is present even with very few requests. What is the real connection count limit?
(and btw. this is already an issue with MySQL, but limit of small number of connections instead of strict 1 is enough and reliable)
There was a problem hiding this comment.
Current limits were:
Oracle-free (23) - 200 processes, 322 sessions
Oracle-xe (18) - 320 processes, 502 sessions
About 60-90 sessions can be already used internally by Oracle background services.
You can see more stats here https://github.com/atk4/data/actions/runs/31742995074/job/94590708915
Check steps Oracle stats before tests, Oracle stats after tests. These steps were added while testing and not included in this final PR. Also listener hitting limits is not visible in these stats because when listener hits limits, then it doesn't reach DB at all, so it's not visible in DB side of stats. It's only visible in listener log file, but it's stored on container which is not accessible from github job.
There was a problem hiding this comment.
Also you can check step "Stress test Oracle connections" in this job https://github.com/atk4/data/actions/runs/31723291851/job/94525337653
You can see there how it hits the limits randomly, then retries one or few times and then continue successfully.
There was a problem hiding this comment.
The proper solution might be to release the connection better. It might be even a bug in php driver.
Does this happen with pdo_oci as well as oci8?
A solution worth to consider might be even to force to release the connection using destructor.
There was a problem hiding this comment.
It's not a bug or something in php.
It's how Oracle Listener works. When you try to connect Oracle, then you first hit the listener and listener keeps track of active sessions too. If listener thinks that there are more active sessions than Oracle DB allows, then it do not pass you trough to database at all.
Listener has a tiny cache of that sessions stats and when you disconnect it doesn't instantly register that in its cache. So in very busy connect/disconnect load listener stays a bit behind.
Simple retrying in few quite small intervals is a way to go and solves everything.
| { | ||
| return new class($driver) extends AbstractDriverMiddleware { | ||
| // ORA-12516: TNS:listener could not find available handler with matching protocol stack | ||
| private const RETRY_ERROR_CODES = [12516]; |
There was a problem hiding this comment.
In Discord you mentioned https://dbamarco.wordpress.com/2023/07/20/suspicous-ora-12516-by-oracle-scan-listener/ . I want to check if this desctribe the issue. In the chat, he wrote they increased the connection limit... Did you see this issue in production somewhen?
There was a problem hiding this comment.
Increasing connection limit is simplest workaround. Of course if you have 1000000 connections limit, then you'll not hit it while testing and all should be fine. But normally there shouldn't be such high limit. I've tested that all here in last few days. It's really that - in case of this failure just retry in 10-100ms (rarely a bit more - up till about 0.5s) and listener cache is updated, connections are freed and tests continue.
That wordpress topic is about slightly different kind of case, but still was the one to give me idea where to look at.
| DB_USER: system | ||
| DB_PASSWORD: atk4_pass | ||
| # connection to portable database | ||
| DB_DSN: "pdo_oci:dbname=oracle/${{ matrix.type == 'Phpunit Lowest' && 'freepdb1' || 'xepdb1' }}" |
There was a problem hiding this comment.
The issues are mostly/solely present with "lowest". Ie. the lowest/oldest deps or the newest db. I would be happy to have this explained - is this because of the newest/never db and did some db limit changed?
There was a problem hiding this comment.
See #1303 (comment)
I think it's because oracle-fee (v23 which is used in lowest tests) have 322 sessions limit and 200 processes limit by default but old oracle xe has 502 and 320 respectively.
There was a problem hiding this comment.
I trigged about 1000 pipelines with all Oracle versions and I have found:
- the issue is present as you have written - with any many connections, no special query needed
- the affected versions are >=23.4. With 23.3 or lower I was not able to trigger the issue at all
The retry solution might be the only one. I want to still do some more experiments.
Also, have the 322/502 200/320 limits changed with version 23.3 -> 23.4? If not, what changed?
There was a problem hiding this comment.
Well. That could be true about 23.3 and 23.4 as from 23.4 Oracle introduced AI Vector Search & Data Types - Version 23.4 officially introduced the native VECTOR data type and AI vector search features, which were not fully packaged or active in 23.3 developer builds.
Also they did Container and Setup Adjustments - Docker and container setup scripts (like network binding for tnsnames.ora) were updated and corrected in 23.4 compared to the 23.3 free/developer images.
I can't guarantee that this AI answer is correct, but that's approximately that.
But anyway, even if any of these solutions work, I still think that retry middleware is good to have as a fallback mechanism. No long retries, just try few times in quite short intervals and that's it.
It could be even helpful for other DB engines too to avoid some small sudden glitches while connecting.
There was a problem hiding this comment.
I cannot agree with the last paragraph. If 1 concurrent connection can exceed the total connections limit of 200, there is something very wrong with the database.
If there is a possibility of fixing this by some Oracle configuration, I might prefer it. I mean of course not by raising the concurrency to sky...
There was a problem hiding this comment.
Yes that would be perfect to fix it by Oracle settings of course.
But that doesn't mean that retry mechanism as such is a bad idea.
Normally you would not use Oracle as DB for busy webpages which does connect/disconnect all the time. Then Mysql or even Postgre is much better choice. Oracles power is more in data consistency, complex data cases, large data amounts etc. Also preferably by using persistent connections or connection pool at least.
P.S. Also Oracle uses like maybe 50 connections internally for its own services. So it's not 200 free connections at a start.
| APP_USER: atk4_test_user | ||
| APP_USER_PASSWORD: atk4_user_pass | ||
| # Provide healthcheck script options for startup | ||
| options: >- |
There was a problem hiding this comment.
Github CI is dummy, all services are started one by one and all healtchecks are busy waited before tests. If a service starts slow, it will slowdown the CI.
Thanks and I will check this on the separate PRs. In past, I opted out for the healtchecks as the tests on other DBs were realiably enough without having to busy wait.
Please do not merge this PR yet, I want to land the other changes first and verify if retry mechanism is the only solution, I personally do not like it but on the other side Oracle is very ugly software and it might be the best working solution.
2 participants
Implements
RetryConnectionMiddlewareand improve action workflow script for Oracle.This fixes issue caused by Oracle listener which we also often see in Atk tests.
On heavier connection load Oracle randomly becomes inaccessible because Oracle listener have some delay before it actually drops connection out of its cache after you disconnect from db. As result listener kind of "overflows" and don't allow to make new connections to DB even if previous connections are all nicely closed and not hit DBs connection limit.
Implementation have no BC breaks as connection retries are done purely on background.