Timeouts for Pooled Connections

[256BitChris]

I'm leveraging the PooledGMService class - my application has a timeout set at 10 seconds - sometimes GM operations take longer (up to several minutes).

When this happens my Java process times out as expected and returns the connection to the pool. However, this does not cause the process to be destroyed and in fact the gm process can continue processing until it's complete. Calls to ensureHealthy() on a connection in this state block until the previous command has finished.

If I detect a timeout on a non pooled connection, the close() operation will destroy the process which will stop it from taking up processing resources as soon as my application has decided it will no longer wait for the result.

Basically I'd like the ability to flag a pooled connection as unhealthy. This way it won't be reused and if I have testOnReturn enabled the process will be destroyed when it's returned to the pool.

Are there any thoughts on how to approach this? I'm willing to do make the changes but wanted to see if there were any ideas on how to best implement it.

[kennethxu]

Terribly sorry to respond late. Wondering how did you implemented the timeout here. If you used a 2nd thread with Future.get(timeout) or anything similar to this. The other thread is actually still running. In this case, what we can do is to add a little logic in the PooledGMService.ConnectionWrapper.

1. introduce a volatile boolean field: isExecuting
2. set isExecuting to true in the beginning of execute and false in finally.
3. in the close, if isExecuting is true, close the real connection before return it back to pool, this will effectively kill the GM process.
4. for the pool to dispose this connection, we'll also need to change ensureHealthy method to throw exception if the connection is also close. (this should probably have been done anyway)

HTH

[kennethxu]

Given another thought, I'm afraid that is a bad idea. According to the doc of GMConnection, its methods are not designed to be thread-safe. If we start to kill the GM process now and then, then it defeat the purpose of process pool, which is to reuse. For you particular use case, have you considered to use simple service instead? That way you get complete control of the connection and decide when to reuse and when to close (destroy).

[256BitChris]

Thanks for the reply.

The problem with the current implementation of the process pool, is that it allows for GM processes to run to completion without the option to kill them. We experienced this in our production system and it was very wasteful of resources (ie. we'd timeout after 2 minutes but certain images could take 15 minutes or more to process). I played around with the various test on return/get methods and they all had problems where they wouldn't do the test while the connection was running.

We need the performance gains of the batch processing/pooled connections but also the ability to destroy the connections. I implemented a fix a while ago and it's been working without a problem. Here's what I did:

First, I created a destroyable interface which had the destroy() function. This is versus the close function which can sometimes have the side effect of destroying the connection, but sometimes just returns it to the pool.

/**
 * A Destroyable version of the GMConnection interface.
 * <p/>
 * You destroy a connection when you know you want to destroy the underlying process behind it.
 * <p/>
 * The use case for this is when we experience a timeout with GraphicsMagick, control returns to Java but the
 * underlying GM process continues to run until completion causing severe resource starvation issues at peak times.
 */
public interface DestroyableGMConnection extends GMConnection {

    /**
     * Destroy this connection.
     */
    public void destroy();
}

I then implemented this in the PooledGMService::ConnectionWrapper class (through copying the implementation and renaming it (a little cleaner than monkey patching)):

   /**
    * This method is called when we know we want to destroy the wrapped connection rather than just close it and go
    * through the logic there.
    */
   public void destroy() {
       real.close();
   }

Here's how my application leverages the code:

final DestroyableGMConnection gmConnection = gmService.getConnection();
final Future<String> timedTask = processThreadPool.submit(new Callable<String>() {
    @Override
    public String call() throws Exception {
      try {
         return gmConnection.execute(cmd);
      } finally {
         gmConnection.close();
      }
    }
});
try {
     timedTask.get(processingTimeout, TimeUnit.MILLISECONDS);
}} catch (TimeoutException e) {
     LOG.warn("Process {} timed out after {}ms", cmd, processingTimeout);
     /**
      * In the case of a timeout, the Java thread returns control here but the GM process continues
      * to execute until its processing is complete.
      *
      * Destroy the connection here to kill the underlying GM process, and then close it so that it gets released from the pool.
      *
      */
      gmConnection.destroy();
      gmConnection.close();
      throw e;
}

So, in the end I was able to fix this in a simple way by just exposing the destroy method as the pooled connection prevents you from accessing this directly. You can then turn on TEST_ON_RETURN and when the pool attempts to return, it will test the connection, fail, and not let the connection again.

Hope that helps!

[digz6666]

@64BitChris Could you please share some more example?

[kennethxu]

The new version of common pools added timeout function, see: https://commons.apache.org/proper/commons-pool/apidocs/org/apache/commons/pool2/impl/AbandonedConfig.html#getRemoveAbandonedTimeout--

There is a pull request #16, which worked on updating to common-pool v2. The pull request was submitted before -safe-mode change, and now need resolve the conflict before merging.

Once we upgrade to common-pool v2, we should be able to enable timeout for the pooled connection.

My bandwidth is very limited, any help is welcome.