Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

File event listener is recreated after deletion

Assignee

Reporter

Labels

Sprint

Description

Recently, some of our CTE test fail (on core, worker, cluster) because of errors logged in all.log. The errors come from a failing file event listener.
Example of the error:

The listener is created during FileEventListenerTest (specifically /cloveretl.server/src/integrationTest/java/com/cloveretl/server/test/events/file/suite/A2_ImportFileEventListenersTest.java). I've seen S3-remote-file-event-listener-test or FTP-slow-file-upload-listener-test. All the listeners are deleted after the test, so this should not happen. It seems that some of the listeners are sometimes recreated after deletion.

Example: https://linda.javlin.eu/jenkins/job/cloveretl.server-vte-test-worker-master/790/CONTAINER=weblogic-12-2-1,DATABASE=db2,OBFUSCATION=obfuscated,jdk=java-1.8/ - listener FTP-slow-file-upload-listener-test is active (and failing) after the tests finished.

Creation of the listener (jenkins build log + all.log):

So the listener has id 92 and taskId 115.

Then the listener is deleted (jenkins build log + server-audit.log):

So the listener should be deleted, but it appears again with different id (from all.log):

Steps to reproduce

None

Attachments

3

Activity

Show:

Filip Reichman August 18, 2020 at 11:38 AM

Added a check: an EventListener is not updated if it has already been deleted.

Filip Reichman August 18, 2020 at 6:42 AM
Edited

It is a race condition

  1. there is a collection of FileEventProducers (NodeDependentEventListenerLifecycleAware.activeEventProducers), each FileEventProducer is holding an instance of FileEventListener

  2. a FileEventListener is deleted from DB

  3. in the meantime FileEventProducer holding reference to the deleted FileEventListener is executed and it updates the FileEventListener in the DB

    1. the FileEventListener is not in the DB anymore, so a brand new FileEventListener is created

  4. the original FileEventProducer is removed from the collection

  5. a new FileEventProducer is added to the collection

The FileEventProducer should be removed from the collection before it is deleted from DB, but there is a job ClusterNodeCheckJob (executing EventListenerManager.checkActiveListeners()) checking if listeners (located in the Cache) are active, so the (not in DB) listener can be initialized again.

Fixed

Details

Priority

Fix versions

Branch

QA Testing

UNDECIDED

Components

Environment

VTE, master

Created August 17, 2020 at 7:45 AM
Updated September 12, 2023 at 8:44 AM
Resolved August 18, 2020 at 11:38 AM
Configure