#435  onedev/server in CrashLoopBackoff after upgrading to 5.3.1
Closed
Shamil opened 1 month ago
--> Wrapper Started as Console
Java Service Wrapper Standard Edition 64-bit 3.5.44
  Copyright (C) 1999-2020 Tanuki Software, Ltd. All Rights Reserved.
    http://wrapper.tanukisoftware.com
  Licensed to OneDev for Service Wrapping

Launching a JVM...
WrapperManager: Initializing...
INFO  - Launching application from '/app'...
INFO  - Initializing dependency injection container...
INFO  - Starting plugin manager...
INFO  - Upgrading /opt/onedev...
ERROR - Please stop server running at "/opt/onedev" before upgrading
<-- Wrapper Stopped

Logs from the container suggest that the upgrade isn't happening properly...

Robin Shen commented 1 month ago

Looks like old version of OneDev is not terminating gracefully upon upgrading. Can you please let me know the reproducing steps?

Shamil commented 1 month ago

@robin

cd onedev-k8s/
kubectl apply -k .
Robin Shen commented 1 month ago

I set up a k8s cluster running OneDev 5.3.0, and then upgrade to 5.3.1 by changing the image version, and everything works fine. If you can comes with a detailed reproducible steps for this issues, it will save me a lot of time to solve the issue.

Shamil commented 1 month ago

@robin

I've tried to debug by the following

cd onedev-k8s/maintenance;
kubectl apply -k .
kubectl exec -it onedev-65675f5b6f-4l7rs -- bash
/app/boot/wrapper-linux-x86-64 /app/conf/wrapper.conf wrapper.pidfile=/app/status/onedev_upgrade.pid -- upgrade /opt/onedev
root@onedev-65675f5b6f-4l7rs:~/bin# /app/boot/wrapper-linux-x86-64 /app/conf/wrapper.conf wrapper.pidfile=/app/status/onedev_upgrade.pid -- upgrade /opt/onedev
--> Wrapper Started as Console
Java Service Wrapper Standard Edition 64-bit 3.5.44
  Copyright (C) 1999-2020 Tanuki Software, Ltd. All Rights Reserved.
    http://wrapper.tanukisoftware.com
  Licensed to OneDev for Service Wrapping

Launching a JVM...
WrapperManager: Initializing...
INFO  - Launching application from '/app'...
INFO  - Cleaning temp directory...
INFO  - Initializing dependency injection container...
INFO  - Starting plugin manager...
INFO  - Upgrading /opt/onedev...
ERROR - Please stop server running at "/opt/onedev" before upgrading
<-- Wrapper Stopped
root@onedev-65675f5b6f-4l7rs:~/bin# ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0  28248 11588 ?        Ss   13:37   0:00 /bin/bash /root/bin/idle.sh
root           7  0.0  0.0  20280  3952 pts/0    Ss   13:37   0:00 bash
root          65  0.0  0.0   6180   736 ?        S    13:38   0:00 sleep 60
root         106  0.0  0.0  36080  3208 pts/0    R+   13:39   0:00 ps aux
Robin Shen commented 1 month ago

When OneDev runs, it will create file /opt/onedev/status/onedev.pid, and this file will be removed if the container is stopped gracefully. The upgrade process will check existence of this file to determine whether or not the server is running. Obviously OneDev is forcibly stopped for some reason and that file is left behind. You may remove this file manually to make upgrade working.

So the problem is how to reproduce the situation OneDev container is forcibly stopped. At my side, whether I perform an image update, or entering into maintenance mode, k8s stops the container gracefully (with a 30s grace shutdown period).

Shamil commented 1 month ago

Ok, it looks like there was a stray pid file, after removing it, it seems to work now :)

Robin Shen commented 1 month ago

OK. Will improve the upgrade logic to make it more robust.

Robin Shen changed state to 'Closed' 1 month ago
Previous Value Current Value
Open
Closed
No comment
issue 1 of 1
Type
Support Request
Priority
Normal
Assignee
Issue Votes (0)
Watchers (3)
Reference
issue onedev/server#435
Please wait...
Page is in error, reload to recover