Restart loops every few days"? Debug pasted

[SepticFrank]

I appreciate any advice. I've tried a reflashing two different SD cards a few times. Every few days after ssh, it reads "Node is syncing, Blockchain progress 6 restarts, Lightning progress 100%. If there is a certain type of troubleshooting, I'd appreciate pointing me in the right direction.

Generating debug logs. Be patient, this should take maximum 2 minutes ..
tail: error reading '/mnt/hdd/bitcoin/debug.log': Input/output error
ls: reading directory '/mnt/hdd': Input/output error
(standard_in) 1: syntax error
/home/admin/config.scripts/blitz.datadrive.sh: line 500: * 1024: syntax error: operand expected (error token is "* 1024")
Redacting ..

redacting file: /var/cache/raspiblitz/debug.log

backup info: network
backup info: chain

---

• RASPIBLITZ DEBUG LOGS

---

blitzversion: 1.10.0
chainnetwork: bitcoin / test
01:29:13 up 2 days, 5:49, 2 users, load average: 2.15, 2.09, 2.17

*** FAILED SERVICES ***
list any servcies with problems: sudo systemctl list-units --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● bitcoind.service loaded failed failed Bitcoin daemon on mainnet
● lnd.service loaded failed failed Lightning Network Daemon
● logrotate.service loaded failed failed Rotate log files

LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
3 loaded units listed.

*** SETUPPHASE / BOOTSTRAP ***
see logs: cat /home/admin/raspiblitz.log
setupPhase--> done
state--> ready

*** BACKGROUNDSERVICE ***
to monitor Background service call: sudo journalctl -f -u background

*** BLOCKCHAIN (MAINNET) SYSTEMD STATUS ***
● bitcoind.service - Bitcoin daemon on mainnet
Loaded: loaded (/etc/systemd/system/bitcoind.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2024-02-09 05:33:37 GMT; 19h ago
Process: 2896698 ExecStartPre=/home/admin/config.scripts/bitcoin.check.sh prestart mainnet (code=exited, status=0/SUCCESS)
Process: 2896728 ExecStart=/usr/local/bin/bitcoind -daemonwait -conf=/mnt/hdd/bitcoin/bitcoin.conf -datadir=/mnt/hdd/bitcoin (code=exited, status=1/FAILURE)
CPU: 239ms

Warning: journal has been rotated since unit was started, output may be incomplete.

*** LAST BLOCKCHAIN (MAINNET) ERROR LOGS ***
sudo journalctl -u bitcoind -b --no-pager -n20
-- Journal begins at Fri 2024-02-09 10:28:36 GMT, ends at Sat 2024-02-10 01:29:13 GMT. --
-- No entries --

*** LAST BLOCKCHAIN (MAINNET) INFO LOGS ***
sudo tail -n 50 /mnt/hdd/bitcoin/debug.log

*** LND (MAINNET) SYSTEMD STATUS ***

• OFF by config -

*** CORE LIGHTNING (MAINNET) SYSTEMD STATUS ***

• not activated -

*** BLOCKCHAIN (TESTNET) SYSTEMD STATUS ***

• OFF by config -
  *** LND (TESTNET) SYSTEMD STATUS ***
• OFF by config -

*** CORE LIGHTNING (TESTNET) SYSTEMD STATUS ***

• not activated -

*** BLOCKCHAIN (SIGNET) SYSTEMD STATUS ***

• OFF by config -
  *** LND (SIGNET) SYSTEMD STATUS ***
• OFF by config -

*** CORE LIGHTNING (SIGNET) SYSTEMD STATUS ***

• not activated -

*** NGINX SYSTEMD STATUS ***
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/nginx.service.d
└─raspiblitz.conf
Active: active (running) since Wed 2024-02-07 19:35:45 GMT; 2 days ago
Docs: man:nginx(8)
Main PID: 1447 (nginx)
Tasks: 5 (limit: 4531)
CPU: 235ms
CGroup: /system.slice/nginx.service
├─1447 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─1448 nginx: worker process
├─1449 nginx: worker process
├─1450 nginx: worker process
└─1451 nginx: worker process

Warning: journal has been rotated since unit was started, output may be incomplete.

*** LAST NGINX LOGS ***
sudo journalctl -u nginx -b --no-pager -n20
-- Journal begins at Fri 2024-02-09 10:28:36 GMT, ends at Sat 2024-02-10 01:29:13 GMT. --
-- No entries --
--> CHECK CONFIG: sudo nginx -t
nginx: [emerg] cannot load certificate "/mnt/hdd/app-data/nginx/tls.cert": BIO_new_file() failed (SSL: error:02001005:system library:fopen:Input/output error:fopen('/mnt/hdd/app-data/nginx/tls.cert','r') error:2006D002:BIO routines:BIO_new_file:system lib)
nginx: configuration file /etc/nginx/nginx.conf test failed

*** BLITZAPI STATUS ***
installed=1
repo='https://github.com/fusion44/blitz_api.git'
branch='blitz-v1.10'
commit='191c77b4d4cc5c9d97dc50da31e5cfb2e5bcb930'
● blitzapi.service - BlitzBackendAPI
Loaded: loaded (/etc/systemd/system/blitzapi.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2024-02-07 19:35:42 GMT; 2 days ago
Main PID: 813 (python)
Tasks: 13 (limit: 4531)
CPU: 1h 9min 58.477s
CGroup: /system.slice/blitzapi.service
└─813 /home/blitzapi/blitz_api/venv/bin/python -m uvicorn app.main:app --port 11111 --host=... --root-path /api

Feb 10 01:29:12 raspberrypi python[813]: 2024-02-10 01:29:12 | ⚠️ | utils.py:69 | Key 'hdd_capacity_bytes' not found in Redis DB.
Feb 10 01:29:13 raspberrypi python[813]: 2024-02-10 01:29:13 | ❌ | service.py:194 | Aiohttp client connection error: Cannot connect to host ...:8332 ssl:default [Connect call failed ('127.0.0.1', 8332)]

*** LAST BLITZAPI LOGS ***
sudo journalctl -u blitzapi -b --no-pager -n20
-- Journal begins at Fri 2024-02-09 10:28:36 GMT, ends at Sat 2024-02-10 01:29:14 GMT. --
Feb 10 01:29:00 raspberrypi python[813]: 2024-02-10 01:29:00 | ⚠️ | utils.py:69 | Key 'hdd_capacity_bytes' not found in Redis DB.
Feb 10 01:29:01 raspberrypi python[813]: 2024-02-10 01:29:01 | ❌ | service.py:194 | Aiohttp client connection error: Cannot connect to host ...:8332 ssl:default [Connect call failed ('127.0.0.1', 8332)]
Feb 10 01:29:02 raspberrypi python[813]: 2024-02-10 01:29:02 | ℹ️ | utils.py:67 | Key 'tor_web_addr' not found in Redis DB.
Feb 10 01:29:02 raspberrypi python[813]: 2024-02-10 01:29:02 | ⚠️ | utils.py:69 | Key 'hdd_capacity_bytes' not found in Redis DB.
Feb 10 01:29:03 raspberrypi python[813]: 2024-02-10 01:29:03 | ❌ | service.py:194 | Aiohttp client connection error: Cannot connect to host ...:8332 ssl:default [Connect call failed ('127.0.0.1', 8332)]
Feb 10 01:29:04 raspberrypi python[813]: 2024-02-10 01:29:04 | ℹ️ | utils.py:67 | Key 'tor_web_addr' not found in Redis DB.
Feb 10 01:29:04 raspberrypi python[813]: 2024-02-10 01:29:04 | ⚠️ | utils.py:69 | Key 'hdd_capacity_bytes' not found in Redis DB.
Feb 10 01:29:05 raspberrypi python[813]: 2024-02-10 01:29:05 | ❌ | service.py:194 | Aiohttp client connection error: Cannot connect to host ...:8332 ssl:default [Connect call failed ('127.0.0.1', 8332)]
Feb 10 01:29:06 raspberrypi python[813]: 2024-02-10 01:29:06 | ℹ️ | utils.py:67 | Key 'tor_web_addr' not found in Redis DB.
Feb 10 01:29:06 raspberrypi python[813]: 2024-02-10 01:29:06 | ⚠️ | utils.py:69 | Key 'hdd_capacity_bytes' not found in Redis DB.
Feb 10 01:29:07 raspberrypi python[813]: 2024-02-10 01:29:07 | ❌ | service.py:194 | Aiohttp client connection error: Cannot connect to host ...:8332 ssl:default [Connect call failed ('127.0.0.1', 8332)]
Feb 10 01:29:08 raspberrypi python[813]: 2024-02-10 01:29:08 | ℹ️ | utils.py:67 | Key 'tor_web_addr' not found in Redis DB.
Feb 10 01:29:08 raspberrypi python[813]: 2024-02-10 01:29:08 | ⚠️ | utils.py:69 | Key 'hdd_capacity_bytes' not found in Redis DB.
Feb 10 01:29:09 raspberrypi python[813]: 2024-02-10 01:29:09 | ❌ | service.py:194 | Aiohttp client connection error: Cannot connect to host ...:8332 ssl:default [Connect call failed ('127.0.0.1', 8332)]
Feb 10 01:29:10 raspberrypi python[813]: 2024-02-10 01:29:10 | ℹ️ | utils.py:67 | Key 'tor_web_addr' not found in Redis DB.
Feb 10 01:29:10 raspberrypi python[813]: 2024-02-10 01:29:10 | ⚠️ | utils.py:69 | Key 'hdd_capacity_bytes' not found in Redis DB.
Feb 10 01:29:11 raspberrypi python[813]: 2024-02-10 01:29:11 | ❌ | service.py:194 | Aiohttp client connection error: Cannot connect to host ...:8332 ssl:default [Connect call failed ('127.0.0.1', 8332)]
Feb 10 01:29:12 raspberrypi python[813]: 2024-02-10 01:29:12 | ℹ️ | utils.py:67 | Key 'tor_web_addr' not found in Redis DB.
Feb 10 01:29:12 raspberrypi python[813]: 2024-02-10 01:29:12 | ⚠️ | utils.py:69 | Key 'hdd_capacity_bytes' not found in Redis DB.
Feb 10 01:29:13 raspberrypi python[813]: 2024-02-10 01:29:13 | ❌ | service.py:194 | Aiohttp client connection error: Cannot connect to host ...:8332 ssl:default [Connect call failed ('127.0.0.1', 8332)]

*** BLITZ WebUI STATUS ***
installed=1
repo='https://github.com/raspiblitz/raspiblitz-web.git'
branch='release/v1.10'
commit='64de1160e5a65ca179b8e41cd0ba1348539dff34'

• TOUCHSCREEN is OFF by config
• Loop is OFF by config
• LND-RTL is OFF by config
• CL-RTL is OFF by config
• Electrum Rust Server is OFF by config
• LIT is OFF by config
• LNDg is OFF by config

• BTCPayServer is OFF by config
• BTC-RPC-Explorer is OFF by config

• LNbits is OFF by config
• Thunderhub is OFF by config
• SPECTER is OFF by config
• SPHINX is OFF by config
• FINTS is OFF by config

*** MOUNTED DRIVES ***
df -T -h
Filesystem Type Size Used Avail Use% Mounted on
/dev/root ext4 30G 18G 11G 63% /
devtmpfs devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs tmpfs 1.9G 16K 1.9G 1% /dev/shm
tmpfs tmpfs 769M 1.1M 768M 1% /run
tmpfs tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs tmpfs 32M 168K 32M 1% /var/cache/raspiblitz
/dev/mmcblk0p1 vfat 255M 31M 225M 13% /boot
/dev/sda1 ext4 916G 767G 103G 89% /mnt/hdd
tmpfs tmpfs 385M 16K 385M 1% /run/user/1000
/dev/sdb1 vfat 29G 4.9M 29G 1% /mnt/backup
tmpfs tmpfs 385M 16K 385M 1% /run/user/117

*** SD CARD HOMES ***
sudo du -ahd1 /home
1.4G /home/mempool
667M /home/electrs
891M /home/bitcoin
35M /home/pi
2.4G /home/btcpay
721M /home/admin
56K /home/lit
180M /home/btcrpcexplorer
450M /home/rtl
586M /home/joinmarket
565M /home/jam
136M /home/root
1.6G /home/thunderhub
663M /home/lnbits
8.6M /home/circuitbreaker
296M /home/bos
849M /home/blitzapi
12G /home

*** LOGFILES ***
Archived and active journals take up 256.0M in the file system.
918M /var/log

*** DATADRIVE ***

RASPIBLITZ DATA DRIVE Status

BASICS

isMounted=1
isBTRFS=0
hddRaspiData=0
hddRaspiVersion=''
isSMART=0
isSSD=0
datadisk='sda'
datapartition='sda1'
hddCandidate='sda'
hddPartitionCandidate='sda1'
hddBlocksBitcoin=0
hddBytes=
hddGigaBytes=
hddUsedInfo='18G
16K
31M
767G
4.9M (63113891%)'
error='unkown command'

*** NETWORK ***
localip=...
localiprange=.../24
dhcp=1
network_device=eth0

*** ZRAM ***
downloaded=1
serviceLoaded=1
serviceActive=1

*** HARDWARE TEST RESULTS ***
UndervoltageReports in Logs: 0

*** SYSTEM CACHE STATUS ***
system_temp_fahrenheit="95"
system_count_start_lightning="2"
system_ups_battery="100"
system_count_longscan="101"
system_vm_vagrant="0"
system_ram_mb="3844"
system_ups_status="OFF"
system_count_start_blockchain="6"
system_temp_celsius="35"
system_ram_gb="4"
system_up="193815"
system_count_undervoltage="0"
system_ram_available_mb="3280"
system_board="rp4"
system_cpu_load=" 2.58, 2.24, 2.21"
system_init_time="1707334549"
ln_default_sync_initial_done="1"
ln_default_channels_inactive="0"
ln_default_wallet_channels_pending="0"
ln_default_error_short=""
ln_default_activated="1"
ln_default_alias=****
ln_default_recovery_mode="0"
ln_default_peers="6"
ln_default_wallet_channels_balance=****
ln_default_channels_pending="0"
ln_default_locked="0"
ln_default_wallet_onchain_pending="297584"
ln_default_channels_active="7"
ln_default_version="0.16.4-beta"
ln_default_error_full=""
ln_default_sync_chain="1"
ln_default_running="0"
ln_default_wallet_onchain_balance=****
ln_default_ready="0"
ln_default_online="0"
ln_default_fees_total="0"
ln_default_recovery_done="0"
ln_default_sync_progress="100.00"
ln_default_tor="0"
ln_default_channels_total="7"
btc_default_port="8333"
btc_default_sync_percentage="100.00"
btc_default_blocks_data_kb="608518548"
btc_default_peers="18"
btc_default_running="0"
btc_default_error_short=""
btc_default_version="v26.0.0"
btc_default_online="0"
btc_default_synced="1"
btc_default_ready="0"
btc_default_error_full=""
btc_default_mempool_transactions="25704"
btc_default_blocks_verified="829615"
btc_default_blocks_headers="829615"
btc_default_blocks_behind="0"
btc_default_sync_progress="0.9999996963071541"
btc_default_activated="1"
btc_default_sync_initial_done="1"
btc_default_sync_initialblockdownload="0"
*** POSSIBLE ERROR REPORTS ***

*** OPTION: SHARE THIS DEBUG OUTPUT ***
An easy way to share this debug output on GitHub or on a support chat
Use the following command and share the resulting link using termbin.com service and tor proxy:
debug -l
If tor is failing and you don't mind leaking your ip address to the termbin service, use without tor:
debug -l -n

Press ENTER to return to main menu.

[rootzoll]

As you dont run any lightning wallets - so there are no funds on the node. You should try:

• flash new image and do a fresh setup (choose "other options" when ask to recover and then go for FRESH SETUP)

On first try you can reuse the old blockchain, but if that also does not work repeat try with also not using the old blockchain

[SepticFrank]

@rootzoll There are actually some funds in LND and on chain. LND stops running when the bitcoin service goes into a restart loop. Are there any options in the repair menu I should try first? Or just follow the steps for recovery on a fresh setup? Am I looking at getting a new SSD and Pi4 to solve this?

Thanks

[rootzoll]

If you have funds with lnd (channles & on-chain) you should backup those first:

Make sure that you have your LND seed words

Get a download of the StaticChannelBackup (SCB) - for example with:
SSH-MAINMENU > REPAIR > REPAIR-LND > GETSCB

Download the LND Rescue-Backup:
SSH-MAINMENU > REPAIR > REPAIR-LND > BACKUP-LND
Node will shutdown after that.

Now you can try to setup the node fresh (replace hardware if needed):

Flash new image and do a fresh setup (choose "other options" when ask to recover and then go for FRESH SETUP)
On first try you can reuse the old blockchain (if old ssd), but if that also does not work repeat try with also not using the old blockchain. On SSH setup choose with LND and upload the old nodes LND Rescue-Backup - just as fallback if that fails try to recover with SEED+StaticChannelBackup.