Can't get this working

[calvinbui]

I'm yet to get this thing working. I'm following the docker-compose file to deploy via Ansible. If I was to translate it to a docker-compose file, it would look like:

version: '3'
services:
  postgres:
    ports:
      - '5432:5432'
    environment:
      - POSTGRES_USER=sponsorblock
      - POSTGRES_PASSWORD=abc123
      - POSTGRES_DB=sponsorblock
    volumes:
      - ./mirror:/mirror
      - ./postgres:/var/lib/postgresql/data:rw
    image: postgres:alpine

  sponsorblock-mirror:
    image: ghcr.io/mchangrh/sb-mirror:latest
    environment:
      - MIRROR="TRUE"
      - SQLITE="FALSE"
      - MIRROR_URL="sponsor.ajay.app"
    volumes:
      - ./mirror:/mirror
      - ./export:/export

  sponsorblock-server:
    ports: 
     - "8080:8080"
    volumes:
      - ./export/SponsorTimesDB.db:/app/database/SponsorTimesDB.db
      - ./postgres-config.json:/app/config.json
    image: ghcr.io/mchangrh/sb-server-runner:latest

config:

{
  "port": 8080,
  "globalSalt": "mirrorsalt",
  "adminUserID": "c132d179bfa6a48f4014c163e4f530ecb401f505cf2e5fd02f5b1bb55ac97f5c",
  "behindProxy": true,
  "postgres": {
    "user": "sponsorblock",
    "password": "abc123",
    "host": "postgres",
    "port": 5432
  },
  "mode": "mirror",
  "dumpDatabase": {
    "postgresExportPath": "/mirror"
  }
}

and the current logs:

❯ docker logs -f sponsorblock-mirror
Uses SponsorBlock data from https://sponsor.ajay.app/
Downloading from sponsor.ajay.app
Validating Downloads
rsync error: timeout waiting for daemon connection (code 35) at socket.c(278) [Receiver=3.2.3]
head: /mirror/*.csv: No such file or directory
awk: /mirror/*.csv: No such file or directory
mv: can't rename 'tmp.csv': No such file or directory
Starting SQLite Conversion
Error: cannot open "/mirror/*.csv"
Starting rsync daemon
Downloading from sponsor.ajay.app
rsync error: timeout waiting for daemon connection (code 35) at socket.c(278) [Receiver=3.2.3]
Validating Downloads
head: /mirror/*.csv: No such file or directory
awk: /mirror/*.csv: No such file or directory
mv: can't rename 'tmp.csv': No such file or directory
Starting SQLite Conversion
Error: cannot open "/mirror/*.csv"

❯ docker logs -f sponsorblock-server
Licenced under the MIT Licence https://github.com/ajayyy/SponsorBlockServer
Already up to date.

> sponsor_block_server@0.1.0 start
> ts-node src/index.ts

 WARN  2021-12-04T08:51:02.929Z:   [dumpDatabase] No tables configured

[mchangrh]

sponsor.ajay.app is currently going through CloudFlare so rsync cannot connect but it should be falling back on cURL

[mchangrh]

overriding MIRROR_URL will also make it download recursively and also not fallback on cURL

[mchangrh]

• awk not printing ','
• base sqlite is out of date
• manual address override

[mchangrh]

I'll push the changes I have now, but the "csv" export format from postgres is invalid and breaks sqlite, that will require an additional awk

[calvinbui]

any updates?

[mchangrh]

last I remember, I got it working by filtering awk, and was blocked by #12 I'll see if I can import CSV directly into postgres instead

[xRealNeon]

The rsync connection with this container won't work for me, but just like this, I get it

rsync -r rsync://rsync.sponsor.ajay.app:31111/sponsorblock/ sponsorblock/ -P