Offline Import Not Working #64

thawtes · 2021-11-10T14:13:56Z

offline import not working

image: openatv 6.4

Traceback (most recent call last): File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/OfflineImport.py", line 13, in <module> File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/EPGImport.py", line 12, in <module> ValueError: Attempted relative import in non-package

The text was updated successfully, but these errors were encountered:

TwolDE2 · 2021-11-10T14:39:10Z

OfflineImport
import EPGConfig
import EPGImport ————->>> change to
Import . EPGConfig
import . EPGImport. ——-> check it then works

thawtes · 2021-11-10T15:01:03Z

import EPGConfig

i try and result
import . EPGConfig ^ SyntaxError: invalid syntax

not work

TwolDE2 · 2021-11-10T17:13:30Z

sorry try:
from . import EPGConfig
from . import EPGImport

thawtes · 2021-11-10T20:03:45Z

from . import EPGConfig

I tried again but it still doesn't work

Traceback (most recent call last): File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/OfflineImport.py", line 12, in <module> from . import EPGConfig ValueError: Attempted relative import in non-package

TwolDE2 · 2021-11-11T12:54:01Z

so have now realised this is run basically as a script and those imports in py3 are therefore much harder to handle ... when I have time to concentrate will look, unless someone else has a quick solution
you might try with python execute and the -m option

thawtes · 2022-03-16T06:40:03Z

so have now realised this is run basically as a script and those imports in py3 are therefore much harder to handle ... when I have time to concentrate will look, unless someone else has a quick solution you might try with python execute and the -m option

is there any progress? were you able to do something?

( -m option not working )

thx

TwolDE2 · 2022-03-16T13:13:21Z

Afraid it was summerged by other issues - Now you have reminded me will look again, but will be next week

thawtes · 2022-03-16T14:10:37Z

Afraid it was summerged by other issues - Now you have reminded me will look again, but will be next week

thanks, good luck...

TwolDE2 · 2022-04-01T12:57:20Z

@thawtes - so by adding to the sys.path I was able to get so far, but still big issues .
So how & on what do you normally run this import

Huevos · 2022-04-01T14:06:41Z

@thawtes can you tell us the exact command you uses on the command line and where the files being processed are located.

Huevos · 2022-04-01T20:21:49Z

@thawtes please see this commit... and please give feedback.
5714422

thawtes · 2022-04-02T04:58:04Z

@thawtes - so by adding to the sys.path I was able to get so far, but still big issues . So how & on what do you normally run this import

python /usr/lib/enigma2/python/Plugins/Extensions/EPGImport/OfflineImport.py /etc/epgimport/mycustom.sources.xml (or rytec.sources.xml)
When I give this command, the import of EPG data starts at that moment and the result is successful. A great feature for me.

thawtes · 2022-04-02T04:59:13Z

@thawtes can you tell us the exact command you uses on the command line and where the files being processed are located.

python /usr/lib/enigma2/python/Plugins/Extensions/EPGImport/OfflineImport.py /etc/epgimport/rytec.sources.xml
When I give this command, the import of EPG data starts at that moment and the result is successful. A great feature for me.
everything was working great before this commit a848f6e

thawtes · 2022-04-02T05:11:33Z

@thawtes please see this commit... and please give feedback. 5714422

now i tried again as you said, so like this commit 5714422

command
cd /usr/lib/enigma2/python/Plugins/Extensions

command
python -m EPGImport.OfflineImport it seems to be working, so it doesn't give an error message

command
python -m EPGImport.OfflineImport /etc/epgimport/rytec.sources.xml unfortunately it doesn't work :(

result

[EPGImport] oudeis patch not detected, using epg.dat instead.
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
  File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/OfflineImport.py", line 76, in <module>
  File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/OfflineImport.py", line 59, in importFrom
  File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/EPGImport.py", line 237, in beginImport
  File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/epgdat_importer.py", line 3, in <module>
  File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/epgdat.py", line 13, in <module>
ModuleNotFoundError: No module named 'boxbranding'

TwolDE2 · 2022-04-02T12:52:12Z

which image are you running that on????

Huevos · 2022-04-02T15:17:43Z

Ok so try this... hopefully there will not be more...

In epgdat.py, lines 12/13:

from boxbranding import getImageDistro
EpgDatV8 = getImageDistro() in ("openvix",)

Change to:

try:
	from boxbranding import getImageDistro
	EpgDatV8 = getImageDistro() in ("openvix",)
except:
	EpgDatV8 = os.path.isfile("/etc/image-version") and "distro=openvix" in open("/etc/image-version").read()

thawtes · 2022-04-02T22:10:20Z

which image are you running that on????
openATV 7.0

thawtes · 2022-04-02T22:12:37Z

Ok so try this... hopefully there will not be more...

In epgdat.py, lines 12/13:

from boxbranding import getImageDistro
EpgDatV8 = getImageDistro() in ("openvix",)

Change to:

try:
	from boxbranding import getImageDistro
	EpgDatV8 = getImageDistro() in ("openvix",)
except:
	EpgDatV8 = os.path.isfile("/etc/image-version") and "distro=openvix" in open("/etc/image-version").read()

command now it works! thanks @Huevos @TwolDE2

python -m EPGImport.OfflineImport /etc/epgimport/rytec.sources.xml
[EPGImport] oudeis patch not detected, using epg.dat instead.
[EPGImport] failed to load C implementation, sorry
[EPGImport] nextImport, source= XYZ_SERVICES
[EPGImport] afterDownload /etc/epgimport/myServices.xml
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/rytec.channels.xml'
[XMLTVConverter] Enumerating event information
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport] ### thread is ready ### Events: 8
[EPGImport] imported 8 events
[EPGImport] no Oudeis patch, load(/media/hdd/epg_new.dat) required
Done, data is in /media/hdd/epg_new.dat
[EPGImport] #### Finished ####

but it doesn't create /media/hdd/epg_new.dat file
therefore, the import could not be performed ??

and is this an error message?

[EPGImport] failed to load C implementation, sorry
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object

Huevos · 2022-04-03T12:56:30Z

Can you add some debug in EPGImport.py?

    def doThreadRead(self, filename):
        """This is used on PLi with threading"""
        for data in self.createIterator(filename):
            if data is not None:
            	print("data", data)
                print("type(data)", type(data))
                self.eventCount += 1
                try:
                    r, d = data
                    print("r", r)
                    print("type(r)", type(r))
                    print("d", d)
                    print("type(d)", type(d))
                    print("d[0]", d[0])
                    print("type(d[0])", type(d[0]))
                    print("self.longDescUntil", self.longDescUntil)
                    print("type(self.longDescUntil)", type(self.longDescUntil))
                    print("d[0] > self.longDescUntil", d[0] > self.longDescUntil)
                    if d[0] > self.longDescUntil:
                        # Remove long description (save RAM memory)
                        d = d[:4] + ('',) + d[5:]
                    self.storage.importEvents(r, (d,))
                except Exception as e:
                    import traceback
                    traceback.print_exc()
                    print("[EPGImport] ### importEvents exception:", e, file=log)
        print("[EPGImport] ### thread is ready ### Events:", self.eventCount, file=log)
        if filename:
            try:
                os.unlink(filename)
            except Exception as e:
                print("[EPGImport] warning: Could not remove '%s' intermediate" % filename, e, file=log)

        return

thawtes · 2022-04-03T21:56:47Z

@Huevos

[EPGImport] oudeis patch not detected, using epg.dat instead.
[EPGImport] failed to load C implementation, sorry
[EPGImport] nextImport, source= MY SERVICES
[EPGImport] afterDownload /etc/epgimport/myServices.xml
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/rytec.channels.xml'
[XMLTVConverter] Enumerating event information
data (['1:0:1:CA59:7B0E:42E:1A40000:0:0:0:'], (1649019600, 43200, 'my service program info', '', 'my service descriptions', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA59:7B0E:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649019600, 43200, 'my service program info', '', 'my service descriptions', 0)
type(d) <class 'tuple'>
d[0] 1649019600
type(d[0]) <class 'int'>
self.longDescUntil 1649454729.7295923
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
data (['1:0:1:CA59:7B0E:42E:1A40000:0:0:0:'], (1649062800, 43200, 'my service program info', '', 'my service descriptions', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA59:7B0E:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649062800, 43200, 'my service program info', '', 'my service descriptions', 0)
type(d) <class 'tuple'>
d[0] 1649062800
type(d[0]) <class 'int'>
self.longDescUntil 1649454729.7295923
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
data (['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1649019600, 43200, 'my service program info', '', 'my service descriptions', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649019600, 43200, 'my service program info', '', 'my service descriptions', 0)
type(d) <class 'tuple'>
d[0] 1649019600
type(d[0]) <class 'int'>
self.longDescUntil 1649454729.7295923
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
data (['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1649062800, 43200, 'my service program info', '', 'my service descriptions', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649062800, 43200, 'my service program info', '', 'my service descriptions', 0)
type(d) <class 'tuple'>
d[0] 1649062800
type(d[0]) <class 'int'>
self.longDescUntil 1649454729.7295923
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
data (['1:0:19:CA01:A926:42E:1A40000:0:0:0:'], (1649019600, 43200, 'my service program info', '', 'my service descriptions', 0))
type(data) <class 'tuple'>
r ['1:0:19:CA01:A926:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649019600, 43200, 'my service program info', '', 'my service descriptions', 0)
type(d) <class 'tuple'>
d[0] 1649019600
type(d[0]) <class 'int'>
self.longDescUntil 1649454729.7295923
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
data (['1:0:19:CA01:A926:42E:1A40000:0:0:0:'], (1649062800, 43200, 'my service program info', '', 'my service descriptions', 0))
type(data) <class 'tuple'>
r ['1:0:19:CA01:A926:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649062800, 43200, 'my service program info', '', 'my service descriptions', 0)
type(d) <class 'tuple'>
d[0] 1649062800
type(d[0]) <class 'int'>
self.longDescUntil 1649454729.7295923
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
data (['1:0:1:EA67:4C5:4C5:1A40000:0:0:0:'], (1649019600, 43200, 'my service program info', '', 'my service descriptions', 0))
type(data) <class 'tuple'>
r ['1:0:1:EA67:4C5:4C5:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649019600, 43200, 'my service program info', '', 'my service descriptions', 0)
type(d) <class 'tuple'>
d[0] 1649019600
type(d[0]) <class 'int'>
self.longDescUntil 1649454729.7295923
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
data (['1:0:1:EA67:4C5:4C5:1A40000:0:0:0:'], (1649062800, 43200, 'my service program info', '', 'my service descriptions', 0))
type(data) <class 'tuple'>
r ['1:0:1:EA67:4C5:4C5:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649062800, 43200, 'my service program info', '', 'my service descriptions', 0)
type(d) <class 'tuple'>
d[0] 1649062800
type(d[0]) <class 'int'>
self.longDescUntil 1649454729.7295923
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport] ### thread is ready ### Events: 8
[EPGImport] imported 8 events
[EPGImport] no Oudeis patch, load(/media/hdd/epg_new.dat) required
Done, data is in /media/hdd/epg_new.dat
[EPGImport] #### Finished ####

Huevos · 2022-04-03T23:07:49Z

You seem to be missing part of the debug from your output:

                    import traceback
                    traceback.print_exc()

thawtes · 2022-04-04T00:27:09Z

You seem to be missing part of the debug from your output:
                    import traceback
                    traceback.print_exc()

there are these lines

EPGImport.py
thawtes@28eee35

Huevos · 2022-04-04T08:44:27Z

Your log is showing [EPGImport] ### importEvents exception but not the output from the traceback.

TwolDE2 · 2022-04-04T12:23:45Z

@thawtes
looking at the log, I believe that you are not using epgcache to import events but epgdat_importer and epgdat------->
[EPGImport] oudeis patch not detected, using epg.dat instead.
[EPGImport] failed to load C implementation, sorry

so can you use the 3 modules updated in my last git update(titled offline changes) - I have added Huevos debug plus a bit more
to see if which modules we hit. see --->
TwolDE2@01f1510

thawtes · 2022-04-04T21:29:11Z

@TwolDE2 @Huevos

after this commit TwolDE2@01f1510

[EPGImport][beginImport] oudeis patch not detected, using epg.dat instead.
[EPGImport][beginImport] using epgdat_importer.epgdatclass.
[EPGImport] failed to load C implementation, sorry
[EPGImport] nextImport, source= MY_CHANNELS
[EPGImport] afterDownload /etc/epgimport/myServices.xml
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/rytec.channels.xml'
[XMLTVConverter] Enumerating event information
data (['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1649106000, 43200, 'program_name', '', 'program_desc', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649106000, 43200, 'program_name', '', 'program_desc', 0)
type(d) <class 'tuple'>
d[0] 1649106000
type(d[0]) <class 'int'>
self.longDescUntil 1649539498.63188
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1649106000, 43200, 'program_name', '', 'program_desc', 0)
[EPGImport][doThreadRead] ### importEvents exception: name 'event_starttime_unix_gmt' is not defined
data (['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1649149200, 43200, 'program_name', '', 'program_desc', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649149200, 43200, 'program_name', '', 'program_desc', 0)
type(d) <class 'tuple'>
d[0] 1649149200
type(d[0]) <class 'int'>
self.longDescUntil 1649539498.63188
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1649149200, 43200, 'program_name', '', 'program_desc', 0)
[EPGImport][doThreadRead] ### importEvents exception: name 'event_starttime_unix_gmt' is not defined
[EPGImport][doThreadRead] ### thread is ready ### Events: 2
[epgdat][preprocess_events_channel] service :  1:0:1:CA02:A926:42E:1A40000:0:0:0:
[epgdat][preprocess_events_channel] service :  1:0:16:CB24:A02D:42E:1A40000:0:0:0:
[EPGImport] imported 2 events
[EPGImport] no Oudeis patch, load(/media/hdd/epg_new.dat) required
Done, data is in /media/hdd/epg_new.dat
[EPGImport] #### Finished ####

TwolDE2 · 2022-04-05T09:38:05Z

@thawtes
My bad on print statement, but at least we know where we are...... epgdat.py!

can you copy dreamcrc.c from your XMLTV git into /usr/lib/enigma2/python/Plugins/Extensions/EPGImport and lets see if we can import the C++ routine.
You said you are using ATV, so I am guessing they are using the V8 format for the epg, so have forced that variable to True.
added the import change, more debug, the EpgDatV8 change, etc to epgdat.py
Hopefully all correct this time, and hopefully we will see where it breaks . could be the pack routines or the fancy crc routine as the C++ code was being simulated in python(possibly for the 1st time).
Anyway here: TwolDE2@7c72478

thawtes · 2022-04-05T21:04:00Z

@thawtes My bad on print statement, but at least we know where we are...... epgdat.py!

can you copy dreamcrc.c from your XMLTV git into /usr/lib/enigma2/python/Plugins/Extensions/EPGImport and lets see if we can import the C++ routine.

You said you are using ATV, so I am guessing they are using the V8 format for the epg, so have forced that variable to True.

added the import change, more debug, the EpgDatV8 change, etc to epgdat.py
Hopefully all correct this time, and hopefully we will see where it breaks . could be the pack routines or the fancy crc routine as the C++ code was being simulated in python(possibly for the 1st time).
Anyway here: TwolDE2@7c72478

after this commit TwolDE2@7c72478

root@vuduo2:/usr/lib/enigma2/python/Plugins/Extensions# python -m EPGImport.OfflineImport /etc/epgimport/myServices.sources.xml
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
  File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/OfflineImport.py", line 76, in <module>
  File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/OfflineImport.py", line 59, in importFrom
  File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/EPGImport.py", line 245, in beginImport
    from . import epgdat_importer
  File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/epgdat_importer.py", line 3, in <module>
    from . import epgdat
  File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/epgdat.py", line 16
    import . dreamcrc
           ^
SyntaxError: invalid syntax
[EPGImport][beginImport] oudeis patch not detected, using epg.dat instead.
[EPGImport][beginImport] using epgdat_importer.epgdatclass.

import . dreamcrc change to import dreamcrc

result ...


[EPGImport][beginImport] oudeis patch not detected, using epg.dat instead.
[EPGImport][beginImport] using epgdat_importer.epgdatclass.
[EPGImport] failed to load C implementation, sorry
[EPGImport] nextImport, source= MY_CHANNELS
[EPGImport] afterDownload /etc/epgimport/myServivces.xml
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/rytec.channels.xml'
[XMLTVConverter] Enumerating event information
data (['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1649192400, 43200, 'program_name', '', 'program_desc', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649192400, 43200, 'program_name', '', 'program_desc', 0)
type(d) <class 'tuple'>
d[0] 1649192400
type(d[0]) <class 'int'>
self.longDescUntil 1649625082.8190384
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1649192400, 43200, 'program_name', '', 'program_desc', 0)
[epgdat][add_event]add event:- starttime, duration, title, description 1649192400 43200 program_name program_desc
[epgdat][short_desc] s :  program_name
[EPGImport][doThreadRead] ### importEvents exception: argument for 's' must be a bytes object
data (['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1649235600, 43200, 'program_name', '', 'program_desc', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649235600, 43200, 'program_name', '', 'program_desc', 0)
type(d) <class 'tuple'>
d[0] 1649235600
type(d[0]) <class 'int'>
self.longDescUntil 1649625082.8190384
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1649235600, 43200, 'program_name', '', 'program_desc', 0)
[epgdat][add_event]add event:- starttime, duration, title, description 1649235600 43200 program_name program_desc
[epgdat][short_desc] s :  program_name
[EPGImport][doThreadRead] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport][doThreadRead] ### thread is ready ### Events: 2
[epgdat][preprocess_events_channel] service :  1:0:1:CA02:A926:42E:1A40000:0:0:0:
[epgdat][preprocess_events_channel] service :  1:0:16:CB24:A02D:42E:1A40000:0:0:0:
[EPGImport] imported 2 events
[EPGImport] no Oudeis patch, load(/media/hdd/epg_new.dat) required
Done, data is in /media/hdd/epg_new.dat
[EPGImport] #### Finished ####

TwolDE2 · 2022-04-06T11:05:43Z

Thanks- for your patience!
so made a couple of changes

fixed hopefully import, so if you copied the module over perhaps it will find it.
I encode s before its used in epgdat.py, so lets see if that gets through the struct.pack routines
see https://github.com/TwolDE2/XMLTV-Import/blob/python3/src/EPGImport/epgdat.py
there are a couple of fields in the struct.pack, I am not sure about, but perhaps its just my lack if understanding.

thawtes · 2022-04-07T20:52:02Z

you're welcome, I really thank you..

New LOG

[EPGImport][beginImport] oudeis patch not detected, using epg.dat instead.
[EPGImport][beginImport] using epgdat_importer.epgdatclass.
[EPGImport] using C module, yay
[EPGImport] nextImport, source= MY_CHANNELS
[EPGImport] afterDownload /etc/epgimport/myServices.xml
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/rytec.channels.xml'
[XMLTVConverter] Enumerating event information
data (['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1648069200, 43200, 'MyProgramName', '', 'MyProgramDesc', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1648069200, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
type(d) <class 'tuple'>
d[0] 1648069200
type(d[0]) <class 'int'>
self.longDescUntil 1649796613.7646136
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1648069200, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
[epgdat][add_event]add event:- starttime, duration, title, description 1648069200 43200 MyProgramName MyProgramDesc
[epgdat][short_desc] s :  MyProgramName
[EPGImport][doThreadRead] ### importEvents exception: argument for 's' must be a bytes object
data (['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1648112400, 43200, 'MyProgramName', '', 'MyProgramDesc', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1648112400, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
type(d) <class 'tuple'>
d[0] 1648112400
type(d[0]) <class 'int'>
self.longDescUntil 1649796613.7646136
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1648112400, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
[epgdat][add_event]add event:- starttime, duration, title, description 1648112400 43200 MyProgramName MyProgramDesc
[epgdat][short_desc] s :  MyProgramName
[EPGImport][doThreadRead] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport][doThreadRead] ### thread is ready ### Events: 2
[epgdat][preprocess_events_channel] service :  1:0:1:CA02:A926:42E:1A40000:0:0:0:
[epgdat][preprocess_events_channel] service :  1:0:16:CB24:A02D:42E:1A40000:0:0:0:
[EPGImport] imported 2 events
[EPGImport] no Oudeis patch, load(/media/hdd/epg_new.dat) required
Done, data is in /media/hdd/epg_new.dat
[EPGImport] #### Finished ####

it keeps giving this error:
[EPGImport][doThreadRead] ### importEvents exception: argument for 's' must be a bytes object

and, I still don't understand why /media/hdd/epg_new.dat file is not created

TwolDE · 2022-04-08T15:08:01Z

So import now works for C++
Talked with Huevos and as a result try change on my git to epgdat.py
TwolDE2@57e7b5c

thawtes · 2022-04-08T21:34:02Z

thanks for your patience!

new LOG :

[EPGImport][beginImport] oudeis patch not detected, using epg.dat instead.
[EPGImport][beginImport] using epgdat_importer.epgdatclass.
[EPGImport] using C module, yay
[EPGImport] nextImport, source= MY_CHANNELS
[EPGImport] afterDownload /etc/epgimport/myServices.xml
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/rytec.channels.xml'
[XMLTVConverter] Enumerating event information
data (['1:0:1:CA02:A926:42E:1A40000:0:0:0:'], (1649451600, 43200, 'MyProgramName', '', 'MyProgramDesc', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA02:A926:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649451600, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
type(d) <class 'tuple'>
d[0] 1649451600
type(d[0]) <class 'int'>
self.longDescUntil 1649884937.8729842
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1649451600, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
[epgdat][add_event]add event:- starttime, duration, title, description 1649451600 43200 MyProgramName MyProgramDesc
[epgdat][short_desc] s :  MyProgramName
[EPGImport][doThreadRead] ### importEvents exception: can't concat str to bytes
data (['1:0:1:CA02:A926:42E:1A40000:0:0:0:'], (1649494800, 43200, 'MyProgramName', '', 'MyProgramDesc', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA02:A926:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649494800, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
type(d) <class 'tuple'>
d[0] 1649494800
type(d[0]) <class 'int'>
self.longDescUntil 1649884937.8729842
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1649494800, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
[epgdat][add_event]add event:- starttime, duration, title, description 1649494800 43200 MyProgramName MyProgramDesc
[epgdat][short_desc] s :  MyProgramName
[EPGImport][doThreadRead] ### importEvents exception: can't concat str to bytes
[EPGImport][doThreadRead] ### thread is ready ### Events: 2
[epgdat][preprocess_events_channel] service :  1:0:1:CA02:A926:42E:1A40000:0:0:0:
[EPGImport] imported 2 events
[EPGImport] no Oudeis patch, load(/media/hdd/epg_new.dat) required
Done, data is in /media/hdd/epg_new.dat
[EPGImport] #### Finished ####

[EPGImport][doThreadRead] ### importEvents exception: can't concat str to bytes

note: /media/hdd/epg_new.dat file still cannot be created.

Could it be a problem with the epgdat_importer.py file?

thx

Huevos · 2022-04-08T22:58:07Z

First we need to get rid of the errors. Then it should work.

TwolDE · 2022-04-09T08:37:07Z

So import now works for C++
Talked with Huevos and as a result try change on my git to epgdat.py
TwolDE2@57e7b5c

TwolDE · 2022-04-09T08:40:11Z

so new update to epgdat - changes variable name "s" to sd & ld to avoid confusion on debug.
also changes in short & long description routines so that hopefully we get binary string(s) in both routines (issue was string added to bytes) ... this is python 3 only, need to modify to support both
TwolDE2@af6f69f

thawtes · 2022-04-09T17:38:30Z

[EPGImport][beginImport] oudeis patch not detected, using epg.dat instead.
[EPGImport][beginImport] using epgdat_importer.epgdatclass.
[EPGImport] using C module, yay
[EPGImport] nextImport, source= MY_CHANNELS
[EPGImport] afterDownload /etc/epgimport/myServices.xml
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/rytec.channels.xml'

[XMLTVConverter] Enumerating event information

data (['1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1649538000, 43200, 'MyProgramName', '', 'MyProgramDesc', 0))
type(data) <class 'tuple'>
r ['1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649538000, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
type(d) <class 'tuple'>
d[0] 1649538000
type(d[0]) <class 'int'>
self.longDescUntil 1649995104.6941166
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1649538000, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
[epgdat][add_event]add event:- starttime, duration, title, description 1649538000 43200 MyProgramName MyProgramDesc
[epgdat][short_desc] sd :  MyProgramName
[epgdat][short_desc] res :  b'eng\x0e\x15MyProgramName\x00'
[epgdat][long_desc] ld :  MyProgramDesc
[epgdat][long_desc] num_tot_desc :  1

data (['1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1649581200, 43200, 'MyProgramName', '', 'MyProgramDesc', 0))
type(data) <class 'tuple'>
r ['1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649581200, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
type(d) <class 'tuple'>
d[0] 1649581200
type(d[0]) <class 'int'>
self.longDescUntil 1649995104.6941166
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1649581200, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
[epgdat][add_event]add event:- starttime, duration, title, description 1649581200 43200 MyProgramName MyProgramDesc
[epgdat][short_desc] sd :  MyProgramName
[epgdat][short_desc] res :  b'eng\x0e\x15MyProgramName\x00'
[epgdat][long_desc] ld :  MyProgramDesc
[epgdat][long_desc] num_tot_desc :  1

[EPGImport][doThreadRead] ### thread is ready ### Events: 2
[epgdat][preprocess_events_channel] service :  1:0:16:CB24:A02D:42E:1A40000:0:0:0:
[epgdat][preprocess_events_channel] event :  (1649538000, 43200, (1729232839, b'eng\x0e\x15MyProgramName\x00'), [(2728014673, b'\x00eng\x00\x00\x0e\x15MyProgramDesc')])
[epgdat][preprocess_events_channel] packing data for event:  (1649538000, 43200, (1729232839, b'eng\x0e\x15MyProgramName\x00'), [(2728014673, b'\x00eng\x00\x00\x0e\x15MyProgramDesc')])

[EPGImport] Failure in epg_done
[EPGImport] imported 2 events
[EPGImport] no Oudeis patch, load(/media/hdd/epg_new.dat) required
Done, data is in /media/hdd/epg_new.dat
[EPGImport] #### Finished ####

[EPGImport] Failure in epg_done > this error msg in epgdat_importer.py

TwolDE · 2022-04-10T07:05:33Z

so better, 90% there? ... hopefully added some more corrections, debug on final packs - all in epgdat.py
TwolDE2@f7bc481

thawtes · 2022-04-10T17:46:53Z

[EPGImport][beginImport] oudeis patch not detected, using epg.dat instead.
[EPGImport][beginImport] using epgdat_importer.epgdatclass.
[EPGImport] using C module, yay
[EPGImport] nextImport, source= MY_CHANNELS
[EPGImport] afterDownload /etc/epgimport/myServices.xml
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/rytec.channels.xml'
[XMLTVConverter] Enumerating event information
data (['1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1649538000, 43200, 'MyProgramName', '', 'MyProgramDesc', 0))
type(data) <class 'tuple'>
r ['1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649538000, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
type(d) <class 'tuple'>
d[0] 1649538000
type(d[0]) <class 'int'>
self.longDescUntil 1650044574.933827
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1649538000, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
[epgdat][add_event]add event:- starttime, duration, title, description 1649538000 43200 MyProgramName MyProgramDesc
[epgdat][short_desc] sd :  MyProgramName
[epgdat][short_desc] res :  b'eng\x0e\x15MyProgramName\x00'
[epgdat][long_desc] ld :  MyProgramDesc
[epgdat][long_desc] num_tot_desc :  1
[EPGImport][doThreadRead] ### thread is ready ### Events: 1
[epgdat][preprocess_events_channel] service :  1:0:16:CB24:A02D:42E:1A40000:0:0:0:
[epgdat][preprocess_events_channel] event :  (1649538000, 43200, (1729232839, b'eng\x0e\x15MyProgramName\x00'), [(2728014673, b'\x00eng\x00\x00\x0e\x15MyProgramDesc')])
[epgdat][preprocess_events_channel] packing data for event:  (1649538000, 43200, (1729232839, b'eng\x0e\x15MyProgramName\x00'), [(2728014673, b'\x00eng\x00\x00\x0e\x15MyProgramDesc')])
[epgdat][preprocess_events_channel] through pack_1 2nd  b'\x00\x01\xe9\x1e'
[epgdat][preprocess_events_channel] through pack_2 b'!\x00\x00'
[epgdat][preprocess_events_channel] through pack_3 b'\x12\x00\x00'
[epgdat][preprocess_events_channel] through pack_4 b'\xc7\xff\x11g'
[epgdat][preprocess_events_channel] through pack_4 - 2nd b'\xc7\xff\x11gQ3\x9a\xa2'
[epgdat][final_process] ENTRY : 
[epgdat][final_process] through pack_1 b'2Tv\x98ENIGMA_EPG_V8\x01\x00\x00\x00'
[epgdat][final_process] through pack_1 - 2nd b'\x02\x00\x00\x00'
[epgdat][final_process] through final pack_1 pack_2 b'\xc7\xff\x11g\x01\x00\x00\x00' [b'M\x13eng\x0e\x15MyProgramName\x00', 1]
[epgdat][final_process] through final pack_1 pack_2 b'Q3\x9a\xa2\x01\x00\x00\x00' [b'N\x15\x00eng\x00\x00\x0e\x15MyProgramDesc', 1]
[EPGImport] imported 1 events
[EPGImport] no Oudeis patch, load(/media/hdd/epg_new.dat) required
Done, data is in /media/hdd/epg_new.dat
[EPGImport] #### Finished ####

**that's it :) it works now. @TwolDE @Huevos thank you so much

now created /media/hdd/epg_new.dat**

TwolDE2 · 2022-04-10T17:58:37Z

Great - will strip out debug and issue pull after some cleanup
Also need to see if we can get the C++ routine automatically added into the epgimport directory have to look at ipk build

thawtes · 2022-04-11T20:52:10Z

@TwolDE @TwolDE2 @Huevos

there is a problem.

/media/hdd/epg_new.dat file is created (after change to epg.dat) but data is not visible in EPG list +infobar EPG

When I restart the device, the epg.dat file remains as epg.dat.loading and enigma2 does not start, the system becomes unusable.

maybe this LOG can be useful
start enigma2 with debug log

[eEPGCache] time updated.. start EPG Mainloop
[eEPGCache] /media/hdd/epg.dat not found, try /epg.dat
[eEPGCache] /epg.dat not found, giving up
[eEPGTransponderDataReader] start Mainloop
[eEPGChannelData] next update in 2 sec
[SEC] set dynamic current limiting
[eDVBServicePlay] timeshift
[eEPGChannelData] start reading events(1649710479)
[eDVBSectionReader] DMX_SET_FILTER pid=18
[eDVBSectionReader] DMX_SET_FILTER pid=18
[eDVBSectionReader] DMX_SET_FILTER pid=18
[eEPGChannelData] abort non avail nownext reading
[eEPGChannelData] abort non avail schedule reading
[eEPGChannelData] abort non avail schedule other reading

thanks

TwolDE · 2022-04-12T11:34:28Z

so having stopped the image, you have renamed the old dat file and copied /media/hdd/epg_new.dat to /media/hdd/epg.dat and then restarted the image?

TwolDE · 2022-04-12T14:37:28Z

... you could also rename/disable the C++ routine and see if you have same issue with the embedded software routine in epgdat.py when you copy dat file

thawtes · 2022-04-12T17:34:30Z

so having stopped the image, you have renamed the old dat file and copied /media/hdd/epg_new.dat to /media/hdd/epg.dat and then restarted the image?

yes i know, i did it this way

thawtes · 2022-04-12T17:36:22Z

... you could also rename/disable the C++ routine and see if you have same issue with the embedded software routine in epgdat.py when you copy dat file

i didn't fully understand it

TwolDE2 · 2022-04-12T19:37:13Z

If the C++ routine (dreamcrc) doesn‘t load, then it uses a “ software simulation“ in python in epgdat.py. I just wondered if that might work better (although probably slower). So perhaps rename dreamcrc to something else, so that the import fails and then it will use the python code instead.

the fact that epg.dat doesn‘t load implies the data is not correct, something I experienced initially with Crossepg

thawtes · 2022-04-12T23:25:13Z

ok, i understand

without dreamcrc :

[EPGImport][beginImport] oudeis patch not detected, using epg.dat instead.
[EPGImport][beginImport] using epgdat_importer.epgdatclass.
[EPGImport] failed to load C implementation, sorry
....
[EPGImport][doThreadRead] ### importEvents exception: ord() expected string of length 1, but int found
....

/media/hdd/epg_new.dat file is not created

with dreamcrc #64 (comment)

Could the problem be due to the characters being this way (b'eng\x0c\x15 ....etc)?

TwolDE · 2022-04-22T11:17:54Z

@thawtes
So I have cleaned up a bit more and everything now runs clean on both C++ crc module or via ,py version.
But epg.dat is corrupt on loading.
So thought I would try python2 version.
runs and produces same number of records.
But epg.dat is corrupt on loading.

So I cannot get either py2 or py3 Ofline_importer versions to produce a valid epg.dat
I am using rytec UK files as import source and running on OpenViX with V8 epgcache (py2 & py3).

TwolDE · 2022-04-26T13:24:16Z

@thawtes - fixed will upload tomorrrow

AbuBaniaz · 2023-01-19T15:28:48Z

@thawtes
Is it OK to close this issue? I have been advised it has been fixed.

thawtes · 2023-01-20T13:47:42Z

@thawtes Is it OK to close this issue? I have been advised it has been fixed.

I will check and let you know

Offline Import Not Working #64

Offline Import Not Working #64

Comments

thawtes commented Nov 10, 2021 • edited Loading

TwolDE2 commented Nov 10, 2021

thawtes commented Nov 10, 2021 • edited Loading

TwolDE2 commented Nov 10, 2021

thawtes commented Nov 10, 2021

TwolDE2 commented Nov 11, 2021

thawtes commented Mar 16, 2022

TwolDE2 commented Mar 16, 2022

thawtes commented Mar 16, 2022 • edited Loading

TwolDE2 commented Apr 1, 2022

Huevos commented Apr 1, 2022

Huevos commented Apr 1, 2022

thawtes commented Apr 2, 2022 • edited Loading

thawtes commented Apr 2, 2022 • edited Loading

thawtes commented Apr 2, 2022

TwolDE2 commented Apr 2, 2022

Huevos commented Apr 2, 2022

thawtes commented Apr 2, 2022

thawtes commented Apr 2, 2022 • edited Loading

Huevos commented Apr 3, 2022 • edited Loading

thawtes commented Apr 3, 2022 • edited Loading

Huevos commented Apr 3, 2022

thawtes commented Apr 4, 2022 • edited Loading

Huevos commented Apr 4, 2022

TwolDE2 commented Apr 4, 2022

thawtes commented Apr 4, 2022

TwolDE2 commented Apr 5, 2022

thawtes commented Apr 5, 2022 • edited Loading

TwolDE2 commented Apr 6, 2022

thawtes commented Apr 7, 2022 • edited Loading

TwolDE commented Apr 8, 2022

thawtes commented Apr 8, 2022 • edited Loading

Huevos commented Apr 8, 2022

TwolDE commented Apr 9, 2022

TwolDE commented Apr 9, 2022

thawtes commented Apr 9, 2022 • edited Loading

TwolDE commented Apr 10, 2022

thawtes commented Apr 10, 2022

TwolDE2 commented Apr 10, 2022

thawtes commented Apr 11, 2022 • edited Loading

TwolDE commented Apr 12, 2022

TwolDE commented Apr 12, 2022

thawtes commented Apr 12, 2022 • edited Loading

thawtes commented Apr 12, 2022

TwolDE2 commented Apr 12, 2022

thawtes commented Apr 12, 2022

TwolDE commented Apr 22, 2022

TwolDE commented Apr 26, 2022

AbuBaniaz commented Jan 19, 2023

thawtes commented Jan 20, 2023

thawtes commented Nov 10, 2021 •

edited

Loading

thawtes commented Nov 10, 2021 •

edited

Loading

thawtes commented Mar 16, 2022 •

edited

Loading

thawtes commented Apr 2, 2022 •

edited

Loading

thawtes commented Apr 2, 2022 •

edited

Loading

thawtes commented Apr 2, 2022 •

edited

Loading

Huevos commented Apr 3, 2022 •

edited

Loading

thawtes commented Apr 3, 2022 •

edited

Loading

thawtes commented Apr 4, 2022 •

edited

Loading

thawtes commented Apr 5, 2022 •

edited

Loading

thawtes commented Apr 7, 2022 •

edited

Loading

thawtes commented Apr 8, 2022 •

edited

Loading

thawtes commented Apr 9, 2022 •

edited

Loading

thawtes commented Apr 11, 2022 •

edited

Loading

thawtes commented Apr 12, 2022 •

edited

Loading