diff options
Diffstat (limited to 'backend/tolData')
28 files changed, 0 insertions, 3388 deletions
diff --git a/backend/tolData/README.md b/backend/tolData/README.md deleted file mode 100644 index 3b78af8..0000000 --- a/backend/tolData/README.md +++ /dev/null @@ -1,149 +0,0 @@ -This directory holds files used to generate the tree-of-life database data.db. - -# Database Tables -## Tree Structure -- `nodes` <br> - Format : `name TEXT PRIMARY KEY, id TEXT UNIQUE, tips INT` <br> - Represents a tree-of-life node. `tips` holds the number of no-child descendants -- `edges` <br> - Format: `parent TEXT, child TEXT, p_support INT, PRIMARY KEY (parent, child)` <br> - `p_support` is 1 if the edge has 'phylogenetic support', and 0 otherwise -## Node Mappings -- `eol_ids` <br> - Format: `name TEXT PRIMARY KEY, id INT` <br> - Associates nodes with EOL IDs -- `wiki_ids` <br> - Format: `name TEXT PRIMARY KEY, id INT` <br> - Associates nodes with wikipedia page IDs -## Node Vernacular Names -- `names` <br> - Format: `name TEXT, alt_name TEXT, pref_alt INT, src TEXT, PRIMARY KEY(name, alt_name)` <br> - Associates a node with alternative names. - `pref_alt` is 1 if the alt-name is the most 'preferred' one. - `src` indicates the dataset the alt-name was obtained from (can be 'eol', 'enwiki', or 'picked'). -## Node Descriptions -- `descs` <br> - Format: `wiki_id INT PRIMARY KEY, desc TEXT, from_dbp INT` <br> - Associates a wikipedia page ID with a short-description. - `from_dbp` is 1 if the description was obtained from DBpedia, and 0 otherwise. -## Node Images -- `node_imgs` <br> - Format: `name TEXT PRIMARY KEY, img_id INT, src TEXT` <br> - Associates a node with an image. -- `images` <br> - Format: `id INT, src TEXT, url TEXT, license TEXT, artist TEXT, credit TEXT, PRIMARY KEY (id, src)` <br> - Represents an image, identified by a source ('eol', 'enwiki', or 'picked'), and a source-specific ID. -- `linked_imgs` <br> - Format: `name TEXT PRIMARY KEY, otol_ids TEXT` <br> - Associates a node with an image from another node. - `otol_ids` can be an otol ID, or (for compound nodes) two comma-separated strings that may be otol IDs or empty. -## Reduced Trees -- `nodes_t`, `nodes_i`, `nodes_p` <br> - These are like `nodes`, but describe nodes of reduced trees. -- `edges_t`, `edges_i`, `edges_p` <br> - Like `edges` but for reduced trees. -## Other -- `node_iucn` <br> - Format: `name TEXT PRIMARY KEY, iucn TEXT` <br> - Associates nodes with IUCN conservation status strings (eg: 'endangered') -- `node_pop` <br> - Format: `name TEXT PRIMARY KEY, pop INT` <br> - Associates nodes with popularity values (higher means more popular) - -# Generating the Database - -As a warning, the whole process takes a lot of time and file space. The -tree will probably have about 2.6 million nodes. Downloading the images -takes several days, and occupies over 200 GB. - -## Environment -Some of the scripts use third-party packages: -- `indexed_bzip2`: For parallelised bzip2 processing. -- `jsonpickle`: For encoding class objects as JSON. -- `requests`: For downloading data. -- `PIL`: For image processing. -- `tkinter`: For providing a basic GUI to review images. -- `mwxml`, `mwparserfromhell`: For parsing Wikipedia dumps. - -## Generate Tree Structure Data -1. Obtain 'tree data files' in otol/, as specified in it's README. -2. Run genOtolData.py, which creates data.db, and adds the `nodes` and `edges` tables, - using data in otol/. It also uses these files, if they exist: - - pickedOtolNames.txt: Has lines of the form `name1|otolId1`. - Can be used to override numeric suffixes added to same-name nodes. - -## Generate Dataset Mappings -1. Obtain 'taxonomy data files' in otol/, 'mapping files' in eol/, - files in wikidata/, and 'dump-index files' in enwiki/, as specified - in their READMEs. -2. Run genMappingData.py, which adds the `eol_ids` and `wiki_ids` tables, - as well as `node_iucn`. It uses the files obtained above, the `nodes` table, - and 'picked mappings' files, if they exist. - - pickedEolIds.txt contains lines like `3785967|405349`, specifying - an otol ID and an eol ID to map it to. The eol ID can be empty, - in which case the otol ID won't be mapped. - - pickedWikiIds.txt and pickedWikiIdsRough.txt contain lines like - `5341349|Human`, specifying an otol ID and an enwiki title, - which may contain spaces. The title can be empty. - -## Generate Node Name Data -1. Obtain 'name data files' in eol/, and 'description database files' in enwiki/, - as specified in their READMEs. -2. Run genNameData.py, which adds the `names` table, using data in eol/ and enwiki/, - along with the `nodes`, `eol_ids`, and `wiki_ids` tables. <br> - It also uses pickedNames.txt, if it exists. This file can hold lines like - `embryophyta|land plant|1`, specifying a node name, an alt-name to add for it, - and a 1 or 0 indicating whether it is a 'preferred' alt-name. The last field - can be empty, which indicates that the alt-name should be removed, or, if the - alt-name is the same as the node name, that no alt-name should be preferred. - -## Generate Node Description Data -1. Obtain files in dbpedia/, as specified in it's README. -2. Run genDescData.py, which adds the `descs` table, using data in dbpedia/ and - enwiki/, and the `nodes` table. - -## Generate Node Images Data -### Get images from EOL -1. Obtain 'image metadata files' in eol/, as specified in it's README. -2. In eol/, run downloadImgs.py, which downloads images (possibly multiple per node), - into eol/imgsForReview, using data in eol/, as well as the `eol_ids` table. -3. In eol/, run reviewImgs.py, which interactively displays the downloaded images for - each node, providing the choice of which to use, moving them to eol/imgs/. - Uses `names` and `eol_ids` to display extra info. -### Get Images from Wikipedia -1. In enwiki/, run genImgData.py, which looks for wikipedia image names for each node, - using the `wiki_ids` table, and stores them in a database. -2. In enwiki/, run downloadImgLicenseInfo.py, which downloads licensing information for - those images, using wikipedia's online API. -3. In enwiki/, run downloadImgs.py, which downloads 'permissively-licensed' - images into enwiki/imgs/. -### Merge the Image Sets -1. Run reviewImgsToGen.py, which displays images from eol/imgs/ and enwiki/imgs/, - and enables choosing, for each node, which image should be used, if any, - and outputs choice information into imgList.txt. Uses the `nodes`, - `eol_ids`, and `wiki_ids` tables (as well as `names` to display extra info). -2. Run genImgs.py, which creates cropped/resized images in img/, from files listed in - imgList.txt and located in eol/ and enwiki/, and creates the `node_imgs` and - `images` tables. If pickedImgs/ is present, images within it are also used. <br> - The outputs might need to be manually created/adjusted: - - An input image might have no output produced, possibly due to - data incompatibilities, memory limits, etc. A few input image files - might actually be html files, containing a 'file not found' page. - - An input x.gif might produce x-1.jpg, x-2.jpg, etc, instead of x.jpg. - - An input image might produce output with unexpected dimensions. - This seems to happen when the image is very large, and triggers a - decompression bomb warning. -### Add more Image Associations -1. Run genLinkedImgs.py, which tries to associate nodes without images to - images of it's children. Adds the `linked_imgs` table, and uses the - `nodes`, `edges`, and `node_imgs` tables. - -## Generate Reduced Trees -1. Run genReducedTrees.py, which generates multiple reduced versions of the tree, - adding the `nodes_*` and `edges_*` tables, using `nodes` and `names`. Reads from - pickedNodes.txt, which lists names of nodes that must be included (1 per line). - -## Generate Node Popularity Data -1. Obtain 'page view files' in enwiki/Run genPopData.py, as specified in it's README. -2. Run genPopData.py, which adds the `node_pop` table, using data in enwiki/, - and the `wiki_ids` table. diff --git a/backend/tolData/dbpedia/README.md b/backend/tolData/dbpedia/README.md deleted file mode 100644 index dd9bda7..0000000 --- a/backend/tolData/dbpedia/README.md +++ /dev/null @@ -1,29 +0,0 @@ -This directory holds files obtained/derived from [Dbpedia](https://www.dbpedia.org). - -# Downloaded Files -- `labels_lang=en.ttl.bz2` <br> - Obtained via https://databus.dbpedia.org/dbpedia/collections/latest-core. - Downloaded from <https://databus.dbpedia.org/dbpedia/generic/labels/2022.03.01/labels_lang=en.ttl.bz2>. -- `page_lang=en_ids.ttl.bz2` <br> - Downloaded from <https://databus.dbpedia.org/dbpedia/generic/page/2022.03.01/page_lang=en_ids.ttl.bz2> -- `redirects_lang=en_transitive.ttl.bz2` <br> - Downloaded from <https://databus.dbpedia.org/dbpedia/generic/redirects/2022.03.01/redirects_lang=en_transitive.ttl.bz2>. -- `disambiguations_lang=en.ttl.bz2` <br> - Downloaded from <https://databus.dbpedia.org/dbpedia/generic/disambiguations/2022.03.01/disambiguations_lang=en.ttl.bz2>. -- `instance-types_lang=en_specific.ttl.bz2` <br> - Downloaded from <https://databus.dbpedia.org/dbpedia/mappings/instance-types/2022.03.01/instance-types_lang=en_specific.ttl.bz2>. -- `short-abstracts_lang=en.ttl.bz2` <br> - Downloaded from <https://databus.dbpedia.org/vehnem/text/short-abstracts/2021.05.01/short-abstracts_lang=en.ttl.bz2>. - -# Other Files -- genDescData.py <br> - Used to generate a database representing data from the ttl files. -- descData.db <br> - Generated by genDescData.py. <br> - Tables: <br> - - `labels`: `iri TEXT PRIMARY KEY, label TEXT ` - - `ids`: `iri TEXT PRIMARY KEY, id INT` - - `redirects`: `iri TEXT PRIMARY KEY, target TEXT` - - `disambiguations`: `iri TEXT PRIMARY KEY` - - `types`: `iri TEXT, type TEXT` - - `abstracts`: `iri TEXT PRIMARY KEY, abstract TEXT` diff --git a/backend/tolData/dbpedia/genDescData.py b/backend/tolData/dbpedia/genDescData.py deleted file mode 100755 index 43ed815..0000000 --- a/backend/tolData/dbpedia/genDescData.py +++ /dev/null @@ -1,128 +0,0 @@ -#!/usr/bin/python3 - -import re -import bz2, sqlite3 - -import argparse -parser = argparse.ArgumentParser(description=""" -Adds DBpedia labels/types/abstracts/etc data into a database -""", formatter_class=argparse.RawDescriptionHelpFormatter) -parser.parse_args() - -labelsFile = 'labels_lang=en.ttl.bz2' # Had about 16e6 entries -idsFile = 'page_lang=en_ids.ttl.bz2' -redirectsFile = 'redirects_lang=en_transitive.ttl.bz2' -disambigFile = 'disambiguations_lang=en.ttl.bz2' -typesFile = 'instance-types_lang=en_specific.ttl.bz2' -abstractsFile = 'short-abstracts_lang=en.ttl.bz2' -dbFile = 'descData.db' -# In testing, this script took a few hours to run, and generated about 10GB - -print('Creating database') -dbCon = sqlite3.connect(dbFile) -dbCur = dbCon.cursor() - -print('Reading/storing label data') -dbCur.execute('CREATE TABLE labels (iri TEXT PRIMARY KEY, label TEXT)') -dbCur.execute('CREATE INDEX labels_idx ON labels(label)') -dbCur.execute('CREATE INDEX labels_idx_nc ON labels(label COLLATE NOCASE)') -labelLineRegex = re.compile(r'<([^>]+)> <[^>]+> "((?:[^"]|\\")+)"@en \.\n') -lineNum = 0 -with bz2.open(labelsFile, mode='rt') as file: - for line in file: - lineNum += 1 - if lineNum % 1e5 == 0: - print(f'At line {lineNum}') - # - match = labelLineRegex.fullmatch(line) - if match is None: - raise Exception(f'ERROR: Line {lineNum} has unexpected format') - dbCur.execute('INSERT INTO labels VALUES (?, ?)', (match.group(1), match.group(2))) - -print('Reading/storing wiki page ids') -dbCur.execute('CREATE TABLE ids (iri TEXT PRIMARY KEY, id INT)') -dbCur.execute('CREATE INDEX ids_idx ON ids(id)') -idLineRegex = re.compile(r'<([^>]+)> <[^>]+> "(\d+)".*\n') -lineNum = 0 -with bz2.open(idsFile, mode='rt') as file: - for line in file: - lineNum += 1 - if lineNum % 1e5 == 0: - print(f'At line {lineNum}') - # - match = idLineRegex.fullmatch(line) - if match is None: - raise Exception(f'ERROR: Line {lineNum} has unexpected format') - try: - dbCur.execute('INSERT INTO ids VALUES (?, ?)', (match.group(1), int(match.group(2)))) - except sqlite3.IntegrityError as e: - # Accounts for certain lines that have the same IRI - print(f'WARNING: Failed to add entry with IRI "{match.group(1)}": {e}') - -print('Reading/storing redirection data') -dbCur.execute('CREATE TABLE redirects (iri TEXT PRIMARY KEY, target TEXT)') -redirLineRegex = re.compile(r'<([^>]+)> <[^>]+> <([^>]+)> \.\n') -lineNum = 0 -with bz2.open(redirectsFile, mode='rt') as file: - for line in file: - lineNum += 1 - if lineNum % 1e5 == 0: - print(f'At line {lineNum}') - # - match = redirLineRegex.fullmatch(line) - if match is None: - raise Exception(f'ERROR: Line {lineNum} has unexpected format') - dbCur.execute('INSERT INTO redirects VALUES (?, ?)', (match.group(1), match.group(2))) - -print('Reading/storing diambiguation-page data') -dbCur.execute('CREATE TABLE disambiguations (iri TEXT PRIMARY KEY)') -disambigLineRegex = redirLineRegex -lineNum = 0 -with bz2.open(disambigFile, mode='rt') as file: - for line in file: - lineNum += 1 - if lineNum % 1e5 == 0: - print(f'At line {lineNum}') - # - match = disambigLineRegex.fullmatch(line) - if match is None: - raise Exception(f'ERROR: Line {lineNum} has unexpected format') - dbCur.execute('INSERT OR IGNORE INTO disambiguations VALUES (?)', (match.group(1),)) - -print('Reading/storing instance-type data') -dbCur.execute('CREATE TABLE types (iri TEXT, type TEXT)') -dbCur.execute('CREATE INDEX types_iri_idx ON types(iri)') -typeLineRegex = redirLineRegex -lineNum = 0 -with bz2.open(typesFile, mode='rt') as file: - for line in file: - lineNum += 1 - if lineNum % 1e5 == 0: - print(f'At line {lineNum}') - # - match = typeLineRegex.fullmatch(line) - if match is None: - raise Exception(f'ERROR: Line {lineNum} has unexpected format') - dbCur.execute('INSERT INTO types VALUES (?, ?)', (match.group(1), match.group(2))) - -print('Reading/storing abstracts') -dbCur.execute('CREATE TABLE abstracts (iri TEXT PRIMARY KEY, abstract TEXT)') -descLineRegex = labelLineRegex -lineNum = 0 -with bz2.open(abstractsFile, mode='rt') as file: - for line in file: - lineNum += 1 - if lineNum % 1e5 == 0: - print(f'At line {lineNum}') - # - if line[0] == '#': - continue - match = descLineRegex.fullmatch(line) - if match is None: - raise Exception(f'ERROR: Line {lineNum} has unexpected format') - dbCur.execute('INSERT INTO abstracts VALUES (?, ?)', - (match.group(1), match.group(2).replace(r'\"', '"'))) - -print('Closing database') -dbCon.commit() -dbCon.close() diff --git a/backend/tolData/enwiki/README.md b/backend/tolData/enwiki/README.md deleted file mode 100644 index 76f9ee5..0000000 --- a/backend/tolData/enwiki/README.md +++ /dev/null @@ -1,63 +0,0 @@ -This directory holds files obtained/derived from [English Wikipedia](https://en.wikipedia.org/wiki/Main_Page). - -# Downloaded Files -- enwiki-20220501-pages-articles-multistream.xml.bz2 <br> - Contains text content and metadata for pages in enwiki. - Obtained via <https://dumps.wikimedia.org/backup-index.html> (site suggests downloading from a mirror). - Some file content and format information was available from - <https://meta.wikimedia.org/wiki/Data_dumps/What%27s_available_for_download>. -- enwiki-20220501-pages-articles-multistream-index.txt.bz2 <br> - Obtained like above. Holds lines of the form offset1:pageId1:title1, - providing, for each page, an offset into the dump file of a chunk of - 100 pages that includes it. - -# Dump-Index Files -- genDumpIndexDb.py <br> - Creates a database version of the enwiki-dump index file. -- dumpIndex.db <br> - Generated by genDumpIndexDb.py. <br> - Tables: <br> - - `offsets`: `title TEXT PRIMARY KEY, id INT UNIQUE, offset INT, next_offset INT` - -# Description Database Files -- genDescData.py <br> - Reads through pages in the dump file, and adds short-description info to a database. -- descData.db <br> - Generated by genDescData.py. <br> - Tables: <br> - - `pages`: `id INT PRIMARY KEY, title TEXT UNIQUE` - - `redirects`: `id INT PRIMARY KEY, target TEXT` - - `descs`: `id INT PRIMARY KEY, desc TEXT` - -# Image Database Files -- genImgData.py <br> - Used to find infobox image names for page IDs, storing them into a database. -- downloadImgLicenseInfo.py <br> - Used to download licensing metadata for image names, via wikipedia's online API, storing them into a database. -- imgData.db <br> - Used to hold metadata about infobox images for a set of pageIDs. - Generated using getEnwikiImgData.py and downloadImgLicenseInfo.py. <br> - Tables: <br> - - `page_imgs`: `page_id INT PRIMAY KEY, img_name TEXT` <br> - `img_name` may be null, which means 'none found', and is used to avoid re-processing page-ids. - - `imgs`: `name TEXT PRIMARY KEY, license TEXT, artist TEXT, credit TEXT, restrictions TEXT, url TEXT` <br> - Might lack some matches for `img_name` in `page_imgs`, due to licensing info unavailability. -- downloadImgs.py <br> - Used to download image files into imgs/. - -# Page View Files -- pageviews/pageviews-*-user.bz2 - Each holds wikimedia article page view data for some month. - Obtained via <https://dumps.wikimedia.org/other/pageview_complete/monthly/>. - Some format info was available from <https://dumps.wikimedia.org/other/pageview_complete/readme.html>. -- genPageviewData.py <br> - Reads pageview/*, and creates a database holding average monthly pageview counts. -- pageviewData.db <br> - Generated using genPageviewData.py. <br> - Tables: <br> - - `views`: `title TEXT PRIMARY KEY, id INT, views INT` - -# Other Files -- lookupPage.py <br> - Running `lookupPage.py title1` looks in the dump for a page with a given title, - and prints the contents to stdout. Uses dumpIndex.db. diff --git a/backend/tolData/enwiki/downloadImgLicenseInfo.py b/backend/tolData/enwiki/downloadImgLicenseInfo.py deleted file mode 100755 index ba6317e..0000000 --- a/backend/tolData/enwiki/downloadImgLicenseInfo.py +++ /dev/null @@ -1,147 +0,0 @@ -#!/usr/bin/python3 - -import re -import sqlite3, urllib.parse, html -import requests -import time, signal - -import argparse -parser = argparse.ArgumentParser(description=""" -Reads image names from a database, and uses enwiki's online API to obtain -licensing information for them, adding the info to the database. - -SIGINT causes the program to finish an ongoing download and exit. -The program can be re-run to continue downloading, and looks -at already-processed names to decide what to skip. -""", formatter_class=argparse.RawDescriptionHelpFormatter) -parser.parse_args() - -imgDb = 'imgData.db' -apiUrl = 'https://en.wikipedia.org/w/api.php' -userAgent = 'terryt.dev (terry06890@gmail.com)' -batchSz = 50 # Max 50 -tagRegex = re.compile(r'<[^<]+>') -whitespaceRegex = re.compile(r'\s+') - -print('Opening database') -dbCon = sqlite3.connect(imgDb) -dbCur = dbCon.cursor() -dbCur2 = dbCon.cursor() -print('Checking for table') -if dbCur.execute('SELECT name FROM sqlite_master WHERE type="table" AND name="imgs"').fetchone() is None: - dbCur.execute('CREATE TABLE imgs(' \ - 'name TEXT PRIMARY KEY, license TEXT, artist TEXT, credit TEXT, restrictions TEXT, url TEXT)') - -print('Reading image names') -imgNames: set[str] = set() -for (imgName,) in dbCur.execute('SELECT DISTINCT img_name FROM page_imgs WHERE img_name NOT NULL'): - imgNames.add(imgName) -print(f'Found {len(imgNames)}') - -print('Checking for already-processed images') -oldSz = len(imgNames) -for (imgName,) in dbCur.execute('SELECT name FROM imgs'): - imgNames.discard(imgName) -print(f'Found {oldSz - len(imgNames)}') - -# Set SIGINT handler -interrupted = False -oldHandler = None -def onSigint(sig, frame): - global interrupted - interrupted = True - signal.signal(signal.SIGINT, oldHandler) -oldHandler = signal.signal(signal.SIGINT, onSigint) - -print('Iterating through image names') -imgNameList = list(imgNames) -iterNum = 0 -for i in range(0, len(imgNameList), batchSz): - iterNum += 1 - if iterNum % 1 == 0: - print(f'At iteration {iterNum} (after {(iterNum - 1) * batchSz} images)') - if interrupted: - print(f'Exiting loop at iteration {iterNum}') - break - # Get batch - imgBatch = imgNameList[i:i+batchSz] - imgBatch = ['File:' + x for x in imgBatch] - # Make request - headers = { - 'user-agent': userAgent, - 'accept-encoding': 'gzip', - } - params = { - 'action': 'query', - 'format': 'json', - 'prop': 'imageinfo', - 'iiprop': 'extmetadata|url', - 'maxlag': '5', - 'titles': '|'.join(imgBatch), - 'iiextmetadatafilter': 'Artist|Credit|LicenseShortName|Restrictions', - } - responseObj = None - try: - response = requests.get(apiUrl, params=params, headers=headers) - responseObj = response.json() - except Exception as e: - print(f'ERROR: Exception while downloading info: {e}') - print('\tImage batch: ' + '|'.join(imgBatch)) - continue - # Parse response-object - if 'query' not in responseObj or 'pages' not in responseObj['query']: - print('WARNING: Response object for doesn\'t have page data') - print('\tImage batch: ' + '|'.join(imgBatch)) - if 'error' in responseObj: - errorCode = responseObj['error']['code'] - print(f'\tError code: {errorCode}') - if errorCode == 'maxlag': - time.sleep(5) - continue - pages = responseObj['query']['pages'] - normalisedToInput: dict[str, str] = {} - if 'normalized' in responseObj['query']: - for entry in responseObj['query']['normalized']: - normalisedToInput[entry['to']] = entry['from'] - for _, page in pages.items(): - # Some fields // More info at https://www.mediawiki.org/wiki/Extension:CommonsMetadata#Returned_data - # LicenseShortName: short human-readable license name, apparently more reliable than 'License', - # Artist: author name (might contain complex html, multiple authors, etc) - # Credit: 'source' - # For image-map-like images, can be quite large/complex html, creditng each sub-image - # May be <a href='text1'>text2</a>, where the text2 might be non-indicative - # Restrictions: specifies non-copyright legal restrictions - title: str = page['title'] - if title in normalisedToInput: - title = normalisedToInput[title] - title = title[5:] # Remove 'File:' - if title not in imgNames: - print(f'WARNING: Got title "{title}" not in image-name list') - continue - if 'imageinfo' not in page: - print(f'WARNING: No imageinfo section for page "{title}"') - continue - metadata = page['imageinfo'][0]['extmetadata'] - url: str = page['imageinfo'][0]['url'] - license: str | None = metadata['LicenseShortName']['value'] if 'LicenseShortName' in metadata else None - artist: str | None = metadata['Artist']['value'] if 'Artist' in metadata else None - credit: str | None = metadata['Credit']['value'] if 'Credit' in metadata else None - restrictions: str | None = metadata['Restrictions']['value'] if 'Restrictions' in metadata else None - # Remove markup - if artist is not None: - artist = tagRegex.sub(' ', artist) - artist = whitespaceRegex.sub(' ', artist) - artist = html.unescape(artist) - artist = urllib.parse.unquote(artist) - if credit is not None: - credit = tagRegex.sub(' ', credit) - credit = whitespaceRegex.sub(' ', credit) - credit = html.unescape(credit) - credit = urllib.parse.unquote(credit) - # Add to db - dbCur2.execute('INSERT INTO imgs VALUES (?, ?, ?, ?, ?, ?)', - (title, license, artist, credit, restrictions, url)) - -print('Closing database') -dbCon.commit() -dbCon.close() diff --git a/backend/tolData/enwiki/downloadImgs.py b/backend/tolData/enwiki/downloadImgs.py deleted file mode 100755 index def4714..0000000 --- a/backend/tolData/enwiki/downloadImgs.py +++ /dev/null @@ -1,88 +0,0 @@ -#!/usr/bin/python3 - -import sys, re, os -import sqlite3 -import urllib.parse, requests -import time, signal - -import argparse -parser = argparse.ArgumentParser(description=""" -Downloads images from URLs in an image database, into an output directory, -with names of the form 'pageId1.ext1'. - -SIGINT causes the program to finish an ongoing download and exit. -The program can be re-run to continue downloading, and looks -in the output directory do decide what to skip. -""", formatter_class=argparse.RawDescriptionHelpFormatter) -parser.parse_args() - -imgDb = 'imgData.db' # About 130k image names -outDir = 'imgs' -licenseRegex = re.compile(r'cc0|cc([ -]by)?([ -]sa)?([ -][1234]\.[05])?( \w\w\w?)?', flags=re.IGNORECASE) -# In testing, this downloaded about 100k images, over several days - -if not os.path.exists(outDir): - os.mkdir(outDir) -print('Checking for already-downloaded images') -fileList = os.listdir(outDir) -pageIdsDone: set[int] = set() -for filename in fileList: - basename, extension = os.path.splitext(filename) - pageIdsDone.add(int(basename)) -print(f'Found {len(pageIdsDone)}') - -# Set SIGINT handler -interrupted = False -oldHandler = None -def onSigint(sig, frame): - global interrupted - interrupted = True - signal.signal(signal.SIGINT, oldHandler) -oldHandler = signal.signal(signal.SIGINT, onSigint) - -print('Opening database') -dbCon = sqlite3.connect(imgDb) -dbCur = dbCon.cursor() -print('Starting downloads') -iterNum = 0 -query = 'SELECT page_id, license, artist, credit, restrictions, url FROM' \ - ' imgs INNER JOIN page_imgs ON imgs.name = page_imgs.img_name' -for pageId, license, artist, credit, restrictions, url in dbCur.execute(query): - if pageId in pageIdsDone: - continue - if interrupted: - print('Exiting loop') - break - # Check for problematic attributes - if license is None or licenseRegex.fullmatch(license) is None: - continue - if artist is None or artist == '' or len(artist) > 100 or re.match(r'(\d\. )?File:', artist) is not None: - continue - if credit is None or len(credit) > 300 or re.match(r'File:', credit) is not None: - continue - if restrictions is not None and restrictions != '': - continue - # Download image - iterNum += 1 - print(f'Iteration {iterNum}: Downloading for page-id {pageId}') - urlParts = urllib.parse.urlparse(url) - extension = os.path.splitext(urlParts.path)[1] - if len(extension) <= 1: - print(f'WARNING: No filename extension found in URL {url}') - sys.exit(1) - outFile = f'{outDir}/{pageId}{extension}' - headers = { - 'user-agent': 'terryt.dev (terry06890@gmail.com)', - 'accept-encoding': 'gzip', - } - try: - response = requests.get(url, headers=headers) - with open(outFile, 'wb') as file: - file.write(response.content) - time.sleep(1) - # https://en.wikipedia.org/wiki/Wikipedia:Database_download says to 'throttle self to 1 cache miss per sec' - # It's unclear how to properly check for cache misses, so this just aims for 1 per sec - except Exception as e: - print(f'Error while downloading to {outFile}: {e}') -print('Closing database') -dbCon.close() diff --git a/backend/tolData/enwiki/genDescData.py b/backend/tolData/enwiki/genDescData.py deleted file mode 100755 index 1698f5c..0000000 --- a/backend/tolData/enwiki/genDescData.py +++ /dev/null @@ -1,124 +0,0 @@ -#!/usr/bin/python3 - -import sys, os, re -import bz2 -import html, mwxml, mwparserfromhell -import sqlite3 - -import argparse -parser = argparse.ArgumentParser(description=""" -Reads through the wiki dump, and attempts to parse short-descriptions, -and add them to a database -""", formatter_class=argparse.RawDescriptionHelpFormatter) -parser.parse_args() - -dumpFile = 'enwiki-20220501-pages-articles-multistream.xml.bz2' # Had about 22e6 pages -enwikiDb = 'descData.db' -# In testing, this script took over 10 hours to run, and generated about 5GB - -descLineRegex = re.compile('^ *[A-Z\'"]') -embeddedHtmlRegex = re.compile(r'<[^<]+/>|<!--[^<]+-->|<[^</]+>([^<]*|[^<]*<[^<]+>[^<]*)</[^<]+>|<[^<]+$') - # Recognises a self-closing HTML tag, a tag with 0 children, tag with 1 child with 0 children, or unclosed tag -convertTemplateRegex = re.compile(r'{{convert\|(\d[^|]*)\|(?:(to|-)\|(\d[^|]*)\|)?([a-z][^|}]*)[^}]*}}') -def convertTemplateReplace(match): - if match.group(2) is None: - return f'{match.group(1)} {match.group(4)}' - else: - return f'{match.group(1)} {match.group(2)} {match.group(3)} {match.group(4)}' -parensGroupRegex = re.compile(r' \([^()]*\)') -leftoverBraceRegex = re.compile(r'(?:{\||{{).*') - -def parseDesc(text: str) -> str | None: - # Find first matching line outside {{...}}, [[...]], and block-html-comment constructs, - # and then accumulate lines until a blank one. - # Some cases not accounted for include: disambiguation pages, abstracts with sentences split-across-lines, - # nested embedded html, 'content significant' embedded-html, markup not removable with mwparsefromhell, - lines: list[str] = [] - openBraceCount = 0 - openBracketCount = 0 - inComment = False - skip = False - for line in text.splitlines(): - line = line.strip() - if not lines: - if line: - if openBraceCount > 0 or line[0] == '{': - openBraceCount += line.count('{') - openBraceCount -= line.count('}') - skip = True - if openBracketCount > 0 or line[0] == '[': - openBracketCount += line.count('[') - openBracketCount -= line.count(']') - skip = True - if inComment or line.find('<!--') != -1: - if line.find('-->') != -1: - if inComment: - inComment = False - skip = True - else: - inComment = True - skip = True - if skip: - skip = False - continue - if line[-1] == ':': # Seems to help avoid disambiguation pages - return None - if descLineRegex.match(line) is not None: - lines.append(line) - else: - if not line: - return removeMarkup(' '.join(lines)) - lines.append(line) - if lines: - return removeMarkup(' '.join(lines)) - return None -def removeMarkup(content: str) -> str: - content = embeddedHtmlRegex.sub('', content) - content = convertTemplateRegex.sub(convertTemplateReplace, content) - content = mwparserfromhell.parse(content).strip_code() # Remove wikitext markup - content = parensGroupRegex.sub('', content) - content = leftoverBraceRegex.sub('', content) - return content -def convertTitle(title: str) -> str: - return html.unescape(title).replace('_', ' ') - -print('Creating database') -if os.path.exists(enwikiDb): - raise Exception(f'ERROR: Existing {enwikiDb}') -dbCon = sqlite3.connect(enwikiDb) -dbCur = dbCon.cursor() -dbCur.execute('CREATE TABLE pages (id INT PRIMARY KEY, title TEXT UNIQUE)') -dbCur.execute('CREATE INDEX pages_title_idx ON pages(title COLLATE NOCASE)') -dbCur.execute('CREATE TABLE redirects (id INT PRIMARY KEY, target TEXT)') -dbCur.execute('CREATE INDEX redirects_idx ON redirects(target)') -dbCur.execute('CREATE TABLE descs (id INT PRIMARY KEY, desc TEXT)') - -print('Iterating through dump file') -with bz2.open(dumpFile, mode='rt') as file: - dump = mwxml.Dump.from_file(file) - pageNum = 0 - for page in dump: - pageNum += 1 - if pageNum % 1e4 == 0: - print(f'At page {pageNum}') - if pageNum > 3e4: - break - # Parse page - if page.namespace == 0: - try: - dbCur.execute('INSERT INTO pages VALUES (?, ?)', (page.id, convertTitle(page.title))) - except sqlite3.IntegrityError as e: - # Accounts for certain pages that have the same title - print(f'Failed to add page with title "{page.title}": {e}', file=sys.stderr) - continue - if page.redirect is not None: - dbCur.execute('INSERT INTO redirects VALUES (?, ?)', (page.id, convertTitle(page.redirect))) - else: - revision = next(page) - desc = parseDesc(revision.text) - if desc is not None: - dbCur.execute('INSERT INTO descs VALUES (?, ?)', (page.id, desc)) - -print('Closing database') -dbCon.commit() -dbCon.close() diff --git a/backend/tolData/enwiki/genDumpIndexDb.py b/backend/tolData/enwiki/genDumpIndexDb.py deleted file mode 100755 index 3bd129f..0000000 --- a/backend/tolData/enwiki/genDumpIndexDb.py +++ /dev/null @@ -1,56 +0,0 @@ -#!/usr/bin/python3 - -import sys, os, re -import bz2 -import sqlite3 - -import argparse -parser = argparse.ArgumentParser(description=""" -Adds data from the wiki dump index-file into a database -""", formatter_class=argparse.RawDescriptionHelpFormatter) -parser.parse_args() - -indexFile = 'enwiki-20220501-pages-articles-multistream-index.txt.bz2' # Had about 22e6 lines -indexDb = 'dumpIndex.db' - -if os.path.exists(indexDb): - raise Exception(f'ERROR: Existing {indexDb}') -print('Creating database') -dbCon = sqlite3.connect(indexDb) -dbCur = dbCon.cursor() -dbCur.execute('CREATE TABLE offsets (title TEXT PRIMARY KEY, id INT UNIQUE, offset INT, next_offset INT)') - -print('Iterating through index file') -lineRegex = re.compile(r'([^:]+):([^:]+):(.*)') -lastOffset = 0 -lineNum = 0 -entriesToAdd: list[tuple[str, str]] = [] -with bz2.open(indexFile, mode='rt') as file: - for line in file: - lineNum += 1 - if lineNum % 1e5 == 0: - print(f'At line {lineNum}') - # - match = lineRegex.fullmatch(line.rstrip()) - assert match is not None - offsetStr, pageId, title = match.group(1,2,3) - offset = int(offsetStr) - if offset > lastOffset: - for t, p in entriesToAdd: - try: - dbCur.execute('INSERT INTO offsets VALUES (?, ?, ?, ?)', (t, int(p), lastOffset, offset)) - except sqlite3.IntegrityError as e: - # Accounts for certain entries in the file that have the same title - print(f'Failed on title "{t}": {e}', file=sys.stderr) - entriesToAdd = [] - lastOffset = offset - entriesToAdd.append((title, pageId)) -for title, pageId in entriesToAdd: - try: - dbCur.execute('INSERT INTO offsets VALUES (?, ?, ?, ?)', (title, int(pageId), lastOffset, -1)) - except sqlite3.IntegrityError as e: - print(f'Failed on title "{t}": {e}', file=sys.stderr) - -print('Closing database') -dbCon.commit() -dbCon.close() diff --git a/backend/tolData/enwiki/genImgData.py b/backend/tolData/enwiki/genImgData.py deleted file mode 100755 index 00140f6..0000000 --- a/backend/tolData/enwiki/genImgData.py +++ /dev/null @@ -1,186 +0,0 @@ -#!/usr/bin/python3 - -import re -import bz2, html, urllib.parse -import sqlite3 - -import argparse -parser = argparse.ArgumentParser(description=""" -For some set of page IDs, looks up their content in the wiki dump, -and tries to parse infobox image names, storing them into a database. - -The program can be re-run with an updated set of page IDs, and -will skip already-processed page IDs. -""", formatter_class=argparse.RawDescriptionHelpFormatter) -parser.parse_args() - -def getInputPageIds(): - pageIds: set[int] = set() - dbCon = sqlite3.connect('../data.db') - dbCur = dbCon.cursor() - for (pageId,) in dbCur.execute('SELECT id from wiki_ids'): - pageIds.add(pageId) - dbCon.close() - return pageIds -dumpFile = 'enwiki-20220501-pages-articles-multistream.xml.bz2' -indexDb = 'dumpIndex.db' -imgDb = 'imgData.db' # The database to create -idLineRegex = re.compile(r'<id>(.*)</id>') -imageLineRegex = re.compile(r'.*\| *image *= *([^|]*)') -bracketImageRegex = re.compile(r'\[\[(File:[^|]*).*]]') -imageNameRegex = re.compile(r'.*\.(jpg|jpeg|png|gif|tiff|tif)', flags=re.IGNORECASE) -cssImgCropRegex = re.compile(r'{{css image crop\|image *= *(.*)', flags=re.IGNORECASE) - -print('Getting input page-ids') -pageIds = getInputPageIds() -print(f'Found {len(pageIds)}') - -print('Opening databases') -indexDbCon = sqlite3.connect(indexDb) -indexDbCur = indexDbCon.cursor() -imgDbCon = sqlite3.connect(imgDb) -imgDbCur = imgDbCon.cursor() -print('Checking tables') -if imgDbCur.execute('SELECT name FROM sqlite_master WHERE type="table" AND name="page_imgs"').fetchone() is None: - # Create tables if not present - imgDbCur.execute('CREATE TABLE page_imgs (page_id INT PRIMARY KEY, img_name TEXT)') # img_name may be NULL - imgDbCur.execute('CREATE INDEX page_imgs_idx ON page_imgs(img_name)') -else: - # Check for already-processed page IDs - numSkipped = 0 - for (pid,) in imgDbCur.execute('SELECT page_id FROM page_imgs'): - if pid in pageIds: - pageIds.remove(pid) - numSkipped += 1 - else: - print(f'WARNING: Found already-processed page ID {pid} which was not in input set') - print(f'Will skip {numSkipped} already-processed page IDs') - -print('Getting dump-file offsets') -offsetToPageids: dict[int, list[int]] = {} -offsetToEnd: dict[int, int] = {} # Maps chunk-start offsets to their chunk-end offsets -iterNum = 0 -for pageId in pageIds: - iterNum += 1 - if iterNum % 1e4 == 0: - print(f'At iteration {iterNum}') - # - query = 'SELECT offset, next_offset FROM offsets WHERE id = ?' - row: tuple[int, int] | None = indexDbCur.execute(query, (pageId,)).fetchone() - if row is None: - print(f'WARNING: Page ID {pageId} not found') - continue - chunkOffset, endOffset = row - offsetToEnd[chunkOffset] = endOffset - if chunkOffset not in offsetToPageids: - offsetToPageids[chunkOffset] = [] - offsetToPageids[chunkOffset].append(pageId) -print(f'Found {len(offsetToEnd)} chunks to check') - -print('Iterating through chunks in dump file') -def getImageName(content: list[str]) -> str | None: - """ Given an array of text-content lines, tries to return an infoxbox image name, or None """ - # Doesn't try and find images in outside-infobox [[File:...]] and <imagemap> sections - for line in content: - match = imageLineRegex.match(line) - if match is not None: - imageName = match.group(1).strip() - if imageName == '': - return None - imageName = html.unescape(imageName) - # Account for {{... - if imageName.startswith('{'): - match = cssImgCropRegex.match(imageName) - if match is None: - return None - imageName = match.group(1) - # Account for [[File:...|...]] - if imageName.startswith('['): - match = bracketImageRegex.match(imageName) - if match is None: - return None - imageName = match.group(1) - # Account for <!-- - if imageName.find('<!--') != -1: - return None - # Remove an initial 'File:' - if imageName.startswith('File:'): - imageName = imageName[5:] - # Remove an initial 'Image:' - if imageName.startswith('Image:'): - imageName = imageName[6:] - # Check for extension - match = imageNameRegex.match(imageName) - if match is not None: - imageName = match.group(0) - imageName = urllib.parse.unquote(imageName) - imageName = html.unescape(imageName) # Intentionally unescaping again (handles some odd cases) - imageName = imageName.replace('_', ' ') - return imageName - # Exclude lines like: | image = <imagemap> - return None - return None -with open(dumpFile, mode='rb') as file: - iterNum = 0 - for pageOffset, endOffset in offsetToEnd.items(): - iterNum += 1 - if iterNum % 100 == 0: - print(f'At iteration {iterNum}') - # - pageIds = offsetToPageids[pageOffset] - # Jump to chunk - file.seek(pageOffset) - compressedData = file.read(None if endOffset == -1 else endOffset - pageOffset) - data = bz2.BZ2Decompressor().decompress(compressedData).decode() - # Look in chunk for pages - lines = data.splitlines() - lineIdx = 0 - while lineIdx < len(lines): - # Look for <page> - if lines[lineIdx].lstrip() != '<page>': - lineIdx += 1 - continue - # Check page id - lineIdx += 3 - idLine = lines[lineIdx].lstrip() - match = idLineRegex.fullmatch(idLine) - if match is None or int(match.group(1)) not in pageIds: - lineIdx += 1 - continue - pageId = int(match.group(1)) - lineIdx += 1 - # Look for <text> in <page> - foundText = False - while lineIdx < len(lines): - if not lines[lineIdx].lstrip().startswith('<text '): - lineIdx += 1 - continue - foundText = True - # Get text content - content: list[str] = [] - line = lines[lineIdx] - content.append(line[line.find('>') + 1:]) - lineIdx += 1 - foundTextEnd = False - while lineIdx < len(lines): - line = lines[lineIdx] - if not line.endswith('</text>'): - content.append(line) - lineIdx += 1 - continue - foundTextEnd = True - content.append(line[:line.rfind('</text>')]) - # Look for image-filename - imageName = getImageName(content) - imgDbCur.execute('INSERT into page_imgs VALUES (?, ?)', (pageId, imageName)) - break - if not foundTextEnd: - print(f'WARNING: Did not find </text> for page id {pageId}') - break - if not foundText: - print(f'WARNING: Did not find <text> for page id {pageId}') - -print('Closing databases') -indexDbCon.close() -imgDbCon.commit() -imgDbCon.close() diff --git a/backend/tolData/enwiki/genPageviewData.py b/backend/tolData/enwiki/genPageviewData.py deleted file mode 100755 index 6a5d79c..0000000 --- a/backend/tolData/enwiki/genPageviewData.py +++ /dev/null @@ -1,62 +0,0 @@ -#!/usr/bin/python3 - -import sys, os, glob, math, re -from collections import defaultdict -import bz2, sqlite3 - -import argparse -parser = argparse.ArgumentParser(description=""" -Reads through wikimedia files containing pageview counts, -computes average counts, and adds them to a database -""", formatter_class=argparse.RawDescriptionHelpFormatter) -args = parser.parse_args() - -pageviewFiles = glob.glob('./pageviews/pageviews-*-user.bz2') -dbFile = 'pageviewData.db' -dumpIndexDb = 'dumpIndex.db' - -# Took about 15min per file (each about 180e6 lines) - -if os.path.exists(dbFile): - print('ERROR: Database already exists') - sys.exit(1) - -# Each pageview file has lines that seem to hold these space-separated fields: - # wiki code (eg: en.wikipedia), article title, page ID (may be: null), - # platform (eg: mobile-web), monthly view count, - # hourly count string (eg: A1B2 means 1 view on day 1 and 2 views on day 2) -namespaceRegex = re.compile(r'[a-zA-Z]+:') -titleToViews: dict[str, int] = defaultdict(int) -linePrefix = b'en.wikipedia ' -for filename in pageviewFiles: - print(f'Reading from {filename}') - with bz2.open(filename, 'rb') as file: - for lineNum, line in enumerate(file, 1): - if lineNum % 1e6 == 0: - print(f'At line {lineNum}') - if not line.startswith(linePrefix): - continue - # Get second and second-last fields - line = line[len(linePrefix):line.rfind(b' ')] # Remove first and last fields - title = line[:line.find(b' ')].decode('utf-8') - viewCount = int(line[line.rfind(b' ')+1:]) - if namespaceRegex.match(title) is not None: - continue - # Update map - titleToViews[title] += viewCount -print(f'Found {len(titleToViews)} titles') - -print('Writing to db') -dbCon = sqlite3.connect(dbFile) -dbCur = dbCon.cursor() -idbCon = sqlite3.connect(dumpIndexDb) -idbCur = idbCon.cursor() -dbCur.execute('CREATE TABLE views (title TEXT PRIMARY KEY, id INT, views INT)') -for title, views in titleToViews.items(): - row = idbCur.execute('SELECT id FROM offsets WHERE title = ?', (title,)).fetchone() - if row is not None: - wikiId = int(row[0]) - dbCur.execute('INSERT INTO views VALUES (?, ?, ?)', (title, wikiId, math.floor(views / len(pageviewFiles)))) -dbCon.commit() -dbCon.close() -idbCon.close() diff --git a/backend/tolData/enwiki/lookupPage.py b/backend/tolData/enwiki/lookupPage.py deleted file mode 100755 index 427aa7a..0000000 --- a/backend/tolData/enwiki/lookupPage.py +++ /dev/null @@ -1,66 +0,0 @@ -#!/usr/bin/python3 - -import sys -import bz2 -import sqlite3 - -import argparse -parser = argparse.ArgumentParser(description=""" -Looks up a page with title title1 in the wiki dump, using the dump-index -db, and prints the corresponding <page>. -""", formatter_class=argparse.RawDescriptionHelpFormatter) -parser.add_argument("title", help="The title to look up") -args = parser.parse_args() - -dumpFile = 'enwiki-20220501-pages-articles-multistream.xml.bz2' -indexDb = 'dumpIndex.db' -pageTitle = args.title.replace('_', ' ') - -print('Looking up offset in index db') -dbCon = sqlite3.connect(indexDb) -dbCur = dbCon.cursor() -query = 'SELECT title, offset, next_offset FROM offsets WHERE title = ?' -row = dbCur.execute(query, (pageTitle,)).fetchone() -if row is None: - print('Title not found') - sys.exit(0) -_, pageOffset, endOffset = row -dbCon.close() -print(f'Found chunk at offset {pageOffset}') - -print('Reading from wiki dump') -content: list[str] = [] -with open(dumpFile, mode='rb') as file: - # Get uncompressed chunk - file.seek(pageOffset) - compressedData = file.read(None if endOffset == -1 else endOffset - pageOffset) - data = bz2.BZ2Decompressor().decompress(compressedData).decode() - # Look in chunk for page - lines = data.splitlines() - lineIdx = 0 - found = False - pageNum = 0 - while not found: - line = lines[lineIdx] - if line.lstrip() == '<page>': - pageNum += 1 - if pageNum > 100: - print('ERROR: Did not find title after 100 pages') - break - lineIdx += 1 - titleLine = lines[lineIdx] - if titleLine.lstrip() == '<title>' + pageTitle + '</title>': - found = True - print(f'Found title in chunk as page {pageNum}') - content.append(line) - content.append(titleLine) - while True: - lineIdx += 1 - line = lines[lineIdx] - content.append(line) - if line.lstrip() == '</page>': - break - lineIdx += 1 - -print('Content: ') -print('\n'.join(content)) diff --git a/backend/tolData/eol/README.md b/backend/tolData/eol/README.md deleted file mode 100644 index c07b48e..0000000 --- a/backend/tolData/eol/README.md +++ /dev/null @@ -1,31 +0,0 @@ -This directory holds files obtained via the [Encyclopedia of Life](https://eol.org/). - -# Mapping Files -- `provider_ids.csv.gz` <br> - Obtained from <https://opendata.eol.org/dataset/identifier-map> on 22/08/22 (says last updated 27/07/22). - Associates EOL IDs with taxon IDs from sources like NCBI and Index Fungorium. - -# Name Data Files -- vernacularNames.csv <br> - Obtained from <https://opendata.eol.org/dataset/vernacular-names> on 24/04/2022 (last updated on 27/10/2020). - Contains alternative-node-names data from EOL. - -# Image Metadata Files -- imagesList.tgz <br> - Obtained from <https://opendata.eol.org/dataset/images-list> on 24/04/2022 (last updated on 05/02/2020). - Contains metadata for images from EOL. -- imagesList/ <br> - Extracted from imagesList.tgz. -- genImagesListDb.py <br> - Creates a database, and imports imagesList/*.csv files into it. -- imagesList.db <br> - Created by running genImagesListDb.py <br> - Tables: <br> - - `images`: - `content_id INT PRIMARY KEY, page_id INT, source_url TEXT, copy_url TEXT, license TEXT, copyright_owner TEXT` - -# Image Generation Files -- downloadImgs.py <br> - Used to download image files into imgsForReview/. -- reviewImgs.py <br> - Used to review images in imgsForReview/, moving acceptable ones into imgs/. diff --git a/backend/tolData/eol/downloadImgs.py b/backend/tolData/eol/downloadImgs.py deleted file mode 100755 index 5213aaf..0000000 --- a/backend/tolData/eol/downloadImgs.py +++ /dev/null @@ -1,142 +0,0 @@ -#!/usr/bin/python3 - -import sys, re, os, random -import sqlite3 -import urllib.parse, requests -import time -from threading import Thread -import signal - -import argparse -parser = argparse.ArgumentParser(description=""" -For some set of EOL IDs, downloads associated images from URLs in -an image-list database. Uses multiple downloading threads. - -May obtain multiple images per ID. The images will get names -with the form 'eolId1 contentId1.ext1'. - -SIGINT causes the program to finish ongoing downloads and exit. -The program can be re-run to continue downloading. It looks for -already-downloaded files, and continues after the one with -highest EOL ID. -""", formatter_class=argparse.RawDescriptionHelpFormatter) -parser.parse_args() - -imagesListDb = 'imagesList.db' -def getInputEolIds() -> set[int]: - eolIds: set[int] = set() - dbCon = sqlite3.connect('../data.db') - dbCur = dbCon.cursor() - for (id,) in dbCur.execute('SELECT id FROM eol_ids'): - eolIds.add(id) - dbCon.close() - return eolIds -outDir = 'imgsForReview/' -MAX_IMGS_PER_ID = 3 -MAX_THREADS = 5 -POST_DL_DELAY_MIN = 2 # Minimum delay in seconds to pause after download before starting another (for each thread) -POST_DL_DELAY_MAX = 3 -LICENSE_REGEX = r'cc-by((-nc)?(-sa)?(-[234]\.[05])?)|cc-publicdomain|cc-0-1\.0|public domain' - -print('Getting input EOL IDs') -eolIds = getInputEolIds() -print('Getting EOL IDs to download for') -# Get IDs from images-list db -imgDbCon = sqlite3.connect(imagesListDb) -imgCur = imgDbCon.cursor() -imgListIds: set[int] = set() -for (pageId,) in imgCur.execute('SELECT DISTINCT page_id FROM images'): - imgListIds.add(pageId) -# Get set intersection, and sort into list -eolIds = eolIds.intersection(imgListIds) -eolIdList = sorted(eolIds) -print(f'Result: {len(eolIdList)} EOL IDs') - -print('Checking output directory') -if not os.path.exists(outDir): - os.mkdir(outDir) -print('Finding next ID to download for') -nextIdx = 0 -fileList = os.listdir(outDir) -ids = [int(filename.split(' ')[0]) for filename in fileList] -if ids: - ids.sort() - nextIdx = eolIdList.index(ids[-1]) + 1 -if nextIdx == len(eolIdList): - print('No IDs left. Exiting...') - sys.exit(0) - -print('Starting download threads') -numThreads = 0 -threadException: Exception | None = None # Used for ending main thread after a non-main thread exception -# Handle SIGINT signals -interrupted = False -oldHandler = None -def onSigint(sig, frame): - global interrupted - interrupted = True - signal.signal(signal.SIGINT, oldHandler) -oldHandler = signal.signal(signal.SIGINT, onSigint) -# Function for threads to execute -def downloadImg(url, outFile): - global numThreads, threadException - try: - data = requests.get(url) - with open(outFile, 'wb') as file: - file.write(data.content) - time.sleep(random.random() * (POST_DL_DELAY_MAX - POST_DL_DELAY_MIN) + POST_DL_DELAY_MIN) - except Exception as e: - print(f'Error while downloading to {outFile}: {str(e)}', file=sys.stderr) - threadException = e - numThreads -= 1 -# Manage downloading -for idx in range(nextIdx, len(eolIdList)): - eolId = eolIdList[idx] - # Get image urls - ownerSet: set[str] = set() # Used to get images from different owners, for variety - exitLoop = False - query = 'SELECT content_id, copy_url, license, copyright_owner FROM images WHERE page_id = ?' - for contentId, url, license, copyrightOwner in imgCur.execute(query, (eolId,)): - if url.startswith('data/'): - url = 'https://content.eol.org/' + url - urlParts = urllib.parse.urlparse(url) - extension = os.path.splitext(urlParts.path)[1] - if len(extension) <= 1: - print(f'WARNING: No filename extension found in URL {url}', file=sys.stderr) - continue - # Check image-quantity limit - if len(ownerSet) == MAX_IMGS_PER_ID: - break - # Check for skip conditions - if re.fullmatch(LICENSE_REGEX, license) is None: - continue - if len(copyrightOwner) > 100: # Avoid certain copyrightOwner fields that seem long and problematic - continue - if copyrightOwner in ownerSet: - continue - ownerSet.add(copyrightOwner) - # Determine output filename - outPath = f'{outDir}{eolId} {contentId}{extension}' - if os.path.exists(outPath): - print(f'WARNING: {outPath} already exists. Skipping download.') - continue - # Check thread limit - while numThreads == MAX_THREADS: - time.sleep(1) - # Wait for threads after an interrupt or thread-exception - if interrupted or threadException is not None: - print('Waiting for existing threads to end') - while numThreads > 0: - time.sleep(1) - exitLoop = True - break - # Perform download - print(f'Downloading image to {outPath}') - numThreads += 1 - thread = Thread(target=downloadImg, args=(url, outPath), daemon=True) - thread.start() - if exitLoop: - break -# Close images-list db -print('Finished downloading') -imgDbCon.close() diff --git a/backend/tolData/eol/genImagesListDb.py b/backend/tolData/eol/genImagesListDb.py deleted file mode 100755 index 808292d..0000000 --- a/backend/tolData/eol/genImagesListDb.py +++ /dev/null @@ -1,34 +0,0 @@ -#!/usr/bin/python3 - -import os, re -import csv -import sqlite3 - -import argparse -parser = argparse.ArgumentParser(description=""" -Generates a sqlite db from a directory of CSV files holding EOL image data -""", formatter_class=argparse.RawDescriptionHelpFormatter) -parser.parse_args() - -imagesListDir = 'imagesList/' -dbFile = 'imagesList.db' - -print('Creating database') -dbCon = sqlite3.connect(dbFile) -dbCur = dbCon.cursor() -dbCur.execute('CREATE TABLE images' \ - ' (content_id INT PRIMARY KEY, page_id INT, source_url TEXT, copy_url TEXT, license TEXT, copyright_owner TEXT)') -dbCur.execute('CREATE INDEX images_pid_idx ON images(page_id)') -print('Reading CSV files') -csvFilenames = os.listdir(imagesListDir) -for filename in csvFilenames: - print(f'Processing {imagesListDir}{filename}') - with open(imagesListDir + filename, newline='') as file: - for contentId, pageId, sourceUrl, copyUrl, license, owner in csv.reader(file): - if re.match(r'^[a-zA-Z]', contentId): # Skip header line - continue - dbCur.execute('INSERT INTO images VALUES (?, ?, ?, ?, ?, ?)', - (int(contentId), int(pageId), sourceUrl, copyUrl, license, owner)) -print('Closing database') -dbCon.commit() -dbCon.close() diff --git a/backend/tolData/eol/reviewImgs.py b/backend/tolData/eol/reviewImgs.py deleted file mode 100755 index e44fb3d..0000000 --- a/backend/tolData/eol/reviewImgs.py +++ /dev/null @@ -1,202 +0,0 @@ -#!/usr/bin/python3 - -import sys, re, os, time -import sqlite3 -import tkinter as tki -from tkinter import ttk -import PIL -from PIL import ImageTk, Image, ImageOps - -import argparse -parser = argparse.ArgumentParser(description=""" -Provides a GUI for reviewing images. Looks in a for-review directory for -images named 'eolId1 contentId1.ext1', and, for each EOL ID, enables the user to -choose an image to keep, or reject all. Also provides image rotation. -Chosen images are placed in another directory, and rejected ones are deleted. -""", formatter_class=argparse.RawDescriptionHelpFormatter) -parser.parse_args() - -imgDir = 'imgsForReview/' -outDir = 'imgs/' -extraInfoDbCon = sqlite3.connect('../data.db') -extraInfoDbCur = extraInfoDbCon.cursor() -def getExtraInfo(eolId: int) -> str: - global extraInfoDbCur - query = 'SELECT names.alt_name FROM' \ - ' names INNER JOIN eol_ids ON eol_ids.name = names.name' \ - ' WHERE id = ? and pref_alt = 1' - row = extraInfoDbCur.execute(query, (eolId,)).fetchone() - if row is not None: - return f'Reviewing EOL ID {eolId}, aka "{row[0]}"' - else: - return f'Reviewing EOL ID {eolId}' -IMG_DISPLAY_SZ = 400 -MAX_IMGS_PER_ID = 3 -IMG_BG_COLOR = (88, 28, 135) -PLACEHOLDER_IMG = Image.new('RGB', (IMG_DISPLAY_SZ, IMG_DISPLAY_SZ), IMG_BG_COLOR) - -print('Checking output directory') -if not os.path.exists(outDir): - os.mkdir(outDir) -print('Getting input image list') -imgList = os.listdir(imgDir) -imgList.sort(key=lambda s: int(s.split(' ')[0])) -if not imgList: - print('No input images found') - sys.exit(0) - -class EolImgReviewer: - """ Provides the GUI for reviewing images """ - def __init__(self, root, imgList): - self.root = root - root.title('EOL Image Reviewer') - # Setup main frame - mainFrame = ttk.Frame(root, padding='5 5 5 5') - mainFrame.grid(column=0, row=0, sticky=(tki.N, tki.W, tki.E, tki.S)) - root.columnconfigure(0, weight=1) - root.rowconfigure(0, weight=1) - # Set up images-to-be-reviewed frames - self.imgs = [PLACEHOLDER_IMG] * MAX_IMGS_PER_ID # Stored as fields for use in rotation - self.photoImgs = list(map(lambda img: ImageTk.PhotoImage(img), self.imgs)) # Image objects usable by tkinter - # These need a persistent reference for some reason (doesn't display otherwise) - self.labels: list[ttk.Label] = [] - for i in range(MAX_IMGS_PER_ID): - frame = ttk.Frame(mainFrame, width=IMG_DISPLAY_SZ, height=IMG_DISPLAY_SZ) - frame.grid(column=i, row=0) - label = ttk.Label(frame, image=self.photoImgs[i]) - label.grid(column=0, row=0) - self.labels.append(label) - # Add padding - for child in mainFrame.winfo_children(): - child.grid_configure(padx=5, pady=5) - # Add keyboard bindings - root.bind('<q>', self.quit) - root.bind('<Key-j>', lambda evt: self.accept(0)) - root.bind('<Key-k>', lambda evt: self.accept(1)) - root.bind('<Key-l>', lambda evt: self.accept(2)) - root.bind('<Key-i>', lambda evt: self.reject()) - root.bind('<Key-a>', lambda evt: self.rotate(0)) - root.bind('<Key-s>', lambda evt: self.rotate(1)) - root.bind('<Key-d>', lambda evt: self.rotate(2)) - root.bind('<Key-A>', lambda evt: self.rotate(0, True)) - root.bind('<Key-S>', lambda evt: self.rotate(1, True)) - root.bind('<Key-D>', lambda evt: self.rotate(2, True)) - # Initialise images to review - self.imgList = imgList - self.imgListIdx = 0 - self.nextEolId = 0 - self.nextImgNames: list[str] = [] - self.rotations: list[int] = [] - self.getNextImgs() - # For displaying extra info - self.numReviewed = 0 - self.startTime = time.time() - def getNextImgs(self): - """ Updates display with new images to review, or ends program """ - # Gather names of next images to review - for i in range(MAX_IMGS_PER_ID): - if self.imgListIdx == len(self.imgList): - if i == 0: - self.quit() - return - break - imgName = self.imgList[self.imgListIdx] - eolId = int(re.match(r'(\d+) (\d+)', imgName).group(1)) - if i == 0: - self.nextEolId = eolId - self.nextImgNames = [imgName] - self.rotations = [0] - else: - if self.nextEolId != eolId: - break - self.nextImgNames.append(imgName) - self.rotations.append(0) - self.imgListIdx += 1 - # Update displayed images - idx = 0 - while idx < MAX_IMGS_PER_ID: - if idx < len(self.nextImgNames): - try: - img = Image.open(imgDir + self.nextImgNames[idx]) - img = ImageOps.exif_transpose(img) - except PIL.UnidentifiedImageError: - os.remove(imgDir + self.nextImgNames[idx]) - del self.nextImgNames[idx] - del self.rotations[idx] - continue - self.imgs[idx] = self.resizeImgForDisplay(img) - else: - self.imgs[idx] = PLACEHOLDER_IMG - self.photoImgs[idx] = ImageTk.PhotoImage(self.imgs[idx]) - self.labels[idx].config(image=self.photoImgs[idx]) - idx += 1 - # Restart if all image files non-recognisable - if not self.nextImgNames: - self.getNextImgs() - return - # Update title - firstImgIdx = self.imgListIdx - len(self.nextImgNames) + 1 - lastImgIdx = self.imgListIdx - title = getExtraInfo(self.nextEolId) - title += f' (imgs {firstImgIdx} to {lastImgIdx} out of {len(self.imgList)})' - self.root.title(title) - def accept(self, imgIdx): - """ React to a user selecting an image """ - if imgIdx >= len(self.nextImgNames): - print('Invalid selection') - return - for i in range(len(self.nextImgNames)): - inFile = imgDir + self.nextImgNames[i] - if i == imgIdx: # Move accepted image, rotating if needed - outFile = outDir + self.nextImgNames[i] - img = Image.open(inFile) - img = ImageOps.exif_transpose(img) - if self.rotations[i] != 0: - img = img.rotate(self.rotations[i], expand=True) - img.save(outFile) - os.remove(inFile) - else: # Delete non-accepted image - os.remove(inFile) - self.numReviewed += 1 - self.getNextImgs() - def reject(self): - """ React to a user rejecting all images of a set """ - for i in range(len(self.nextImgNames)): - os.remove(imgDir + self.nextImgNames[i]) - self.numReviewed += 1 - self.getNextImgs() - def rotate(self, imgIdx, anticlockwise = False): - """ Respond to a user rotating an image """ - deg = -90 if not anticlockwise else 90 - self.imgs[imgIdx] = self.imgs[imgIdx].rotate(deg) - self.photoImgs[imgIdx] = ImageTk.PhotoImage(self.imgs[imgIdx]) - self.labels[imgIdx].config(image=self.photoImgs[imgIdx]) - self.rotations[imgIdx] = (self.rotations[imgIdx] + deg) % 360 - def quit(self, e = None): - global extraInfoDbCon - print(f'Number reviewed: {self.numReviewed}') - timeElapsed = time.time() - self.startTime - print(f'Time elapsed: {timeElapsed:.2f} seconds') - if self.numReviewed > 0: - print(f'Avg time per review: {timeElapsed/self.numReviewed:.2f} seconds') - extraInfoDbCon.close() - self.root.destroy() - def resizeImgForDisplay(self, img): - """ Returns a copy of an image, shrunk to fit in it's frame (keeps aspect ratio), and with a background """ - if max(img.width, img.height) > IMG_DISPLAY_SZ: - if (img.width > img.height): - newHeight = int(img.height * IMG_DISPLAY_SZ/img.width) - img = img.resize((IMG_DISPLAY_SZ, newHeight)) - else: - newWidth = int(img.width * IMG_DISPLAY_SZ / img.height) - img = img.resize((newWidth, IMG_DISPLAY_SZ)) - bgImg = PLACEHOLDER_IMG.copy() - bgImg.paste(img, box=( - int((IMG_DISPLAY_SZ - img.width) / 2), - int((IMG_DISPLAY_SZ - img.height) / 2))) - return bgImg -# Create GUI and defer control -print('Starting GUI') -root = tki.Tk() -EolImgReviewer(root, imgList) -root.mainloop() diff --git a/backend/tolData/genDescData.py b/backend/tolData/genDescData.py deleted file mode 100755 index bb1cbc8..0000000 --- a/backend/tolData/genDescData.py +++ /dev/null @@ -1,90 +0,0 @@ -#!/usr/bin/python3 - -import sqlite3 - -import argparse -parser = argparse.ArgumentParser(description=""" -Maps nodes to short descriptions, using data from DBpedia and -Wikipedia, and stores results in the database. -""", formatter_class=argparse.RawDescriptionHelpFormatter) -args = parser.parse_args() - -dbpediaDb = 'dbpedia/descData.db' -enwikiDb = 'enwiki/descData.db' -dbFile = 'data.db' - -print('Creating table') -dbCon = sqlite3.connect(dbFile) -dbCur = dbCon.cursor() -dbCur.execute('CREATE TABLE descs (wiki_id INT PRIMARY KEY, desc TEXT, from_dbp INT)') - -print('Getting node mappings') -nodeToWikiId: dict[str, int] = {} -for name, wikiId in dbCur.execute('SELECT name, id from wiki_ids'): - nodeToWikiId[name] = wikiId - -print('Reading data from DBpedia') -dbpCon = sqlite3.connect(dbpediaDb) -dbpCur = dbpCon.cursor() -print('Getting node IRIs') -nodeToIri: dict[str, str] = {} -iterNum = 0 -for name, wikiId in nodeToWikiId.items(): - iterNum += 1 - if iterNum % 1e5 == 0: - print(f'At iteration {iterNum}') - # - row = dbpCur.execute('SELECT iri FROM ids where id = ?', (wikiId,)).fetchone() - if row is not None: - nodeToIri[name] = row[0] -print('Resolving redirects') -iterNum = 0 -for name, iri in nodeToIri.items(): - iterNum += 1 - if iterNum % 1e5 == 0: - print(f'At iteration {iterNum}') - # - row = dbpCur.execute('SELECT target FROM redirects where iri = ?', (iri,)).fetchone() - if row is not None: - nodeToIri[name] = row[0] -print('Adding descriptions') -iterNum = 0 -for name, iri in nodeToIri.items(): - iterNum += 1 - if iterNum % 1e4 == 0: - print(f'At iteration {iterNum}') - # - row = dbpCur.execute('SELECT abstract FROM abstracts WHERE iri = ?', (iri,)).fetchone() - if row is not None: - dbCur.execute('INSERT OR IGNORE INTO descs VALUES (?, ?, ?)', (nodeToWikiId[name], row[0], 1)) - del nodeToWikiId[name] -dbpCon.close() - -print('Reading data from Wikipedia') -enwikiCon = sqlite3.connect(enwikiDb) -enwikiCur = enwikiCon.cursor() -print('Resolving redirects') -iterNum = 0 -for name, wikiId in nodeToWikiId.items(): - iterNum += 1 - if iterNum % 1e4 == 0: - print(f'At iteration {iterNum}') - # - query = 'SELECT pages.id FROM redirects INNER JOIN pages ON redirects.target = pages.title WHERE redirects.id = ?' - row = enwikiCur.execute(query, (wikiId,)).fetchone() - if row is not None: - nodeToWikiId[name] = row[0] -print('Adding descriptions') -iterNum = 0 -for name, wikiId in nodeToWikiId.items(): - iterNum += 1 - if iterNum % 1e3 == 0: - print(f'At iteration {iterNum}') - # - row = enwikiCur.execute('SELECT desc FROM descs where id = ?', (wikiId,)).fetchone() - if row is not None: - dbCur.execute('INSERT OR IGNORE INTO descs VALUES (?, ?, ?)', (wikiId, row[0], 0)) - -print('Closing databases') -dbCon.commit() -dbCon.close() diff --git a/backend/tolData/genImgs.py b/backend/tolData/genImgs.py deleted file mode 100755 index 6f72b49..0000000 --- a/backend/tolData/genImgs.py +++ /dev/null @@ -1,196 +0,0 @@ -#!/usr/bin/python3 - -import sys, os, subprocess -import sqlite3, urllib.parse -import signal - -import argparse -parser = argparse.ArgumentParser(description=""" -Reads node IDs and image paths from a file, and possibly from a directory, -and generates cropped/resized versions of those images into a directory, -with names of the form 'nodeId1.jpg'. Also adds image metadata to the -database. - -SIGINT can be used to stop, and the program can be re-run to continue -processing. It uses already-existing database entries to decide what -to skip. -""", formatter_class=argparse.RawDescriptionHelpFormatter) -parser.parse_args() - -imgListFile = 'imgList.txt' -outDir = 'img/' -eolImgDb = 'eol/imagesList.db' -enwikiImgDb = 'enwiki/imgData.db' -pickedImgsDir = 'pickedImgs/' -pickedImgsFilename = 'imgData.txt' -dbFile = 'data.db' -IMG_OUT_SZ = 200 -genImgFiles = True # Usable for debugging - -class PickedImg: - """ Represents a picked-image from pickedImgsDir """ - def __init__(self, nodeName: str, id: int, filename: str, url: str, license: str, artist: str, credit: str): - self.nodeName = nodeName - self.id = id - self.filename = filename - self.url = url - self.license = license - self.artist = artist - self.credit = credit - -if not os.path.exists(outDir): - os.mkdir(outDir) - -print('Opening databases') -dbCon = sqlite3.connect(dbFile) -dbCur = dbCon.cursor() -eolCon = sqlite3.connect(eolImgDb) -eolCur = eolCon.cursor() -enwikiCon = sqlite3.connect(enwikiImgDb) -enwikiCur = enwikiCon.cursor() -print('Checking for picked-images') -nodeToPickedImg: dict[str, PickedImg] = {} -if os.path.exists(pickedImgsDir + pickedImgsFilename): - lineNum = 0 - with open(pickedImgsDir + pickedImgsFilename) as file: - for line in file: - lineNum += 1 - filename, url, license, artist, credit = line.rstrip().split('|') - nodeName = os.path.splitext(filename)[0] # Remove extension - (otolId,) = dbCur.execute('SELECT id FROM nodes WHERE name = ?', (nodeName,)).fetchone() - nodeToPickedImg[otolId] = PickedImg(nodeName, lineNum, filename, url, license, artist, credit) - -print('Checking for image tables') -nodesDone: set[str] = set() -imgsDone: set[tuple[int, str]] = set() -if dbCur.execute('SELECT name FROM sqlite_master WHERE type="table" AND name="node_imgs"').fetchone() is None: - # Add image tables if not present - dbCur.execute('CREATE TABLE node_imgs (name TEXT PRIMARY KEY, img_id INT, src TEXT)') - dbCur.execute('CREATE TABLE images' \ - ' (id INT, src TEXT, url TEXT, license TEXT, artist TEXT, credit TEXT, PRIMARY KEY (id, src))') -else: - # Get existing image-associated nodes - for (otolId,) in dbCur.execute('SELECT nodes.id FROM node_imgs INNER JOIN nodes ON node_imgs.name = nodes.name'): - nodesDone.add(otolId) - # Get existing node-associated images - for imgId, imgSrc in dbCur.execute('SELECT id, src from images'): - imgsDone.add((imgId, imgSrc)) - print(f'Found {len(nodesDone)} nodes and {len(imgsDone)} images to skip') - -# Set SIGINT handler -interrupted = False -def onSigint(sig, frame): - global interrupted - interrupted = True -signal.signal(signal.SIGINT, onSigint) - -print('Iterating through input images') -def quit(): - print('Closing databases') - dbCon.commit() - dbCon.close() - eolCon.close() - enwikiCon.close() - sys.exit(0) -def convertImage(imgPath, outPath): - print(f'Converting {imgPath} to {outPath}') - if os.path.exists(outPath): - print('ERROR: Output image already exists') - return False - try: - completedProcess = subprocess.run( - ['npx', 'smartcrop-cli', '--width', str(IMG_OUT_SZ), '--height', str(IMG_OUT_SZ), imgPath, outPath], - stdout=subprocess.DEVNULL - ) - except Exception as e: - print(f'ERROR: Exception while attempting to run smartcrop: {e}') - return False - if completedProcess.returncode != 0: - print(f'ERROR: smartcrop had exit status {completedProcess.returncode}') - return False - return True -print('Processing picked-images') -for otolId, imgData in nodeToPickedImg.items(): - # Check for SIGINT event - if interrupted: - print('Exiting') - quit() - # Skip if already processed - if otolId in nodesDone: - continue - # Convert image - if genImgFiles: - success = convertImage(pickedImgsDir + imgData.filename, outDir + otolId + '.jpg') - if not success: - quit() - else: - print(f'Processing {imgData.nodeName}: {otolId}.jpg') - # Add entry to db - if (imgData.id, 'picked') not in imgsDone: - dbCur.execute('INSERT INTO images VALUES (?, ?, ?, ?, ?, ?)', - (imgData.id, 'picked', imgData.url, imgData.license, imgData.artist, imgData.credit)) - imgsDone.add((imgData.id, 'picked')) - dbCur.execute('INSERT INTO node_imgs VALUES (?, ?, ?)', (imgData.nodeName, imgData.id, 'picked')) - nodesDone.add(otolId) -print('Processing images from eol and enwiki') -iterNum = 0 -with open(imgListFile) as file: - for line in file: - iterNum += 1 - # Check for SIGINT event - if interrupted: - print('Exiting') - break - # Skip lines without an image path - if line.find(' ') == -1: - continue - # Get filenames - otolId, _, imgPath = line.rstrip().partition(' ') - # Skip if already processed - if otolId in nodesDone: - continue - # Convert image - if genImgFiles: - success = convertImage(imgPath, outDir + otolId + '.jpg') - if not success: - break - else: - if iterNum % 1e4 == 0: - print(f'At iteration {iterNum}') - # Add entry to db - (nodeName,) = dbCur.execute('SELECT name FROM nodes WHERE id = ?', (otolId,)).fetchone() - fromEol = imgPath.startswith('eol/') - imgName = os.path.basename(os.path.normpath(imgPath)) # Get last path component - imgName = os.path.splitext(imgName)[0] # Remove extension - if fromEol: - eolIdStr, _, contentIdStr = imgName.partition(' ') - eolId, contentId = (int(eolIdStr), int(contentIdStr)) - if (eolId, 'eol') not in imgsDone: - query = 'SELECT source_url, license, copyright_owner FROM images WHERE content_id = ?' - row = eolCur.execute(query, (contentId,)).fetchone() - if row is None: - print(f'ERROR: No image record for EOL ID {eolId}, content ID {contentId}') - break - url, license, owner = row - dbCur.execute('INSERT INTO images VALUES (?, ?, ?, ?, ?, ?)', - (eolId, 'eol', url, license, owner, '')) - imgsDone.add((eolId, 'eol')) - dbCur.execute('INSERT INTO node_imgs VALUES (?, ?, ?)', (nodeName, eolId, 'eol')) - else: - enwikiId = int(imgName) - if (enwikiId, 'enwiki') not in imgsDone: - query = 'SELECT name, license, artist, credit FROM' \ - ' page_imgs INNER JOIN imgs ON page_imgs.img_name = imgs.name' \ - ' WHERE page_imgs.page_id = ?' - row = enwikiCur.execute(query, (enwikiId,)).fetchone() - if row is None: - print(f'ERROR: No image record for enwiki ID {enwikiId}') - break - name, license, artist, credit = row - url = 'https://en.wikipedia.org/wiki/File:' + urllib.parse.quote(name) - dbCur.execute('INSERT INTO images VALUES (?, ?, ?, ?, ?, ?)', - (enwikiId, 'enwiki', url, license, artist, credit)) - imgsDone.add((enwikiId, 'enwiki')) - dbCur.execute('INSERT INTO node_imgs VALUES (?, ?, ?)', (nodeName, enwikiId, 'enwiki')) -# Close dbs -quit() diff --git a/backend/tolData/genLinkedImgs.py b/backend/tolData/genLinkedImgs.py deleted file mode 100755 index 6d2feff..0000000 --- a/backend/tolData/genLinkedImgs.py +++ /dev/null @@ -1,124 +0,0 @@ -#!/usr/bin/python3 - -import re -import sqlite3 - -import argparse -parser = argparse.ArgumentParser(description=""" -Look for nodes without images in the database, and tries to -associate them with images from their children -""", formatter_class=argparse.RawDescriptionHelpFormatter) -parser.parse_args() - -dbFile = 'data.db' -compoundNameRegex = re.compile(r'\[(.+) \+ (.+)]') -upPropagateCompoundImgs = False - -print('Opening databases') -dbCon = sqlite3.connect(dbFile) -dbCur = dbCon.cursor() -dbCur.execute('CREATE TABLE linked_imgs (name TEXT PRIMARY KEY, otol_ids TEXT)') - -print('Getting nodes with images') -resolvedNodes: dict[str, str] = {} # Will map node names to otol IDs with a usable image -query = 'SELECT nodes.name, nodes.id FROM nodes INNER JOIN node_imgs ON nodes.name = node_imgs.name' -for name, otolId in dbCur.execute(query): - resolvedNodes[name] = otolId -print(f'Found {len(resolvedNodes)}') - -print('Iterating through nodes, trying to resolve images for ancestors') -nodesToResolve: dict[str, list[dict[str, str | int | None]]] = {} - # Maps a node name to a list of objects that represent possible child images -processedNodes: dict[str, str] = {} # Map a node name to an OTOL ID, representing a child node whose image is to be used -parentToChosenTips: dict[str, int] = {} # Used to prefer images from children with more tips -iterNum = 0 -while resolvedNodes: - iterNum += 1 - if iterNum % 1e3 == 0: - print(f'At iteration {iterNum}') - # Get next node - nodeName, otolId = resolvedNodes.popitem() - processedNodes[nodeName] = otolId - # Traverse upwards, resolving ancestors if able - while True: - # Get parent - row = dbCur.execute('SELECT parent FROM edges WHERE child = ?', (nodeName,)).fetchone() - if row is None or row[0] in processedNodes or row[0] in resolvedNodes: - break - parent: str = row[0] - # Get parent data - if parent not in nodesToResolve: - childNames: list[str] = [ - row[0] for row in dbCur.execute('SELECT child FROM edges WHERE parent = ?', (parent,))] - query = 'SELECT name, tips FROM nodes WHERE name IN ({})'.format(','.join(['?'] * len(childNames))) - childObjs = [{'name': row[0], 'tips': row[1], 'otolId': None} for row in dbCur.execute(query, childNames)] - childObjs.sort(key=lambda x: x['tips'], reverse=True) - nodesToResolve[parent] = childObjs - else: - childObjs = nodesToResolve[parent] - # Check if highest-tips child - if childObjs[0]['name'] == nodeName: - # Resolve parent, and continue from it - dbCur.execute('INSERT INTO linked_imgs VALUES (?, ?)', (parent, otolId)) - del nodesToResolve[parent] - processedNodes[parent] = otolId - parentToChosenTips[parent] = childObjs[0]['tips'] - nodeName = parent - continue - else: - # Mark child as a potential choice - childObj = next(c for c in childObjs if c['name'] == nodeName) - childObj['otolId'] = otolId - break - # When out of resolved nodes, resolve nodesToResolve nodes, possibly adding more nodes to resolve - if not resolvedNodes: - for name, childObjs in nodesToResolve.items(): - childObj = next(c for c in childObjs if c['otolId'] is not None) - resolvedNodes[name] = childObj['otolId'] - parentToChosenTips[name] = childObj['tips'] - dbCur.execute('INSERT INTO linked_imgs VALUES (?, ?)', (name, childObj['otolId'])) - nodesToResolve.clear() - -print('Replacing linked-images for compound nodes') -iterNum = 0 -for nodeName in processedNodes.keys(): - iterNum += 1 - if iterNum % 1e4 == 0: - print(f'At iteration {iterNum}') - # - match = compoundNameRegex.fullmatch(nodeName) - if match is not None: - # Replace associated image with subname images - subName1, subName2 = match.group(1,2) - otolIdPair = ['', ''] - if subName1 in processedNodes: - otolIdPair[0] = processedNodes[subName1] - if subName2 in processedNodes: - otolIdPair[1] = processedNodes[subName2] - # Use no image if both subimages not found - if otolIdPair[0] == '' and otolIdPair[1] == '': - dbCur.execute('DELETE FROM linked_imgs WHERE name = ?', (nodeName,)) - continue - # Add to db - dbCur.execute('UPDATE linked_imgs SET otol_ids = ? WHERE name = ?', - (otolIdPair[0] + ',' + otolIdPair[1], nodeName)) - # Possibly repeat operation upon parent/ancestors - if upPropagateCompoundImgs: - while True: - # Get parent - row = dbCur.execute('SELECT parent FROM edges WHERE child = ?', (nodeName,)).fetchone() - if row is not None: - parent = row[0] - # Check num tips - (numTips,) = dbCur.execute('SELECT tips from nodes WHERE name = ?', (nodeName,)).fetchone() - if parent in parentToChosenTips and parentToChosenTips[parent] <= numTips: - # Replace associated image - dbCur.execute('UPDATE linked_imgs SET otol_ids = ? WHERE name = ?', - (otolIdPair[0] + ',' + otolIdPair[1], parent)) - nodeName = parent - continue - break - -print('Closing databases') -dbCon.commit() -dbCon.close() diff --git a/backend/tolData/genMappingData.py b/backend/tolData/genMappingData.py deleted file mode 100755 index 5339c4e..0000000 --- a/backend/tolData/genMappingData.py +++ /dev/null @@ -1,229 +0,0 @@ -#!/usr/bin/python3 - -import os -from collections import defaultdict -import gzip, csv, sqlite3 - -import argparse -parser = argparse.ArgumentParser(description=""" -Maps otol IDs to EOL and enwiki titles, using IDs from various -other sources (like NCBI). - -Reads otol taxonomy data to get source IDs for otol IDs, -then looks up those IDs in an EOL provider_ids file, -and in a wikidata dump, and stores results in the database. - -Based on code from https://github.com/OneZoom/OZtree, located in -OZprivate/ServerScripts/TaxonMappingAndPopularity/ (22 Aug 2022). -""", formatter_class=argparse.RawDescriptionHelpFormatter) -args = parser.parse_args() - -taxonomyFile = 'otol/taxonomy.tsv' -eolIdsFile = 'eol/provider_ids.csv.gz' -wikidataDb = 'wikidata/taxonSrcs.db' -enwikiDumpIndexDb = 'enwiki/dumpIndex.db' -pickedMappings = { - 'eol': ['pickedEolIds.txt'], - 'enwiki': ['pickedWikiIds.txt', 'pickedWikiIdsRough.txt'] -} -dbFile = 'data.db' - -print('Reading taxonomy file') -# The file has a header line, then lines that hold these fields (each is followed by a tab-pipe-tab sequence): - # uid (otol-id, eg: 93302), parent_uid, name, rank, - # sourceinfo (comma-separated source specifiers, eg: ncbi:2952,gbif:3207147), uniqueName, flags -OTOL_SRCS = ['ncbi', 'if', 'worms', 'irmng', 'gbif'] # Earlier sources will get higher priority -nodeToSrcIds: dict[int, dict[str, int]] = defaultdict(dict) # Maps otol ID to {src1: id1, src2: id2, ...} -usedSrcIds: set[tuple[str, int]] = set() # {(src1, id1), ...} (used to avoid storing IDs that won't be used) -with open(taxonomyFile) as file: # Had about 4.5e6 lines - lineNum = 0 - for line in file: - lineNum += 1 - if lineNum % 1e5 == 0: - print(f'At line {lineNum}') - # Skip header line - if lineNum == 1: - continue - # Parse line - fields = line.split('\t|\t') - try: - otolId = int(fields[0]) - except ValueError: - print(f'Skipping non-integral ID {fields[0]} on line {lineNum}') - continue - srcsField = fields[4] - # Add source IDs - for srcPair in srcsField.split(','): - src, srcIdStr = srcPair.split(':', 1) - if srcIdStr.isdecimal() and src in OTOL_SRCS and src not in nodeToSrcIds[otolId]: - srcId = int(srcIdStr) - nodeToSrcIds[otolId][src] = srcId - usedSrcIds.add((src, srcId)) -print(f'- Result has {sum([len(v) for v in nodeToSrcIds.values()]):,} entries') # Was about 6.7e6 - -print('Reading EOL provider_ids file') -# The CSV file has a header line, then lines that hold these fields: - # node_id, resource_pk (ID from external source), resource_id (int denoting external-source), - # page_id (eol ID), preferred_canonical_for_page -EOL_SRCS = {676: 'ncbi', 459: 'worms', 767: 'gbif'} # Maps ints to external-source names -srcToEolId: dict[str, dict[int, int]] = {src: {} for src in EOL_SRCS.values()} # Maps src1 to {id1: eolId1, ...} -with gzip.open(eolIdsFile, mode='rt') as file: # Had about 13e6 lines - for lineNum, row in enumerate(csv.reader(file), 1): - if lineNum % 1e6 == 0: - print(f'At line {lineNum}') - # Skip header line - if lineNum == 1: - continue - # Parse line - eolId = int(row[3]) - srcVal = int(row[2]) - srcIdStr = row[1] - if srcIdStr.isdecimal() and srcVal in EOL_SRCS: - srcId = int(srcIdStr) - src = EOL_SRCS[srcVal] - if (src, srcId) not in usedSrcIds: - continue - if srcId in srcToEolId[src]: - print(f'Found {src} ID {srcId} with multiple EOL IDs {srcToEolId[src][srcId]} and {eolId}') - continue - srcToEolId[src][srcId] = eolId -print(f'- Result has {sum([len(v) for v in srcToEolId.values()]):,} entries') - # Was about 3.5e6 (4.2e6 without usedSrcIds) - -print('Resolving candidate EOL IDs') -# For each otol ID, find eol IDs with matching sources, and choose the 'best' one -nodeToEolId: dict[int, int] = {} # Maps otol ID to eol ID -for otolId, srcInfo in nodeToSrcIds.items(): - eolIdToCount: dict[int, int] = defaultdict(int) - for src, srcId in srcInfo.items(): - if src in srcToEolId and srcId in srcToEolId[src]: - eolId = srcToEolId[src][srcId] - eolIdToCount[eolId] += 1 - if len(eolIdToCount) == 1: - nodeToEolId[otolId] = list(eolIdToCount)[0] - elif len(eolIdToCount) > 1: - # For multiple candidates, prefer those with most sources, and break ties by picking the lowest - maxCount = max(eolIdToCount.values()) - eolIds = [eolId for eolId, count in eolIdToCount.items() if count == maxCount] - nodeToEolId[otolId] = min(eolIds) -print(f'- Result has {len(nodeToEolId):,} entries') # Was about 2.7e6 - -print('Reading from Wikidata db') -srcToWikiTitle: dict[str, dict[int, str]] = defaultdict(dict) # Maps 'eol'/etc to {srcId1: title1, ...} -wikiTitles = set() -titleToIucnStatus: dict[str, str] = {} -dbCon = sqlite3.connect(wikidataDb) -dbCur = dbCon.cursor() -for src, srcId, title in dbCur.execute('SELECT src, id, title from src_id_to_title'): - if (src, srcId) not in usedSrcIds and src != 'eol': # Keep EOL IDs for later use - continue - srcToWikiTitle[src][srcId] = title - wikiTitles.add(title) -for title, status in dbCur.execute('SELECT title, status from title_iucn'): - if title in wikiTitles: - titleToIucnStatus[title] = status -print(f'- Source-to-title map has {sum([len(v) for v in srcToWikiTitle.values()]):,} entries') - # Was about 1.1e6 (1.2e6 without usedSrcIds) -print(f'- IUCN map has {len(titleToIucnStatus):,} entries') # Was about 7e4 (7.2e4 without usedSrcIds) -dbCon.close() - -print('Resolving candidate Wikidata items') -# For each otol ID, find wikidata titles with matching sources, and choose the 'best' one -nodeToWikiTitle: dict[int, str] = {} -for otolId, srcInfo in nodeToSrcIds.items(): - titleToSrcs: dict[str, list[str]] = defaultdict(list) # Maps candidate titles to list of sources - for src, srcId in srcInfo.items(): - if src in srcToWikiTitle and srcId in srcToWikiTitle[src]: - title = srcToWikiTitle[src][srcId] - titleToSrcs[title].append(src) - # Choose title to use - if len(titleToSrcs) == 1: - nodeToWikiTitle[otolId] = list(titleToSrcs)[0] - elif len(titleToSrcs) > 1: # Test example: otol ID 621052 - # Get titles with most sources - maxSrcCnt = max([len(srcs) for srcs in titleToSrcs.values()]) - titleToSrcs = {t: s for t, s in titleToSrcs.items() if len(s) == maxSrcCnt} - if len(titleToSrcs) == 1: - nodeToWikiTitle[otolId] = list(titleToSrcs)[0] - else: # Test example: otol ID 4235272 - # Get a title with a source with highest priority - srcToTitle = {s: t for t in titleToSrcs for s in titleToSrcs[t]} - for src in OTOL_SRCS: - if src in srcToTitle: - nodeToWikiTitle[otolId] = srcToTitle[src] - break -print(f'- Result has {len(nodeToWikiTitle):,} entries') # Was about 4e5 - -print('Adding extra EOL mappings from Wikidata') -eolIdToNode = {eolId: node for node, eolId in nodeToEolId.items()} -wikiTitleToNode = {title: node for node, title in nodeToWikiTitle.items()} -addedEntries: dict[int, int] = {} -for eolId, title in srcToWikiTitle['eol'].items(): - if title in wikiTitleToNode: - otolId = wikiTitleToNode[title] - if otolId not in nodeToEolId: # Only add if the otol ID has no EOL ID - nodeToEolId[otolId] = eolId - addedEntries[otolId] = eolId -print(f'- Added {len(addedEntries):,} entries') # Was about 3e3 - -print('Reading picked mappings') -for src in pickedMappings: - for filename in pickedMappings[src]: - if not os.path.exists(filename): - continue - with open(filename) as file: - for line in file: - otolIdStr, mappedVal = line.rstrip().split('|') - otolId = int(otolIdStr) - if src == 'eol': - if mappedVal: - nodeToEolId[otolId] = int(mappedVal) - else: - if otolId in nodeToEolId: - del nodeToEolId[otolId] - else: # src == 'enwiki' - if mappedVal: - nodeToWikiTitle[otolId] = mappedVal - else: - if otolId in nodeToWikiTitle: - del nodeToWikiTitle[otolId] - -print('Getting enwiki page IDs') -titleToPageId: dict[str, int] = {} -numNotFound = 0 -dbCon = sqlite3.connect(enwikiDumpIndexDb) -dbCur = dbCon.cursor() -for title in nodeToWikiTitle.values(): - record = dbCur.execute('SELECT id FROM offsets WHERE title = ?', (title,)).fetchone() - if record != None: - titleToPageId[title] = record[0] - else: - numNotFound += 1 -dbCon.close() -print(f'Unable to find IDs for {numNotFound} titles') # Was 2913 - -print('Writing to db') -dbCon = sqlite3.connect(dbFile) -dbCur = dbCon.cursor() -# Get otol id-to-name map -otolIdToName: dict[int, str] = {} -for nodeName, nodeId in dbCur.execute('SELECT name, id from nodes'): - if nodeId.startswith('ott'): - otolIdToName[int(nodeId[3:])] = nodeName -# Add eol mappings -dbCur.execute('CREATE TABLE eol_ids (name TEXT PRIMARY KEY, id INT)') -dbCur.execute('CREATE INDEX eol_id_idx ON eol_ids(id)') -for otolId, eolId in nodeToEolId.items(): - if otolId in otolIdToName: - dbCur.execute('INSERT INTO eol_ids VALUES (?, ?)', (otolIdToName[otolId], eolId)) -# Add enwiki mappings -dbCur.execute('CREATE TABLE wiki_ids (name TEXT PRIMARY KEY, id INT)') -dbCur.execute('CREATE INDEX wiki_id_idx ON wiki_ids(id)') -dbCur.execute('CREATE TABLE node_iucn (name TEXT PRIMARY KEY, iucn TEXT)') -for otolId, title in nodeToWikiTitle.items(): - if otolId in otolIdToName and title in titleToPageId: - dbCur.execute('INSERT INTO wiki_ids VALUES (?, ?)', (otolIdToName[otolId], titleToPageId[title])) - if title in titleToIucnStatus: - dbCur.execute('INSERT INTO node_iucn VALUES (?, ?)', (otolIdToName[otolId], titleToIucnStatus[title])) -dbCon.commit() -dbCon.close() diff --git a/backend/tolData/genNameData.py b/backend/tolData/genNameData.py deleted file mode 100755 index 2df144d..0000000 --- a/backend/tolData/genNameData.py +++ /dev/null @@ -1,113 +0,0 @@ -#!/usr/bin/python3 - -import re, os -import html, csv, sqlite3 - -import argparse -parser = argparse.ArgumentParser(description=""" -Maps nodes to vernacular names, using data from EOL, enwiki, and a -picked-names file, and stores results in the database. -""", formatter_class=argparse.RawDescriptionHelpFormatter) -args = parser.parse_args() - -eolNamesFile = 'eol/vernacularNames.csv' -enwikiDb = 'enwiki/descData.db' -pickedNamesFile = 'pickedNames.txt' -dbFile = 'data.db' - -dbCon = sqlite3.connect(dbFile) -dbCur = dbCon.cursor() - -print('Getting node mappings') -nodeToTips: dict[str, int] = {} -eolIdToNode: dict[int, str] = {} # Maps eol ID to node name (if there are multiple, choose one with most tips) -wikiIdToNode: dict[int, str] = {} -for name, tips in dbCur.execute('SELECT name, tips from nodes'): - nodeToTips[name] = tips -for name, eolId in dbCur.execute('SELECT name, id from eol_ids'): - if eolId not in eolIdToNode or nodeToTips[eolIdToNode[eolId]] < nodeToTips[name]: - eolIdToNode[eolId] = name -for name, wikiId in dbCur.execute('SELECT name, id from wiki_ids'): - if wikiId not in wikiIdToNode or nodeToTips[wikiIdToNode[wikiId]] < nodeToTips[name]: - wikiIdToNode[wikiId] = name - -print('Creating table') -dbCur.execute('CREATE TABLE names(name TEXT, alt_name TEXT, pref_alt INT, src TEXT, PRIMARY KEY(name, alt_name))') -dbCur.execute('CREATE INDEX names_idx ON names(name)') -dbCur.execute('CREATE INDEX names_alt_idx ON names(alt_name)') -dbCur.execute('CREATE INDEX names_alt_idx_nc ON names(alt_name COLLATE NOCASE)') - -print('Getting names from EOL') -# The CSV file has a header line, then lines with these fields: - # page_id, canonical_form (canonical name, not always unique to page ID), - # vernacular_string (vernacular name), language_code, - # resource_name, is_preferred_by_resource, is_preferred_by_eol -namesToSkip = {'unknown', 'unknown species', 'unidentified species'} -with open(eolNamesFile, newline='') as file: - for lineNum, fields in enumerate(csv.reader(file), 1): - if lineNum % 1e5 == 0: - print(f'At line {lineNum}') # Reached about 2.8e6 - # Skip header line - if lineNum == 1: - continue - # Parse line - eolId = int(fields[0]) - name = html.unescape(fields[2]).lower() - lang = fields[3] - isPreferred = 1 if fields[6] == 'preferred' else 0 - # Add to db - if eolId in eolIdToNode and name not in namesToSkip and name not in nodeToTips \ - and lang == 'eng' and len(name.split(' ')) <= 3: # Ignore names with >3 words - cmd = 'INSERT OR IGNORE INTO names VALUES (?, ?, ?, \'eol\')' - # The 'OR IGNORE' accounts for duplicate lines - dbCur.execute(cmd, (eolIdToNode[eolId], name, isPreferred)) - -print('Getting names from Wikipedia') -altNameRegex = re.compile(r'[a-z]+') # Avoids names like 'evolution of elephants', 'banana fiber', 'fish (zoology)', -enwikiCon = sqlite3.connect(enwikiDb) -enwikiCur = enwikiCon.cursor() -iterNum = 0 -for wikiId, nodeName in wikiIdToNode.items(): - iterNum += 1 - if iterNum % 1e4 == 0: - print(f'At iteration {iterNum}') # Reached about 3.6e5 - # - query = 'SELECT p1.title FROM pages p1' \ - ' INNER JOIN redirects r1 ON p1.id = r1.id' \ - ' INNER JOIN pages p2 ON r1.target = p2.title WHERE p2.id = ?' - for (name,) in enwikiCur.execute(query, (wikiId,)): - name = name.lower() - if altNameRegex.fullmatch(name) is not None and name != nodeName and name not in nodeToTips: - dbCur.execute('INSERT OR IGNORE INTO names VALUES (?, ?, ?, \'enwiki\')', (nodeName, name, 0)) - -print('Getting picked names') -# File format: - # nodename1|altName1|isPreferred1 -> Add an alt-name - # nodename1|altName1| -> Remove an alt-name - # nodename1|nodeName1| -> Remove any preferred-alt status -if os.path.exists(pickedNamesFile): - with open(pickedNamesFile) as file: - for line in file: - nodeName, altName, isPreferredStr = line.lower().rstrip().split('|') - if nodeName not in nodeToTips: - print(f'Skipping "{nodeName}", as no such node exists') - continue - if isPreferredStr: - isPreferred = 1 if isPreferredStr == '1' else 0 - if isPreferred == 1: - # Remove any existing preferred-alt status - cmd = 'UPDATE names SET pref_alt = 0 WHERE name = ? AND alt_name = ? AND pref_alt = 1' - dbCur.execute(cmd, (nodeName, altName)) - # Remove any existing record - dbCur.execute('DELETE FROM names WHERE name = ? AND alt_name = ?', (nodeName, altName)) - # Add record - dbCur.execute('INSERT INTO names VALUES (?, ?, ?, "picked")', (nodeName, altName, isPreferred)) - elif nodeName != altName: # Remove any matching record - dbCur.execute('DELETE FROM names WHERE name = ? AND alt_name = ?', (nodeName, altName)) - else: # Remove any preferred-alt status - cmd = 'UPDATE names SET pref_alt = 0 WHERE name = ? AND alt_name = ? AND pref_alt = 1' - dbCur.execute(cmd, (nodeName, altName)) - -print('Closing database') -dbCon.commit() -dbCon.close() diff --git a/backend/tolData/genOtolData.py b/backend/tolData/genOtolData.py deleted file mode 100755 index d4d6ee8..0000000 --- a/backend/tolData/genOtolData.py +++ /dev/null @@ -1,246 +0,0 @@ -#!/usr/bin/python3 - -import re, os -import json, sqlite3 - -import argparse -parser = argparse.ArgumentParser(description=""" -Reads files describing a tree-of-life from an 'Open Tree of Life' release, -and stores tree info in a database. - -Reads a labelled_supertree_ottnames.tre file, which is assumed to have this format: - The tree-of-life is represented in Newick format, which looks like: (n1,n2,(n3,n4)n5)n6 - The root node is named n6, and has children n1, n2, and n5. - Name examples include: Homo_sapiens_ott770315, mrcaott6ott22687, 'Oxalis san-miguelii ott5748753', - 'ott770315' and 'mrcaott6ott22687' are node IDs. The latter is for a 'compound node'. - The node with ID 'ott770315' will get the name 'homo sapiens'. - A compound node will get a name composed from it's sub-nodes (eg: [name1 + name2]). - It is possible for multiple nodes to have the same name. - In these cases, extra nodes will be named sequentially, as 'name1 [2]', 'name1 [3]', etc. -Reads an annotations.json file, which is assumed to have this format: - Holds a JSON object, whose 'nodes' property maps node IDs to objects holding information about that node, - such as the properties 'supported_by' and 'conflicts_with', which list phylogenetic trees that - support/conflict with the node's placement. -Reads from a picked-names file, if present, which specifies name and node ID pairs. - These help resolve cases where multiple nodes share the same name. -""", formatter_class=argparse.RawDescriptionHelpFormatter) -parser.parse_args() - -class Node: - ' Represents a tree-of-life node ' - def __init__(self, name, childIds, parentId, tips, pSupport): - self.name = name - self.childIds = childIds - self.parentId = parentId - self.tips = tips - self.pSupport = pSupport - -treeFile = 'otol/labelled_supertree_ottnames.tre' # Had about 2.5e9 nodes -annFile = 'otol/annotations.json' -dbFile = 'data.db' -nodeMap: dict[str, Node] = {} # Maps node IDs to node objects -nameToFirstId: dict[str, str] = {} # Maps node names to first found ID (names might have multiple IDs) -dupNameToIds: dict[str, list[str]] = {} # Maps names of nodes with multiple IDs to those IDs -pickedNamesFile = 'pickedOtolNames.txt' - -print('Parsing tree file') -# Read file -data: str -with open(treeFile) as file: - data = file.read() -dataIdx = 0 -# Parse content -iterNum = 0 -def parseNewick() -> str: - """ Parses a node using 'data' and 'dataIdx', updates nodeMap accordingly, and returns the node's ID """ - global data, dataIdx, iterNum - iterNum += 1 - if iterNum % 1e5 == 0: - print(f'At iteration {iterNum}') - # Check for EOF - if dataIdx == len(data): - raise Exception(f'ERROR: Unexpected EOF at index {dataIdx}') - # Check for node - if data[dataIdx] == '(': # parse inner node - dataIdx += 1 - childIds: list[str] = [] - while True: - # Read child - childId = parseNewick() - childIds.append(childId) - if (dataIdx == len(data)): - raise Exception(f'ERROR: Unexpected EOF at index {dataIdx}') - # Check for next child - if (data[dataIdx] == ','): - dataIdx += 1 - continue - else: - # Get node name and id - dataIdx += 1 # Consume an expected ')' - name, id = parseNewickName() - updateNameMaps(name, id) - # Get child num-tips total - tips = 0 - for childId in childIds: - tips += nodeMap[childId].tips - # Add node to nodeMap - nodeMap[id] = Node(name, childIds, None, tips, False) - # Update childrens' parent reference - for childId in childIds: - nodeMap[childId].parentId = id - return id - else: # Parse node name - name, id = parseNewickName() - updateNameMaps(name, id) - nodeMap[id] = Node(name, [], None, 1, False) - return id -def parseNewickName() -> tuple[str, str]: - """ Parses a node name using 'data' and 'dataIdx', and returns a (name, id) pair """ - global data, dataIdx - name: str - end = dataIdx - # Get name - if (end < len(data) and data[end] == "'"): # Check for quoted name - end += 1 - inQuote = True - while end < len(data): - if (data[end] == "'"): - if end + 1 < len(data) and data[end + 1] == "'": # Account for '' as escaped-quote - end += 2 - continue - else: - end += 1 - inQuote = False - break - end += 1 - if inQuote: - raise Exception(f'ERROR: Unexpected EOF at index {dataIdx}') - name = data[dataIdx:end] - dataIdx = end - else: - while end < len(data) and not re.match(r'[(),]', data[end]): - end += 1 - if (end == dataIdx): - raise Exception(f'ERROR: Unexpected EOF at index {dataIdx}') - name = data[dataIdx:end].rstrip() - if end == len(data): # Ignore trailing input semicolon - name = name[:-1] - dataIdx = end - # Convert to (name, id) - name = name.lower() - if name.startswith('mrca'): - return (name, name) - elif name[0] == "'": - match = re.fullmatch(r"'([^\\\']+) (ott\d+)'", name) - if match is None: - raise Exception(f'ERROR: invalid name \'{name}\'') - name = match.group(1).replace("''", "'") - return (name, match.group(2)) - else: - match = re.fullmatch(r"([^\\\']+)_(ott\d+)", name) - if match is None: - raise Exception(f'ERROR: invalid name \'{name}\'') - return (match.group(1).replace('_', ' '), match.group(2)) -def updateNameMaps(name, id): - global nameToFirstId, dupNameToIds - if name not in nameToFirstId: - nameToFirstId[name] = id - else: - if name not in dupNameToIds: - dupNameToIds[name] = [nameToFirstId[name], id] - else: - dupNameToIds[name].append(id) -rootId: str = parseNewick() - -print('Resolving duplicate names') -# Read picked-names file -nameToPickedId: dict[str, str] = {} -if os.path.exists(pickedNamesFile): - with open(pickedNamesFile) as file: - for line in file: - name, _, otolId = line.rstrip().partition('|') - nameToPickedId[name] = otolId -# Resolve duplicates -for dupName, ids in dupNameToIds.items(): - # Check for picked id - if dupName in nameToPickedId: - idToUse = nameToPickedId[dupName] - else: - # Get conflicting node with most tips - tipNums = [nodeMap[id].tips for id in ids] - maxIdx = tipNums.index(max(tipNums)) - idToUse = ids[maxIdx] - # Adjust name of other conflicting nodes - counter = 2 - for id in ids: - if id != idToUse: - nodeMap[id].name += f' [{counter}]' - counter += 1 - -print('Changing mrca* names') -def convertMrcaName(id: str): - node = nodeMap[id] - name = node.name - childIds = node.childIds - if len(childIds) < 2: - print(f'WARNING: MRCA node \'{name}\' has less than 2 children') - return - # Get 2 children with most tips - childTips = [nodeMap[id].tips for id in childIds] - maxIdx1 = childTips.index(max(childTips)) - childTips[maxIdx1] = 0 - maxIdx2 = childTips.index(max(childTips)) - childId1 = childIds[maxIdx1] - childId2 = childIds[maxIdx2] - childName1 = nodeMap[childId1].name - childName2 = nodeMap[childId2].name - # Check for mrca* child names - if childName1.startswith('mrca'): - childName1 = convertMrcaName(childId1) - if childName2.startswith('mrca'): - childName2 = convertMrcaName(childId2) - # Check for composite names - match = re.fullmatch(r'\[(.+) \+ (.+)]', childName1) - if match is not None: - childName1 = match.group(1) - match = re.fullmatch(r'\[(.+) \+ (.+)]', childName2) - if match is not None: - childName2 = match.group(1) - # Create composite name - node.name = f'[{childName1} + {childName2}]' - return childName1 -for id, node in nodeMap.items(): - if node.name.startswith('mrca'): - convertMrcaName(id) - -print('Parsing annotations file') -# Read file -with open(annFile) as file: - data = file.read() -obj = json.loads(data) -nodeAnnsMap = obj['nodes'] -# Find relevant annotations -for id, node in nodeMap.items(): - # Set has-support value using annotations - if id in nodeAnnsMap: - nodeAnns = nodeAnnsMap[id] - supportQty = len(nodeAnns['supported_by']) if 'supported_by' in nodeAnns else 0 - conflictQty = len(nodeAnns['conflicts_with']) if 'conflicts_with' in nodeAnns else 0 - node.pSupport = supportQty > 0 and conflictQty == 0 - -print('Creating nodes and edges tables') -dbCon = sqlite3.connect(dbFile) -dbCur = dbCon.cursor() -dbCur.execute('CREATE TABLE nodes (name TEXT PRIMARY KEY, id TEXT UNIQUE, tips INT)') -dbCur.execute('CREATE INDEX nodes_idx_nc ON nodes(name COLLATE NOCASE)') -dbCur.execute('CREATE TABLE edges (parent TEXT, child TEXT, p_support INT, PRIMARY KEY (parent, child))') -dbCur.execute('CREATE INDEX edges_child_idx ON edges(child)') -for otolId, node in nodeMap.items(): - dbCur.execute('INSERT INTO nodes VALUES (?, ?, ?)', (node.name, otolId, node.tips)) - for childId in node.childIds: - childNode = nodeMap[childId] - dbCur.execute('INSERT INTO edges VALUES (?, ?, ?)', - (node.name, childNode.name, 1 if childNode.pSupport else 0)) -print('Closing database') -dbCon.commit() -dbCon.close() diff --git a/backend/tolData/genPopData.py b/backend/tolData/genPopData.py deleted file mode 100755 index 3bb1325..0000000 --- a/backend/tolData/genPopData.py +++ /dev/null @@ -1,39 +0,0 @@ -#!/usr/bin/python3 - -import sqlite3 - -import argparse -parser = argparse.ArgumentParser(description=""" -Reads enwiki page view info from a database, and stores it -as node popularity values in the database. -""", formatter_class=argparse.RawDescriptionHelpFormatter) -args = parser.parse_args() - -pageviewsDb = 'enwiki/pageviewData.db' -dbFile = 'data.db' - -dbCon = sqlite3.connect(dbFile) -dbCur = dbCon.cursor() - -print('Getting view counts') -pdbCon = sqlite3.connect(pageviewsDb) -pdbCur = pdbCon.cursor() -nodeToViews: dict[str, int] = {} # Maps node names to counts -iterNum = 0 -for wikiId, views in pdbCur.execute('SELECT id, views from views'): - iterNum += 1 - if iterNum % 1e4 == 0: - print(f'At iteration {iterNum}') # Reached 1.6e6 - # - row = dbCur.execute('SELECT name FROM wiki_ids WHERE id = ?', (wikiId,)).fetchone() - if row is not None: - nodeToViews[row[0]] = views -pdbCon.close() - -print(f'Writing {len(nodeToViews)} entries to db') -dbCur.execute('CREATE TABLE node_pop (name TEXT PRIMARY KEY, pop INT)') -for nodeName, views in nodeToViews.items(): - dbCur.execute('INSERT INTO node_pop VALUES (?, ?)', (nodeName, views)) - -dbCon.commit() -dbCon.close() diff --git a/backend/tolData/genReducedTrees.py b/backend/tolData/genReducedTrees.py deleted file mode 100755 index 66fef40..0000000 --- a/backend/tolData/genReducedTrees.py +++ /dev/null @@ -1,334 +0,0 @@ -#!/usr/bin/python3 - -import sys, re -import sqlite3 - -import argparse -parser = argparse.ArgumentParser(description=""" -Creates reduced versions of the tree in the database: -- A 'picked nodes' tree: - Created from a minimal set of node names read from a file, - possibly with some extra randmly-picked children. -- An 'images only' tree: - Created by removing nodes without an image or presence in the - 'picked' tree. -- A 'weakly trimmed' tree: - Created by removing nodes that lack an image or description, or - presence in the 'picked' tree. And, for nodes with 'many' children, - removing some more, despite any node descriptions. -""", formatter_class=argparse.RawDescriptionHelpFormatter) -parser.add_argument('--tree', choices=['picked', 'images', 'trimmed'], help='Only generate the specified tree') -args = parser.parse_args() - -tree = args.tree -dbFile = 'data.db' -pickedNodesFile = 'pickedNodes.txt' -COMP_NAME_REGEX = re.compile(r'\[.+ \+ .+]') # Used to recognise composite nodes - -class Node: - def __init__(self, id, children, parent, tips, pSupport): - self.id = id - self.children = children - self.parent = parent - self.tips = tips - self.pSupport = pSupport - -print('Opening database') -dbCon = sqlite3.connect(dbFile) -dbCur = dbCon.cursor() - -def genPickedNodeTree(dbCur: sqlite3.Cursor, pickedNames: set[str], rootName: str) -> None: - global COMP_NAME_REGEX - PREF_NUM_CHILDREN = 3 # Include extra children up to this limit - print('Getting ancestors') - nodeMap = genNodeMap(dbCur, pickedNames, 100) - print(f'Result has {len(nodeMap)} nodes') - print('Removing composite nodes') - removedNames = removeCompositeNodes(nodeMap) - print(f'Result has {len(nodeMap)} nodes') - print('Removing \'collapsible\' nodes') - temp = removeCollapsibleNodes(nodeMap, pickedNames) - removedNames.update(temp) - print(f'Result has {len(nodeMap)} nodes') - print('Adding some additional nearby children') - namesToAdd: list[str] = [] - iterNum = 0 - for name, node in nodeMap.items(): - iterNum += 1 - if iterNum % 100 == 0: - print(f'At iteration {iterNum}') - # - numChildren = len(node.children) - if numChildren < PREF_NUM_CHILDREN: - children = [row[0] for row in dbCur.execute('SELECT child FROM edges where parent = ?', (name,))] - newChildren: list[str] = [] - for n in children: - if n in nodeMap or n in removedNames: - continue - if COMP_NAME_REGEX.fullmatch(n) is not None: - continue - if dbCur.execute('SELECT name from node_imgs WHERE name = ?', (n,)).fetchone() is None and \ - dbCur.execute('SELECT name from linked_imgs WHERE name = ?', (n,)).fetchone() is None: - continue - newChildren.append(n) - newChildNames = newChildren[:(PREF_NUM_CHILDREN - numChildren)] - node.children.extend(newChildNames) - namesToAdd.extend(newChildNames) - for name in namesToAdd: - parent, pSupport = dbCur.execute('SELECT parent, p_support from edges WHERE child = ?', (name,)).fetchone() - (id,) = dbCur.execute('SELECT id FROM nodes WHERE name = ?', (name,)).fetchone() - parent = None if parent == '' else parent - nodeMap[name] = Node(id, [], parent, 0, pSupport == 1) - print(f'Result has {len(nodeMap)} nodes') - print('Updating \'tips\' values') - updateTips(rootName, nodeMap) - print('Creating table') - addTreeTables(nodeMap, dbCur, 'p') -def genImagesOnlyTree( - dbCur: sqlite3.Cursor, - nodesWithImgOrPicked: set[str], - pickedNames: set[str], - rootName: str) -> None: - print('Getting ancestors') - nodeMap = genNodeMap(dbCur, nodesWithImgOrPicked, 1e4) - print(f'Result has {len(nodeMap)} nodes') - print('Removing composite nodes') - removeCompositeNodes(nodeMap) - print(f'Result has {len(nodeMap)} nodes') - print('Removing \'collapsible\' nodes') - removeCollapsibleNodes(nodeMap, pickedNames) - print(f'Result has {len(nodeMap)} nodes') - print('Updating \'tips\' values') # Needed for next trimming step - updateTips(rootName, nodeMap) - print('Trimming from nodes with \'many\' children') - trimIfManyChildren(nodeMap, rootName, 300, pickedNames) - print(f'Result has {len(nodeMap)} nodes') - print('Updating \'tips\' values') - updateTips(rootName, nodeMap) - print('Creating table') - addTreeTables(nodeMap, dbCur, 'i') -def genWeaklyTrimmedTree( - dbCur: sqlite3.Cursor, - nodesWithImgDescOrPicked: set[str], - nodesWithImgOrPicked: set[str], - rootName: str) -> None: - print('Getting ancestors') - nodeMap = genNodeMap(dbCur, nodesWithImgDescOrPicked, 1e5) - print(f'Result has {len(nodeMap)} nodes') - print('Getting nodes to \'strongly keep\'') - iterNum = 0 - nodesFromImgOrPicked: set[str] = set() - for name in nodesWithImgOrPicked: - iterNum += 1 - if iterNum % 1e4 == 0: - print(f'At iteration {iterNum}') - # - while name is not None: - if name not in nodesFromImgOrPicked: - nodesFromImgOrPicked.add(name) - name = nodeMap[name].parent - else: - break - print(f'Node set has {len(nodesFromImgOrPicked)} nodes') - print('Removing \'collapsible\' nodes') - removeCollapsibleNodes(nodeMap, nodesWithImgDescOrPicked) - print(f'Result has {len(nodeMap)} nodes') - print('Updating \'tips\' values') # Needed for next trimming step - updateTips(rootName, nodeMap) - print('Trimming from nodes with \'many\' children') - trimIfManyChildren(nodeMap, rootName, 600, nodesFromImgOrPicked) - print(f'Result has {len(nodeMap)} nodes') - print('Updating \'tips\' values') - updateTips(rootName, nodeMap) - print('Creating table') - addTreeTables(nodeMap, dbCur, 't') -# Helper functions -def genNodeMap(dbCur: sqlite3.Cursor, nameSet: set[str], itersBeforePrint = 1) -> dict[str, Node]: - """ Returns a subtree that includes nodes in 'nameSet', as a name-to-Node map """ - nodeMap: dict[str, Node] = {} - iterNum = 0 - name: str | None - for name in nameSet: - iterNum += 1 - if iterNum % itersBeforePrint == 0: - print(f'At iteration {iterNum}') - # - prevName: str | None = None - while name is not None: - if name not in nodeMap: - # Add node - id, tips = dbCur.execute('SELECT id, tips from nodes where name = ?', (name,)).fetchone() - row: None | tuple[str, int] = dbCur.execute( - 'SELECT parent, p_support from edges where child = ?', (name,)).fetchone() - parent = None if row is None or row[0] == '' else row[0] - pSupport = row is None or row[1] == 1 - children = [] if prevName is None else [prevName] - nodeMap[name] = Node(id, children, parent, 0, pSupport) - # Iterate to parent - prevName = name - name = parent - else: - # Just add as child - if prevName is not None: - nodeMap[name].children.append(prevName) - break - return nodeMap -def removeCompositeNodes(nodeMap: dict[str, Node]) -> set[str]: - """ Given a tree, removes composite-name nodes, and returns the removed nodes' names """ - global COMP_NAME_REGEX - namesToRemove: set[str] = set() - for name, node in nodeMap.items(): - parent = node.parent - if parent is not None and COMP_NAME_REGEX.fullmatch(name) is not None: - # Connect children to parent - nodeMap[parent].children.remove(name) - nodeMap[parent].children.extend(node.children) - for n in node.children: - nodeMap[n].parent = parent - nodeMap[n].pSupport &= node.pSupport - # Remember for removal - namesToRemove.add(name) - for name in namesToRemove: - del nodeMap[name] - return namesToRemove -def removeCollapsibleNodes(nodeMap: dict[str, Node], nodesToKeep: set[str] = set()) -> set[str]: - """ Given a tree, removes single-child parents, then only-childs, - with given exceptions, and returns the set of removed nodes' names """ - namesToRemove: set[str] = set() - # Remove single-child parents - for name, node in nodeMap.items(): - if len(node.children) == 1 and node.parent is not None and name not in nodesToKeep: - # Connect parent and children - parent = node.parent - child = node.children[0] - nodeMap[parent].children.remove(name) - nodeMap[parent].children.append(child) - nodeMap[child].parent = parent - nodeMap[child].pSupport &= node.pSupport - # Remember for removal - namesToRemove.add(name) - for name in namesToRemove: - del nodeMap[name] - # Remove only-childs (not redundant because 'nodesToKeep' can cause single-child parents to be kept) - namesToRemove.clear() - for name, node in nodeMap.items(): - isOnlyChild = node.parent is not None and len(nodeMap[node.parent].children) == 1 - if isOnlyChild and name not in nodesToKeep: - # Connect parent and children - parent = node.parent - nodeMap[parent].children = node.children - for n in node.children: - nodeMap[n].parent = parent - nodeMap[n].pSupport &= node.pSupport - # Remember for removal - namesToRemove.add(name) - for name in namesToRemove: - del nodeMap[name] - # - return namesToRemove -def trimIfManyChildren( - nodeMap: dict[str, Node], rootName: str, childThreshold: int, nodesToKeep: set[str] = set()) -> None: - namesToRemove: set[str] = set() - def findTrimmables(nodeName: str) -> None: - nonlocal nodeMap, nodesToKeep - node = nodeMap[nodeName] - if len(node.children) > childThreshold: - numToTrim = len(node.children) - childThreshold - # Try removing nodes, preferring those with less tips - candidatesToTrim = [n for n in node.children if n not in nodesToKeep] - childToTips = {n: nodeMap[n].tips for n in candidatesToTrim} - candidatesToTrim.sort(key=lambda n: childToTips[n], reverse=True) - childrenToRemove = set(candidatesToTrim[-numToTrim:]) - node.children = [n for n in node.children if n not in childrenToRemove] - # Mark nodes for deletion - for n in childrenToRemove: - markForRemoval(n) - # Recurse on children - for n in node.children: - findTrimmables(n) - def markForRemoval(nodeName: str) -> None: - nonlocal nodeMap, namesToRemove - namesToRemove.add(nodeName) - for child in nodeMap[nodeName].children: - markForRemoval(child) - findTrimmables(rootName) - for nodeName in namesToRemove: - del nodeMap[nodeName] -def updateTips(nodeName: str, nodeMap: dict[str, Node]) -> int: - """ Updates the 'tips' values for a node and it's descendants, returning the node's new 'tips' value """ - node = nodeMap[nodeName] - tips = sum([updateTips(childName, nodeMap) for childName in node.children]) - tips = max(1, tips) - node.tips = tips - return tips -def addTreeTables(nodeMap: dict[str, Node], dbCur: sqlite3.Cursor, suffix: str): - """ Adds a tree to the database, as tables nodes_X and edges_X, where X is the given suffix """ - nodesTbl = f'nodes_{suffix}' - edgesTbl = f'edges_{suffix}' - dbCur.execute(f'CREATE TABLE {nodesTbl} (name TEXT PRIMARY KEY, id TEXT UNIQUE, tips INT)') - dbCur.execute(f'CREATE INDEX {nodesTbl}_idx_nc ON {nodesTbl}(name COLLATE NOCASE)') - dbCur.execute(f'CREATE TABLE {edgesTbl} (parent TEXT, child TEXT, p_support INT, PRIMARY KEY (parent, child))') - dbCur.execute(f'CREATE INDEX {edgesTbl}_child_idx ON {edgesTbl}(child)') - for name, node in nodeMap.items(): - dbCur.execute(f'INSERT INTO {nodesTbl} VALUES (?, ?, ?)', (name, node.id, node.tips)) - for childName in node.children: - pSupport = 1 if nodeMap[childName].pSupport else 0 - dbCur.execute(f'INSERT INTO {edgesTbl} VALUES (?, ?, ?)', (name, childName, pSupport)) - -print('Finding root node') -query = 'SELECT name FROM nodes LEFT JOIN edges ON nodes.name = edges.child WHERE edges.parent IS NULL LIMIT 1' -(rootName,) = dbCur.execute(query).fetchone() -print(f'Found \'{rootName}\'') - -print('=== Getting picked-nodes ===') -pickedNames: set[str] = set() -pickedTreeExists = False -if dbCur.execute('SELECT name FROM sqlite_master WHERE type="table" AND name="nodes_p"').fetchone() is None: - print(f'Reading from {pickedNodesFile}') - with open(pickedNodesFile) as file: - for line in file: - name = line.rstrip() - row = dbCur.execute('SELECT name from nodes WHERE name = ?', (name,)).fetchone() - if row is None: - row = dbCur.execute('SELECT name from names WHERE alt_name = ?', (name,)).fetchone() - if row is not None: - pickedNames.add(row[0]) - if not pickedNames: - raise Exception('ERROR: No picked names found') -else: - pickedTreeExists = True - print('Picked-node tree already exists') - if tree == 'picked': - sys.exit() - for (name,) in dbCur.execute('SELECT name FROM nodes_p'): - pickedNames.add(name) -print(f'Found {len(pickedNames)} names') - -if (tree == 'picked' or tree is None) and not pickedTreeExists: - print('=== Generating picked-nodes tree ===') - genPickedNodeTree(dbCur, pickedNames, rootName) -if tree != 'picked': - print('=== Finding \'non-low significance\' nodes ===') - nodesWithImgOrPicked: set[str] = set() - nodesWithImgDescOrPicked: set[str] = set() - print('Finding nodes with descs') - for (name,) in dbCur.execute('SELECT name FROM wiki_ids'): # Can assume the wiki_id has a desc - nodesWithImgDescOrPicked.add(name) - print('Finding nodes with images') - for (name,) in dbCur.execute('SELECT name FROM node_imgs'): - nodesWithImgDescOrPicked.add(name) - nodesWithImgOrPicked.add(name) - print('Adding picked nodes') - for name in pickedNames: - nodesWithImgDescOrPicked.add(name) - nodesWithImgOrPicked.add(name) - if tree == 'images' or tree is None: - print('=== Generating images-only tree ===') - genImagesOnlyTree(dbCur, nodesWithImgOrPicked, pickedNames, rootName) - if tree == 'trimmed' or tree is None: - print('=== Generating weakly-trimmed tree ===') - genWeaklyTrimmedTree(dbCur, nodesWithImgDescOrPicked, nodesWithImgOrPicked, rootName) - -print('Closing database') -dbCon.commit() -dbCon.close() diff --git a/backend/tolData/otol/README.md b/backend/tolData/otol/README.md deleted file mode 100644 index e018369..0000000 --- a/backend/tolData/otol/README.md +++ /dev/null @@ -1,19 +0,0 @@ -This directory holds files obtained via the -[Open Tree of Life](https://tree.opentreeoflife.org/about/open-tree-of-life). - -# Tree Data Files -- `opentree13.4tree.tgz` <br> - Obtained from <https://tree.opentreeoflife.org/about/synthesis-release/v13.4>. - Contains tree data from the [Open Tree of Life](https://tree.opentreeoflife.org/about/open-tree-of-life). -- `labelled_supertree_ottnames.tre` <br> - Extracted from the .tgz file. Describes the structure of the tree. -- `annotations.json` <br> - Extracted from the .tgz file. Contains additional attributes of tree - nodes. Used for finding out which nodes have 'phylogenetic support'. - -# Taxonomy Data Files -- `ott3.3.tgz` <br> - Obtained from <https://tree.opentreeoflife.org/about/taxonomy-version/ott3.3>. - Contains taxonomy data from the Open Tree of Life. -- `otol/taxonomy.tsv` <br> - Extracted from the .tgz file. Holds taxon IDs from sources like NCBI, used to map between datasets. diff --git a/backend/tolData/pickedImgs/README.md b/backend/tolData/pickedImgs/README.md deleted file mode 100644 index dfe192b..0000000 --- a/backend/tolData/pickedImgs/README.md +++ /dev/null @@ -1,10 +0,0 @@ -This directory holds additional image files to use for tree-of-life nodes, -on top of those from EOL and Wikipedia. - -Possible Files -============== -- (Image files) -- imgData.txt <br> - Contains lines with the format `filename|url|license|artist|credit`. - The filename should consist of a node name, with an image extension. - Other fields correspond to those in the `images` table (see ../README.md). diff --git a/backend/tolData/reviewImgsToGen.py b/backend/tolData/reviewImgsToGen.py deleted file mode 100755 index f3791bc..0000000 --- a/backend/tolData/reviewImgsToGen.py +++ /dev/null @@ -1,223 +0,0 @@ -#!/usr/bin/python3 - -import os, time -import sqlite3 -import tkinter as tki -from tkinter import ttk -import PIL -from PIL import ImageTk, Image, ImageOps - -import argparse -parser = argparse.ArgumentParser(description=""" -Provides a GUI that displays, for each node in the database, associated -images from EOL and Wikipedia, and allows choosing which to use. Writes -choice data to a text file with lines of the form 'otolId1 imgPath1', or -'otolId1', where no path indicates a choice of no image. - -The program can be closed, and run again to continue from the last choice. -The program looks for an existing output file to determine what choices -have already been made. -""", formatter_class=argparse.RawDescriptionHelpFormatter) -parser.parse_args() - -eolImgDir = 'eol/imgs/' -enwikiImgDir = 'enwiki/imgs/' -dbFile = 'data.db' -outFile = 'imgList.txt' -IMG_DISPLAY_SZ = 400 -PLACEHOLDER_IMG = Image.new('RGB', (IMG_DISPLAY_SZ, IMG_DISPLAY_SZ), (88, 28, 135)) -onlyReviewPairs = True - -print('Opening database') -dbCon = sqlite3.connect(dbFile) -dbCur = dbCon.cursor() - -nodeToImgs: dict[str, list[str]] = {} # Maps otol-ids to arrays of image paths -print('Iterating through images from EOL') -if os.path.exists(eolImgDir): - for filename in os.listdir(eolImgDir): - # Get associated EOL ID - eolId, _, _ = filename.partition(' ') - query = 'SELECT nodes.id FROM nodes INNER JOIN eol_ids ON nodes.name = eol_ids.name WHERE eol_ids.id = ?' - # Get associated node IDs - found = False - for (otolId,) in dbCur.execute(query, (int(eolId),)): - if otolId not in nodeToImgs: - nodeToImgs[otolId] = [] - nodeToImgs[otolId].append(eolImgDir + filename) - found = True - if not found: - print(f'WARNING: No node found for {eolImgDir}{filename}') -print(f'Result: {len(nodeToImgs)} nodes with images') -print('Iterating through images from Wikipedia') -if os.path.exists(enwikiImgDir): - for filename in os.listdir(enwikiImgDir): - # Get associated page ID - wikiId, _, _ = filename.partition('.') - # Get associated node IDs - query = 'SELECT nodes.id FROM nodes INNER JOIN wiki_ids ON nodes.name = wiki_ids.name WHERE wiki_ids.id = ?' - found = False - for (otolId,) in dbCur.execute(query, (int(wikiId),)): - if otolId not in nodeToImgs: - nodeToImgs[otolId] = [] - nodeToImgs[otolId].append(enwikiImgDir + filename) - found = True - if not found: - print(f'WARNING: No node found for {enwikiImgDir}{filename}') -print(f'Result: {len(nodeToImgs)} nodes with images') -print('Filtering out already-made image choices') - -oldSz = len(nodeToImgs) -if os.path.exists(outFile): - with open(outFile) as file: - for line in file: - line = line.rstrip() - if ' ' in line: - line = line[:line.find(' ')] - del nodeToImgs[line] -print(f'Filtered out {oldSz - len(nodeToImgs)} entries') - -class ImgReviewer: - """ Provides the GUI for reviewing images """ - def __init__(self, root, nodeToImgs): - self.root = root - root.title('Image Reviewer') - # Setup main frame - mainFrame = ttk.Frame(root, padding='5 5 5 5') - mainFrame.grid(column=0, row=0, sticky=(tki.N, tki.W, tki.E, tki.S)) - root.columnconfigure(0, weight=1) - root.rowconfigure(0, weight=1) - # Set up images-to-be-reviewed frames - self.eolImg = ImageTk.PhotoImage(PLACEHOLDER_IMG) - self.enwikiImg = ImageTk.PhotoImage(PLACEHOLDER_IMG) - self.labels: list[ttk.Label] = [] - for i in (0, 1): - frame = ttk.Frame(mainFrame, width=IMG_DISPLAY_SZ, height=IMG_DISPLAY_SZ) - frame.grid(column=i, row=0) - label = ttk.Label(frame, image=self.eolImg if i == 0 else self.enwikiImg) - label.grid(column=0, row=0) - self.labels.append(label) - # Add padding - for child in mainFrame.winfo_children(): - child.grid_configure(padx=5, pady=5) - # Add keyboard bindings - root.bind('<q>', self.quit) - root.bind('<Key-j>', lambda evt: self.accept(0)) - root.bind('<Key-k>', lambda evt: self.accept(1)) - root.bind('<Key-l>', lambda evt: self.reject()) - # Set fields - self.nodeImgsList = list(nodeToImgs.items()) - self.listIdx = -1 - self.otolId = None - self.eolImgPath = None - self.enwikiImgPath = None - self.numReviewed = 0 - self.startTime = time.time() - # Initialise images to review - self.getNextImgs() - def getNextImgs(self): - """ Updates display with new images to review, or ends program """ - # Get next image paths - while True: - self.listIdx += 1 - if self.listIdx == len(self.nodeImgsList): - print('No more images to review. Exiting program.') - self.quit() - return - self.otolId, imgPaths = self.nodeImgsList[self.listIdx] - # Potentially skip user choice - if onlyReviewPairs and len(imgPaths) == 1: - with open(outFile, 'a') as file: - file.write(f'{self.otolId} {imgPaths[0]}\n') - continue - break - # Update displayed images - self.eolImgPath = self.enwikiImgPath = None - imageOpenError = False - for imgPath in imgPaths: - img: Image - try: - img = Image.open(imgPath) - img = ImageOps.exif_transpose(img) - except PIL.UnidentifiedImageError: - print(f'UnidentifiedImageError for {imgPath}') - imageOpenError = True - continue - if imgPath.startswith('eol/'): - self.eolImgPath = imgPath - self.eolImg = ImageTk.PhotoImage(self.resizeImgForDisplay(img)) - elif imgPath.startswith('enwiki/'): - self.enwikiImgPath = imgPath - self.enwikiImg = ImageTk.PhotoImage(self.resizeImgForDisplay(img)) - else: - print(f'Unexpected image path {imgPath}') - self.quit() - return - # Re-iterate if all image paths invalid - if self.eolImgPath is None and self.enwikiImgPath is None: - if imageOpenError: - self.reject() - self.getNextImgs() - return - # Add placeholder images - if self.eolImgPath is None: - self.eolImg = ImageTk.PhotoImage(self.resizeImgForDisplay(PLACEHOLDER_IMG)) - elif self.enwikiImgPath is None: - self.enwikiImg = ImageTk.PhotoImage(self.resizeImgForDisplay(PLACEHOLDER_IMG)) - # Update image-frames - self.labels[0].config(image=self.eolImg) - self.labels[1].config(image=self.enwikiImg) - # Update title - title = f'Images for otol ID {self.otolId}' - query = 'SELECT names.alt_name FROM' \ - ' nodes INNER JOIN names ON nodes.name = names.name' \ - ' WHERE nodes.id = ? and pref_alt = 1' - row = dbCur.execute(query, (self.otolId,)).fetchone() - if row is not None: - title += f', aka {row[0]}' - title += f' ({self.listIdx + 1} out of {len(self.nodeImgsList)})' - self.root.title(title) - def accept(self, imgIdx): - """ React to a user selecting an image """ - imgPath = self.eolImgPath if imgIdx == 0 else self.enwikiImgPath - if imgPath is None: - print('Invalid selection') - return - with open(outFile, 'a') as file: - file.write(f'{self.otolId} {imgPath}\n') - self.numReviewed += 1 - self.getNextImgs() - def reject(self): - """"" React to a user rejecting all images of a set """ - with open(outFile, 'a') as file: - file.write(f'{self.otolId}\n') - self.numReviewed += 1 - self.getNextImgs() - def quit(self, e = None): - global dbCon - print(f'Number reviewed: {self.numReviewed}') - timeElapsed = time.time() - self.startTime - print(f'Time elapsed: {timeElapsed:.2f} seconds') - if self.numReviewed > 0: - print(f'Avg time per review: {timeElapsed/self.numReviewed:.2f} seconds') - dbCon.close() - self.root.destroy() - def resizeImgForDisplay(self, img): - """ Returns a copy of an image, shrunk to fit it's frame (keeps aspect ratio), and with a background """ - if max(img.width, img.height) > IMG_DISPLAY_SZ: - if (img.width > img.height): - newHeight = int(img.height * IMG_DISPLAY_SZ/img.width) - img = img.resize((IMG_DISPLAY_SZ, newHeight)) - else: - newWidth = int(img.width * IMG_DISPLAY_SZ / img.height) - img = img.resize((newWidth, IMG_DISPLAY_SZ)) - bgImg = PLACEHOLDER_IMG.copy() - bgImg.paste(img, box=( - int((IMG_DISPLAY_SZ - img.width) / 2), - int((IMG_DISPLAY_SZ - img.height) / 2))) - return bgImg -# Create GUI and defer control -print('Starting GUI') -root = tki.Tk() -ImgReviewer(root, nodeToImgs) -root.mainloop() diff --git a/backend/tolData/wikidata/README.md b/backend/tolData/wikidata/README.md deleted file mode 100644 index db45b3c..0000000 --- a/backend/tolData/wikidata/README.md +++ /dev/null @@ -1,18 +0,0 @@ -This directory holds files obtained via [Wikidata](https://www.wikidata.org/). - -# Downloaded Files -- `latest-all.json.bz2` <br> - Obtained from <https://dumps.wikimedia.org/wikidatawiki/entities/> (on 23/08/22). - Format info can be found at <https://doc.wikimedia.org/Wikibase/master/php/md_docs_topics_json.html>. - -# Other Files -- genTaxonSrcData.py <br> - Used to generate a database holding taxon information from the dump. -- offsets.dat <br> - Holds bzip2 block offsets for the dump. Generated and used by - genTaxonSrcData.py for parallel processing of the dump. -- taxonSrcs.db <br> - Generated by genTaxonSrcData.py. <br> - Tables: <br> - - `src_id_to_title`: `src TEXT, id INT, title TEXT, PRIMARY KEY(src, id)` - - `title_iucn`: `title TEXT PRIMARY KEY, status TEXT` diff --git a/backend/tolData/wikidata/genTaxonSrcData.py b/backend/tolData/wikidata/genTaxonSrcData.py deleted file mode 100755 index 5d10c71..0000000 --- a/backend/tolData/wikidata/genTaxonSrcData.py +++ /dev/null @@ -1,240 +0,0 @@ -#!/usr/bin/python3 - -import sys, os, re, math, io -from collections import defaultdict -import bz2, json, sqlite3 -import multiprocessing, indexed_bzip2, pickle, tempfile - -import argparse -parser = argparse.ArgumentParser(description=""" -Reads a wikidata JSON dump, looking for enwiki taxon items, and associated -IDs from sources like GBIF/etc, and IUCN conservation status. Writes results -into a database. - -The JSON dump contains an array of objects, each of which describes a -Wikidata item item1, and takes up it's own line. -- Getting item1's Wikidata ID: item1['id'] (eg: "Q144") -- Checking if item1 is a taxon: item1['claims']['P31'][idx1]['mainsnak']['datavalue']['value']['numeric-id'] == id1 - 'idx1' indexes an array of statements - 'id1' is a Wikidata ID denoting a taxon item type (eg: 310890 means 'monotypic taxon') -- Checking if item1 is a taxon-alt: item1['claims']['P31'][idx1]['mainsnak']['datavalue']['value']['numeric-id'] == id1 - 'id1' denotes a common-name-alternative item type (eg: 55983715 means 'organisms known by a particular common name') - Getting the ID of the item that item1 is an alternative for: - item1['claims']['P31'][idx1]['qualifiers']['P642'][idx2]['datavalue']['value']['numeric-id'] -- Checking for an EOL/NCBI/etc ID: item['claims'][prop1][idx1]['mainsnak']['datavalue']['value'] (eg: "328672") - 'prop1' denotes a 'has ID from source *' property (eg: 'P830' means 'has EOL ID') -- Checking for an IUCN status: item['claims']['P141'][idx1]['mainsnak']['datavalue']['value']['id'] (eg: "Q219127") - -Based on code from https://github.com/OneZoom/OZtree, located in -OZprivate/ServerScripts/TaxonMappingAndPopularity/ (22 Aug 2022). -""", formatter_class=argparse.RawDescriptionHelpFormatter) -args = parser.parse_args() - -# On Linux, running on the full dataset caused the processes to hang after processing. This was resolved by: -# - Storing subprocess results in temp files. Apparently passing large objects through pipes can cause deadlock. -# - Using set_start_method('spawn'). Apparently 'fork' can cause unexpected copying of lock/semaphore/etc state. -# Related: https://bugs.python.org/issue6721 -# - Using pool.map() instead of pool.imap_unordered(), which seems to hang in some cases (was using python 3.8). -# Possibly related: https://github.com/python/cpython/issues/72882 - -WD_FILE = 'latest-all.json.bz2' -OFFSETS_FILE = 'offsets.dat' -DB_FILE = 'taxonSrcs.db' -N_PROCS = 6 # Took about 3 hours (probably would've taken 6-12 with N_PROCS=1) - -# Wikidata entity IDs -TAXON_IDS = ['Q16521', 'Q310890', 'Q23038290', 'Q713623'] # 'taxon', 'monotypic taxon', 'fossil taxon', 'clade' -TAXON_ALT_IDS = ['Q55983715', 'Q502895'] # 'organisms known by a particular common name', 'common name' -SRC_PROP_IDS = {'P830': 'eol', 'P685': 'ncbi', 'P1391': 'if', 'P850': 'worms', 'P5055': 'irmng', 'P846': 'gbif'} -IUCN_STATUS_IDS = { - 'Q211005': 'least concern', 'Q719675': 'near threatened', 'Q278113': 'vulnerable', - 'Q11394': 'endangered', 'Q219127': 'critically endangered', 'Q239509': 'extinct in the wild', - 'Q237350': 'extinct species', 'Q3245245': 'data deficient' -} -# For filtering lines before parsing JSON -LINE_REGEX = re.compile(('"numeric-id":(?:' + '|'.join([s[1:] for s in TAXON_IDS + TAXON_ALT_IDS]) + ')\D').encode()) - -def main() -> None: - # Maps to populate - srcIdToId: dict[str, dict[int, int]] = defaultdict(dict) # Maps 'eol'/etc to {srcId1: wikidataId1, ...} - idToTitle: dict[int, str] = {} # Maps wikidata ID to enwiki title - idToAltId: dict[int, int] = {} # Maps taxon-item wikidata ID to taxon-alt ID (eg: 'canis lupus familiaris' -> 'dog') - idToIucnStatus: dict[int, str] = {} # Maps wikidata ID to iucn-status string ('least concern', etc) - # Check db - if os.path.exists(DB_FILE): - print('ERROR: Database already exists') - sys.exit(1) - # Read dump - if N_PROCS == 1: - with bz2.open(WD_FILE, mode='rb') as file: - for lineNum, line in enumerate(file, 1): - if lineNum % 1e4 == 0: - print(f'At line {lineNum}') - readDumpLine(line, srcIdToId, idToTitle, idToAltId, idToIucnStatus) - else: - if not os.path.exists(OFFSETS_FILE): - print('Creating offsets file') # For indexed access for multiprocessing (creation took about 6.7 hours) - with indexed_bzip2.open(WD_FILE) as file: - with open(OFFSETS_FILE, 'wb') as file2: - pickle.dump(file.block_offsets(), file2) - print('Allocating file into chunks') - fileSz: int # About 1.4 TB - with indexed_bzip2.open(WD_FILE) as file: - with open(OFFSETS_FILE, 'rb') as file2: - file.set_block_offsets(pickle.load(file2)) - fileSz = file.seek(0, io.SEEK_END) - chunkSz = math.floor(fileSz / N_PROCS) - chunkIdxs = [-1] + [chunkSz * i for i in range(1, N_PROCS)] + [fileSz-1] - # Each adjacent pair specifies a start+end byte index for readDumpChunk() - print(f'- Chunk size: {chunkSz:,}') - print('Starting processes to read dump') - with tempfile.TemporaryDirectory() as tempDirName: - # Using maxtasksperchild=1 to free resources on task completion - with multiprocessing.Pool(processes=N_PROCS, maxtasksperchild=1) as pool: - for outFilename in pool.map(readDumpChunkOneParam, - ((i, chunkIdxs[i], chunkIdxs[i+1], f'{tempDirName}/{i}.pickle') for i in range(N_PROCS))): - # Get map data from subprocess output file - with open(outFilename, 'rb') as file: - maps = pickle.load(file) - # Add to maps - for src, idMap in maps[0].items(): - srcIdToId[src].update(idMap) - idToTitle.update(maps[1]) - idToAltId.update(maps[2]) - idToIucnStatus.update(maps[3]) - # - print('Writing to db') - dbCon = sqlite3.connect(DB_FILE) - dbCur = dbCon.cursor() - dbCur.execute('CREATE TABLE src_id_to_title (src TEXT, id INT, title TEXT, PRIMARY KEY(src, id))') - for src, submap in srcIdToId.items(): - for srcId, wId in submap.items(): - if wId not in idToTitle: # Check for a title, possibly via an alt-taxon - if wId in idToAltId: - wId = idToAltId[wId] - else: - continue - dbCur.execute('INSERT INTO src_id_to_title VALUES (?, ?, ?)', (src, srcId, idToTitle[wId])) - dbCur.execute('CREATE TABLE title_iucn (title TEXT PRIMARY KEY, status TEXT)') - for wId, status in idToIucnStatus.items(): - if wId not in idToTitle: # Check for a title, possibly via an alt-taxon - if wId in idToAltId and idToAltId[wId] not in idToIucnStatus: - wId = idToAltId[wId] - else: - continue - dbCur.execute('INSERT OR IGNORE INTO title_iucn VALUES (?, ?)', (idToTitle[wId], status)) - # The 'OR IGNORE' allows for multiple taxons using the same alt - #dbCur.execute('CREATE TABLE id_to_alt_title (id TEXT PRIMARY KEY, title TEXT, alt TEXT)') - #for wId, altId in idToAltId.items(): - # dbCur.execute('INSERT INTO id_to_alt_title VALUES (?, ?, ?)', - # (wId, idToTitle[wId] if wId in idToTitle else None, idToTitle[altId])) - dbCon.commit() - dbCon.close() - -def readDumpLine( - lineBytes: bytes, - srcIdToId: dict[str, dict[int, int]], - idToTitle: dict[int, str], - idToAltId: dict[int, int], - idToIucnStatus: dict[int, str]) -> None: - # Check if taxon item - if LINE_REGEX.search(lineBytes) is None: - return - try: - line = lineBytes.decode('utf-8').rstrip().rstrip(',') - jsonItem = json.loads(line) - except json.JSONDecodeError: - print(f'Unable to parse line {line} as JSON') - return - isTaxon = False - altTaxa: list[int] = [] # For a taxon-alt item, holds associated taxon-item IDs - claims = None - try: - claims = jsonItem['claims'] - for statement in claims['P31']: # Check for 'instance of' statements - typeId: str = statement['mainsnak']['datavalue']['value']['id'] - if typeId in TAXON_IDS: - isTaxon = True - break - elif typeId in TAXON_ALT_IDS: - snaks = statement['qualifiers']['P642'] # Check for 'of' qualifiers - altTaxa.extend([int(s['datavalue']['value']['numeric-id']) for s in snaks]) - break - except (KeyError, ValueError): - return - if not isTaxon and not altTaxa: - return - # Get wikidata ID and enwiki title - itemId: int | None = None - itemTitle: str | None = None - try: - itemId = int(jsonItem['id'][1:]) # Skips initial 'Q' - itemTitle = jsonItem['sitelinks']['enwiki']['title'] - except KeyError: - # Allow taxon-items without titles (they might get one via a taxon-alt) - if itemId is not None and isTaxon: - itemTitle = None - else: - return - # Update maps - if itemTitle is not None: - idToTitle[itemId] = itemTitle - for altId in altTaxa: - idToAltId[altId] = itemId - # Check for source IDs - for srcPropId, src in SRC_PROP_IDS.items(): - if srcPropId in claims: - try: - srcId = int(claims[srcPropId][0]['mainsnak']['datavalue']['value']) - srcIdToId[src][srcId] = itemId - except (KeyError, ValueError): - continue - # Check for IUCN status - if 'P141' in claims: # Check for 'iucn conservation status' statement - try: - iucnStatusId: str = claims['P141'][0]['mainsnak']['datavalue']['value']['id'] - idToIucnStatus[itemId] = IUCN_STATUS_IDS[iucnStatusId] - except KeyError: - pass - -def readDumpChunkOneParam(params: tuple[int, int, int, str]) -> str: - """ Forwards to readDumpChunk(), for use with pool.map() """ - return readDumpChunk(*params) - -# Reads lines in the dump that begin after a start-byte, and not after an end byte - # If startByte is -1, start at the first line -def readDumpChunk(procId: int, startByte: int, endByte: int, outFilename: str) -> str: - # Maps to populate - maps: tuple[ - dict[str, dict[int, int]], - dict[int, str], - dict[int, int], - dict[int, str]] = (defaultdict(dict), {}, {}, {}) - # Read dump - with indexed_bzip2.open(WD_FILE) as file: - # Load offsets file - with open(OFFSETS_FILE, 'rb') as file2: - offsets = pickle.load(file2) - file.set_block_offsets(offsets) - # Seek to chunk - if startByte != -1: - file.seek(startByte) - file.readline() - else: - startByte = 0 # Used for progress calculation - # Read lines - count = 0 - while file.tell() <= endByte: - count += 1 - if count % 1e4 == 0: - perc = (file.tell() - startByte) / (endByte - startByte) * 100 - print(f'Thread {procId}: {perc:.2f}%') - readDumpLine(file.readline(), *maps) - # Output results into file - with open(outFilename, 'wb') as file: - pickle.dump(maps, file) - return outFilename - -if __name__ == '__main__': # Guard needed for multiprocessing - multiprocessing.set_start_method('spawn') - main() |