diff options
Diffstat (limited to 'backend/data/README.md')
| -rw-r--r-- | backend/data/README.md | 27 |
1 files changed, 15 insertions, 12 deletions
diff --git a/backend/data/README.md b/backend/data/README.md index fb3ca16..e622832 100644 --- a/backend/data/README.md +++ b/backend/data/README.md @@ -12,12 +12,13 @@ File Generation Process genEolNameDataPickedIds.txt. 3 Node Description Data 1 Obtain data in dbpedia/ and enwiki/, as specified in their README files. - 2 Run genDbpData.py, which adds a 'descs' table to data.db, using - data in dbpedia/dbpData.db, the 'nodes' table, and possibly + 2 Run genDbpData.py, which adds 'wiki\_ids' and 'descs' tables to data.db, + using data in dbpedia/dbpData.db, the 'nodes' table, and possibly genDescNamesToSkip.txt and dbpPickedLabels.txt. - 3 Run genEnwikiDescData.py, which adds to the 'descs' table, using data in - enwiki/enwikiData.db, and the 'nodes' table. Also uses genDescNamesToSkip.txt and - genEnwikiDescTitlesToUse.txt for skipping/resolving some name-page associations. + 3 Run genEnwikiDescData.py, which adds to the 'wiki\_ids' and 'descs' tables, + using data in enwiki/enwikiData.db, and the 'nodes' table. + Also uses genDescNamesToSkip.txt and genEnwikiDescTitlesToUse.txt for + skipping/resolving some name-page associations. 3 Image Data 1 In eol/, run downloadImgs.py to download EOL images into eol/imgsForReview/. It uses data in eol/imagesList.db, and the 'eol\_ids' table. @@ -25,15 +26,16 @@ File Generation Process images in eol/imgsReviewed/ (uses 'names' and 'eol\_ids' to display extra info). 3 In enwiki/, run getEnwikiImgData.py, which generates a list of tol-node images, and creates enwiki/enwikiImgs.db to store it. - Uses the 'descs' table to get tol-node wiki-ids. + Uses the 'wiki\_ids' table to get tol-node wiki-ids. 4 In enwiki/, run downloadImgLicenseInfo.py, which downloads licensing information for images listed in enwiki/enwikiImgs.db, and stores it in that db. 5 In enwiki/, run downloadEnwikiImgs.py, which downloads 'permissively-licensed' images in listed in enwiki/enwikiImgs.db, storing them in enwiki/imgs/. 6 Run reviewImgsToMerge.py, which displays images from eol/ and enwiki/, - enables choosing, for each tol-node, which image should be used, if any, - and outputs choice information into mergedImgList.txt. + and enables choosing, for each tol-node, which image should be used, if any, + and outputs choice information into mergedImgList.txt. Uses the 'nodes', + 'eol\_ids', and 'wiki\_ids' tables (as well as 'names' for info-display). 7 Run genImgsForWeb.py, which creates cropped/resized images in img/, using mergedImgList.txt, and adds 'images' and 'node_imgs' tables to data.db. Smartcrop's outputs might need to be manually created/adjusted: <br> @@ -47,14 +49,14 @@ File Generation Process The result might have as many as 150k images, with about 2/3 of them being from wikipedia. 8 Run genLinkedImgs.py to add a 'linked_imgs' table to data.db, - which uses 'nodes', 'edges', 'eol\_ids', and 'node_imgs', to associate + which uses 'nodes', 'edges', 'eol\_ids', and 'node\_imgs', to associate nodes without images to child images. 5 Reduced Tree Structure Data 1 Run genReducedTreeData.py, which adds 'r_nodes' and 'r_edges' tables to data.db, using reducedTol/names.txt, and the 'nodes' and 'names' tables. 6 Other - Optionally run genEnwikiNameData.py, which adds more entries to the 'names' table, - using data in enwiki/enwikiData.db, and the 'names' and 'descs' tables. + using data in enwiki/enwikiData.db, and the 'names' and 'wiki\_ids' tables. - Optionally run trimTree.py, which tries to remove some 'low-significance' nodes, for the sake of performance and result-relevance. Without this, jumping to certain nodes within the fungi and moths can take over a minute to render. @@ -63,9 +65,10 @@ data.db Tables ============== - nodes: name TEXT PRIMARY KEY, id TEXT UNIQUE, tips INT - edges: node TEXT, child TEXT, p\_support INT, PRIMARY KEY (node, child) -- names: name TEXT, alt\_name TEXT, pref\_alt INT, src TEXT, PRIMARY KEY(name, alt\_name) - eol\_ids: id INT PRIMARY KEY, name TEXT -- descs: name TEXT PRIMARY KEY, desc TEXT, redirected INT, wiki\_id INT, from\_dbp INT +- names: name TEXT, alt\_name TEXT, pref\_alt INT, src TEXT, PRIMARY KEY(name, alt\_name) +- wiki\_ids: name TEXT PRIMARY KEY, id INT, redirected INT +- descs: wiki\_id INT PRIMARY KEY, desc TEXT, from\_dbp INT - node\_imgs: name TEXT PRIMARY KEY, img\_id INT, src TEXT - images: id INT, src TEXT, url TEXT, license TEXT, artist TEXT, credit TEXT, PRIMARY KEY (id, src) - linked\_imgs: name TEXT PRIMARY KEY, otol\_id INT, otol\_id2 INT |